Oct 30 11:53:21.053367 ip-10-0-141-251 systemd[1]: Starting Kubernetes Kubelet... Oct 30 11:53:21.502272 ip-10-0-141-251 kubenswrapper[2545]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 11:53:21.502272 ip-10-0-141-251 kubenswrapper[2545]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 11:53:21.502272 ip-10-0-141-251 kubenswrapper[2545]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 11:53:21.502272 ip-10-0-141-251 kubenswrapper[2545]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 11:53:21.505861 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.505781 2545 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 11:53:21.505861 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.505861 2545 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 11:53:21.505861 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.505864 2545 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 11:53:21.505953 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.505867 2545 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 11:53:21.505953 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.505870 2545 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 11:53:21.505953 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.505873 2545 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 11:53:21.505953 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.505876 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 11:53:21.505953 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.505878 2545 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 11:53:21.505953 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.505881 2545 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 11:53:21.505953 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.505883 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 11:53:21.505953 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.505886 2545 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 11:53:21.505953 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.505888 2545 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 11:53:21.505953 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.505891 2545 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 11:53:21.505953 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.505893 2545 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 11:53:21.505953 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.505896 2545 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 11:53:21.505953 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.505898 2545 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 11:53:21.505953 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.505900 2545 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 11:53:21.505953 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.505903 2545 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 11:53:21.505953 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.505905 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 11:53:21.505953 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.505920 2545 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 11:53:21.505953 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.505923 2545 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 11:53:21.505953 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.505925 2545 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 11:53:21.506408 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.505928 2545 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 11:53:21.506408 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.505930 2545 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 11:53:21.506408 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.505933 2545 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 11:53:21.506408 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.505936 2545 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 11:53:21.506408 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.505939 2545 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 11:53:21.506408 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.505941 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 11:53:21.506408 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.505944 2545 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 11:53:21.506408 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.505946 2545 feature_gate.go:386] unrecognized feature gate: Example Oct 30 11:53:21.506408 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.505949 2545 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 11:53:21.506408 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.505951 2545 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 11:53:21.506408 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.505954 2545 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 11:53:21.506408 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.505957 2545 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 11:53:21.506408 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.505960 2545 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 11:53:21.506408 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.505963 2545 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 11:53:21.506408 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.505966 2545 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 11:53:21.506408 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.505968 2545 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 11:53:21.506408 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.505971 2545 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 11:53:21.506408 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.505974 2545 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 11:53:21.506408 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.505976 2545 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 11:53:21.506408 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.505978 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 11:53:21.506858 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.505981 2545 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 11:53:21.506858 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.505983 2545 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 11:53:21.506858 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.505986 2545 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 11:53:21.506858 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.505988 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 11:53:21.506858 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.505990 2545 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 11:53:21.506858 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.505993 2545 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 11:53:21.506858 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.505995 2545 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 11:53:21.506858 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.505998 2545 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 11:53:21.506858 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.506000 2545 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 11:53:21.506858 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.506002 2545 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 11:53:21.506858 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.506005 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 11:53:21.506858 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.506008 2545 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 11:53:21.506858 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.506010 2545 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 11:53:21.506858 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.506014 2545 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 11:53:21.506858 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.506017 2545 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 11:53:21.506858 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.506019 2545 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 11:53:21.506858 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.506022 2545 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 11:53:21.506858 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.506024 2545 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 11:53:21.506858 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.506027 2545 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 11:53:21.507299 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.506029 2545 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 11:53:21.507299 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.506032 2545 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 11:53:21.507299 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.506034 2545 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 11:53:21.507299 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.506036 2545 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 11:53:21.507299 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.506039 2545 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 11:53:21.507299 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.506041 2545 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 11:53:21.507299 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.506043 2545 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 11:53:21.507299 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.506046 2545 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 11:53:21.507299 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.506048 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 11:53:21.507299 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.506051 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 11:53:21.507299 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.506053 2545 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 11:53:21.507299 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.506058 2545 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 11:53:21.507299 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.506062 2545 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 11:53:21.507299 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.506065 2545 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 11:53:21.507299 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.506068 2545 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 11:53:21.507299 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.506071 2545 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 11:53:21.507299 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.506075 2545 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 11:53:21.507299 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.506077 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 11:53:21.507299 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.506080 2545 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 11:53:21.507730 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.506084 2545 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 11:53:21.507730 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.506088 2545 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 11:53:21.507730 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.506091 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 11:53:21.507730 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.506093 2545 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 11:53:21.507730 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.506096 2545 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 11:53:21.507730 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.506099 2545 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 11:53:21.507730 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.506102 2545 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 11:53:21.507730 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.506105 2545 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 11:53:21.507730 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.506120 2545 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 11:53:21.507730 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.506123 2545 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 11:53:21.507730 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.506126 2545 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 11:53:21.507730 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.506128 2545 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 11:53:21.507730 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.506131 2545 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 11:53:21.507730 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.506133 2545 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 11:53:21.507730 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.506136 2545 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 11:53:21.507730 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.506138 2545 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 11:53:21.507730 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.506142 2545 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 11:53:21.507730 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.506144 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 11:53:21.507730 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.506147 2545 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 11:53:21.508172 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.506150 2545 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 11:53:21.508172 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.506153 2545 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 11:53:21.508172 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.506155 2545 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 11:53:21.508172 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.506158 2545 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 11:53:21.508172 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.506160 2545 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 11:53:21.508172 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.506163 2545 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 11:53:21.508172 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.506165 2545 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 11:53:21.508172 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.506940 2545 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 11:53:21.508172 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.506948 2545 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 11:53:21.508172 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.506952 2545 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 11:53:21.508172 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.506954 2545 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 11:53:21.508172 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.506957 2545 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 11:53:21.508172 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.506959 2545 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 11:53:21.508172 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.506964 2545 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 11:53:21.508172 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.506967 2545 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 11:53:21.508172 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.506970 2545 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 11:53:21.508172 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.506973 2545 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 11:53:21.508172 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.506976 2545 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 11:53:21.508172 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.506978 2545 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 11:53:21.508650 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.506982 2545 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 11:53:21.508650 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.506986 2545 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 11:53:21.508650 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.506988 2545 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 11:53:21.508650 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.506991 2545 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 11:53:21.508650 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.506993 2545 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 11:53:21.508650 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.506995 2545 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 11:53:21.508650 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.506998 2545 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 11:53:21.508650 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507000 2545 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 11:53:21.508650 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507003 2545 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 11:53:21.508650 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507005 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 11:53:21.508650 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507008 2545 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 11:53:21.508650 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507010 2545 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 11:53:21.508650 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507013 2545 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 11:53:21.508650 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507015 2545 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 11:53:21.508650 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507017 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 11:53:21.508650 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507020 2545 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 11:53:21.508650 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507022 2545 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 11:53:21.508650 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507025 2545 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 11:53:21.508650 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507027 2545 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 11:53:21.509094 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507030 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 11:53:21.509094 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507032 2545 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 11:53:21.509094 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507035 2545 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 11:53:21.509094 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507037 2545 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 11:53:21.509094 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507040 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 11:53:21.509094 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507042 2545 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 11:53:21.509094 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507044 2545 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 11:53:21.509094 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507047 2545 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 11:53:21.509094 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507049 2545 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 11:53:21.509094 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507052 2545 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 11:53:21.509094 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507054 2545 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 11:53:21.509094 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507057 2545 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 11:53:21.509094 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507059 2545 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 11:53:21.509094 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507062 2545 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 11:53:21.509094 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507066 2545 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 11:53:21.509094 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507069 2545 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 11:53:21.509094 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507072 2545 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 11:53:21.509094 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507074 2545 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 11:53:21.509094 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507077 2545 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 11:53:21.509549 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507079 2545 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 11:53:21.509549 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507082 2545 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 11:53:21.509549 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507084 2545 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 11:53:21.509549 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507087 2545 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 11:53:21.509549 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507089 2545 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 11:53:21.509549 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507092 2545 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 11:53:21.509549 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507094 2545 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 11:53:21.509549 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507097 2545 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 11:53:21.509549 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507099 2545 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 11:53:21.509549 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507102 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 11:53:21.509549 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507104 2545 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 11:53:21.509549 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507121 2545 feature_gate.go:386] unrecognized feature gate: Example Oct 30 11:53:21.509549 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507124 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 11:53:21.509549 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507126 2545 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 11:53:21.509549 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507129 2545 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 11:53:21.509549 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507132 2545 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 11:53:21.509549 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507134 2545 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 11:53:21.509549 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507137 2545 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 11:53:21.509549 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507139 2545 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 11:53:21.509549 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507142 2545 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 11:53:21.510015 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507144 2545 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 11:53:21.510015 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507147 2545 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 11:53:21.510015 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507149 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 11:53:21.510015 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507152 2545 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 11:53:21.510015 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507154 2545 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 11:53:21.510015 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507157 2545 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 11:53:21.510015 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507160 2545 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 11:53:21.510015 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507164 2545 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 11:53:21.510015 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507169 2545 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 11:53:21.510015 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507172 2545 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 11:53:21.510015 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507175 2545 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 11:53:21.510015 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507178 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 11:53:21.510015 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507180 2545 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 11:53:21.510015 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507183 2545 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 11:53:21.510015 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507186 2545 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 11:53:21.510015 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507188 2545 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 11:53:21.510015 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507191 2545 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 11:53:21.510015 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507194 2545 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 11:53:21.510015 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507196 2545 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 11:53:21.510015 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507199 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 11:53:21.510485 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507202 2545 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 11:53:21.510485 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507204 2545 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 11:53:21.510485 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507207 2545 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 11:53:21.510485 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507209 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 11:53:21.510485 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507211 2545 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 11:53:21.510485 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507214 2545 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 11:53:21.510485 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507217 2545 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 11:53:21.510485 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507219 2545 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 11:53:21.510485 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507221 2545 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 11:53:21.510485 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507224 2545 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 11:53:21.510485 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507226 2545 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 11:53:21.510485 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507228 2545 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 11:53:21.510485 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507231 2545 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 11:53:21.510485 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507233 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 11:53:21.510485 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507236 2545 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 11:53:21.510485 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507238 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 11:53:21.510485 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507709 2545 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 11:53:21.510485 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507714 2545 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 11:53:21.510485 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507717 2545 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 11:53:21.510485 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507720 2545 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 11:53:21.510943 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507723 2545 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 11:53:21.510943 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507726 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 11:53:21.510943 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507729 2545 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 11:53:21.510943 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507731 2545 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 11:53:21.510943 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507734 2545 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 11:53:21.510943 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507737 2545 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 11:53:21.510943 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507741 2545 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 11:53:21.510943 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507744 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 11:53:21.510943 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507748 2545 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 11:53:21.510943 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507751 2545 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 11:53:21.510943 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507753 2545 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 11:53:21.510943 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507756 2545 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 11:53:21.510943 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507758 2545 feature_gate.go:386] unrecognized feature gate: Example Oct 30 11:53:21.510943 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507761 2545 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 11:53:21.510943 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507763 2545 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 11:53:21.510943 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507766 2545 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 11:53:21.510943 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507768 2545 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 11:53:21.510943 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507771 2545 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 11:53:21.510943 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507773 2545 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 11:53:21.510943 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507776 2545 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 11:53:21.511406 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507778 2545 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 11:53:21.511406 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507781 2545 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 11:53:21.511406 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507783 2545 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 11:53:21.511406 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507786 2545 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 11:53:21.511406 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507788 2545 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 11:53:21.511406 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507791 2545 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 11:53:21.511406 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507793 2545 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 11:53:21.511406 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507798 2545 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 11:53:21.511406 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507801 2545 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 11:53:21.511406 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507803 2545 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 11:53:21.511406 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507806 2545 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 11:53:21.511406 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507808 2545 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 11:53:21.511406 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507811 2545 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 11:53:21.511406 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507814 2545 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 11:53:21.511406 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507816 2545 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 11:53:21.511406 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507818 2545 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 11:53:21.511406 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507821 2545 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 11:53:21.511406 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507824 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 11:53:21.511406 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507826 2545 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 11:53:21.511870 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507828 2545 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 11:53:21.511870 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507831 2545 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 11:53:21.511870 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507833 2545 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 11:53:21.511870 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507836 2545 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 11:53:21.511870 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507838 2545 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 11:53:21.511870 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507840 2545 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 11:53:21.511870 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507843 2545 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 11:53:21.511870 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507845 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 11:53:21.511870 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507848 2545 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 11:53:21.511870 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507851 2545 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 11:53:21.511870 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507855 2545 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 11:53:21.511870 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507859 2545 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 11:53:21.511870 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507862 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 11:53:21.511870 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507864 2545 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 11:53:21.511870 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507868 2545 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 11:53:21.511870 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507871 2545 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 11:53:21.511870 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507873 2545 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 11:53:21.511870 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507876 2545 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 11:53:21.511870 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507878 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 11:53:21.512316 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507881 2545 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 11:53:21.512316 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507884 2545 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 11:53:21.512316 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507887 2545 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 11:53:21.512316 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507889 2545 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 11:53:21.512316 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507892 2545 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 11:53:21.512316 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507894 2545 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 11:53:21.512316 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507897 2545 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 11:53:21.512316 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507900 2545 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 11:53:21.512316 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507903 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 11:53:21.512316 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507905 2545 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 11:53:21.512316 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507908 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 11:53:21.512316 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507910 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 11:53:21.512316 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507913 2545 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 11:53:21.512316 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507916 2545 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 11:53:21.512316 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507918 2545 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 11:53:21.512316 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507920 2545 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 11:53:21.512316 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507923 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 11:53:21.512316 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507925 2545 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 11:53:21.512316 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507928 2545 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 11:53:21.512779 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507930 2545 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 11:53:21.512779 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507933 2545 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 11:53:21.512779 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507935 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 11:53:21.512779 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507938 2545 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 11:53:21.512779 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507940 2545 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 11:53:21.512779 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507943 2545 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 11:53:21.512779 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507945 2545 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 11:53:21.512779 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507948 2545 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 11:53:21.512779 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507951 2545 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 11:53:21.512779 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507953 2545 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 11:53:21.512779 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507956 2545 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 11:53:21.512779 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507958 2545 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 11:53:21.512779 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507960 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 11:53:21.512779 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507963 2545 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 11:53:21.512779 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507967 2545 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 11:53:21.512779 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507969 2545 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 11:53:21.512779 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507972 2545 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 11:53:21.512779 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507974 2545 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 11:53:21.512779 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507976 2545 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 11:53:21.512779 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507979 2545 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 11:53:21.513279 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507982 2545 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 11:53:21.513279 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507984 2545 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 11:53:21.513279 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507987 2545 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 11:53:21.513279 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507991 2545 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 11:53:21.513279 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.507993 2545 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 11:53:21.513279 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.508092 2545 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 11:53:21.513279 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.508100 2545 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 11:53:21.513279 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.508119 2545 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 11:53:21.513279 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.508124 2545 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 11:53:21.513279 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.508129 2545 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 11:53:21.513279 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.508132 2545 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 11:53:21.513279 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.508139 2545 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 11:53:21.513279 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.508143 2545 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 11:53:21.513279 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.508146 2545 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 11:53:21.513279 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.508149 2545 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 11:53:21.513279 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.508152 2545 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 11:53:21.513279 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.508156 2545 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 11:53:21.513279 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.508159 2545 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 11:53:21.513279 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.508161 2545 flags.go:64] FLAG: --cgroup-root="" Oct 30 11:53:21.513279 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.508164 2545 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 11:53:21.513279 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.508167 2545 flags.go:64] FLAG: --client-ca-file="" Oct 30 11:53:21.513279 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.508170 2545 flags.go:64] FLAG: --cloud-provider="external" Oct 30 11:53:21.513279 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.508173 2545 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 11:53:21.513796 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.508177 2545 flags.go:64] FLAG: --cluster-domain="" Oct 30 11:53:21.513796 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.508180 2545 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 11:53:21.513796 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.508183 2545 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 11:53:21.513796 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.508187 2545 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 11:53:21.513796 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.508191 2545 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 11:53:21.513796 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.508195 2545 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 11:53:21.513796 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.508198 2545 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 11:53:21.513796 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.508201 2545 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 11:53:21.513796 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.508204 2545 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 11:53:21.513796 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.508207 2545 flags.go:64] FLAG: --contention-profiling="false" Oct 30 11:53:21.513796 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.508210 2545 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 11:53:21.513796 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.508213 2545 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 11:53:21.513796 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.508216 2545 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 11:53:21.513796 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.508219 2545 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 11:53:21.513796 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.508223 2545 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 11:53:21.513796 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.508226 2545 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 11:53:21.513796 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.508229 2545 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 11:53:21.513796 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.508232 2545 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 11:53:21.513796 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.508235 2545 flags.go:64] FLAG: --enable-server="true" Oct 30 11:53:21.513796 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.508237 2545 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 11:53:21.513796 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.508242 2545 flags.go:64] FLAG: --event-burst="100" Oct 30 11:53:21.513796 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.508245 2545 flags.go:64] FLAG: --event-qps="50" Oct 30 11:53:21.513796 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.508248 2545 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 11:53:21.513796 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.508251 2545 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 11:53:21.514376 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.508254 2545 flags.go:64] FLAG: --eviction-hard="" Oct 30 11:53:21.514376 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.508258 2545 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 11:53:21.514376 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.508261 2545 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 11:53:21.514376 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.508264 2545 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 11:53:21.514376 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.508267 2545 flags.go:64] FLAG: --eviction-soft="" Oct 30 11:53:21.514376 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.508270 2545 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 11:53:21.514376 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.508273 2545 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 11:53:21.514376 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.508275 2545 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 11:53:21.514376 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.508278 2545 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 11:53:21.514376 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.508281 2545 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 11:53:21.514376 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.508284 2545 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 11:53:21.514376 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.508287 2545 flags.go:64] FLAG: --feature-gates="" Oct 30 11:53:21.514376 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.508291 2545 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 11:53:21.514376 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.508294 2545 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 11:53:21.514376 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.508297 2545 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 11:53:21.514376 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.508300 2545 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 11:53:21.514376 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.508303 2545 flags.go:64] FLAG: --healthz-port="10248" Oct 30 11:53:21.514376 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.508307 2545 flags.go:64] FLAG: --help="false" Oct 30 11:53:21.514376 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.508310 2545 flags.go:64] FLAG: --hostname-override="ip-10-0-141-251.ec2.internal" Oct 30 11:53:21.514376 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.508313 2545 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 11:53:21.514376 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.508316 2545 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 11:53:21.514376 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.508319 2545 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 11:53:21.514376 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.508322 2545 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 11:53:21.514376 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.508326 2545 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 11:53:21.514919 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.508328 2545 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 11:53:21.514919 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.508331 2545 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 11:53:21.514919 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.508334 2545 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 11:53:21.514919 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.508337 2545 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 11:53:21.514919 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.508339 2545 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 11:53:21.514919 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.508343 2545 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 11:53:21.514919 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.508346 2545 flags.go:64] FLAG: --kube-reserved="" Oct 30 11:53:21.514919 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.508350 2545 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 11:53:21.514919 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.508353 2545 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 11:53:21.514919 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.508356 2545 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 11:53:21.514919 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.508359 2545 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 11:53:21.514919 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.508362 2545 flags.go:64] FLAG: --lock-file="" Oct 30 11:53:21.514919 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.508364 2545 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 11:53:21.514919 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.508367 2545 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 11:53:21.514919 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.508370 2545 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 11:53:21.514919 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.508376 2545 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 11:53:21.514919 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.508379 2545 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 11:53:21.514919 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.508382 2545 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 11:53:21.514919 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.508384 2545 flags.go:64] FLAG: --logging-format="text" Oct 30 11:53:21.514919 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.508388 2545 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 11:53:21.514919 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.508391 2545 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 11:53:21.514919 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.508394 2545 flags.go:64] FLAG: --manifest-url="" Oct 30 11:53:21.514919 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.508397 2545 flags.go:64] FLAG: --manifest-url-header="" Oct 30 11:53:21.514919 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.508401 2545 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 11:53:21.514919 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.508404 2545 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 11:53:21.515510 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.508408 2545 flags.go:64] FLAG: --max-pods="110" Oct 30 11:53:21.515510 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.508411 2545 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 11:53:21.515510 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.508413 2545 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 11:53:21.515510 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.508417 2545 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 11:53:21.515510 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.508420 2545 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 11:53:21.515510 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.508423 2545 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 11:53:21.515510 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.508426 2545 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 11:53:21.515510 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.508429 2545 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 11:53:21.515510 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.508435 2545 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 11:53:21.515510 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.508439 2545 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 11:53:21.515510 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.508441 2545 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 11:53:21.515510 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.508444 2545 flags.go:64] FLAG: --pod-cidr="" Oct 30 11:53:21.515510 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.508447 2545 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 11:53:21.515510 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.508450 2545 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 11:53:21.515510 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.508454 2545 flags.go:64] FLAG: --pods-per-core="0" Oct 30 11:53:21.515510 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.508458 2545 flags.go:64] FLAG: --port="10250" Oct 30 11:53:21.515510 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.508461 2545 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 11:53:21.515510 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.508464 2545 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-01af67c8e3389319a" Oct 30 11:53:21.515510 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.508467 2545 flags.go:64] FLAG: --qos-reserved="" Oct 30 11:53:21.515510 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.508470 2545 flags.go:64] FLAG: --read-only-port="10255" Oct 30 11:53:21.515510 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.508473 2545 flags.go:64] FLAG: --register-node="true" Oct 30 11:53:21.515510 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.508476 2545 flags.go:64] FLAG: --register-with-taints="" Oct 30 11:53:21.515510 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.508479 2545 flags.go:64] FLAG: --registry-burst="10" Oct 30 11:53:21.515510 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.508482 2545 flags.go:64] FLAG: --registry-qps="5" Oct 30 11:53:21.515510 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.508485 2545 flags.go:64] FLAG: --reserved-cpus="" Oct 30 11:53:21.515510 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.508488 2545 flags.go:64] FLAG: --reserved-memory="" Oct 30 11:53:21.516319 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.508493 2545 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 11:53:21.516319 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.508496 2545 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 11:53:21.516319 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.508499 2545 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 11:53:21.516319 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.508502 2545 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 11:53:21.516319 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.508505 2545 flags.go:64] FLAG: --runonce="false" Oct 30 11:53:21.516319 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.508508 2545 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 11:53:21.516319 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.508511 2545 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 11:53:21.516319 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.508514 2545 flags.go:64] FLAG: --seccomp-default="false" Oct 30 11:53:21.516319 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.508517 2545 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 11:53:21.516319 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.508520 2545 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 11:53:21.516319 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.508523 2545 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 11:53:21.516319 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.508526 2545 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 11:53:21.516319 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.508530 2545 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 11:53:21.516319 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.508533 2545 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 11:53:21.516319 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.508536 2545 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 11:53:21.516319 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.508539 2545 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 11:53:21.516319 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.508542 2545 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 11:53:21.516319 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.508545 2545 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 11:53:21.516319 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.508548 2545 flags.go:64] FLAG: --system-cgroups="" Oct 30 11:53:21.516319 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.508550 2545 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 11:53:21.516319 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.508557 2545 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 11:53:21.516319 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.508559 2545 flags.go:64] FLAG: --tls-cert-file="" Oct 30 11:53:21.516319 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.508562 2545 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 11:53:21.516319 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.508566 2545 flags.go:64] FLAG: --tls-min-version="" Oct 30 11:53:21.516319 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.508569 2545 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 11:53:21.517290 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.508571 2545 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 11:53:21.517290 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.508574 2545 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 11:53:21.517290 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.508577 2545 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 11:53:21.517290 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.508580 2545 flags.go:64] FLAG: --v="2" Oct 30 11:53:21.517290 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.508584 2545 flags.go:64] FLAG: --version="false" Oct 30 11:53:21.517290 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.508588 2545 flags.go:64] FLAG: --vmodule="" Oct 30 11:53:21.517290 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.508592 2545 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 11:53:21.517290 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.508596 2545 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 11:53:21.517290 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.508747 2545 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 11:53:21.517290 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.508753 2545 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 11:53:21.517290 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.508756 2545 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 11:53:21.517290 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.508759 2545 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 11:53:21.517290 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.508762 2545 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 11:53:21.517290 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.508765 2545 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 11:53:21.517290 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.508768 2545 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 11:53:21.517290 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.508770 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 11:53:21.517290 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.508773 2545 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 11:53:21.517290 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.508776 2545 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 11:53:21.517290 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.508778 2545 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 11:53:21.517290 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.508781 2545 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 11:53:21.517290 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.508785 2545 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 11:53:21.517777 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.508787 2545 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 11:53:21.517777 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.508790 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 11:53:21.517777 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.508793 2545 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 11:53:21.517777 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.508795 2545 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 11:53:21.517777 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.508798 2545 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 11:53:21.517777 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.508800 2545 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 11:53:21.517777 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.508806 2545 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 11:53:21.517777 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.508809 2545 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 11:53:21.517777 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.508812 2545 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 11:53:21.517777 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.508814 2545 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 11:53:21.517777 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.508817 2545 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 11:53:21.517777 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.508819 2545 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 11:53:21.517777 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.508822 2545 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 11:53:21.517777 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.508825 2545 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 11:53:21.517777 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.508827 2545 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 11:53:21.517777 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.508830 2545 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 11:53:21.517777 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.508832 2545 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 11:53:21.517777 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.508835 2545 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 11:53:21.517777 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.508838 2545 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 11:53:21.517777 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.508841 2545 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 11:53:21.518320 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.508844 2545 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 11:53:21.518320 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.508846 2545 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 11:53:21.518320 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.508849 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 11:53:21.518320 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.508851 2545 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 11:53:21.518320 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.508854 2545 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 11:53:21.518320 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.508856 2545 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 11:53:21.518320 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.508859 2545 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 11:53:21.518320 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.508861 2545 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 11:53:21.518320 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.508864 2545 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 11:53:21.518320 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.508866 2545 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 11:53:21.518320 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.508869 2545 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 11:53:21.518320 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.508871 2545 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 11:53:21.518320 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.508875 2545 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 11:53:21.518320 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.508878 2545 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 11:53:21.518320 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.508880 2545 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 11:53:21.518320 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.508883 2545 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 11:53:21.518320 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.508885 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 11:53:21.518320 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.508887 2545 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 11:53:21.518320 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.508891 2545 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 11:53:21.518785 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.508895 2545 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 11:53:21.518785 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.508899 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 11:53:21.518785 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.508902 2545 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 11:53:21.518785 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.508905 2545 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 11:53:21.518785 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.508909 2545 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 11:53:21.518785 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.508912 2545 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 11:53:21.518785 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.508915 2545 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 11:53:21.518785 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.508917 2545 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 11:53:21.518785 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.508920 2545 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 11:53:21.518785 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.508923 2545 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 11:53:21.518785 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.508925 2545 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 11:53:21.518785 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.508930 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 11:53:21.518785 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.508933 2545 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 11:53:21.518785 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.508935 2545 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 11:53:21.518785 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.508938 2545 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 11:53:21.518785 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.508940 2545 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 11:53:21.518785 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.508943 2545 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 11:53:21.518785 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.508945 2545 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 11:53:21.518785 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.508948 2545 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 11:53:21.519231 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.508950 2545 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 11:53:21.519231 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.508953 2545 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 11:53:21.519231 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.508956 2545 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 11:53:21.519231 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.508958 2545 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 11:53:21.519231 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.508961 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 11:53:21.519231 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.508963 2545 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 11:53:21.519231 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.508966 2545 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 11:53:21.519231 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.508969 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 11:53:21.519231 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.508971 2545 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 11:53:21.519231 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.508974 2545 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 11:53:21.519231 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.508976 2545 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 11:53:21.519231 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.508979 2545 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 11:53:21.519231 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.508983 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 11:53:21.519231 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.508985 2545 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 11:53:21.519231 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.508988 2545 feature_gate.go:386] unrecognized feature gate: Example Oct 30 11:53:21.519231 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.508990 2545 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 11:53:21.519231 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.508993 2545 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 11:53:21.519231 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.508995 2545 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 11:53:21.519231 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.508998 2545 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 11:53:21.519231 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.509000 2545 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 11:53:21.519718 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.509002 2545 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 11:53:21.519718 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.509005 2545 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 11:53:21.519718 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.509007 2545 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 11:53:21.519718 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.509010 2545 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 11:53:21.519718 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.509013 2545 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 11:53:21.519718 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.509016 2545 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 11:53:21.519718 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.509018 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 11:53:21.519718 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.509021 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 11:53:21.519718 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.509023 2545 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 11:53:21.519718 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.509026 2545 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 11:53:21.519718 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.509028 2545 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 11:53:21.519718 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.509031 2545 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 11:53:21.519718 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.509033 2545 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 11:53:21.519718 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.509036 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 11:53:21.519718 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.509038 2545 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 11:53:21.519718 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.509063 2545 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 11:53:21.520087 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.509095 2545 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 11:53:21.520087 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.509100 2545 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 11:53:21.520087 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.509103 2545 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 11:53:21.520087 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.509118 2545 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 11:53:21.520087 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.509122 2545 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 11:53:21.520087 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.509125 2545 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 11:53:21.520087 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.509127 2545 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 11:53:21.520087 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.509130 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 11:53:21.520087 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.509135 2545 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 11:53:21.520087 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.509138 2545 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 11:53:21.520087 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.509141 2545 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 11:53:21.520087 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.509143 2545 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 11:53:21.520087 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.509146 2545 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 11:53:21.520087 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.509149 2545 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 11:53:21.520087 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.509151 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 11:53:21.520087 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.509154 2545 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 11:53:21.520087 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.509156 2545 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 11:53:21.520087 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.509159 2545 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 11:53:21.520087 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.509162 2545 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 11:53:21.520536 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.509164 2545 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 11:53:21.520536 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.509166 2545 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 11:53:21.520536 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.509170 2545 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 11:53:21.520536 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.509173 2545 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 11:53:21.520536 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.509175 2545 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 11:53:21.520536 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.509178 2545 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 11:53:21.520536 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.509180 2545 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 11:53:21.520536 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.509183 2545 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 11:53:21.520536 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.509185 2545 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 11:53:21.520536 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.509188 2545 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 11:53:21.520536 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.509190 2545 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 11:53:21.520536 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.509193 2545 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 11:53:21.520536 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.509195 2545 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 11:53:21.520536 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.509198 2545 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 11:53:21.520536 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.509200 2545 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 11:53:21.520536 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.509203 2545 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 11:53:21.520536 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.509206 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 11:53:21.520536 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.509208 2545 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 11:53:21.520536 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.509215 2545 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 11:53:21.520536 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.509218 2545 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 11:53:21.521000 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.509221 2545 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 11:53:21.521000 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.509225 2545 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 11:53:21.521000 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.509227 2545 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 11:53:21.521000 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.509230 2545 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 11:53:21.521000 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.509232 2545 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 11:53:21.521000 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.509235 2545 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 11:53:21.521000 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.509237 2545 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 11:53:21.521000 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.509240 2545 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 11:53:21.521000 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.509242 2545 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 11:53:21.521000 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.509245 2545 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 11:53:21.521000 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.509247 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 11:53:21.521000 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.509250 2545 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 11:53:21.521000 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.509252 2545 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 11:53:21.521000 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.509256 2545 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 11:53:21.521000 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.509260 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 11:53:21.521000 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.509264 2545 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 11:53:21.521000 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.509267 2545 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 11:53:21.521000 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.509271 2545 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 11:53:21.521000 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.509275 2545 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 11:53:21.521459 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.509278 2545 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 11:53:21.521459 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.509281 2545 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 11:53:21.521459 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.509284 2545 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 11:53:21.521459 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.509287 2545 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 11:53:21.521459 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.509289 2545 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 11:53:21.521459 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.509292 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 11:53:21.521459 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.509294 2545 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 11:53:21.521459 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.509297 2545 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 11:53:21.521459 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.509299 2545 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 11:53:21.521459 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.509302 2545 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 11:53:21.521459 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.509305 2545 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 11:53:21.521459 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.509307 2545 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 11:53:21.521459 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.509310 2545 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 11:53:21.521459 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.509312 2545 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 11:53:21.521459 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.509316 2545 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 11:53:21.521459 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.509319 2545 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 11:53:21.521459 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.509322 2545 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 11:53:21.521459 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.509324 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 11:53:21.521459 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.509327 2545 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 11:53:21.521459 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.509329 2545 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 11:53:21.521911 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.509332 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 11:53:21.521911 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.509334 2545 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 11:53:21.521911 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.509336 2545 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 11:53:21.521911 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.509339 2545 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 11:53:21.521911 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.509341 2545 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 11:53:21.521911 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.509344 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 11:53:21.521911 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.509347 2545 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 11:53:21.521911 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.509350 2545 feature_gate.go:386] unrecognized feature gate: Example Oct 30 11:53:21.521911 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.509353 2545 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 11:53:21.521911 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.509355 2545 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 11:53:21.521911 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.509358 2545 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 11:53:21.521911 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.509361 2545 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 11:53:21.521911 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.509364 2545 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 11:53:21.521911 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.509366 2545 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 11:53:21.521911 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.509369 2545 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 11:53:21.521911 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.509371 2545 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 11:53:21.521911 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.509374 2545 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 11:53:21.521911 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.509376 2545 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 11:53:21.521911 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.509379 2545 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 11:53:21.521911 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.509381 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 11:53:21.522409 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.509384 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 11:53:21.522409 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.509386 2545 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 11:53:21.522409 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.509389 2545 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 11:53:21.522409 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.509391 2545 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 11:53:21.522409 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.509394 2545 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 11:53:21.522409 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.509397 2545 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 11:53:21.522409 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.509401 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 11:53:21.522409 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.509403 2545 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 11:53:21.522409 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.517310 2545 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 11:53:21.522409 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.517325 2545 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 11:53:21.522409 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.518232 2545 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 11:53:21.522409 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.518239 2545 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 11:53:21.522409 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518294 2545 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 11:53:21.522409 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518299 2545 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 11:53:21.522409 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518302 2545 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 11:53:21.522409 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518306 2545 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 11:53:21.522409 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518309 2545 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 11:53:21.522409 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518312 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 11:53:21.522409 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518314 2545 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 11:53:21.522833 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518317 2545 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 11:53:21.522833 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518320 2545 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 11:53:21.522833 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518323 2545 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 11:53:21.522833 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518326 2545 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 11:53:21.522833 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518329 2545 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 11:53:21.522833 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518331 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 11:53:21.522833 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518334 2545 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 11:53:21.522833 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518337 2545 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 11:53:21.522833 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518340 2545 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 11:53:21.522833 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518342 2545 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 11:53:21.522833 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518345 2545 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 11:53:21.522833 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518348 2545 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 11:53:21.522833 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518351 2545 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 11:53:21.522833 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518353 2545 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 11:53:21.522833 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518356 2545 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 11:53:21.522833 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518359 2545 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 11:53:21.522833 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518361 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 11:53:21.522833 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518364 2545 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 11:53:21.522833 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518367 2545 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 11:53:21.523388 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518369 2545 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 11:53:21.523388 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518372 2545 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 11:53:21.523388 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518375 2545 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 11:53:21.523388 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518377 2545 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 11:53:21.523388 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518380 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 11:53:21.523388 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518384 2545 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 11:53:21.523388 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518388 2545 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 11:53:21.523388 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518391 2545 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 11:53:21.523388 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518394 2545 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 11:53:21.523388 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518396 2545 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 11:53:21.523388 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518400 2545 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 11:53:21.523388 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518402 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 11:53:21.523388 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518405 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 11:53:21.523388 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518407 2545 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 11:53:21.523388 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518410 2545 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 11:53:21.523388 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518413 2545 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 11:53:21.523388 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518415 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 11:53:21.523388 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518418 2545 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 11:53:21.523388 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518420 2545 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 11:53:21.523818 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518423 2545 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 11:53:21.523818 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518426 2545 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 11:53:21.523818 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518429 2545 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 11:53:21.523818 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518432 2545 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 11:53:21.523818 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518435 2545 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 11:53:21.523818 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518437 2545 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 11:53:21.523818 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518440 2545 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 11:53:21.523818 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518443 2545 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 11:53:21.523818 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518445 2545 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 11:53:21.523818 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518448 2545 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 11:53:21.523818 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518450 2545 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 11:53:21.523818 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518453 2545 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 11:53:21.523818 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518455 2545 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 11:53:21.523818 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518458 2545 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 11:53:21.523818 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518460 2545 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 11:53:21.523818 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518463 2545 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 11:53:21.523818 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518465 2545 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 11:53:21.523818 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518468 2545 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 11:53:21.523818 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518470 2545 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 11:53:21.523818 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518473 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 11:53:21.524299 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518475 2545 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 11:53:21.524299 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518478 2545 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 11:53:21.524299 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518480 2545 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 11:53:21.524299 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518483 2545 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 11:53:21.524299 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518486 2545 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 11:53:21.524299 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518490 2545 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 11:53:21.524299 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518493 2545 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 11:53:21.524299 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518496 2545 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 11:53:21.524299 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518498 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 11:53:21.524299 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518501 2545 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 11:53:21.524299 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518503 2545 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 11:53:21.524299 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518506 2545 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 11:53:21.524299 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518509 2545 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 11:53:21.524299 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518511 2545 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 11:53:21.524299 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518514 2545 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 11:53:21.524299 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518517 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 11:53:21.524299 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518520 2545 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 11:53:21.524299 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518523 2545 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 11:53:21.524299 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518526 2545 feature_gate.go:386] unrecognized feature gate: Example Oct 30 11:53:21.524743 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518528 2545 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 11:53:21.524743 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518531 2545 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 11:53:21.524743 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518533 2545 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 11:53:21.524743 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518536 2545 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 11:53:21.524743 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518538 2545 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 11:53:21.524743 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518541 2545 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 11:53:21.524743 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518543 2545 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 11:53:21.524743 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518546 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 11:53:21.524743 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518549 2545 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 11:53:21.524743 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518551 2545 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 11:53:21.524743 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518554 2545 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 11:53:21.524743 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518556 2545 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 11:53:21.524743 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518559 2545 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 11:53:21.524743 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518561 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 11:53:21.524743 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518564 2545 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 11:53:21.524743 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518566 2545 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 11:53:21.524743 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518569 2545 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 11:53:21.524743 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518571 2545 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 11:53:21.524743 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518573 2545 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 11:53:21.524743 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518576 2545 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 11:53:21.525204 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518579 2545 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 11:53:21.525204 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518582 2545 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 11:53:21.525204 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.518609 2545 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 11:53:21.525204 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518741 2545 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 11:53:21.525204 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518746 2545 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 11:53:21.525204 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518749 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 11:53:21.525204 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518751 2545 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 11:53:21.525204 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518754 2545 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 11:53:21.525204 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518757 2545 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 11:53:21.525204 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518760 2545 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 11:53:21.525204 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518763 2545 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 11:53:21.525204 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518765 2545 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 11:53:21.525204 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518768 2545 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 11:53:21.525204 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518771 2545 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 11:53:21.525204 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518773 2545 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 11:53:21.525204 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518775 2545 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 11:53:21.525615 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518778 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 11:53:21.525615 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518780 2545 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 11:53:21.525615 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518783 2545 feature_gate.go:386] unrecognized feature gate: Example Oct 30 11:53:21.525615 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518786 2545 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 11:53:21.525615 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518790 2545 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 11:53:21.525615 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518793 2545 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 11:53:21.525615 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518796 2545 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 11:53:21.525615 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518798 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 11:53:21.525615 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518801 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 11:53:21.525615 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518803 2545 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 11:53:21.525615 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518806 2545 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 11:53:21.525615 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518808 2545 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 11:53:21.525615 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518810 2545 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 11:53:21.525615 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518813 2545 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 11:53:21.525615 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518815 2545 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 11:53:21.525615 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518818 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 11:53:21.525615 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518820 2545 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 11:53:21.525615 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518823 2545 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 11:53:21.525615 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518825 2545 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 11:53:21.526039 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518828 2545 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 11:53:21.526039 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518830 2545 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 11:53:21.526039 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518832 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 11:53:21.526039 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518835 2545 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 11:53:21.526039 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518837 2545 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 11:53:21.526039 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518840 2545 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 11:53:21.526039 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518843 2545 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 11:53:21.526039 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518845 2545 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 11:53:21.526039 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518848 2545 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 11:53:21.526039 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518850 2545 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 11:53:21.526039 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518854 2545 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 11:53:21.526039 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518857 2545 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 11:53:21.526039 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518861 2545 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 11:53:21.526039 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518863 2545 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 11:53:21.526039 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518866 2545 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 11:53:21.526039 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518868 2545 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 11:53:21.526039 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518871 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 11:53:21.526039 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518873 2545 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 11:53:21.526039 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518875 2545 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 11:53:21.526520 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518878 2545 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 11:53:21.526520 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518880 2545 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 11:53:21.526520 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518883 2545 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 11:53:21.526520 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518885 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 11:53:21.526520 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518888 2545 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 11:53:21.526520 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518890 2545 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 11:53:21.526520 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518892 2545 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 11:53:21.526520 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518895 2545 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 11:53:21.526520 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518897 2545 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 11:53:21.526520 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518900 2545 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 11:53:21.526520 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518902 2545 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 11:53:21.526520 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518904 2545 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 11:53:21.526520 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518907 2545 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 11:53:21.526520 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518909 2545 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 11:53:21.526520 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518912 2545 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 11:53:21.526520 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518914 2545 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 11:53:21.526520 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518917 2545 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 11:53:21.526520 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518919 2545 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 11:53:21.526520 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518922 2545 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 11:53:21.526520 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518925 2545 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 11:53:21.526977 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518928 2545 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 11:53:21.526977 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518930 2545 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 11:53:21.526977 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518933 2545 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 11:53:21.526977 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518935 2545 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 11:53:21.526977 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518937 2545 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 11:53:21.526977 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518940 2545 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 11:53:21.526977 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518943 2545 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 11:53:21.526977 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518946 2545 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 11:53:21.526977 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518948 2545 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 11:53:21.526977 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518951 2545 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 11:53:21.526977 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518953 2545 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 11:53:21.526977 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518955 2545 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 11:53:21.526977 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518958 2545 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 11:53:21.526977 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518960 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 11:53:21.526977 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518962 2545 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 11:53:21.526977 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518967 2545 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 11:53:21.526977 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518970 2545 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 11:53:21.526977 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518972 2545 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 11:53:21.526977 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518975 2545 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 11:53:21.526977 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518977 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 11:53:21.527443 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518980 2545 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 11:53:21.527443 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518982 2545 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 11:53:21.527443 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518985 2545 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 11:53:21.527443 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518987 2545 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 11:53:21.527443 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518990 2545 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 11:53:21.527443 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518992 2545 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 11:53:21.527443 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518994 2545 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 11:53:21.527443 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518997 2545 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 11:53:21.527443 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.518999 2545 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 11:53:21.527443 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.519002 2545 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 11:53:21.527443 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.519004 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 11:53:21.527443 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.519006 2545 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 11:53:21.527443 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.519009 2545 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 11:53:21.527443 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.519012 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 11:53:21.527443 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.519014 2545 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 11:53:21.527443 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.519035 2545 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 11:53:21.527807 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.519066 2545 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 11:53:21.527807 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.519070 2545 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 11:53:21.527807 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.519072 2545 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 11:53:21.527807 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.519075 2545 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 11:53:21.527807 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.519078 2545 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 11:53:21.527807 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.519080 2545 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 11:53:21.527807 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.519083 2545 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 11:53:21.527807 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.519085 2545 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 11:53:21.527807 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.519088 2545 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 11:53:21.527807 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.519090 2545 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 11:53:21.527807 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.519092 2545 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 11:53:21.527807 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.519096 2545 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 11:53:21.527807 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.519099 2545 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 11:53:21.527807 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.519102 2545 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 11:53:21.527807 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.519104 2545 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 11:53:21.527807 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.519118 2545 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 11:53:21.527807 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.519121 2545 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 11:53:21.527807 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.519124 2545 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 11:53:21.527807 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.519126 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 11:53:21.527807 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.519129 2545 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 11:53:21.528262 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.519131 2545 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 11:53:21.528262 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.519134 2545 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 11:53:21.528262 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.519136 2545 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 11:53:21.528262 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.519138 2545 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 11:53:21.528262 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.519141 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 11:53:21.528262 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.519143 2545 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 11:53:21.528262 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.519146 2545 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 11:53:21.528262 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.519148 2545 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 11:53:21.528262 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.519151 2545 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 11:53:21.528262 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.519154 2545 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 11:53:21.528262 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.519157 2545 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 11:53:21.528262 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.519159 2545 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 11:53:21.528262 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.519162 2545 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 11:53:21.528262 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.519164 2545 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 11:53:21.528262 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.519166 2545 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 11:53:21.528262 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.519169 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 11:53:21.528262 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.519171 2545 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 11:53:21.528262 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.519173 2545 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 11:53:21.528262 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.519176 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 11:53:21.528262 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.519178 2545 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 11:53:21.528777 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.519181 2545 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 11:53:21.528777 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.519183 2545 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 11:53:21.528777 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.519186 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 11:53:21.528777 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.519190 2545 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 11:53:21.528777 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.519193 2545 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 11:53:21.528777 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.519195 2545 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 11:53:21.528777 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.519198 2545 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 11:53:21.528777 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.519200 2545 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 11:53:21.528777 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.519202 2545 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 11:53:21.528777 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.519205 2545 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 11:53:21.528777 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.519209 2545 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 11:53:21.528777 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.519211 2545 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 11:53:21.528777 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.519214 2545 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 11:53:21.528777 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.519216 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 11:53:21.528777 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.519218 2545 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 11:53:21.528777 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.519221 2545 feature_gate.go:386] unrecognized feature gate: Example Oct 30 11:53:21.528777 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.519225 2545 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 11:53:21.528777 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.519228 2545 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 11:53:21.528777 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.519231 2545 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 11:53:21.528777 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.519234 2545 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 11:53:21.529248 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.519237 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 11:53:21.529248 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.519240 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 11:53:21.529248 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.519242 2545 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 11:53:21.529248 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.519246 2545 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 11:53:21.529248 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.519248 2545 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 11:53:21.529248 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.519251 2545 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 11:53:21.529248 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.519253 2545 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 11:53:21.529248 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.519256 2545 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 11:53:21.529248 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.519258 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 11:53:21.529248 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.519261 2545 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 11:53:21.529248 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.519263 2545 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 11:53:21.529248 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.519266 2545 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 11:53:21.529248 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.519268 2545 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 11:53:21.529248 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.519271 2545 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 11:53:21.529248 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.519273 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 11:53:21.529248 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.519277 2545 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 11:53:21.529248 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.519279 2545 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 11:53:21.529248 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.519282 2545 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 11:53:21.529248 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.519284 2545 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 11:53:21.529248 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.519287 2545 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 11:53:21.529698 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.519289 2545 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 11:53:21.529698 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.519292 2545 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 11:53:21.529698 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.519295 2545 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 11:53:21.529698 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.519298 2545 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 11:53:21.529698 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.519301 2545 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 11:53:21.529698 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.519304 2545 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 11:53:21.529698 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.519307 2545 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 11:53:21.529698 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.519310 2545 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 11:53:21.529698 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.519313 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 11:53:21.529698 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.519316 2545 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 11:53:21.529698 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.519319 2545 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 11:53:21.529698 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.519321 2545 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 11:53:21.529698 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.519324 2545 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 11:53:21.529698 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.519326 2545 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 11:53:21.529698 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.519329 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 11:53:21.529698 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.519333 2545 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 11:53:21.529698 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.519336 2545 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 11:53:21.529698 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.519339 2545 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 11:53:21.530145 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.519341 2545 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 11:53:21.530145 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.519343 2545 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 11:53:21.530145 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.519346 2545 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 11:53:21.530145 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.519349 2545 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 11:53:21.530145 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.519351 2545 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 11:53:21.530145 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.519354 2545 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 11:53:21.530145 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.519356 2545 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 11:53:21.530145 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:21.519358 2545 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 11:53:21.530145 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.519413 2545 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 11:53:21.530145 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.525251 2545 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 30 11:53:21.530145 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.526266 2545 server.go:1014] "Starting client certificate rotation" Oct 30 11:53:21.530145 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.526359 2545 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 11:53:21.530145 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.526393 2545 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 11:53:21.530145 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.528522 2545 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 11:53:21.530145 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.529586 2545 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 11:53:21.539128 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.539095 2545 log.go:25] "Validated CRI v1 runtime API" Oct 30 11:53:21.543765 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.543750 2545 log.go:25] "Validated CRI v1 image API" Oct 30 11:53:21.544568 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.544528 2545 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 11:53:21.551480 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.551461 2545 fs.go:135] Filesystem UUIDs: map[19a6a40c-0284-4256-b5ae-cc8b2af1b706:/dev/nvme0n1p3 1ae0e2cc-2f11-4a0f-8b2c-453b430f35c7:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2] Oct 30 11:53:21.551536 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.551480 2545 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 30 11:53:21.557585 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.557479 2545 manager.go:217] Machine: {Timestamp:2026-10-30 11:53:21.556123765 +0000 UTC m=+0.375295296 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3198319 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2ee154a9a13382148e60952589ab09 SystemUUID:ec2ee154-a9a1-3382-148e-60952589ab09 BootID:8b2fe4e4-2c68-4dc5-aa57-9651d2a3939e Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:f2:db:09:25 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:f2:db:09:25 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ea:ff:c1:c8:ec:02 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 11:53:21.557585 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.557583 2545 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 11:53:21.557686 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.557652 2545 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 11:53:21.560594 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.560574 2545 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 11:53:21.560713 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.560595 2545 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-251.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 11:53:21.560777 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.560715 2545 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 11:53:21.560777 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.560724 2545 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 11:53:21.560777 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.560753 2545 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 11:53:21.562520 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.562510 2545 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 11:53:21.563788 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.563779 2545 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 11:53:21.568397 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.568386 2545 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 11:53:21.568505 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.568495 2545 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 11:53:21.569363 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.569354 2545 kubelet.go:493] "Attempting to sync node with API server" Oct 30 11:53:21.569402 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.569366 2545 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 11:53:21.569402 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.569375 2545 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 11:53:21.569402 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.569384 2545 kubelet.go:405] "Adding apiserver pod source" Oct 30 11:53:21.569402 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.569391 2545 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 11:53:21.569735 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.569715 2545 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 11:53:21.573589 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.573570 2545 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 11:53:21.581079 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.581066 2545 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 11:53:21.582043 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.582031 2545 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 11:53:21.582082 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.582059 2545 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 11:53:21.582215 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.582205 2545 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 11:53:21.582253 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.582220 2545 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 11:53:21.582253 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.582226 2545 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 11:53:21.582253 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.582231 2545 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 11:53:21.582253 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.582236 2545 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 11:53:21.582253 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.582246 2545 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 11:53:21.582869 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.582860 2545 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 11:53:21.582869 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.582869 2545 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 11:53:21.582922 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.582876 2545 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 11:53:21.582922 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.582882 2545 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 11:53:21.582922 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.582890 2545 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 11:53:21.582922 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.582900 2545 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 11:53:21.584368 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.584360 2545 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 11:53:21.584368 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.584368 2545 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 11:53:21.588648 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.588635 2545 server.go:1263] "Started kubelet" Oct 30 11:53:21.588796 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.588719 2545 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 11:53:21.588890 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.588820 2545 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 11:53:21.588890 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.588818 2545 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 11:53:21.589465 ip-10-0-141-251 systemd[1]: Started Kubernetes Kubelet. Oct 30 11:53:21.590332 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.590301 2545 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 11:53:21.592520 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.592507 2545 server.go:317] "Adding debug handlers to kubelet server" Oct 30 11:53:21.595070 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:21.595042 2545 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 30 11:53:21.598954 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.598941 2545 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 11:53:21.599009 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.598952 2545 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 11:53:21.599055 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:21.599012 2545 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-251.ec2.internal\" not found" Oct 30 11:53:21.599738 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.599723 2545 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 11:53:21.599738 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.599725 2545 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 11:53:21.599865 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.599751 2545 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 11:53:21.599865 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.599839 2545 reconstruct.go:97] "Volume reconstruction finished" Oct 30 11:53:21.599865 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.599848 2545 reconciler.go:29] "Reconciler: start to sync state" Oct 30 11:53:21.600154 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.600142 2545 factory.go:153] Registering CRI-O factory Oct 30 11:53:21.600206 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.600157 2545 factory.go:223] Registration of the crio container factory successfully Oct 30 11:53:21.600239 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.600215 2545 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 11:53:21.600239 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.600224 2545 factory.go:55] Registering systemd factory Oct 30 11:53:21.600239 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.600232 2545 factory.go:223] Registration of the systemd container factory successfully Oct 30 11:53:21.600339 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.600259 2545 factory.go:103] Registering Raw factory Oct 30 11:53:21.600339 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.600274 2545 manager.go:1199] Started watching for new ooms in manager Oct 30 11:53:21.601328 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.601309 2545 manager.go:319] Starting recovery of all containers Oct 30 11:53:21.613994 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.613977 2545 manager.go:324] Recovery completed Oct 30 11:53:21.618207 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.618193 2545 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 11:53:21.619834 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.619813 2545 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-251.ec2.internal" event="NodeHasSufficientMemory" Oct 30 11:53:21.619888 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.619855 2545 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-251.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 11:53:21.619888 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.619874 2545 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-251.ec2.internal" event="NodeHasSufficientPID" Oct 30 11:53:21.620385 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.620372 2545 cpu_manager.go:226] "Starting" policy="none" Oct 30 11:53:21.620385 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.620384 2545 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 11:53:21.620474 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.620405 2545 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 11:53:21.620474 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.620405 2545 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 11:53:21.621120 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.621092 2545 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-251.ec2.internal" event="NodeHasSufficientMemory" Oct 30 11:53:21.621182 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.621130 2545 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-251.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 11:53:21.621182 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.621147 2545 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-251.ec2.internal" event="NodeHasSufficientPID" Oct 30 11:53:21.622540 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.622527 2545 policy_none.go:50] "Start" Oct 30 11:53:21.622540 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.622542 2545 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 11:53:21.622639 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.622552 2545 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 11:53:21.623605 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.623572 2545 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-141-251.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 30 11:53:21.623673 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:21.623656 2545 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-141-251.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 30 11:53:21.624301 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.624284 2545 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 11:53:21.624700 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:21.623590 2545 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-251.ec2.internal.18e34c82dea827dd default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-251.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-141-251.ec2.internal,},FirstTimestamp:2026-10-30 11:53:21.588611037 +0000 UTC m=+0.407782566,LastTimestamp:2026-10-30 11:53:21.588611037 +0000 UTC m=+0.407782566,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-251.ec2.internal,}" Oct 30 11:53:21.638406 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:21.638382 2545 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Oct 30 11:53:21.638493 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:21.638395 2545 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Oct 30 11:53:21.638598 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:21.638575 2545 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-141-251.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Oct 30 11:53:21.653287 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:21.653219 2545 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-251.ec2.internal.18e34c82e0849b54 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-251.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-141-251.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-141-251.ec2.internal,},FirstTimestamp:2026-10-30 11:53:21.619835732 +0000 UTC m=+0.439007268,LastTimestamp:2026-10-30 11:53:21.619835732 +0000 UTC m=+0.439007268,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-251.ec2.internal,}" Oct 30 11:53:21.669963 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.669940 2545 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-dczrc" Oct 30 11:53:21.671262 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:21.671054 2545 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-251.ec2.internal.18e34c82e08506a3 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-251.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-141-251.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-141-251.ec2.internal,},FirstTimestamp:2026-10-30 11:53:21.619863203 +0000 UTC m=+0.439034735,LastTimestamp:2026-10-30 11:53:21.619863203 +0000 UTC m=+0.439034735,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-251.ec2.internal,}" Oct 30 11:53:21.761040 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.686615 2545 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-dczrc" Oct 30 11:53:21.761040 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:21.693835 2545 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-251.ec2.internal.18e34c82e08547c8 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-251.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-141-251.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-141-251.ec2.internal,},FirstTimestamp:2026-10-30 11:53:21.61987988 +0000 UTC m=+0.439051414,LastTimestamp:2026-10-30 11:53:21.61987988 +0000 UTC m=+0.439051414,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-251.ec2.internal,}" Oct 30 11:53:21.761040 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:21.699205 2545 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-251.ec2.internal\" not found" Oct 30 11:53:21.761040 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.719230 2545 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 11:53:21.761040 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.720316 2545 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 11:53:21.761040 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.720333 2545 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 11:53:21.761040 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.720350 2545 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 11:53:21.761040 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:21.720384 2545 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 30 11:53:21.761040 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:21.734426 2545 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-141-251.ec2.internal.18e34c82e0849b54\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-251.ec2.internal.18e34c82e0849b54 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-251.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-141-251.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-141-251.ec2.internal,},FirstTimestamp:2026-10-30 11:53:21.619835732 +0000 UTC m=+0.439007268,LastTimestamp:2026-10-30 11:53:21.621103831 +0000 UTC m=+0.440275363,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-251.ec2.internal,}" Oct 30 11:53:21.761040 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.759970 2545 policy_none.go:44] "Start" Oct 30 11:53:21.774225 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:21.774198 2545 reflector.go:204] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" Oct 30 11:53:21.774302 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:21.774219 2545 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-141-251.ec2.internal.18e34c82e08506a3\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-251.ec2.internal.18e34c82e08506a3 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-251.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-141-251.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-141-251.ec2.internal,},FirstTimestamp:2026-10-30 11:53:21.619863203 +0000 UTC m=+0.439034735,LastTimestamp:2026-10-30 11:53:21.621137762 +0000 UTC m=+0.440309291,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-251.ec2.internal,}" Oct 30 11:53:21.799317 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:21.799300 2545 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-251.ec2.internal\" not found" Oct 30 11:53:21.801613 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.801593 2545 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 11:53:21.803871 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.803858 2545 manager.go:341] "Starting Device Plugin manager" Oct 30 11:53:21.803948 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:21.803886 2545 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 11:53:21.803948 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.803896 2545 server.go:87] "Starting device plugin registration server" Oct 30 11:53:21.804144 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.804131 2545 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 11:53:21.804213 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.804146 2545 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 11:53:21.804383 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.804227 2545 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 11:53:21.804500 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.804446 2545 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 11:53:21.804500 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.804454 2545 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 11:53:21.804743 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:21.804721 2545 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 11:53:21.804823 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:21.804782 2545 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-251.ec2.internal\" not found" Oct 30 11:53:21.820449 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.820429 2545 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-141-251.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-251.ec2.internal"] Oct 30 11:53:21.820505 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.820478 2545 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 11:53:21.821244 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.821229 2545 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-251.ec2.internal" event="NodeHasSufficientMemory" Oct 30 11:53:21.821317 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.821257 2545 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-251.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 11:53:21.821317 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.821270 2545 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-251.ec2.internal" event="NodeHasSufficientPID" Oct 30 11:53:21.823487 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.823474 2545 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 11:53:21.823622 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.823607 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-251.ec2.internal" Oct 30 11:53:21.823670 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.823647 2545 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 11:53:21.824203 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.824185 2545 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-251.ec2.internal" event="NodeHasSufficientMemory" Oct 30 11:53:21.824292 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.824214 2545 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-251.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 11:53:21.824292 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.824224 2545 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-251.ec2.internal" event="NodeHasSufficientPID" Oct 30 11:53:21.824579 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.824558 2545 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-251.ec2.internal" event="NodeHasSufficientMemory" Oct 30 11:53:21.824643 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.824587 2545 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-251.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 11:53:21.824643 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.824601 2545 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-251.ec2.internal" event="NodeHasSufficientPID" Oct 30 11:53:21.826181 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.826169 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-251.ec2.internal" Oct 30 11:53:21.826255 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.826195 2545 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 11:53:21.826861 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.826846 2545 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-251.ec2.internal" event="NodeHasSufficientMemory" Oct 30 11:53:21.826928 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.826877 2545 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-251.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 11:53:21.826928 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.826892 2545 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-251.ec2.internal" event="NodeHasSufficientPID" Oct 30 11:53:21.855536 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:21.855513 2545 nodelease.go:50] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-141-251.ec2.internal\" not found" node="ip-10-0-141-251.ec2.internal" Oct 30 11:53:21.856388 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:21.856375 2545 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-251.ec2.internal\" not found" node="ip-10-0-141-251.ec2.internal" Oct 30 11:53:21.862323 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:21.862307 2545 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-251.ec2.internal\" not found" node="ip-10-0-141-251.ec2.internal" Oct 30 11:53:21.901075 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.901057 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a73cae35bcc0a4b145486ce8e9e460c1-config\") pod \"kube-apiserver-proxy-ip-10-0-141-251.ec2.internal\" (UID: \"a73cae35bcc0a4b145486ce8e9e460c1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-251.ec2.internal" Oct 30 11:53:21.901165 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.901083 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f140bf37195964da2b6ead5845b42a26-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-251.ec2.internal\" (UID: \"f140bf37195964da2b6ead5845b42a26\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-251.ec2.internal" Oct 30 11:53:21.901165 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.901099 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f140bf37195964da2b6ead5845b42a26-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-251.ec2.internal\" (UID: \"f140bf37195964da2b6ead5845b42a26\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-251.ec2.internal" Oct 30 11:53:21.904721 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.904707 2545 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 11:53:21.905379 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.905366 2545 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-251.ec2.internal" event="NodeHasSufficientMemory" Oct 30 11:53:21.905445 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.905388 2545 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-251.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 11:53:21.905445 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.905398 2545 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-251.ec2.internal" event="NodeHasSufficientPID" Oct 30 11:53:21.905445 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.905418 2545 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-141-251.ec2.internal" Oct 30 11:53:21.927957 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:21.927942 2545 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-141-251.ec2.internal" Oct 30 11:53:21.928000 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:21.927961 2545 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-141-251.ec2.internal\": node \"ip-10-0-141-251.ec2.internal\" not found" Oct 30 11:53:21.970044 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:21.970023 2545 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-251.ec2.internal\" not found" Oct 30 11:53:22.002254 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:22.002230 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a73cae35bcc0a4b145486ce8e9e460c1-config\") pod \"kube-apiserver-proxy-ip-10-0-141-251.ec2.internal\" (UID: \"a73cae35bcc0a4b145486ce8e9e460c1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-251.ec2.internal" Oct 30 11:53:22.002332 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:22.002260 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f140bf37195964da2b6ead5845b42a26-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-251.ec2.internal\" (UID: \"f140bf37195964da2b6ead5845b42a26\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-251.ec2.internal" Oct 30 11:53:22.002332 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:22.002276 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f140bf37195964da2b6ead5845b42a26-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-251.ec2.internal\" (UID: \"f140bf37195964da2b6ead5845b42a26\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-251.ec2.internal" Oct 30 11:53:22.002332 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:22.002321 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f140bf37195964da2b6ead5845b42a26-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-251.ec2.internal\" (UID: \"f140bf37195964da2b6ead5845b42a26\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-251.ec2.internal" Oct 30 11:53:22.002425 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:22.002337 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f140bf37195964da2b6ead5845b42a26-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-251.ec2.internal\" (UID: \"f140bf37195964da2b6ead5845b42a26\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-251.ec2.internal" Oct 30 11:53:22.002425 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:22.002337 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a73cae35bcc0a4b145486ce8e9e460c1-config\") pod \"kube-apiserver-proxy-ip-10-0-141-251.ec2.internal\" (UID: \"a73cae35bcc0a4b145486ce8e9e460c1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-251.ec2.internal" Oct 30 11:53:22.071152 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:22.071102 2545 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-251.ec2.internal\" not found" Oct 30 11:53:22.158827 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:22.158810 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-251.ec2.internal" Oct 30 11:53:22.165255 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:22.165244 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-251.ec2.internal" Oct 30 11:53:22.171400 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:22.171384 2545 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-251.ec2.internal\" not found" Oct 30 11:53:22.272482 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:22.272452 2545 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-251.ec2.internal\" not found" Oct 30 11:53:22.372995 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:22.372950 2545 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-251.ec2.internal\" not found" Oct 30 11:53:22.473590 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:22.473572 2545 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-251.ec2.internal\" not found" Oct 30 11:53:22.477803 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:22.477788 2545 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 11:53:22.526146 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:22.526127 2545 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 30 11:53:22.526497 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:22.526249 2545 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 11:53:22.574515 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:22.574497 2545 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-251.ec2.internal\" not found" Oct 30 11:53:24.054304 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:24.054273 2545 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-251.ec2.internal\" not found" Oct 30 11:53:24.082708 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:24.082667 2545 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf140bf37195964da2b6ead5845b42a26.slice/crio-c7e539754dfdda2f33506f8d6a7dc8acd027ef96d8d03b975eff48d2c6fc59bd WatchSource:0}: Error finding container c7e539754dfdda2f33506f8d6a7dc8acd027ef96d8d03b975eff48d2c6fc59bd: Status 404 returned error can't find the container with id c7e539754dfdda2f33506f8d6a7dc8acd027ef96d8d03b975eff48d2c6fc59bd Oct 30 11:53:24.083385 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:24.083363 2545 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda73cae35bcc0a4b145486ce8e9e460c1.slice/crio-1b6e44060e4775c8234fb5f6b195c11bc0f4beb0b731852b60d9d07f2b70e864 WatchSource:0}: Error finding container 1b6e44060e4775c8234fb5f6b195c11bc0f4beb0b731852b60d9d07f2b70e864: Status 404 returned error can't find the container with id 1b6e44060e4775c8234fb5f6b195c11bc0f4beb0b731852b60d9d07f2b70e864 Oct 30 11:53:24.088129 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:24.088111 2545 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 11:53:24.102685 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:24.102647 2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-251.ec2.internal" event={"ID":"f140bf37195964da2b6ead5845b42a26","Type":"ContainerStarted","Data":"c7e539754dfdda2f33506f8d6a7dc8acd027ef96d8d03b975eff48d2c6fc59bd"} Oct 30 11:53:24.103594 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:24.103573 2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-251.ec2.internal" event={"ID":"a73cae35bcc0a4b145486ce8e9e460c1","Type":"ContainerStarted","Data":"1b6e44060e4775c8234fb5f6b195c11bc0f4beb0b731852b60d9d07f2b70e864"} Oct 30 11:53:24.140160 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:24.140132 2545 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 11:48:23 +0000 UTC" deadline="2028-07-12 05:56:44.982758688 +0000 UTC" Oct 30 11:53:24.140160 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:24.140158 2545 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14898h3m20.842605456s" Oct 30 11:53:24.155294 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:24.155269 2545 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-251.ec2.internal\" not found" Oct 30 11:53:24.160406 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:24.160397 2545 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 11:53:24.161461 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:24.161446 2545 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-251.ec2.internal" event="NodeHasSufficientMemory" Oct 30 11:53:24.161515 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:24.161472 2545 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-251.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 11:53:24.161515 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:24.161482 2545 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-251.ec2.internal" event="NodeHasSufficientPID" Oct 30 11:53:24.256131 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:24.256103 2545 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-251.ec2.internal\" not found" Oct 30 11:53:24.281767 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:24.281723 2545 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 11:53:24.357001 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:24.356981 2545 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-251.ec2.internal\" not found" Oct 30 11:53:24.407018 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:24.406997 2545 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 11:53:24.478490 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:24.478354 2545 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-251.ec2.internal" Oct 30 11:53:24.500231 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:24.500211 2545 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 11:53:24.501839 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:24.501806 2545 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-251.ec2.internal" Oct 30 11:53:24.526788 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:24.526766 2545 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 11:53:24.590712 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:24.590629 2545 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 11:53:24.949449 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:24.949378 2545 apiserver.go:52] "Watching apiserver" Oct 30 11:53:24.977628 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:24.977605 2545 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 11:53:24.977997 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:24.977976 2545 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/node-ca-hb8j7","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-251.ec2.internal","openshift-multus/multus-additional-cni-plugins-s6k8x","openshift-cluster-node-tuning-operator/tuned-587rx","openshift-dns/node-resolver-xmrdj","openshift-multus/multus-tlwp4","openshift-multus/network-metrics-daemon-dbqhj","openshift-ovn-kubernetes/ovnkube-node-b8bxz","kube-system/kube-apiserver-proxy-ip-10-0-141-251.ec2.internal","openshift-network-diagnostics/network-check-target-bz65k","openshift-network-operator/iptables-alerter-8m2w9","kube-system/global-pull-secret-syncer-fxjlt","kube-system/konnectivity-agent-tdf8d","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h6mlf"] Oct 30 11:53:24.978495 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:24.978476 2545 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 30 11:53:24.980555 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:24.980535 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dbqhj" Oct 30 11:53:24.980667 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:24.980613 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dbqhj" podUID="1d7f8396-e57c-45e5-ae74-a2244db10364" Oct 30 11:53:24.982859 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:24.982823 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-tdf8d" Oct 30 11:53:24.984921 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:24.984900 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-s6k8x" Oct 30 11:53:24.987462 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:24.987436 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-587rx" Oct 30 11:53:24.989709 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:24.989688 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-xmrdj" Oct 30 11:53:24.991119 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:24.990084 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d7f8396-e57c-45e5-ae74-a2244db10364-metrics-certs\") pod \"network-metrics-daemon-dbqhj\" (UID: \"1d7f8396-e57c-45e5-ae74-a2244db10364\") " pod="openshift-multus/network-metrics-daemon-dbqhj" Oct 30 11:53:24.991119 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:24.990116 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzwcn\" (UniqueName: \"kubernetes.io/projected/1d7f8396-e57c-45e5-ae74-a2244db10364-kube-api-access-bzwcn\") pod \"network-metrics-daemon-dbqhj\" (UID: \"1d7f8396-e57c-45e5-ae74-a2244db10364\") " pod="openshift-multus/network-metrics-daemon-dbqhj" Oct 30 11:53:24.992622 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:24.992603 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-tlwp4" Oct 30 11:53:24.994762 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:24.994746 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h6mlf" Oct 30 11:53:24.997499 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:24.997011 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-hb8j7" Oct 30 11:53:24.999335 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:24.999317 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-b8bxz" Oct 30 11:53:25.001974 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.001873 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bz65k" Oct 30 11:53:25.001974 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:25.001928 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bz65k" podUID="ff9feaa5-70cc-47fd-a3d1-a9ac6e7eaace" Oct 30 11:53:25.004320 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.004304 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-8m2w9" Oct 30 11:53:25.007075 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.007055 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fxjlt" Oct 30 11:53:25.007145 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:25.007120 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fxjlt" podUID="6b350710-c48c-4c7e-bce9-e021099e98ca" Oct 30 11:53:25.014053 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.012872 2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 11:53:25.014053 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.013102 2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 11:53:25.014053 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.013320 2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 11:53:25.014053 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.013561 2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 11:53:25.014053 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.013846 2545 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 11:53:25.014053 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.014013 2545 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-lrxbh\"" Oct 30 11:53:25.014343 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.014070 2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 11:53:25.014343 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.014187 2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 11:53:25.014404 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.014394 2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 11:53:25.014574 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.014559 2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 11:53:25.015914 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.015894 2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 11:53:25.016146 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.016130 2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 11:53:25.041443 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.041425 2545 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 11:53:25.041525 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.041474 2545 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-mzvrp" Oct 30 11:53:25.051534 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.051516 2545 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-mzvrp" Oct 30 11:53:25.079652 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.079632 2545 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 11:53:25.091122 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.091101 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92smt\" (UniqueName: \"kubernetes.io/projected/7e8ec7ff-015c-4049-9c5f-aae73dc06e4b-kube-api-access-92smt\") pod \"iptables-alerter-8m2w9\" (UID: \"7e8ec7ff-015c-4049-9c5f-aae73dc06e4b\") " pod="openshift-network-operator/iptables-alerter-8m2w9" Oct 30 11:53:25.091218 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.091187 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3699031f-ee9e-46cc-b37e-4dfc76b9c0b1-etc-sysconfig\") pod \"tuned-587rx\" (UID: \"3699031f-ee9e-46cc-b37e-4dfc76b9c0b1\") " pod="openshift-cluster-node-tuning-operator/tuned-587rx" Oct 30 11:53:25.091281 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.091219 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4tdc\" (UniqueName: \"kubernetes.io/projected/0df1a118-36a5-4c57-8cec-2c8236f1c4de-kube-api-access-r4tdc\") pod \"multus-tlwp4\" (UID: \"0df1a118-36a5-4c57-8cec-2c8236f1c4de\") " pod="openshift-multus/multus-tlwp4" Oct 30 11:53:25.091281 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.091248 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d659d471-f019-4853-97d0-ee6e948e6c1c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-s6k8x\" (UID: \"d659d471-f019-4853-97d0-ee6e948e6c1c\") " pod="openshift-multus/multus-additional-cni-plugins-s6k8x" Oct 30 11:53:25.091281 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.091272 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stff7\" (UniqueName: \"kubernetes.io/projected/d659d471-f019-4853-97d0-ee6e948e6c1c-kube-api-access-stff7\") pod \"multus-additional-cni-plugins-s6k8x\" (UID: \"d659d471-f019-4853-97d0-ee6e948e6c1c\") " pod="openshift-multus/multus-additional-cni-plugins-s6k8x" Oct 30 11:53:25.091416 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.091377 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/644e19b3-10ad-4145-9ed3-595d19ed9b07-systemd-units\") pod \"ovnkube-node-b8bxz\" (UID: \"644e19b3-10ad-4145-9ed3-595d19ed9b07\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8bxz" Oct 30 11:53:25.091465 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.091416 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0df1a118-36a5-4c57-8cec-2c8236f1c4de-host-var-lib-kubelet\") pod \"multus-tlwp4\" (UID: \"0df1a118-36a5-4c57-8cec-2c8236f1c4de\") " pod="openshift-multus/multus-tlwp4" Oct 30 11:53:25.091520 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.091472 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3699031f-ee9e-46cc-b37e-4dfc76b9c0b1-var-lib-kubelet\") pod \"tuned-587rx\" (UID: \"3699031f-ee9e-46cc-b37e-4dfc76b9c0b1\") " pod="openshift-cluster-node-tuning-operator/tuned-587rx" Oct 30 11:53:25.091566 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.091523 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3699031f-ee9e-46cc-b37e-4dfc76b9c0b1-tmp\") pod \"tuned-587rx\" (UID: \"3699031f-ee9e-46cc-b37e-4dfc76b9c0b1\") " pod="openshift-cluster-node-tuning-operator/tuned-587rx" Oct 30 11:53:25.091616 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.091575 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1b735438-5947-4374-bd4c-66ac3a299770-hosts-file\") pod \"node-resolver-xmrdj\" (UID: \"1b735438-5947-4374-bd4c-66ac3a299770\") " pod="openshift-dns/node-resolver-xmrdj" Oct 30 11:53:25.091664 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.091643 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0df1a118-36a5-4c57-8cec-2c8236f1c4de-multus-cni-dir\") pod \"multus-tlwp4\" (UID: \"0df1a118-36a5-4c57-8cec-2c8236f1c4de\") " pod="openshift-multus/multus-tlwp4" Oct 30 11:53:25.091718 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.091672 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0df1a118-36a5-4c57-8cec-2c8236f1c4de-host-run-k8s-cni-cncf-io\") pod \"multus-tlwp4\" (UID: \"0df1a118-36a5-4c57-8cec-2c8236f1c4de\") " pod="openshift-multus/multus-tlwp4" Oct 30 11:53:25.091718 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.091699 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c54f4c1b-f08e-4495-9a14-17469a792ae5-etc-selinux\") pod \"aws-ebs-csi-driver-node-h6mlf\" (UID: \"c54f4c1b-f08e-4495-9a14-17469a792ae5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h6mlf" Oct 30 11:53:25.091808 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.091723 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c54f4c1b-f08e-4495-9a14-17469a792ae5-sys-fs\") pod \"aws-ebs-csi-driver-node-h6mlf\" (UID: \"c54f4c1b-f08e-4495-9a14-17469a792ae5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h6mlf" Oct 30 11:53:25.091808 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.091746 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/644e19b3-10ad-4145-9ed3-595d19ed9b07-host-kubelet\") pod \"ovnkube-node-b8bxz\" (UID: \"644e19b3-10ad-4145-9ed3-595d19ed9b07\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8bxz" Oct 30 11:53:25.091808 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.091767 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3699031f-ee9e-46cc-b37e-4dfc76b9c0b1-lib-modules\") pod \"tuned-587rx\" (UID: \"3699031f-ee9e-46cc-b37e-4dfc76b9c0b1\") " pod="openshift-cluster-node-tuning-operator/tuned-587rx" Oct 30 11:53:25.091808 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.091789 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3699031f-ee9e-46cc-b37e-4dfc76b9c0b1-etc-tuned\") pod \"tuned-587rx\" (UID: \"3699031f-ee9e-46cc-b37e-4dfc76b9c0b1\") " pod="openshift-cluster-node-tuning-operator/tuned-587rx" Oct 30 11:53:25.092021 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.091814 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d659d471-f019-4853-97d0-ee6e948e6c1c-system-cni-dir\") pod \"multus-additional-cni-plugins-s6k8x\" (UID: \"d659d471-f019-4853-97d0-ee6e948e6c1c\") " pod="openshift-multus/multus-additional-cni-plugins-s6k8x" Oct 30 11:53:25.092021 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.091866 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0df1a118-36a5-4c57-8cec-2c8236f1c4de-multus-daemon-config\") pod \"multus-tlwp4\" (UID: \"0df1a118-36a5-4c57-8cec-2c8236f1c4de\") " pod="openshift-multus/multus-tlwp4" Oct 30 11:53:25.092021 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.091889 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6b350710-c48c-4c7e-bce9-e021099e98ca-original-pull-secret\") pod \"global-pull-secret-syncer-fxjlt\" (UID: \"6b350710-c48c-4c7e-bce9-e021099e98ca\") " pod="kube-system/global-pull-secret-syncer-fxjlt" Oct 30 11:53:25.092021 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.091912 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/644e19b3-10ad-4145-9ed3-595d19ed9b07-host-run-netns\") pod \"ovnkube-node-b8bxz\" (UID: \"644e19b3-10ad-4145-9ed3-595d19ed9b07\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8bxz" Oct 30 11:53:25.092021 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.091940 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/644e19b3-10ad-4145-9ed3-595d19ed9b07-var-lib-openvswitch\") pod \"ovnkube-node-b8bxz\" (UID: \"644e19b3-10ad-4145-9ed3-595d19ed9b07\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8bxz" Oct 30 11:53:25.092021 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.091964 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/644e19b3-10ad-4145-9ed3-595d19ed9b07-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-b8bxz\" (UID: \"644e19b3-10ad-4145-9ed3-595d19ed9b07\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8bxz" Oct 30 11:53:25.092021 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.091988 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/644e19b3-10ad-4145-9ed3-595d19ed9b07-ovn-node-metrics-cert\") pod \"ovnkube-node-b8bxz\" (UID: \"644e19b3-10ad-4145-9ed3-595d19ed9b07\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8bxz" Oct 30 11:53:25.092021 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.092010 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3699031f-ee9e-46cc-b37e-4dfc76b9c0b1-run\") pod \"tuned-587rx\" (UID: \"3699031f-ee9e-46cc-b37e-4dfc76b9c0b1\") " pod="openshift-cluster-node-tuning-operator/tuned-587rx" Oct 30 11:53:25.092374 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.092034 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d659d471-f019-4853-97d0-ee6e948e6c1c-cni-binary-copy\") pod \"multus-additional-cni-plugins-s6k8x\" (UID: \"d659d471-f019-4853-97d0-ee6e948e6c1c\") " pod="openshift-multus/multus-additional-cni-plugins-s6k8x" Oct 30 11:53:25.092374 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.092058 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rh4tb\" (UniqueName: \"kubernetes.io/projected/c54f4c1b-f08e-4495-9a14-17469a792ae5-kube-api-access-rh4tb\") pod \"aws-ebs-csi-driver-node-h6mlf\" (UID: \"c54f4c1b-f08e-4495-9a14-17469a792ae5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h6mlf" Oct 30 11:53:25.092374 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.092082 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/644e19b3-10ad-4145-9ed3-595d19ed9b07-run-systemd\") pod \"ovnkube-node-b8bxz\" (UID: \"644e19b3-10ad-4145-9ed3-595d19ed9b07\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8bxz" Oct 30 11:53:25.092374 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.092104 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3699031f-ee9e-46cc-b37e-4dfc76b9c0b1-etc-sysctl-d\") pod \"tuned-587rx\" (UID: \"3699031f-ee9e-46cc-b37e-4dfc76b9c0b1\") " pod="openshift-cluster-node-tuning-operator/tuned-587rx" Oct 30 11:53:25.092374 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.092140 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3699031f-ee9e-46cc-b37e-4dfc76b9c0b1-etc-systemd\") pod \"tuned-587rx\" (UID: \"3699031f-ee9e-46cc-b37e-4dfc76b9c0b1\") " pod="openshift-cluster-node-tuning-operator/tuned-587rx" Oct 30 11:53:25.092374 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.092161 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0df1a118-36a5-4c57-8cec-2c8236f1c4de-hostroot\") pod \"multus-tlwp4\" (UID: \"0df1a118-36a5-4c57-8cec-2c8236f1c4de\") " pod="openshift-multus/multus-tlwp4" Oct 30 11:53:25.092374 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.092182 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/644e19b3-10ad-4145-9ed3-595d19ed9b07-run-ovn\") pod \"ovnkube-node-b8bxz\" (UID: \"644e19b3-10ad-4145-9ed3-595d19ed9b07\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8bxz" Oct 30 11:53:25.092374 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.092206 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/644e19b3-10ad-4145-9ed3-595d19ed9b07-ovnkube-script-lib\") pod \"ovnkube-node-b8bxz\" (UID: \"644e19b3-10ad-4145-9ed3-595d19ed9b07\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8bxz" Oct 30 11:53:25.092374 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.092229 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3699031f-ee9e-46cc-b37e-4dfc76b9c0b1-etc-modprobe-d\") pod \"tuned-587rx\" (UID: \"3699031f-ee9e-46cc-b37e-4dfc76b9c0b1\") " pod="openshift-cluster-node-tuning-operator/tuned-587rx" Oct 30 11:53:25.092374 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.092279 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0df1a118-36a5-4c57-8cec-2c8236f1c4de-host-var-lib-cni-bin\") pod \"multus-tlwp4\" (UID: \"0df1a118-36a5-4c57-8cec-2c8236f1c4de\") " pod="openshift-multus/multus-tlwp4" Oct 30 11:53:25.092374 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.092302 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4e18f771-273b-49e6-872c-6cbe47ab68c4-agent-certs\") pod \"konnectivity-agent-tdf8d\" (UID: \"4e18f771-273b-49e6-872c-6cbe47ab68c4\") " pod="kube-system/konnectivity-agent-tdf8d" Oct 30 11:53:25.092374 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.092325 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/644e19b3-10ad-4145-9ed3-595d19ed9b07-host-run-ovn-kubernetes\") pod \"ovnkube-node-b8bxz\" (UID: \"644e19b3-10ad-4145-9ed3-595d19ed9b07\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8bxz" Oct 30 11:53:25.092374 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.092349 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/644e19b3-10ad-4145-9ed3-595d19ed9b07-ovnkube-config\") pod \"ovnkube-node-b8bxz\" (UID: \"644e19b3-10ad-4145-9ed3-595d19ed9b07\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8bxz" Oct 30 11:53:25.092374 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.092372 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0df1a118-36a5-4c57-8cec-2c8236f1c4de-cnibin\") pod \"multus-tlwp4\" (UID: \"0df1a118-36a5-4c57-8cec-2c8236f1c4de\") " pod="openshift-multus/multus-tlwp4" Oct 30 11:53:25.093021 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.092394 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0df1a118-36a5-4c57-8cec-2c8236f1c4de-cni-binary-copy\") pod \"multus-tlwp4\" (UID: \"0df1a118-36a5-4c57-8cec-2c8236f1c4de\") " pod="openshift-multus/multus-tlwp4" Oct 30 11:53:25.093021 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.092416 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0df1a118-36a5-4c57-8cec-2c8236f1c4de-multus-conf-dir\") pod \"multus-tlwp4\" (UID: \"0df1a118-36a5-4c57-8cec-2c8236f1c4de\") " pod="openshift-multus/multus-tlwp4" Oct 30 11:53:25.093021 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.092440 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0df1a118-36a5-4c57-8cec-2c8236f1c4de-etc-kubernetes\") pod \"multus-tlwp4\" (UID: \"0df1a118-36a5-4c57-8cec-2c8236f1c4de\") " pod="openshift-multus/multus-tlwp4" Oct 30 11:53:25.093021 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.092467 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/644e19b3-10ad-4145-9ed3-595d19ed9b07-run-openvswitch\") pod \"ovnkube-node-b8bxz\" (UID: \"644e19b3-10ad-4145-9ed3-595d19ed9b07\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8bxz" Oct 30 11:53:25.093021 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.092490 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/644e19b3-10ad-4145-9ed3-595d19ed9b07-host-cni-netd\") pod \"ovnkube-node-b8bxz\" (UID: \"644e19b3-10ad-4145-9ed3-595d19ed9b07\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8bxz" Oct 30 11:53:25.093021 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.092523 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b735438-5947-4374-bd4c-66ac3a299770-tmp-dir\") pod \"node-resolver-xmrdj\" (UID: \"1b735438-5947-4374-bd4c-66ac3a299770\") " pod="openshift-dns/node-resolver-xmrdj" Oct 30 11:53:25.093021 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.092547 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0df1a118-36a5-4c57-8cec-2c8236f1c4de-system-cni-dir\") pod \"multus-tlwp4\" (UID: \"0df1a118-36a5-4c57-8cec-2c8236f1c4de\") " pod="openshift-multus/multus-tlwp4" Oct 30 11:53:25.093021 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.092569 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6b350710-c48c-4c7e-bce9-e021099e98ca-kubelet-config\") pod \"global-pull-secret-syncer-fxjlt\" (UID: \"6b350710-c48c-4c7e-bce9-e021099e98ca\") " pod="kube-system/global-pull-secret-syncer-fxjlt" Oct 30 11:53:25.093021 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.092592 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d659d471-f019-4853-97d0-ee6e948e6c1c-cnibin\") pod \"multus-additional-cni-plugins-s6k8x\" (UID: \"d659d471-f019-4853-97d0-ee6e948e6c1c\") " pod="openshift-multus/multus-additional-cni-plugins-s6k8x" Oct 30 11:53:25.093021 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.092613 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/644e19b3-10ad-4145-9ed3-595d19ed9b07-log-socket\") pod \"ovnkube-node-b8bxz\" (UID: \"644e19b3-10ad-4145-9ed3-595d19ed9b07\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8bxz" Oct 30 11:53:25.093021 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.092638 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/644e19b3-10ad-4145-9ed3-595d19ed9b07-host-cni-bin\") pod \"ovnkube-node-b8bxz\" (UID: \"644e19b3-10ad-4145-9ed3-595d19ed9b07\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8bxz" Oct 30 11:53:25.093021 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.092661 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nzgq\" (UniqueName: \"kubernetes.io/projected/644e19b3-10ad-4145-9ed3-595d19ed9b07-kube-api-access-8nzgq\") pod \"ovnkube-node-b8bxz\" (UID: \"644e19b3-10ad-4145-9ed3-595d19ed9b07\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8bxz" Oct 30 11:53:25.093021 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.092693 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzc5h\" (UniqueName: \"kubernetes.io/projected/1b735438-5947-4374-bd4c-66ac3a299770-kube-api-access-gzc5h\") pod \"node-resolver-xmrdj\" (UID: \"1b735438-5947-4374-bd4c-66ac3a299770\") " pod="openshift-dns/node-resolver-xmrdj" Oct 30 11:53:25.093021 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.092720 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0df1a118-36a5-4c57-8cec-2c8236f1c4de-host-run-netns\") pod \"multus-tlwp4\" (UID: \"0df1a118-36a5-4c57-8cec-2c8236f1c4de\") " pod="openshift-multus/multus-tlwp4" Oct 30 11:53:25.093021 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.092744 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0df1a118-36a5-4c57-8cec-2c8236f1c4de-host-var-lib-cni-multus\") pod \"multus-tlwp4\" (UID: \"0df1a118-36a5-4c57-8cec-2c8236f1c4de\") " pod="openshift-multus/multus-tlwp4" Oct 30 11:53:25.093021 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.092769 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d659d471-f019-4853-97d0-ee6e948e6c1c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-s6k8x\" (UID: \"d659d471-f019-4853-97d0-ee6e948e6c1c\") " pod="openshift-multus/multus-additional-cni-plugins-s6k8x" Oct 30 11:53:25.093734 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.092795 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c54f4c1b-f08e-4495-9a14-17469a792ae5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-h6mlf\" (UID: \"c54f4c1b-f08e-4495-9a14-17469a792ae5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h6mlf" Oct 30 11:53:25.093734 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.092844 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzwcn\" (UniqueName: \"kubernetes.io/projected/1d7f8396-e57c-45e5-ae74-a2244db10364-kube-api-access-bzwcn\") pod \"network-metrics-daemon-dbqhj\" (UID: \"1d7f8396-e57c-45e5-ae74-a2244db10364\") " pod="openshift-multus/network-metrics-daemon-dbqhj" Oct 30 11:53:25.093734 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.092871 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3699031f-ee9e-46cc-b37e-4dfc76b9c0b1-host\") pod \"tuned-587rx\" (UID: \"3699031f-ee9e-46cc-b37e-4dfc76b9c0b1\") " pod="openshift-cluster-node-tuning-operator/tuned-587rx" Oct 30 11:53:25.093734 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.092905 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c54f4c1b-f08e-4495-9a14-17469a792ae5-registration-dir\") pod \"aws-ebs-csi-driver-node-h6mlf\" (UID: \"c54f4c1b-f08e-4495-9a14-17469a792ae5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h6mlf" Oct 30 11:53:25.093734 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.092939 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c54f4c1b-f08e-4495-9a14-17469a792ae5-device-dir\") pod \"aws-ebs-csi-driver-node-h6mlf\" (UID: \"c54f4c1b-f08e-4495-9a14-17469a792ae5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h6mlf" Oct 30 11:53:25.093734 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.092963 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/644e19b3-10ad-4145-9ed3-595d19ed9b07-node-log\") pod \"ovnkube-node-b8bxz\" (UID: \"644e19b3-10ad-4145-9ed3-595d19ed9b07\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8bxz" Oct 30 11:53:25.093734 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.092986 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/644e19b3-10ad-4145-9ed3-595d19ed9b07-etc-openvswitch\") pod \"ovnkube-node-b8bxz\" (UID: \"644e19b3-10ad-4145-9ed3-595d19ed9b07\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8bxz" Oct 30 11:53:25.093734 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.093012 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0df1a118-36a5-4c57-8cec-2c8236f1c4de-multus-socket-dir-parent\") pod \"multus-tlwp4\" (UID: \"0df1a118-36a5-4c57-8cec-2c8236f1c4de\") " pod="openshift-multus/multus-tlwp4" Oct 30 11:53:25.093734 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.093035 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0df1a118-36a5-4c57-8cec-2c8236f1c4de-host-run-multus-certs\") pod \"multus-tlwp4\" (UID: \"0df1a118-36a5-4c57-8cec-2c8236f1c4de\") " pod="openshift-multus/multus-tlwp4" Oct 30 11:53:25.093734 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.093059 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6b350710-c48c-4c7e-bce9-e021099e98ca-dbus\") pod \"global-pull-secret-syncer-fxjlt\" (UID: \"6b350710-c48c-4c7e-bce9-e021099e98ca\") " pod="kube-system/global-pull-secret-syncer-fxjlt" Oct 30 11:53:25.093734 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.093087 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4e18f771-273b-49e6-872c-6cbe47ab68c4-konnectivity-ca\") pod \"konnectivity-agent-tdf8d\" (UID: \"4e18f771-273b-49e6-872c-6cbe47ab68c4\") " pod="kube-system/konnectivity-agent-tdf8d" Oct 30 11:53:25.093734 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.093110 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5aee536a-8942-4b48-9b2c-3e36089f3ac5-serviceca\") pod \"node-ca-hb8j7\" (UID: \"5aee536a-8942-4b48-9b2c-3e36089f3ac5\") " pod="openshift-image-registry/node-ca-hb8j7" Oct 30 11:53:25.093734 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.093133 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmx84\" (UniqueName: \"kubernetes.io/projected/5aee536a-8942-4b48-9b2c-3e36089f3ac5-kube-api-access-vmx84\") pod \"node-ca-hb8j7\" (UID: \"5aee536a-8942-4b48-9b2c-3e36089f3ac5\") " pod="openshift-image-registry/node-ca-hb8j7" Oct 30 11:53:25.093734 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.093160 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d7f8396-e57c-45e5-ae74-a2244db10364-metrics-certs\") pod \"network-metrics-daemon-dbqhj\" (UID: \"1d7f8396-e57c-45e5-ae74-a2244db10364\") " pod="openshift-multus/network-metrics-daemon-dbqhj" Oct 30 11:53:25.093734 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.093184 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/644e19b3-10ad-4145-9ed3-595d19ed9b07-host-slash\") pod \"ovnkube-node-b8bxz\" (UID: \"644e19b3-10ad-4145-9ed3-595d19ed9b07\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8bxz" Oct 30 11:53:25.093734 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.093208 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/644e19b3-10ad-4145-9ed3-595d19ed9b07-env-overrides\") pod \"ovnkube-node-b8bxz\" (UID: \"644e19b3-10ad-4145-9ed3-595d19ed9b07\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8bxz" Oct 30 11:53:25.094453 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.093232 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3699031f-ee9e-46cc-b37e-4dfc76b9c0b1-etc-kubernetes\") pod \"tuned-587rx\" (UID: \"3699031f-ee9e-46cc-b37e-4dfc76b9c0b1\") " pod="openshift-cluster-node-tuning-operator/tuned-587rx" Oct 30 11:53:25.094453 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.093257 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3699031f-ee9e-46cc-b37e-4dfc76b9c0b1-etc-sysctl-conf\") pod \"tuned-587rx\" (UID: \"3699031f-ee9e-46cc-b37e-4dfc76b9c0b1\") " pod="openshift-cluster-node-tuning-operator/tuned-587rx" Oct 30 11:53:25.094453 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.093285 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d659d471-f019-4853-97d0-ee6e948e6c1c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-s6k8x\" (UID: \"d659d471-f019-4853-97d0-ee6e948e6c1c\") " pod="openshift-multus/multus-additional-cni-plugins-s6k8x" Oct 30 11:53:25.094453 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.093310 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c54f4c1b-f08e-4495-9a14-17469a792ae5-socket-dir\") pod \"aws-ebs-csi-driver-node-h6mlf\" (UID: \"c54f4c1b-f08e-4495-9a14-17469a792ae5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h6mlf" Oct 30 11:53:25.094453 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.093334 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5aee536a-8942-4b48-9b2c-3e36089f3ac5-host\") pod \"node-ca-hb8j7\" (UID: \"5aee536a-8942-4b48-9b2c-3e36089f3ac5\") " pod="openshift-image-registry/node-ca-hb8j7" Oct 30 11:53:25.094453 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.093360 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7e8ec7ff-015c-4049-9c5f-aae73dc06e4b-iptables-alerter-script\") pod \"iptables-alerter-8m2w9\" (UID: \"7e8ec7ff-015c-4049-9c5f-aae73dc06e4b\") " pod="openshift-network-operator/iptables-alerter-8m2w9" Oct 30 11:53:25.094453 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.093389 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7e8ec7ff-015c-4049-9c5f-aae73dc06e4b-host-slash\") pod \"iptables-alerter-8m2w9\" (UID: \"7e8ec7ff-015c-4049-9c5f-aae73dc06e4b\") " pod="openshift-network-operator/iptables-alerter-8m2w9" Oct 30 11:53:25.094453 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.093418 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3699031f-ee9e-46cc-b37e-4dfc76b9c0b1-sys\") pod \"tuned-587rx\" (UID: \"3699031f-ee9e-46cc-b37e-4dfc76b9c0b1\") " pod="openshift-cluster-node-tuning-operator/tuned-587rx" Oct 30 11:53:25.094453 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.093443 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7vnf\" (UniqueName: \"kubernetes.io/projected/3699031f-ee9e-46cc-b37e-4dfc76b9c0b1-kube-api-access-r7vnf\") pod \"tuned-587rx\" (UID: \"3699031f-ee9e-46cc-b37e-4dfc76b9c0b1\") " pod="openshift-cluster-node-tuning-operator/tuned-587rx" Oct 30 11:53:25.094453 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.093472 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zh8h9\" (UniqueName: \"kubernetes.io/projected/ff9feaa5-70cc-47fd-a3d1-a9ac6e7eaace-kube-api-access-zh8h9\") pod \"network-check-target-bz65k\" (UID: \"ff9feaa5-70cc-47fd-a3d1-a9ac6e7eaace\") " pod="openshift-network-diagnostics/network-check-target-bz65k" Oct 30 11:53:25.094921 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:25.094545 2545 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 11:53:25.094921 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:25.094607 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d7f8396-e57c-45e5-ae74-a2244db10364-metrics-certs podName:1d7f8396-e57c-45e5-ae74-a2244db10364 nodeName:}" failed. No retries permitted until 2026-10-30 11:53:25.594586267 +0000 UTC m=+3.035039488 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1d7f8396-e57c-45e5-ae74-a2244db10364-metrics-certs") pod "network-metrics-daemon-dbqhj" (UID: "1d7f8396-e57c-45e5-ae74-a2244db10364") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 11:53:25.108445 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.108369 2545 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 11:53:25.108540 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.108497 2545 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 11:53:25.108610 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.108587 2545 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-8n4dr\"" Oct 30 11:53:25.108720 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.108630 2545 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-sw2hc\"" Oct 30 11:53:25.108720 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.108662 2545 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-tz47t\"" Oct 30 11:53:25.108720 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.108696 2545 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-6hs8x\"" Oct 30 11:53:25.108839 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.108723 2545 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-llkx2\"" Oct 30 11:53:25.108839 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.108703 2545 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-pxw6w\"" Oct 30 11:53:25.108839 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.108738 2545 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-7cm7r\"" Oct 30 11:53:25.108839 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.108770 2545 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-jxkrq\"" Oct 30 11:53:25.109591 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.109573 2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 11:53:25.109674 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.109648 2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 11:53:25.109741 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.109723 2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 11:53:25.109823 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.109807 2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 11:53:25.109914 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.109900 2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 11:53:25.109967 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.109921 2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 11:53:25.110709 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.110626 2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 11:53:25.110709 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.110650 2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 11:53:25.110709 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.110675 2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 11:53:25.110709 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.110681 2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 11:53:25.110926 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.110732 2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 11:53:25.110926 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.110777 2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 11:53:25.111005 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.110964 2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 11:53:25.111005 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.110994 2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 11:53:25.116233 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.115930 2545 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 11:53:25.120208 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.120189 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzwcn\" (UniqueName: \"kubernetes.io/projected/1d7f8396-e57c-45e5-ae74-a2244db10364-kube-api-access-bzwcn\") pod \"network-metrics-daemon-dbqhj\" (UID: \"1d7f8396-e57c-45e5-ae74-a2244db10364\") " pod="openshift-multus/network-metrics-daemon-dbqhj" Oct 30 11:53:25.194637 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.194613 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4e18f771-273b-49e6-872c-6cbe47ab68c4-agent-certs\") pod \"konnectivity-agent-tdf8d\" (UID: \"4e18f771-273b-49e6-872c-6cbe47ab68c4\") " pod="kube-system/konnectivity-agent-tdf8d" Oct 30 11:53:25.194637 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.194642 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/644e19b3-10ad-4145-9ed3-595d19ed9b07-host-run-ovn-kubernetes\") pod \"ovnkube-node-b8bxz\" (UID: \"644e19b3-10ad-4145-9ed3-595d19ed9b07\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8bxz" Oct 30 11:53:25.194840 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.194657 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/644e19b3-10ad-4145-9ed3-595d19ed9b07-ovnkube-config\") pod \"ovnkube-node-b8bxz\" (UID: \"644e19b3-10ad-4145-9ed3-595d19ed9b07\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8bxz" Oct 30 11:53:25.194840 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.194672 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0df1a118-36a5-4c57-8cec-2c8236f1c4de-cnibin\") pod \"multus-tlwp4\" (UID: \"0df1a118-36a5-4c57-8cec-2c8236f1c4de\") " pod="openshift-multus/multus-tlwp4" Oct 30 11:53:25.194840 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.194686 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0df1a118-36a5-4c57-8cec-2c8236f1c4de-cni-binary-copy\") pod \"multus-tlwp4\" (UID: \"0df1a118-36a5-4c57-8cec-2c8236f1c4de\") " pod="openshift-multus/multus-tlwp4" Oct 30 11:53:25.194840 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.194756 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0df1a118-36a5-4c57-8cec-2c8236f1c4de-cnibin\") pod \"multus-tlwp4\" (UID: \"0df1a118-36a5-4c57-8cec-2c8236f1c4de\") " pod="openshift-multus/multus-tlwp4" Oct 30 11:53:25.194840 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.194795 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/644e19b3-10ad-4145-9ed3-595d19ed9b07-host-run-ovn-kubernetes\") pod \"ovnkube-node-b8bxz\" (UID: \"644e19b3-10ad-4145-9ed3-595d19ed9b07\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8bxz" Oct 30 11:53:25.195083 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.194844 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0df1a118-36a5-4c57-8cec-2c8236f1c4de-multus-conf-dir\") pod \"multus-tlwp4\" (UID: \"0df1a118-36a5-4c57-8cec-2c8236f1c4de\") " pod="openshift-multus/multus-tlwp4" Oct 30 11:53:25.195083 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.194882 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0df1a118-36a5-4c57-8cec-2c8236f1c4de-etc-kubernetes\") pod \"multus-tlwp4\" (UID: \"0df1a118-36a5-4c57-8cec-2c8236f1c4de\") " pod="openshift-multus/multus-tlwp4" Oct 30 11:53:25.195083 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.194891 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0df1a118-36a5-4c57-8cec-2c8236f1c4de-multus-conf-dir\") pod \"multus-tlwp4\" (UID: \"0df1a118-36a5-4c57-8cec-2c8236f1c4de\") " pod="openshift-multus/multus-tlwp4" Oct 30 11:53:25.195083 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.194909 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/644e19b3-10ad-4145-9ed3-595d19ed9b07-run-openvswitch\") pod \"ovnkube-node-b8bxz\" (UID: \"644e19b3-10ad-4145-9ed3-595d19ed9b07\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8bxz" Oct 30 11:53:25.195083 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.194937 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/644e19b3-10ad-4145-9ed3-595d19ed9b07-host-cni-netd\") pod \"ovnkube-node-b8bxz\" (UID: \"644e19b3-10ad-4145-9ed3-595d19ed9b07\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8bxz" Oct 30 11:53:25.195083 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.194946 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0df1a118-36a5-4c57-8cec-2c8236f1c4de-etc-kubernetes\") pod \"multus-tlwp4\" (UID: \"0df1a118-36a5-4c57-8cec-2c8236f1c4de\") " pod="openshift-multus/multus-tlwp4" Oct 30 11:53:25.195083 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.194961 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b735438-5947-4374-bd4c-66ac3a299770-tmp-dir\") pod \"node-resolver-xmrdj\" (UID: \"1b735438-5947-4374-bd4c-66ac3a299770\") " pod="openshift-dns/node-resolver-xmrdj" Oct 30 11:53:25.195083 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.194978 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/644e19b3-10ad-4145-9ed3-595d19ed9b07-run-openvswitch\") pod \"ovnkube-node-b8bxz\" (UID: \"644e19b3-10ad-4145-9ed3-595d19ed9b07\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8bxz" Oct 30 11:53:25.195083 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.194985 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0df1a118-36a5-4c57-8cec-2c8236f1c4de-system-cni-dir\") pod \"multus-tlwp4\" (UID: \"0df1a118-36a5-4c57-8cec-2c8236f1c4de\") " pod="openshift-multus/multus-tlwp4" Oct 30 11:53:25.195083 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.195004 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/644e19b3-10ad-4145-9ed3-595d19ed9b07-host-cni-netd\") pod \"ovnkube-node-b8bxz\" (UID: \"644e19b3-10ad-4145-9ed3-595d19ed9b07\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8bxz" Oct 30 11:53:25.195083 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.195009 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6b350710-c48c-4c7e-bce9-e021099e98ca-kubelet-config\") pod \"global-pull-secret-syncer-fxjlt\" (UID: \"6b350710-c48c-4c7e-bce9-e021099e98ca\") " pod="kube-system/global-pull-secret-syncer-fxjlt" Oct 30 11:53:25.195083 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.195032 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d659d471-f019-4853-97d0-ee6e948e6c1c-cnibin\") pod \"multus-additional-cni-plugins-s6k8x\" (UID: \"d659d471-f019-4853-97d0-ee6e948e6c1c\") " pod="openshift-multus/multus-additional-cni-plugins-s6k8x" Oct 30 11:53:25.195083 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.195055 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/644e19b3-10ad-4145-9ed3-595d19ed9b07-log-socket\") pod \"ovnkube-node-b8bxz\" (UID: \"644e19b3-10ad-4145-9ed3-595d19ed9b07\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8bxz" Oct 30 11:53:25.195083 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.195079 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/644e19b3-10ad-4145-9ed3-595d19ed9b07-host-cni-bin\") pod \"ovnkube-node-b8bxz\" (UID: \"644e19b3-10ad-4145-9ed3-595d19ed9b07\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8bxz" Oct 30 11:53:25.195710 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.195103 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nzgq\" (UniqueName: \"kubernetes.io/projected/644e19b3-10ad-4145-9ed3-595d19ed9b07-kube-api-access-8nzgq\") pod \"ovnkube-node-b8bxz\" (UID: \"644e19b3-10ad-4145-9ed3-595d19ed9b07\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8bxz" Oct 30 11:53:25.195710 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.195129 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzc5h\" (UniqueName: \"kubernetes.io/projected/1b735438-5947-4374-bd4c-66ac3a299770-kube-api-access-gzc5h\") pod \"node-resolver-xmrdj\" (UID: \"1b735438-5947-4374-bd4c-66ac3a299770\") " pod="openshift-dns/node-resolver-xmrdj" Oct 30 11:53:25.195710 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.195153 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0df1a118-36a5-4c57-8cec-2c8236f1c4de-host-run-netns\") pod \"multus-tlwp4\" (UID: \"0df1a118-36a5-4c57-8cec-2c8236f1c4de\") " pod="openshift-multus/multus-tlwp4" Oct 30 11:53:25.195710 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.195177 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0df1a118-36a5-4c57-8cec-2c8236f1c4de-host-var-lib-cni-multus\") pod \"multus-tlwp4\" (UID: \"0df1a118-36a5-4c57-8cec-2c8236f1c4de\") " pod="openshift-multus/multus-tlwp4" Oct 30 11:53:25.195710 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.195213 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d659d471-f019-4853-97d0-ee6e948e6c1c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-s6k8x\" (UID: \"d659d471-f019-4853-97d0-ee6e948e6c1c\") " pod="openshift-multus/multus-additional-cni-plugins-s6k8x" Oct 30 11:53:25.195710 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.195226 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b735438-5947-4374-bd4c-66ac3a299770-tmp-dir\") pod \"node-resolver-xmrdj\" (UID: \"1b735438-5947-4374-bd4c-66ac3a299770\") " pod="openshift-dns/node-resolver-xmrdj" Oct 30 11:53:25.195710 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.195238 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c54f4c1b-f08e-4495-9a14-17469a792ae5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-h6mlf\" (UID: \"c54f4c1b-f08e-4495-9a14-17469a792ae5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h6mlf" Oct 30 11:53:25.195710 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.195265 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3699031f-ee9e-46cc-b37e-4dfc76b9c0b1-host\") pod \"tuned-587rx\" (UID: \"3699031f-ee9e-46cc-b37e-4dfc76b9c0b1\") " pod="openshift-cluster-node-tuning-operator/tuned-587rx" Oct 30 11:53:25.195710 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.195292 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c54f4c1b-f08e-4495-9a14-17469a792ae5-registration-dir\") pod \"aws-ebs-csi-driver-node-h6mlf\" (UID: \"c54f4c1b-f08e-4495-9a14-17469a792ae5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h6mlf" Oct 30 11:53:25.195710 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.195345 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c54f4c1b-f08e-4495-9a14-17469a792ae5-device-dir\") pod \"aws-ebs-csi-driver-node-h6mlf\" (UID: \"c54f4c1b-f08e-4495-9a14-17469a792ae5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h6mlf" Oct 30 11:53:25.195710 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.195352 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/644e19b3-10ad-4145-9ed3-595d19ed9b07-ovnkube-config\") pod \"ovnkube-node-b8bxz\" (UID: \"644e19b3-10ad-4145-9ed3-595d19ed9b07\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8bxz" Oct 30 11:53:25.195710 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.195368 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/644e19b3-10ad-4145-9ed3-595d19ed9b07-node-log\") pod \"ovnkube-node-b8bxz\" (UID: \"644e19b3-10ad-4145-9ed3-595d19ed9b07\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8bxz" Oct 30 11:53:25.195710 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.195393 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/644e19b3-10ad-4145-9ed3-595d19ed9b07-etc-openvswitch\") pod \"ovnkube-node-b8bxz\" (UID: \"644e19b3-10ad-4145-9ed3-595d19ed9b07\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8bxz" Oct 30 11:53:25.195710 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.195402 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0df1a118-36a5-4c57-8cec-2c8236f1c4de-system-cni-dir\") pod \"multus-tlwp4\" (UID: \"0df1a118-36a5-4c57-8cec-2c8236f1c4de\") " pod="openshift-multus/multus-tlwp4" Oct 30 11:53:25.195710 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.195418 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0df1a118-36a5-4c57-8cec-2c8236f1c4de-multus-socket-dir-parent\") pod \"multus-tlwp4\" (UID: \"0df1a118-36a5-4c57-8cec-2c8236f1c4de\") " pod="openshift-multus/multus-tlwp4" Oct 30 11:53:25.195710 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.195442 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0df1a118-36a5-4c57-8cec-2c8236f1c4de-host-run-multus-certs\") pod \"multus-tlwp4\" (UID: \"0df1a118-36a5-4c57-8cec-2c8236f1c4de\") " pod="openshift-multus/multus-tlwp4" Oct 30 11:53:25.195710 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.195448 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/644e19b3-10ad-4145-9ed3-595d19ed9b07-host-cni-bin\") pod \"ovnkube-node-b8bxz\" (UID: \"644e19b3-10ad-4145-9ed3-595d19ed9b07\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8bxz" Oct 30 11:53:25.196500 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.195472 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6b350710-c48c-4c7e-bce9-e021099e98ca-dbus\") pod \"global-pull-secret-syncer-fxjlt\" (UID: \"6b350710-c48c-4c7e-bce9-e021099e98ca\") " pod="kube-system/global-pull-secret-syncer-fxjlt" Oct 30 11:53:25.196500 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.195495 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4e18f771-273b-49e6-872c-6cbe47ab68c4-konnectivity-ca\") pod \"konnectivity-agent-tdf8d\" (UID: \"4e18f771-273b-49e6-872c-6cbe47ab68c4\") " pod="kube-system/konnectivity-agent-tdf8d" Oct 30 11:53:25.196500 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.195501 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d659d471-f019-4853-97d0-ee6e948e6c1c-cnibin\") pod \"multus-additional-cni-plugins-s6k8x\" (UID: \"d659d471-f019-4853-97d0-ee6e948e6c1c\") " pod="openshift-multus/multus-additional-cni-plugins-s6k8x" Oct 30 11:53:25.196500 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.195520 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5aee536a-8942-4b48-9b2c-3e36089f3ac5-serviceca\") pod \"node-ca-hb8j7\" (UID: \"5aee536a-8942-4b48-9b2c-3e36089f3ac5\") " pod="openshift-image-registry/node-ca-hb8j7" Oct 30 11:53:25.196500 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.195540 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/644e19b3-10ad-4145-9ed3-595d19ed9b07-log-socket\") pod \"ovnkube-node-b8bxz\" (UID: \"644e19b3-10ad-4145-9ed3-595d19ed9b07\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8bxz" Oct 30 11:53:25.196500 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.195543 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmx84\" (UniqueName: \"kubernetes.io/projected/5aee536a-8942-4b48-9b2c-3e36089f3ac5-kube-api-access-vmx84\") pod \"node-ca-hb8j7\" (UID: \"5aee536a-8942-4b48-9b2c-3e36089f3ac5\") " pod="openshift-image-registry/node-ca-hb8j7" Oct 30 11:53:25.196500 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.195595 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/644e19b3-10ad-4145-9ed3-595d19ed9b07-host-slash\") pod \"ovnkube-node-b8bxz\" (UID: \"644e19b3-10ad-4145-9ed3-595d19ed9b07\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8bxz" Oct 30 11:53:25.196500 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.195608 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6b350710-c48c-4c7e-bce9-e021099e98ca-kubelet-config\") pod \"global-pull-secret-syncer-fxjlt\" (UID: \"6b350710-c48c-4c7e-bce9-e021099e98ca\") " pod="kube-system/global-pull-secret-syncer-fxjlt" Oct 30 11:53:25.196500 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.195621 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/644e19b3-10ad-4145-9ed3-595d19ed9b07-env-overrides\") pod \"ovnkube-node-b8bxz\" (UID: \"644e19b3-10ad-4145-9ed3-595d19ed9b07\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8bxz" Oct 30 11:53:25.196500 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.195673 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3699031f-ee9e-46cc-b37e-4dfc76b9c0b1-etc-kubernetes\") pod \"tuned-587rx\" (UID: \"3699031f-ee9e-46cc-b37e-4dfc76b9c0b1\") " pod="openshift-cluster-node-tuning-operator/tuned-587rx" Oct 30 11:53:25.196500 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.195693 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/644e19b3-10ad-4145-9ed3-595d19ed9b07-node-log\") pod \"ovnkube-node-b8bxz\" (UID: \"644e19b3-10ad-4145-9ed3-595d19ed9b07\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8bxz" Oct 30 11:53:25.196500 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.195707 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3699031f-ee9e-46cc-b37e-4dfc76b9c0b1-etc-sysctl-conf\") pod \"tuned-587rx\" (UID: \"3699031f-ee9e-46cc-b37e-4dfc76b9c0b1\") " pod="openshift-cluster-node-tuning-operator/tuned-587rx" Oct 30 11:53:25.196500 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.195729 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c54f4c1b-f08e-4495-9a14-17469a792ae5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-h6mlf\" (UID: \"c54f4c1b-f08e-4495-9a14-17469a792ae5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h6mlf" Oct 30 11:53:25.196500 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.195735 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d659d471-f019-4853-97d0-ee6e948e6c1c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-s6k8x\" (UID: \"d659d471-f019-4853-97d0-ee6e948e6c1c\") " pod="openshift-multus/multus-additional-cni-plugins-s6k8x" Oct 30 11:53:25.196500 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.195763 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c54f4c1b-f08e-4495-9a14-17469a792ae5-socket-dir\") pod \"aws-ebs-csi-driver-node-h6mlf\" (UID: \"c54f4c1b-f08e-4495-9a14-17469a792ae5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h6mlf" Oct 30 11:53:25.196500 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.195792 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c54f4c1b-f08e-4495-9a14-17469a792ae5-registration-dir\") pod \"aws-ebs-csi-driver-node-h6mlf\" (UID: \"c54f4c1b-f08e-4495-9a14-17469a792ae5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h6mlf" Oct 30 11:53:25.196500 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.195792 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5aee536a-8942-4b48-9b2c-3e36089f3ac5-host\") pod \"node-ca-hb8j7\" (UID: \"5aee536a-8942-4b48-9b2c-3e36089f3ac5\") " pod="openshift-image-registry/node-ca-hb8j7" Oct 30 11:53:25.197335 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.195818 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7e8ec7ff-015c-4049-9c5f-aae73dc06e4b-iptables-alerter-script\") pod \"iptables-alerter-8m2w9\" (UID: \"7e8ec7ff-015c-4049-9c5f-aae73dc06e4b\") " pod="openshift-network-operator/iptables-alerter-8m2w9" Oct 30 11:53:25.197335 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.195842 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5aee536a-8942-4b48-9b2c-3e36089f3ac5-host\") pod \"node-ca-hb8j7\" (UID: \"5aee536a-8942-4b48-9b2c-3e36089f3ac5\") " pod="openshift-image-registry/node-ca-hb8j7" Oct 30 11:53:25.197335 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.195855 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7e8ec7ff-015c-4049-9c5f-aae73dc06e4b-host-slash\") pod \"iptables-alerter-8m2w9\" (UID: \"7e8ec7ff-015c-4049-9c5f-aae73dc06e4b\") " pod="openshift-network-operator/iptables-alerter-8m2w9" Oct 30 11:53:25.197335 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.195895 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7e8ec7ff-015c-4049-9c5f-aae73dc06e4b-host-slash\") pod \"iptables-alerter-8m2w9\" (UID: \"7e8ec7ff-015c-4049-9c5f-aae73dc06e4b\") " pod="openshift-network-operator/iptables-alerter-8m2w9" Oct 30 11:53:25.197335 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.195886 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c54f4c1b-f08e-4495-9a14-17469a792ae5-device-dir\") pod \"aws-ebs-csi-driver-node-h6mlf\" (UID: \"c54f4c1b-f08e-4495-9a14-17469a792ae5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h6mlf" Oct 30 11:53:25.197335 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.195428 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0df1a118-36a5-4c57-8cec-2c8236f1c4de-host-var-lib-cni-multus\") pod \"multus-tlwp4\" (UID: \"0df1a118-36a5-4c57-8cec-2c8236f1c4de\") " pod="openshift-multus/multus-tlwp4" Oct 30 11:53:25.197335 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.195345 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0df1a118-36a5-4c57-8cec-2c8236f1c4de-cni-binary-copy\") pod \"multus-tlwp4\" (UID: \"0df1a118-36a5-4c57-8cec-2c8236f1c4de\") " pod="openshift-multus/multus-tlwp4" Oct 30 11:53:25.197335 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.195942 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3699031f-ee9e-46cc-b37e-4dfc76b9c0b1-etc-kubernetes\") pod \"tuned-587rx\" (UID: \"3699031f-ee9e-46cc-b37e-4dfc76b9c0b1\") " pod="openshift-cluster-node-tuning-operator/tuned-587rx" Oct 30 11:53:25.197335 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.196006 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/644e19b3-10ad-4145-9ed3-595d19ed9b07-env-overrides\") pod \"ovnkube-node-b8bxz\" (UID: \"644e19b3-10ad-4145-9ed3-595d19ed9b07\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8bxz" Oct 30 11:53:25.197335 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.196027 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0df1a118-36a5-4c57-8cec-2c8236f1c4de-multus-socket-dir-parent\") pod \"multus-tlwp4\" (UID: \"0df1a118-36a5-4c57-8cec-2c8236f1c4de\") " pod="openshift-multus/multus-tlwp4" Oct 30 11:53:25.197335 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.196066 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3699031f-ee9e-46cc-b37e-4dfc76b9c0b1-etc-sysctl-conf\") pod \"tuned-587rx\" (UID: \"3699031f-ee9e-46cc-b37e-4dfc76b9c0b1\") " pod="openshift-cluster-node-tuning-operator/tuned-587rx" Oct 30 11:53:25.197335 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.196100 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d659d471-f019-4853-97d0-ee6e948e6c1c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-s6k8x\" (UID: \"d659d471-f019-4853-97d0-ee6e948e6c1c\") " pod="openshift-multus/multus-additional-cni-plugins-s6k8x" Oct 30 11:53:25.197335 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.196199 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d659d471-f019-4853-97d0-ee6e948e6c1c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-s6k8x\" (UID: \"d659d471-f019-4853-97d0-ee6e948e6c1c\") " pod="openshift-multus/multus-additional-cni-plugins-s6k8x" Oct 30 11:53:25.197335 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.196258 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/644e19b3-10ad-4145-9ed3-595d19ed9b07-host-slash\") pod \"ovnkube-node-b8bxz\" (UID: \"644e19b3-10ad-4145-9ed3-595d19ed9b07\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8bxz" Oct 30 11:53:25.197335 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.195763 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3699031f-ee9e-46cc-b37e-4dfc76b9c0b1-host\") pod \"tuned-587rx\" (UID: \"3699031f-ee9e-46cc-b37e-4dfc76b9c0b1\") " pod="openshift-cluster-node-tuning-operator/tuned-587rx" Oct 30 11:53:25.197335 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.196326 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0df1a118-36a5-4c57-8cec-2c8236f1c4de-host-run-multus-certs\") pod \"multus-tlwp4\" (UID: \"0df1a118-36a5-4c57-8cec-2c8236f1c4de\") " pod="openshift-multus/multus-tlwp4" Oct 30 11:53:25.197335 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.196350 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5aee536a-8942-4b48-9b2c-3e36089f3ac5-serviceca\") pod \"node-ca-hb8j7\" (UID: \"5aee536a-8942-4b48-9b2c-3e36089f3ac5\") " pod="openshift-image-registry/node-ca-hb8j7" Oct 30 11:53:25.197335 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.196422 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3699031f-ee9e-46cc-b37e-4dfc76b9c0b1-sys\") pod \"tuned-587rx\" (UID: \"3699031f-ee9e-46cc-b37e-4dfc76b9c0b1\") " pod="openshift-cluster-node-tuning-operator/tuned-587rx" Oct 30 11:53:25.197994 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.196436 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c54f4c1b-f08e-4495-9a14-17469a792ae5-socket-dir\") pod \"aws-ebs-csi-driver-node-h6mlf\" (UID: \"c54f4c1b-f08e-4495-9a14-17469a792ae5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h6mlf" Oct 30 11:53:25.197994 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.196450 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7vnf\" (UniqueName: \"kubernetes.io/projected/3699031f-ee9e-46cc-b37e-4dfc76b9c0b1-kube-api-access-r7vnf\") pod \"tuned-587rx\" (UID: \"3699031f-ee9e-46cc-b37e-4dfc76b9c0b1\") " pod="openshift-cluster-node-tuning-operator/tuned-587rx" Oct 30 11:53:25.197994 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.196456 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6b350710-c48c-4c7e-bce9-e021099e98ca-dbus\") pod \"global-pull-secret-syncer-fxjlt\" (UID: \"6b350710-c48c-4c7e-bce9-e021099e98ca\") " pod="kube-system/global-pull-secret-syncer-fxjlt" Oct 30 11:53:25.197994 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.196474 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/644e19b3-10ad-4145-9ed3-595d19ed9b07-etc-openvswitch\") pod \"ovnkube-node-b8bxz\" (UID: \"644e19b3-10ad-4145-9ed3-595d19ed9b07\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8bxz" Oct 30 11:53:25.197994 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.196476 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4e18f771-273b-49e6-872c-6cbe47ab68c4-konnectivity-ca\") pod \"konnectivity-agent-tdf8d\" (UID: \"4e18f771-273b-49e6-872c-6cbe47ab68c4\") " pod="kube-system/konnectivity-agent-tdf8d" Oct 30 11:53:25.197994 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.196515 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zh8h9\" (UniqueName: \"kubernetes.io/projected/ff9feaa5-70cc-47fd-a3d1-a9ac6e7eaace-kube-api-access-zh8h9\") pod \"network-check-target-bz65k\" (UID: \"ff9feaa5-70cc-47fd-a3d1-a9ac6e7eaace\") " pod="openshift-network-diagnostics/network-check-target-bz65k" Oct 30 11:53:25.197994 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.196553 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3699031f-ee9e-46cc-b37e-4dfc76b9c0b1-sys\") pod \"tuned-587rx\" (UID: \"3699031f-ee9e-46cc-b37e-4dfc76b9c0b1\") " pod="openshift-cluster-node-tuning-operator/tuned-587rx" Oct 30 11:53:25.197994 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.196535 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0df1a118-36a5-4c57-8cec-2c8236f1c4de-host-run-netns\") pod \"multus-tlwp4\" (UID: \"0df1a118-36a5-4c57-8cec-2c8236f1c4de\") " pod="openshift-multus/multus-tlwp4" Oct 30 11:53:25.197994 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.196587 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92smt\" (UniqueName: \"kubernetes.io/projected/7e8ec7ff-015c-4049-9c5f-aae73dc06e4b-kube-api-access-92smt\") pod \"iptables-alerter-8m2w9\" (UID: \"7e8ec7ff-015c-4049-9c5f-aae73dc06e4b\") " pod="openshift-network-operator/iptables-alerter-8m2w9" Oct 30 11:53:25.197994 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.196612 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3699031f-ee9e-46cc-b37e-4dfc76b9c0b1-etc-sysconfig\") pod \"tuned-587rx\" (UID: \"3699031f-ee9e-46cc-b37e-4dfc76b9c0b1\") " pod="openshift-cluster-node-tuning-operator/tuned-587rx" Oct 30 11:53:25.197994 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.196685 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7e8ec7ff-015c-4049-9c5f-aae73dc06e4b-iptables-alerter-script\") pod \"iptables-alerter-8m2w9\" (UID: \"7e8ec7ff-015c-4049-9c5f-aae73dc06e4b\") " pod="openshift-network-operator/iptables-alerter-8m2w9" Oct 30 11:53:25.197994 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.196717 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4tdc\" (UniqueName: \"kubernetes.io/projected/0df1a118-36a5-4c57-8cec-2c8236f1c4de-kube-api-access-r4tdc\") pod \"multus-tlwp4\" (UID: \"0df1a118-36a5-4c57-8cec-2c8236f1c4de\") " pod="openshift-multus/multus-tlwp4" Oct 30 11:53:25.197994 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.196749 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3699031f-ee9e-46cc-b37e-4dfc76b9c0b1-etc-sysconfig\") pod \"tuned-587rx\" (UID: \"3699031f-ee9e-46cc-b37e-4dfc76b9c0b1\") " pod="openshift-cluster-node-tuning-operator/tuned-587rx" Oct 30 11:53:25.197994 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.196797 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d659d471-f019-4853-97d0-ee6e948e6c1c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-s6k8x\" (UID: \"d659d471-f019-4853-97d0-ee6e948e6c1c\") " pod="openshift-multus/multus-additional-cni-plugins-s6k8x" Oct 30 11:53:25.197994 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.196843 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stff7\" (UniqueName: \"kubernetes.io/projected/d659d471-f019-4853-97d0-ee6e948e6c1c-kube-api-access-stff7\") pod \"multus-additional-cni-plugins-s6k8x\" (UID: \"d659d471-f019-4853-97d0-ee6e948e6c1c\") " pod="openshift-multus/multus-additional-cni-plugins-s6k8x" Oct 30 11:53:25.197994 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.196880 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/644e19b3-10ad-4145-9ed3-595d19ed9b07-systemd-units\") pod \"ovnkube-node-b8bxz\" (UID: \"644e19b3-10ad-4145-9ed3-595d19ed9b07\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8bxz" Oct 30 11:53:25.197994 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.196904 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0df1a118-36a5-4c57-8cec-2c8236f1c4de-host-var-lib-kubelet\") pod \"multus-tlwp4\" (UID: \"0df1a118-36a5-4c57-8cec-2c8236f1c4de\") " pod="openshift-multus/multus-tlwp4" Oct 30 11:53:25.198771 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.196952 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0df1a118-36a5-4c57-8cec-2c8236f1c4de-host-var-lib-kubelet\") pod \"multus-tlwp4\" (UID: \"0df1a118-36a5-4c57-8cec-2c8236f1c4de\") " pod="openshift-multus/multus-tlwp4" Oct 30 11:53:25.198771 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.196964 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/644e19b3-10ad-4145-9ed3-595d19ed9b07-systemd-units\") pod \"ovnkube-node-b8bxz\" (UID: \"644e19b3-10ad-4145-9ed3-595d19ed9b07\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8bxz" Oct 30 11:53:25.198771 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.196988 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3699031f-ee9e-46cc-b37e-4dfc76b9c0b1-var-lib-kubelet\") pod \"tuned-587rx\" (UID: \"3699031f-ee9e-46cc-b37e-4dfc76b9c0b1\") " pod="openshift-cluster-node-tuning-operator/tuned-587rx" Oct 30 11:53:25.198771 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.197013 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3699031f-ee9e-46cc-b37e-4dfc76b9c0b1-tmp\") pod \"tuned-587rx\" (UID: \"3699031f-ee9e-46cc-b37e-4dfc76b9c0b1\") " pod="openshift-cluster-node-tuning-operator/tuned-587rx" Oct 30 11:53:25.198771 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.197070 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3699031f-ee9e-46cc-b37e-4dfc76b9c0b1-var-lib-kubelet\") pod \"tuned-587rx\" (UID: \"3699031f-ee9e-46cc-b37e-4dfc76b9c0b1\") " pod="openshift-cluster-node-tuning-operator/tuned-587rx" Oct 30 11:53:25.198771 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.197099 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1b735438-5947-4374-bd4c-66ac3a299770-hosts-file\") pod \"node-resolver-xmrdj\" (UID: \"1b735438-5947-4374-bd4c-66ac3a299770\") " pod="openshift-dns/node-resolver-xmrdj" Oct 30 11:53:25.198771 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.197126 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0df1a118-36a5-4c57-8cec-2c8236f1c4de-multus-cni-dir\") pod \"multus-tlwp4\" (UID: \"0df1a118-36a5-4c57-8cec-2c8236f1c4de\") " pod="openshift-multus/multus-tlwp4" Oct 30 11:53:25.198771 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.197179 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0df1a118-36a5-4c57-8cec-2c8236f1c4de-host-run-k8s-cni-cncf-io\") pod \"multus-tlwp4\" (UID: \"0df1a118-36a5-4c57-8cec-2c8236f1c4de\") " pod="openshift-multus/multus-tlwp4" Oct 30 11:53:25.198771 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.197183 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0df1a118-36a5-4c57-8cec-2c8236f1c4de-multus-cni-dir\") pod \"multus-tlwp4\" (UID: \"0df1a118-36a5-4c57-8cec-2c8236f1c4de\") " pod="openshift-multus/multus-tlwp4" Oct 30 11:53:25.198771 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.197186 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1b735438-5947-4374-bd4c-66ac3a299770-hosts-file\") pod \"node-resolver-xmrdj\" (UID: \"1b735438-5947-4374-bd4c-66ac3a299770\") " pod="openshift-dns/node-resolver-xmrdj" Oct 30 11:53:25.198771 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.197210 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c54f4c1b-f08e-4495-9a14-17469a792ae5-etc-selinux\") pod \"aws-ebs-csi-driver-node-h6mlf\" (UID: \"c54f4c1b-f08e-4495-9a14-17469a792ae5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h6mlf" Oct 30 11:53:25.198771 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.197249 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c54f4c1b-f08e-4495-9a14-17469a792ae5-sys-fs\") pod \"aws-ebs-csi-driver-node-h6mlf\" (UID: \"c54f4c1b-f08e-4495-9a14-17469a792ae5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h6mlf" Oct 30 11:53:25.198771 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.197287 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/644e19b3-10ad-4145-9ed3-595d19ed9b07-host-kubelet\") pod \"ovnkube-node-b8bxz\" (UID: \"644e19b3-10ad-4145-9ed3-595d19ed9b07\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8bxz" Oct 30 11:53:25.198771 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.197292 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c54f4c1b-f08e-4495-9a14-17469a792ae5-etc-selinux\") pod \"aws-ebs-csi-driver-node-h6mlf\" (UID: \"c54f4c1b-f08e-4495-9a14-17469a792ae5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h6mlf" Oct 30 11:53:25.198771 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.197323 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3699031f-ee9e-46cc-b37e-4dfc76b9c0b1-lib-modules\") pod \"tuned-587rx\" (UID: \"3699031f-ee9e-46cc-b37e-4dfc76b9c0b1\") " pod="openshift-cluster-node-tuning-operator/tuned-587rx" Oct 30 11:53:25.198771 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.197333 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0df1a118-36a5-4c57-8cec-2c8236f1c4de-host-run-k8s-cni-cncf-io\") pod \"multus-tlwp4\" (UID: \"0df1a118-36a5-4c57-8cec-2c8236f1c4de\") " pod="openshift-multus/multus-tlwp4" Oct 30 11:53:25.198771 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.197336 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c54f4c1b-f08e-4495-9a14-17469a792ae5-sys-fs\") pod \"aws-ebs-csi-driver-node-h6mlf\" (UID: \"c54f4c1b-f08e-4495-9a14-17469a792ae5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h6mlf" Oct 30 11:53:25.198771 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.197371 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3699031f-ee9e-46cc-b37e-4dfc76b9c0b1-etc-tuned\") pod \"tuned-587rx\" (UID: \"3699031f-ee9e-46cc-b37e-4dfc76b9c0b1\") " pod="openshift-cluster-node-tuning-operator/tuned-587rx" Oct 30 11:53:25.199637 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.197380 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/644e19b3-10ad-4145-9ed3-595d19ed9b07-host-kubelet\") pod \"ovnkube-node-b8bxz\" (UID: \"644e19b3-10ad-4145-9ed3-595d19ed9b07\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8bxz" Oct 30 11:53:25.199637 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.197401 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d659d471-f019-4853-97d0-ee6e948e6c1c-system-cni-dir\") pod \"multus-additional-cni-plugins-s6k8x\" (UID: \"d659d471-f019-4853-97d0-ee6e948e6c1c\") " pod="openshift-multus/multus-additional-cni-plugins-s6k8x" Oct 30 11:53:25.199637 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.197441 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0df1a118-36a5-4c57-8cec-2c8236f1c4de-multus-daemon-config\") pod \"multus-tlwp4\" (UID: \"0df1a118-36a5-4c57-8cec-2c8236f1c4de\") " pod="openshift-multus/multus-tlwp4" Oct 30 11:53:25.199637 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.197466 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6b350710-c48c-4c7e-bce9-e021099e98ca-original-pull-secret\") pod \"global-pull-secret-syncer-fxjlt\" (UID: \"6b350710-c48c-4c7e-bce9-e021099e98ca\") " pod="kube-system/global-pull-secret-syncer-fxjlt" Oct 30 11:53:25.199637 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.197477 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3699031f-ee9e-46cc-b37e-4dfc76b9c0b1-lib-modules\") pod \"tuned-587rx\" (UID: \"3699031f-ee9e-46cc-b37e-4dfc76b9c0b1\") " pod="openshift-cluster-node-tuning-operator/tuned-587rx" Oct 30 11:53:25.199637 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.197490 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/644e19b3-10ad-4145-9ed3-595d19ed9b07-host-run-netns\") pod \"ovnkube-node-b8bxz\" (UID: \"644e19b3-10ad-4145-9ed3-595d19ed9b07\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8bxz" Oct 30 11:53:25.199637 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.197510 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/644e19b3-10ad-4145-9ed3-595d19ed9b07-var-lib-openvswitch\") pod \"ovnkube-node-b8bxz\" (UID: \"644e19b3-10ad-4145-9ed3-595d19ed9b07\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8bxz" Oct 30 11:53:25.199637 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.197533 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/644e19b3-10ad-4145-9ed3-595d19ed9b07-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-b8bxz\" (UID: \"644e19b3-10ad-4145-9ed3-595d19ed9b07\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8bxz" Oct 30 11:53:25.199637 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.197553 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/644e19b3-10ad-4145-9ed3-595d19ed9b07-ovn-node-metrics-cert\") pod \"ovnkube-node-b8bxz\" (UID: \"644e19b3-10ad-4145-9ed3-595d19ed9b07\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8bxz" Oct 30 11:53:25.199637 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:25.197567 2545 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 11:53:25.199637 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.197569 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3699031f-ee9e-46cc-b37e-4dfc76b9c0b1-run\") pod \"tuned-587rx\" (UID: \"3699031f-ee9e-46cc-b37e-4dfc76b9c0b1\") " pod="openshift-cluster-node-tuning-operator/tuned-587rx" Oct 30 11:53:25.199637 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.197604 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3699031f-ee9e-46cc-b37e-4dfc76b9c0b1-run\") pod \"tuned-587rx\" (UID: \"3699031f-ee9e-46cc-b37e-4dfc76b9c0b1\") " pod="openshift-cluster-node-tuning-operator/tuned-587rx" Oct 30 11:53:25.199637 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.197601 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d659d471-f019-4853-97d0-ee6e948e6c1c-cni-binary-copy\") pod \"multus-additional-cni-plugins-s6k8x\" (UID: \"d659d471-f019-4853-97d0-ee6e948e6c1c\") " pod="openshift-multus/multus-additional-cni-plugins-s6k8x" Oct 30 11:53:25.199637 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.197629 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/644e19b3-10ad-4145-9ed3-595d19ed9b07-var-lib-openvswitch\") pod \"ovnkube-node-b8bxz\" (UID: \"644e19b3-10ad-4145-9ed3-595d19ed9b07\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8bxz" Oct 30 11:53:25.199637 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.197630 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rh4tb\" (UniqueName: \"kubernetes.io/projected/c54f4c1b-f08e-4495-9a14-17469a792ae5-kube-api-access-rh4tb\") pod \"aws-ebs-csi-driver-node-h6mlf\" (UID: \"c54f4c1b-f08e-4495-9a14-17469a792ae5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h6mlf" Oct 30 11:53:25.199637 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.197650 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/644e19b3-10ad-4145-9ed3-595d19ed9b07-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-b8bxz\" (UID: \"644e19b3-10ad-4145-9ed3-595d19ed9b07\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8bxz" Oct 30 11:53:25.199637 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.197655 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/644e19b3-10ad-4145-9ed3-595d19ed9b07-run-systemd\") pod \"ovnkube-node-b8bxz\" (UID: \"644e19b3-10ad-4145-9ed3-595d19ed9b07\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8bxz" Oct 30 11:53:25.200416 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.197681 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3699031f-ee9e-46cc-b37e-4dfc76b9c0b1-etc-sysctl-d\") pod \"tuned-587rx\" (UID: \"3699031f-ee9e-46cc-b37e-4dfc76b9c0b1\") " pod="openshift-cluster-node-tuning-operator/tuned-587rx" Oct 30 11:53:25.200416 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:25.197689 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b350710-c48c-4c7e-bce9-e021099e98ca-original-pull-secret podName:6b350710-c48c-4c7e-bce9-e021099e98ca nodeName:}" failed. No retries permitted until 2026-10-30 11:53:25.697674787 +0000 UTC m=+3.138128014 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6b350710-c48c-4c7e-bce9-e021099e98ca-original-pull-secret") pod "global-pull-secret-syncer-fxjlt" (UID: "6b350710-c48c-4c7e-bce9-e021099e98ca") : object "kube-system"/"original-pull-secret" not registered Oct 30 11:53:25.200416 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.197776 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3699031f-ee9e-46cc-b37e-4dfc76b9c0b1-etc-sysctl-d\") pod \"tuned-587rx\" (UID: \"3699031f-ee9e-46cc-b37e-4dfc76b9c0b1\") " pod="openshift-cluster-node-tuning-operator/tuned-587rx" Oct 30 11:53:25.200416 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.197819 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/644e19b3-10ad-4145-9ed3-595d19ed9b07-host-run-netns\") pod \"ovnkube-node-b8bxz\" (UID: \"644e19b3-10ad-4145-9ed3-595d19ed9b07\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8bxz" Oct 30 11:53:25.200416 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.197851 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/644e19b3-10ad-4145-9ed3-595d19ed9b07-run-systemd\") pod \"ovnkube-node-b8bxz\" (UID: \"644e19b3-10ad-4145-9ed3-595d19ed9b07\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8bxz" Oct 30 11:53:25.200416 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.197869 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3699031f-ee9e-46cc-b37e-4dfc76b9c0b1-etc-systemd\") pod \"tuned-587rx\" (UID: \"3699031f-ee9e-46cc-b37e-4dfc76b9c0b1\") " pod="openshift-cluster-node-tuning-operator/tuned-587rx" Oct 30 11:53:25.200416 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.197893 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0df1a118-36a5-4c57-8cec-2c8236f1c4de-hostroot\") pod \"multus-tlwp4\" (UID: \"0df1a118-36a5-4c57-8cec-2c8236f1c4de\") " pod="openshift-multus/multus-tlwp4" Oct 30 11:53:25.200416 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.197938 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/644e19b3-10ad-4145-9ed3-595d19ed9b07-run-ovn\") pod \"ovnkube-node-b8bxz\" (UID: \"644e19b3-10ad-4145-9ed3-595d19ed9b07\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8bxz" Oct 30 11:53:25.200416 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.197961 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/644e19b3-10ad-4145-9ed3-595d19ed9b07-ovnkube-script-lib\") pod \"ovnkube-node-b8bxz\" (UID: \"644e19b3-10ad-4145-9ed3-595d19ed9b07\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8bxz" Oct 30 11:53:25.200416 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.197983 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3699031f-ee9e-46cc-b37e-4dfc76b9c0b1-etc-modprobe-d\") pod \"tuned-587rx\" (UID: \"3699031f-ee9e-46cc-b37e-4dfc76b9c0b1\") " pod="openshift-cluster-node-tuning-operator/tuned-587rx" Oct 30 11:53:25.200416 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.197982 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d659d471-f019-4853-97d0-ee6e948e6c1c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-s6k8x\" (UID: \"d659d471-f019-4853-97d0-ee6e948e6c1c\") " pod="openshift-multus/multus-additional-cni-plugins-s6k8x" Oct 30 11:53:25.200416 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.198007 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0df1a118-36a5-4c57-8cec-2c8236f1c4de-host-var-lib-cni-bin\") pod \"multus-tlwp4\" (UID: \"0df1a118-36a5-4c57-8cec-2c8236f1c4de\") " pod="openshift-multus/multus-tlwp4" Oct 30 11:53:25.200416 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.198084 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0df1a118-36a5-4c57-8cec-2c8236f1c4de-hostroot\") pod \"multus-tlwp4\" (UID: \"0df1a118-36a5-4c57-8cec-2c8236f1c4de\") " pod="openshift-multus/multus-tlwp4" Oct 30 11:53:25.200416 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.198199 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d659d471-f019-4853-97d0-ee6e948e6c1c-cni-binary-copy\") pod \"multus-additional-cni-plugins-s6k8x\" (UID: \"d659d471-f019-4853-97d0-ee6e948e6c1c\") " pod="openshift-multus/multus-additional-cni-plugins-s6k8x" Oct 30 11:53:25.200416 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.198212 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d659d471-f019-4853-97d0-ee6e948e6c1c-system-cni-dir\") pod \"multus-additional-cni-plugins-s6k8x\" (UID: \"d659d471-f019-4853-97d0-ee6e948e6c1c\") " pod="openshift-multus/multus-additional-cni-plugins-s6k8x" Oct 30 11:53:25.200416 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.198519 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0df1a118-36a5-4c57-8cec-2c8236f1c4de-multus-daemon-config\") pod \"multus-tlwp4\" (UID: \"0df1a118-36a5-4c57-8cec-2c8236f1c4de\") " pod="openshift-multus/multus-tlwp4" Oct 30 11:53:25.200416 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.198543 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0df1a118-36a5-4c57-8cec-2c8236f1c4de-host-var-lib-cni-bin\") pod \"multus-tlwp4\" (UID: \"0df1a118-36a5-4c57-8cec-2c8236f1c4de\") " pod="openshift-multus/multus-tlwp4" Oct 30 11:53:25.201251 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.198561 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3699031f-ee9e-46cc-b37e-4dfc76b9c0b1-etc-systemd\") pod \"tuned-587rx\" (UID: \"3699031f-ee9e-46cc-b37e-4dfc76b9c0b1\") " pod="openshift-cluster-node-tuning-operator/tuned-587rx" Oct 30 11:53:25.201251 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.198588 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/644e19b3-10ad-4145-9ed3-595d19ed9b07-run-ovn\") pod \"ovnkube-node-b8bxz\" (UID: \"644e19b3-10ad-4145-9ed3-595d19ed9b07\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8bxz" Oct 30 11:53:25.201251 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.198674 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/644e19b3-10ad-4145-9ed3-595d19ed9b07-ovnkube-script-lib\") pod \"ovnkube-node-b8bxz\" (UID: \"644e19b3-10ad-4145-9ed3-595d19ed9b07\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8bxz" Oct 30 11:53:25.201251 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.198955 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3699031f-ee9e-46cc-b37e-4dfc76b9c0b1-etc-modprobe-d\") pod \"tuned-587rx\" (UID: \"3699031f-ee9e-46cc-b37e-4dfc76b9c0b1\") " pod="openshift-cluster-node-tuning-operator/tuned-587rx" Oct 30 11:53:25.201251 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.199011 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4e18f771-273b-49e6-872c-6cbe47ab68c4-agent-certs\") pod \"konnectivity-agent-tdf8d\" (UID: \"4e18f771-273b-49e6-872c-6cbe47ab68c4\") " pod="kube-system/konnectivity-agent-tdf8d" Oct 30 11:53:25.201251 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.200522 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/644e19b3-10ad-4145-9ed3-595d19ed9b07-ovn-node-metrics-cert\") pod \"ovnkube-node-b8bxz\" (UID: \"644e19b3-10ad-4145-9ed3-595d19ed9b07\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8bxz" Oct 30 11:53:25.201251 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.201222 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3699031f-ee9e-46cc-b37e-4dfc76b9c0b1-tmp\") pod \"tuned-587rx\" (UID: \"3699031f-ee9e-46cc-b37e-4dfc76b9c0b1\") " pod="openshift-cluster-node-tuning-operator/tuned-587rx" Oct 30 11:53:25.201860 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.201823 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3699031f-ee9e-46cc-b37e-4dfc76b9c0b1-etc-tuned\") pod \"tuned-587rx\" (UID: \"3699031f-ee9e-46cc-b37e-4dfc76b9c0b1\") " pod="openshift-cluster-node-tuning-operator/tuned-587rx" Oct 30 11:53:25.216992 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:25.216973 2545 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 11:53:25.216992 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:25.216994 2545 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 11:53:25.217144 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:25.217006 2545 projected.go:196] Error preparing data for projected volume kube-api-access-zh8h9 for pod openshift-network-diagnostics/network-check-target-bz65k: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 11:53:25.217144 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:25.217063 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ff9feaa5-70cc-47fd-a3d1-a9ac6e7eaace-kube-api-access-zh8h9 podName:ff9feaa5-70cc-47fd-a3d1-a9ac6e7eaace nodeName:}" failed. No retries permitted until 2026-10-30 11:53:25.717047138 +0000 UTC m=+3.157500365 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-zh8h9" (UniqueName: "kubernetes.io/projected/ff9feaa5-70cc-47fd-a3d1-a9ac6e7eaace-kube-api-access-zh8h9") pod "network-check-target-bz65k" (UID: "ff9feaa5-70cc-47fd-a3d1-a9ac6e7eaace") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 11:53:25.219930 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.219906 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzc5h\" (UniqueName: \"kubernetes.io/projected/1b735438-5947-4374-bd4c-66ac3a299770-kube-api-access-gzc5h\") pod \"node-resolver-xmrdj\" (UID: \"1b735438-5947-4374-bd4c-66ac3a299770\") " pod="openshift-dns/node-resolver-xmrdj" Oct 30 11:53:25.220290 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.220269 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4tdc\" (UniqueName: \"kubernetes.io/projected/0df1a118-36a5-4c57-8cec-2c8236f1c4de-kube-api-access-r4tdc\") pod \"multus-tlwp4\" (UID: \"0df1a118-36a5-4c57-8cec-2c8236f1c4de\") " pod="openshift-multus/multus-tlwp4" Oct 30 11:53:25.221854 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.221482 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmx84\" (UniqueName: \"kubernetes.io/projected/5aee536a-8942-4b48-9b2c-3e36089f3ac5-kube-api-access-vmx84\") pod \"node-ca-hb8j7\" (UID: \"5aee536a-8942-4b48-9b2c-3e36089f3ac5\") " pod="openshift-image-registry/node-ca-hb8j7" Oct 30 11:53:25.221854 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.221520 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-stff7\" (UniqueName: \"kubernetes.io/projected/d659d471-f019-4853-97d0-ee6e948e6c1c-kube-api-access-stff7\") pod \"multus-additional-cni-plugins-s6k8x\" (UID: \"d659d471-f019-4853-97d0-ee6e948e6c1c\") " pod="openshift-multus/multus-additional-cni-plugins-s6k8x" Oct 30 11:53:25.221854 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.221804 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7vnf\" (UniqueName: \"kubernetes.io/projected/3699031f-ee9e-46cc-b37e-4dfc76b9c0b1-kube-api-access-r7vnf\") pod \"tuned-587rx\" (UID: \"3699031f-ee9e-46cc-b37e-4dfc76b9c0b1\") " pod="openshift-cluster-node-tuning-operator/tuned-587rx" Oct 30 11:53:25.226724 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.223650 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92smt\" (UniqueName: \"kubernetes.io/projected/7e8ec7ff-015c-4049-9c5f-aae73dc06e4b-kube-api-access-92smt\") pod \"iptables-alerter-8m2w9\" (UID: \"7e8ec7ff-015c-4049-9c5f-aae73dc06e4b\") " pod="openshift-network-operator/iptables-alerter-8m2w9" Oct 30 11:53:25.226724 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.224054 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nzgq\" (UniqueName: \"kubernetes.io/projected/644e19b3-10ad-4145-9ed3-595d19ed9b07-kube-api-access-8nzgq\") pod \"ovnkube-node-b8bxz\" (UID: \"644e19b3-10ad-4145-9ed3-595d19ed9b07\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8bxz" Oct 30 11:53:25.229684 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.229663 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rh4tb\" (UniqueName: \"kubernetes.io/projected/c54f4c1b-f08e-4495-9a14-17469a792ae5-kube-api-access-rh4tb\") pod \"aws-ebs-csi-driver-node-h6mlf\" (UID: \"c54f4c1b-f08e-4495-9a14-17469a792ae5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h6mlf" Oct 30 11:53:25.295884 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.295859 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-tdf8d" Oct 30 11:53:25.307231 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:25.307201 2545 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e18f771_273b_49e6_872c_6cbe47ab68c4.slice/crio-4fe7f906d45dc1117fc963f19feb9094a9508021f874ec361136e810911456ae WatchSource:0}: Error finding container 4fe7f906d45dc1117fc963f19feb9094a9508021f874ec361136e810911456ae: Status 404 returned error can't find the container with id 4fe7f906d45dc1117fc963f19feb9094a9508021f874ec361136e810911456ae Oct 30 11:53:25.317970 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.317948 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-587rx" Oct 30 11:53:25.323924 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.323901 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-s6k8x" Oct 30 11:53:25.325064 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.325043 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-xmrdj" Oct 30 11:53:25.326878 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.326855 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-tlwp4" Oct 30 11:53:25.328871 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:25.328845 2545 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3699031f_ee9e_46cc_b37e_4dfc76b9c0b1.slice/crio-0d22c6c8ba7455c94555e3009017bec97a78d66f90e173ab4a3cae69d93caab5 WatchSource:0}: Error finding container 0d22c6c8ba7455c94555e3009017bec97a78d66f90e173ab4a3cae69d93caab5: Status 404 returned error can't find the container with id 0d22c6c8ba7455c94555e3009017bec97a78d66f90e173ab4a3cae69d93caab5 Oct 30 11:53:25.334204 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:25.334175 2545 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd659d471_f019_4853_97d0_ee6e948e6c1c.slice/crio-e1e676a6d3c93a0071e009327272d842160f1207839b7a25a676a8f5e4d83d25 WatchSource:0}: Error finding container e1e676a6d3c93a0071e009327272d842160f1207839b7a25a676a8f5e4d83d25: Status 404 returned error can't find the container with id e1e676a6d3c93a0071e009327272d842160f1207839b7a25a676a8f5e4d83d25 Oct 30 11:53:25.337856 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:25.337797 2545 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b735438_5947_4374_bd4c_66ac3a299770.slice/crio-81d2c98212fb5e239a5750b14b2032945223d71a039bfef6fd50039af7d88632 WatchSource:0}: Error finding container 81d2c98212fb5e239a5750b14b2032945223d71a039bfef6fd50039af7d88632: Status 404 returned error can't find the container with id 81d2c98212fb5e239a5750b14b2032945223d71a039bfef6fd50039af7d88632 Oct 30 11:53:25.338477 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.338332 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h6mlf" Oct 30 11:53:25.340867 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:25.340849 2545 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0df1a118_36a5_4c57_8cec_2c8236f1c4de.slice/crio-37fb31a12739fbc2204e29a20a6a16c2a38ecb9fc54c663aaa811e6d264a8655 WatchSource:0}: Error finding container 37fb31a12739fbc2204e29a20a6a16c2a38ecb9fc54c663aaa811e6d264a8655: Status 404 returned error can't find the container with id 37fb31a12739fbc2204e29a20a6a16c2a38ecb9fc54c663aaa811e6d264a8655 Oct 30 11:53:25.341493 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.341475 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-hb8j7" Oct 30 11:53:25.349453 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:25.349434 2545 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc54f4c1b_f08e_4495_9a14_17469a792ae5.slice/crio-59286100c47dfffc745aeaf1101d1b002609f107813dd3febe7c1a355fbe11db WatchSource:0}: Error finding container 59286100c47dfffc745aeaf1101d1b002609f107813dd3febe7c1a355fbe11db: Status 404 returned error can't find the container with id 59286100c47dfffc745aeaf1101d1b002609f107813dd3febe7c1a355fbe11db Oct 30 11:53:25.352253 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:25.352231 2545 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5aee536a_8942_4b48_9b2c_3e36089f3ac5.slice/crio-ac1ca6630282423463ddd56cf4dd5bd7e452ffaef60997a055a518a3d193ae1f WatchSource:0}: Error finding container ac1ca6630282423463ddd56cf4dd5bd7e452ffaef60997a055a518a3d193ae1f: Status 404 returned error can't find the container with id ac1ca6630282423463ddd56cf4dd5bd7e452ffaef60997a055a518a3d193ae1f Oct 30 11:53:25.362323 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.362305 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-8m2w9" Oct 30 11:53:25.366637 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.366620 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-b8bxz" Oct 30 11:53:25.378362 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:25.378342 2545 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod644e19b3_10ad_4145_9ed3_595d19ed9b07.slice/crio-1f5ac21f834c419c1d2b1b6d14292b5bd03d2741030a178f2a45cdb3262508dd WatchSource:0}: Error finding container 1f5ac21f834c419c1d2b1b6d14292b5bd03d2741030a178f2a45cdb3262508dd: Status 404 returned error can't find the container with id 1f5ac21f834c419c1d2b1b6d14292b5bd03d2741030a178f2a45cdb3262508dd Oct 30 11:53:25.380303 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:25.380285 2545 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e8ec7ff_015c_4049_9c5f_aae73dc06e4b.slice/crio-1f2563b5c5e8e08274b16bd9bbbccba45366b0aed1b60df792c35b2ac3e13f6f WatchSource:0}: Error finding container 1f2563b5c5e8e08274b16bd9bbbccba45366b0aed1b60df792c35b2ac3e13f6f: Status 404 returned error can't find the container with id 1f2563b5c5e8e08274b16bd9bbbccba45366b0aed1b60df792c35b2ac3e13f6f Oct 30 11:53:25.600693 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.600673 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d7f8396-e57c-45e5-ae74-a2244db10364-metrics-certs\") pod \"network-metrics-daemon-dbqhj\" (UID: \"1d7f8396-e57c-45e5-ae74-a2244db10364\") " pod="openshift-multus/network-metrics-daemon-dbqhj" Oct 30 11:53:25.600836 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:25.600809 2545 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 11:53:25.600891 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:25.600877 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d7f8396-e57c-45e5-ae74-a2244db10364-metrics-certs podName:1d7f8396-e57c-45e5-ae74-a2244db10364 nodeName:}" failed. No retries permitted until 2026-10-30 11:53:26.600863899 +0000 UTC m=+4.041317106 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1d7f8396-e57c-45e5-ae74-a2244db10364-metrics-certs") pod "network-metrics-daemon-dbqhj" (UID: "1d7f8396-e57c-45e5-ae74-a2244db10364") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 11:53:25.702186 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.701966 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6b350710-c48c-4c7e-bce9-e021099e98ca-original-pull-secret\") pod \"global-pull-secret-syncer-fxjlt\" (UID: \"6b350710-c48c-4c7e-bce9-e021099e98ca\") " pod="kube-system/global-pull-secret-syncer-fxjlt" Oct 30 11:53:25.702284 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:25.702131 2545 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 11:53:25.702323 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:25.702309 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b350710-c48c-4c7e-bce9-e021099e98ca-original-pull-secret podName:6b350710-c48c-4c7e-bce9-e021099e98ca nodeName:}" failed. No retries permitted until 2026-10-30 11:53:26.702289735 +0000 UTC m=+4.142742957 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6b350710-c48c-4c7e-bce9-e021099e98ca-original-pull-secret") pod "global-pull-secret-syncer-fxjlt" (UID: "6b350710-c48c-4c7e-bce9-e021099e98ca") : object "kube-system"/"original-pull-secret" not registered Oct 30 11:53:25.803018 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:25.802983 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zh8h9\" (UniqueName: \"kubernetes.io/projected/ff9feaa5-70cc-47fd-a3d1-a9ac6e7eaace-kube-api-access-zh8h9\") pod \"network-check-target-bz65k\" (UID: \"ff9feaa5-70cc-47fd-a3d1-a9ac6e7eaace\") " pod="openshift-network-diagnostics/network-check-target-bz65k" Oct 30 11:53:25.803169 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:25.803127 2545 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 11:53:25.803169 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:25.803145 2545 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 11:53:25.803169 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:25.803156 2545 projected.go:196] Error preparing data for projected volume kube-api-access-zh8h9 for pod openshift-network-diagnostics/network-check-target-bz65k: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 11:53:25.803310 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:25.803210 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ff9feaa5-70cc-47fd-a3d1-a9ac6e7eaace-kube-api-access-zh8h9 podName:ff9feaa5-70cc-47fd-a3d1-a9ac6e7eaace nodeName:}" failed. No retries permitted until 2026-10-30 11:53:26.803192371 +0000 UTC m=+4.243645596 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-zh8h9" (UniqueName: "kubernetes.io/projected/ff9feaa5-70cc-47fd-a3d1-a9ac6e7eaace-kube-api-access-zh8h9") pod "network-check-target-bz65k" (UID: "ff9feaa5-70cc-47fd-a3d1-a9ac6e7eaace") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 11:53:26.054776 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:26.054638 2545 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 11:48:25 +0000 UTC" deadline="2028-06-12 21:00:40.394482008 +0000 UTC" Oct 30 11:53:26.054776 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:26.054682 2545 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14193h7m14.33980409s" Oct 30 11:53:26.100345 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:26.100318 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fxjlt" Oct 30 11:53:26.100515 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:26.100446 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fxjlt" podUID="6b350710-c48c-4c7e-bce9-e021099e98ca" Oct 30 11:53:26.129854 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:26.121703 2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h6mlf" event={"ID":"c54f4c1b-f08e-4495-9a14-17469a792ae5","Type":"ContainerStarted","Data":"59286100c47dfffc745aeaf1101d1b002609f107813dd3febe7c1a355fbe11db"} Oct 30 11:53:26.129854 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:26.125150 2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-xmrdj" event={"ID":"1b735438-5947-4374-bd4c-66ac3a299770","Type":"ContainerStarted","Data":"81d2c98212fb5e239a5750b14b2032945223d71a039bfef6fd50039af7d88632"} Oct 30 11:53:26.140585 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:26.140556 2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-s6k8x" event={"ID":"d659d471-f019-4853-97d0-ee6e948e6c1c","Type":"ContainerStarted","Data":"e1e676a6d3c93a0071e009327272d842160f1207839b7a25a676a8f5e4d83d25"} Oct 30 11:53:26.148168 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:26.148144 2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-587rx" event={"ID":"3699031f-ee9e-46cc-b37e-4dfc76b9c0b1","Type":"ContainerStarted","Data":"0d22c6c8ba7455c94555e3009017bec97a78d66f90e173ab4a3cae69d93caab5"} Oct 30 11:53:26.164865 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:26.162598 2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-tdf8d" event={"ID":"4e18f771-273b-49e6-872c-6cbe47ab68c4","Type":"ContainerStarted","Data":"4fe7f906d45dc1117fc963f19feb9094a9508021f874ec361136e810911456ae"} Oct 30 11:53:26.167559 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:26.167539 2545 generic.go:356] "Generic (PLEG): container finished" podID="f140bf37195964da2b6ead5845b42a26" containerID="c2e4fd77c6ca53931d448c42822c6f59bf2ff55f29062c62ae4d6869151cbeea" exitCode=0 Oct 30 11:53:26.167657 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:26.167593 2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-251.ec2.internal" event={"ID":"f140bf37195964da2b6ead5845b42a26","Type":"ContainerDied","Data":"c2e4fd77c6ca53931d448c42822c6f59bf2ff55f29062c62ae4d6869151cbeea"} Oct 30 11:53:26.176216 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:26.176191 2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-251.ec2.internal" event={"ID":"a73cae35bcc0a4b145486ce8e9e460c1","Type":"ContainerStarted","Data":"f638e0c45c2e95a85b16efaef0676cb14b6e83ca7bb639fed7a9ea6e571ee59c"} Oct 30 11:53:26.184215 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:26.184192 2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b8bxz" event={"ID":"644e19b3-10ad-4145-9ed3-595d19ed9b07","Type":"ContainerStarted","Data":"1f5ac21f834c419c1d2b1b6d14292b5bd03d2741030a178f2a45cdb3262508dd"} Oct 30 11:53:26.197841 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:26.194274 2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-tlwp4" event={"ID":"0df1a118-36a5-4c57-8cec-2c8236f1c4de","Type":"ContainerStarted","Data":"37fb31a12739fbc2204e29a20a6a16c2a38ecb9fc54c663aaa811e6d264a8655"} Oct 30 11:53:26.232428 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:26.232331 2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-8m2w9" event={"ID":"7e8ec7ff-015c-4049-9c5f-aae73dc06e4b","Type":"ContainerStarted","Data":"1f2563b5c5e8e08274b16bd9bbbccba45366b0aed1b60df792c35b2ac3e13f6f"} Oct 30 11:53:26.241126 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:26.241101 2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-hb8j7" event={"ID":"5aee536a-8942-4b48-9b2c-3e36089f3ac5","Type":"ContainerStarted","Data":"ac1ca6630282423463ddd56cf4dd5bd7e452ffaef60997a055a518a3d193ae1f"} Oct 30 11:53:26.254225 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:26.254052 2545 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-251.ec2.internal" podStartSLOduration=2.254039138 podStartE2EDuration="2.254039138s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 11:53:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:53:26.253061207 +0000 UTC m=+3.693514439" watchObservedRunningTime="2026-10-30 11:53:26.254039138 +0000 UTC m=+3.694492368" Oct 30 11:53:26.609672 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:26.609545 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d7f8396-e57c-45e5-ae74-a2244db10364-metrics-certs\") pod \"network-metrics-daemon-dbqhj\" (UID: \"1d7f8396-e57c-45e5-ae74-a2244db10364\") " pod="openshift-multus/network-metrics-daemon-dbqhj" Oct 30 11:53:26.609888 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:26.609703 2545 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 11:53:26.609888 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:26.609773 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d7f8396-e57c-45e5-ae74-a2244db10364-metrics-certs podName:1d7f8396-e57c-45e5-ae74-a2244db10364 nodeName:}" failed. No retries permitted until 2026-10-30 11:53:28.609753232 +0000 UTC m=+6.050206453 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1d7f8396-e57c-45e5-ae74-a2244db10364-metrics-certs") pod "network-metrics-daemon-dbqhj" (UID: "1d7f8396-e57c-45e5-ae74-a2244db10364") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 11:53:26.710512 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:26.710474 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6b350710-c48c-4c7e-bce9-e021099e98ca-original-pull-secret\") pod \"global-pull-secret-syncer-fxjlt\" (UID: \"6b350710-c48c-4c7e-bce9-e021099e98ca\") " pod="kube-system/global-pull-secret-syncer-fxjlt" Oct 30 11:53:26.710675 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:26.710600 2545 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 11:53:26.710675 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:26.710652 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b350710-c48c-4c7e-bce9-e021099e98ca-original-pull-secret podName:6b350710-c48c-4c7e-bce9-e021099e98ca nodeName:}" failed. No retries permitted until 2026-10-30 11:53:28.710637398 +0000 UTC m=+6.151090607 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6b350710-c48c-4c7e-bce9-e021099e98ca-original-pull-secret") pod "global-pull-secret-syncer-fxjlt" (UID: "6b350710-c48c-4c7e-bce9-e021099e98ca") : object "kube-system"/"original-pull-secret" not registered Oct 30 11:53:26.796554 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:26.796511 2545 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 11:53:26.811076 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:26.810963 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zh8h9\" (UniqueName: \"kubernetes.io/projected/ff9feaa5-70cc-47fd-a3d1-a9ac6e7eaace-kube-api-access-zh8h9\") pod \"network-check-target-bz65k\" (UID: \"ff9feaa5-70cc-47fd-a3d1-a9ac6e7eaace\") " pod="openshift-network-diagnostics/network-check-target-bz65k" Oct 30 11:53:26.811216 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:26.811124 2545 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 11:53:26.811216 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:26.811143 2545 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 11:53:26.811216 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:26.811156 2545 projected.go:196] Error preparing data for projected volume kube-api-access-zh8h9 for pod openshift-network-diagnostics/network-check-target-bz65k: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 11:53:26.811216 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:26.811208 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ff9feaa5-70cc-47fd-a3d1-a9ac6e7eaace-kube-api-access-zh8h9 podName:ff9feaa5-70cc-47fd-a3d1-a9ac6e7eaace nodeName:}" failed. No retries permitted until 2026-10-30 11:53:28.811192067 +0000 UTC m=+6.251645289 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-zh8h9" (UniqueName: "kubernetes.io/projected/ff9feaa5-70cc-47fd-a3d1-a9ac6e7eaace-kube-api-access-zh8h9") pod "network-check-target-bz65k" (UID: "ff9feaa5-70cc-47fd-a3d1-a9ac6e7eaace") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 11:53:27.055085 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:27.054972 2545 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 11:48:25 +0000 UTC" deadline="2028-05-20 17:15:49.474520115 +0000 UTC" Oct 30 11:53:27.055085 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:27.055030 2545 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13637h22m22.419494232s" Oct 30 11:53:27.102145 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:27.100946 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bz65k" Oct 30 11:53:27.102145 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:27.101064 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bz65k" podUID="ff9feaa5-70cc-47fd-a3d1-a9ac6e7eaace" Oct 30 11:53:27.102145 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:27.101523 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dbqhj" Oct 30 11:53:27.102145 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:27.101626 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dbqhj" podUID="1d7f8396-e57c-45e5-ae74-a2244db10364" Oct 30 11:53:27.273935 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:27.273854 2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-251.ec2.internal" event={"ID":"f140bf37195964da2b6ead5845b42a26","Type":"ContainerStarted","Data":"ad4d99f88459a1f4749ea8972b817f7cb94acf7cba423ba8b3e0cf06c3e4176e"} Oct 30 11:53:28.107431 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:28.099229 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fxjlt" Oct 30 11:53:28.107431 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:28.099358 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fxjlt" podUID="6b350710-c48c-4c7e-bce9-e021099e98ca" Oct 30 11:53:28.282880 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:28.282188 2545 generic.go:356] "Generic (PLEG): container finished" podID="d659d471-f019-4853-97d0-ee6e948e6c1c" containerID="0e465df6f612805073206b30b3905501fa2f2d3a254412f170c0d262bfa82f73" exitCode=0 Oct 30 11:53:28.283309 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:28.283275 2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-s6k8x" event={"ID":"d659d471-f019-4853-97d0-ee6e948e6c1c","Type":"ContainerDied","Data":"0e465df6f612805073206b30b3905501fa2f2d3a254412f170c0d262bfa82f73"} Oct 30 11:53:28.416687 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:28.416590 2545 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-251.ec2.internal" podStartSLOduration=4.416570665 podStartE2EDuration="4.416570665s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 11:53:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:53:27.314255049 +0000 UTC m=+4.754708279" watchObservedRunningTime="2026-10-30 11:53:28.416570665 +0000 UTC m=+5.857023897" Oct 30 11:53:28.623920 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:28.623883 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d7f8396-e57c-45e5-ae74-a2244db10364-metrics-certs\") pod \"network-metrics-daemon-dbqhj\" (UID: \"1d7f8396-e57c-45e5-ae74-a2244db10364\") " pod="openshift-multus/network-metrics-daemon-dbqhj" Oct 30 11:53:28.624124 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:28.624105 2545 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 11:53:28.624189 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:28.624176 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d7f8396-e57c-45e5-ae74-a2244db10364-metrics-certs podName:1d7f8396-e57c-45e5-ae74-a2244db10364 nodeName:}" failed. No retries permitted until 2026-10-30 11:53:32.624157843 +0000 UTC m=+10.064611070 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1d7f8396-e57c-45e5-ae74-a2244db10364-metrics-certs") pod "network-metrics-daemon-dbqhj" (UID: "1d7f8396-e57c-45e5-ae74-a2244db10364") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 11:53:28.725200 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:28.725126 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6b350710-c48c-4c7e-bce9-e021099e98ca-original-pull-secret\") pod \"global-pull-secret-syncer-fxjlt\" (UID: \"6b350710-c48c-4c7e-bce9-e021099e98ca\") " pod="kube-system/global-pull-secret-syncer-fxjlt" Oct 30 11:53:28.725347 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:28.725290 2545 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 11:53:28.725413 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:28.725354 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b350710-c48c-4c7e-bce9-e021099e98ca-original-pull-secret podName:6b350710-c48c-4c7e-bce9-e021099e98ca nodeName:}" failed. No retries permitted until 2026-10-30 11:53:32.725336578 +0000 UTC m=+10.165789789 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6b350710-c48c-4c7e-bce9-e021099e98ca-original-pull-secret") pod "global-pull-secret-syncer-fxjlt" (UID: "6b350710-c48c-4c7e-bce9-e021099e98ca") : object "kube-system"/"original-pull-secret" not registered Oct 30 11:53:28.825569 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:28.825531 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zh8h9\" (UniqueName: \"kubernetes.io/projected/ff9feaa5-70cc-47fd-a3d1-a9ac6e7eaace-kube-api-access-zh8h9\") pod \"network-check-target-bz65k\" (UID: \"ff9feaa5-70cc-47fd-a3d1-a9ac6e7eaace\") " pod="openshift-network-diagnostics/network-check-target-bz65k" Oct 30 11:53:28.825747 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:28.825703 2545 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 11:53:28.825747 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:28.825726 2545 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 11:53:28.825747 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:28.825740 2545 projected.go:196] Error preparing data for projected volume kube-api-access-zh8h9 for pod openshift-network-diagnostics/network-check-target-bz65k: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 11:53:28.825934 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:28.825794 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ff9feaa5-70cc-47fd-a3d1-a9ac6e7eaace-kube-api-access-zh8h9 podName:ff9feaa5-70cc-47fd-a3d1-a9ac6e7eaace nodeName:}" failed. No retries permitted until 2026-10-30 11:53:32.825776699 +0000 UTC m=+10.266229911 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-zh8h9" (UniqueName: "kubernetes.io/projected/ff9feaa5-70cc-47fd-a3d1-a9ac6e7eaace-kube-api-access-zh8h9") pod "network-check-target-bz65k" (UID: "ff9feaa5-70cc-47fd-a3d1-a9ac6e7eaace") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 11:53:29.105188 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:29.105165 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dbqhj" Oct 30 11:53:29.105318 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:29.105297 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dbqhj" podUID="1d7f8396-e57c-45e5-ae74-a2244db10364" Oct 30 11:53:29.105374 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:29.105356 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bz65k" Oct 30 11:53:29.105432 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:29.105416 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bz65k" podUID="ff9feaa5-70cc-47fd-a3d1-a9ac6e7eaace" Oct 30 11:53:29.287660 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:29.287630 2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-tdf8d" event={"ID":"4e18f771-273b-49e6-872c-6cbe47ab68c4","Type":"ContainerStarted","Data":"a87a93739c084eb21205194e79d98baf9614c93a9f2df1bc6a4b8c2b76de0449"} Oct 30 11:53:29.330040 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:29.329953 2545 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-tdf8d" podStartSLOduration=2.730006296 podStartE2EDuration="6.329936418s" totalImagesPullingTime="3.599930122s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:53:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:53:29.329816307 +0000 UTC m=+6.770269537" watchObservedRunningTime="2026-10-30 11:53:29.329936418 +0000 UTC m=+6.770389646" Oct 30 11:53:30.099944 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:30.099911 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fxjlt" Oct 30 11:53:30.100107 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:30.100044 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fxjlt" podUID="6b350710-c48c-4c7e-bce9-e021099e98ca" Oct 30 11:53:30.295248 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:30.295123 2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-hb8j7" event={"ID":"5aee536a-8942-4b48-9b2c-3e36089f3ac5","Type":"ContainerStarted","Data":"bd38899929377fb74c89328160cada48938f08a73dc08956b6e0367f9603e266"} Oct 30 11:53:31.101473 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:31.100941 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bz65k" Oct 30 11:53:31.101473 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:31.101036 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bz65k" podUID="ff9feaa5-70cc-47fd-a3d1-a9ac6e7eaace" Oct 30 11:53:31.101473 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:31.101345 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dbqhj" Oct 30 11:53:31.101473 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:31.101438 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dbqhj" podUID="1d7f8396-e57c-45e5-ae74-a2244db10364" Oct 30 11:53:31.303203 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:31.303167 2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h6mlf" event={"ID":"c54f4c1b-f08e-4495-9a14-17469a792ae5","Type":"ContainerStarted","Data":"45ad98ea4f2cc6519849bab642bc2c988f7add67b881338134004a0155add1cb"} Oct 30 11:53:31.891336 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:31.890658 2545 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-tdf8d" Oct 30 11:53:31.892245 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:31.892066 2545 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-tdf8d" Oct 30 11:53:31.934263 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:31.933424 2545 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-hb8j7" podStartSLOduration=4.273236001 podStartE2EDuration="8.933411834s" totalImagesPullingTime="4.660175833s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 11:53:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:53:30.334295192 +0000 UTC m=+7.774748422" watchObservedRunningTime="2026-10-30 11:53:31.933411834 +0000 UTC m=+9.373865063" Oct 30 11:53:32.099720 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:32.099693 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fxjlt" Oct 30 11:53:32.099902 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:32.099795 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fxjlt" podUID="6b350710-c48c-4c7e-bce9-e021099e98ca" Oct 30 11:53:32.289821 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:32.289802 2545 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 11:53:32.307619 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:32.307591 2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h6mlf" event={"ID":"c54f4c1b-f08e-4495-9a14-17469a792ae5","Type":"ContainerStarted","Data":"6461e3d2290360da63e467f33fb25b7d0e0eaf026f7af39bdde7b6cff5813e30"} Oct 30 11:53:32.658034 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:32.657484 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d7f8396-e57c-45e5-ae74-a2244db10364-metrics-certs\") pod \"network-metrics-daemon-dbqhj\" (UID: \"1d7f8396-e57c-45e5-ae74-a2244db10364\") " pod="openshift-multus/network-metrics-daemon-dbqhj" Oct 30 11:53:32.658034 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:32.657637 2545 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 11:53:32.658034 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:32.657696 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d7f8396-e57c-45e5-ae74-a2244db10364-metrics-certs podName:1d7f8396-e57c-45e5-ae74-a2244db10364 nodeName:}" failed. No retries permitted until 2026-10-30 11:53:40.65768002 +0000 UTC m=+18.098133244 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1d7f8396-e57c-45e5-ae74-a2244db10364-metrics-certs") pod "network-metrics-daemon-dbqhj" (UID: "1d7f8396-e57c-45e5-ae74-a2244db10364") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 11:53:32.758687 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:32.758664 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6b350710-c48c-4c7e-bce9-e021099e98ca-original-pull-secret\") pod \"global-pull-secret-syncer-fxjlt\" (UID: \"6b350710-c48c-4c7e-bce9-e021099e98ca\") " pod="kube-system/global-pull-secret-syncer-fxjlt" Oct 30 11:53:32.758808 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:32.758794 2545 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 11:53:32.758908 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:32.758860 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b350710-c48c-4c7e-bce9-e021099e98ca-original-pull-secret podName:6b350710-c48c-4c7e-bce9-e021099e98ca nodeName:}" failed. No retries permitted until 2026-10-30 11:53:40.758844052 +0000 UTC m=+18.199297275 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6b350710-c48c-4c7e-bce9-e021099e98ca-original-pull-secret") pod "global-pull-secret-syncer-fxjlt" (UID: "6b350710-c48c-4c7e-bce9-e021099e98ca") : object "kube-system"/"original-pull-secret" not registered Oct 30 11:53:32.860026 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:32.859991 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zh8h9\" (UniqueName: \"kubernetes.io/projected/ff9feaa5-70cc-47fd-a3d1-a9ac6e7eaace-kube-api-access-zh8h9\") pod \"network-check-target-bz65k\" (UID: \"ff9feaa5-70cc-47fd-a3d1-a9ac6e7eaace\") " pod="openshift-network-diagnostics/network-check-target-bz65k" Oct 30 11:53:32.860246 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:32.860228 2545 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 11:53:32.860314 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:32.860253 2545 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 11:53:32.860314 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:32.860273 2545 projected.go:196] Error preparing data for projected volume kube-api-access-zh8h9 for pod openshift-network-diagnostics/network-check-target-bz65k: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 11:53:32.860424 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:32.860342 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ff9feaa5-70cc-47fd-a3d1-a9ac6e7eaace-kube-api-access-zh8h9 podName:ff9feaa5-70cc-47fd-a3d1-a9ac6e7eaace nodeName:}" failed. No retries permitted until 2026-10-30 11:53:40.86032178 +0000 UTC m=+18.300775011 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-zh8h9" (UniqueName: "kubernetes.io/projected/ff9feaa5-70cc-47fd-a3d1-a9ac6e7eaace-kube-api-access-zh8h9") pod "network-check-target-bz65k" (UID: "ff9feaa5-70cc-47fd-a3d1-a9ac6e7eaace") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 11:53:33.100329 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:33.100291 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bz65k" Oct 30 11:53:33.100493 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:33.100382 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bz65k" podUID="ff9feaa5-70cc-47fd-a3d1-a9ac6e7eaace" Oct 30 11:53:33.100552 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:33.100487 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dbqhj" Oct 30 11:53:33.100672 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:33.100603 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dbqhj" podUID="1d7f8396-e57c-45e5-ae74-a2244db10364" Oct 30 11:53:33.191536 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:33.191405 2545 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T11:53:32.2898175Z","UUID":"882ec6f9-2938-4b7d-b185-2483d05d6448","Handler":null,"Name":"","Endpoint":""} Oct 30 11:53:33.195973 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:33.195951 2545 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 11:53:33.196061 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:33.195983 2545 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 11:53:33.310299 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:33.310225 2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-tlwp4" event={"ID":"0df1a118-36a5-4c57-8cec-2c8236f1c4de","Type":"ContainerStarted","Data":"58cfdbb49b5927996316b84c1d3a218dff63b6257c0b68f96e56d6b26be300fd"} Oct 30 11:53:33.311810 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:33.311785 2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-xmrdj" event={"ID":"1b735438-5947-4374-bd4c-66ac3a299770","Type":"ContainerStarted","Data":"298d72ab789e20dcb027039142d93467f9177b5424e11efc9311f94ba829b1bc"} Oct 30 11:53:33.313520 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:33.313493 2545 generic.go:356] "Generic (PLEG): container finished" podID="d659d471-f019-4853-97d0-ee6e948e6c1c" containerID="8a6272e6ebb582eb3df7452766fa897f55e19ef3a221218bca60e352ba32c733" exitCode=0 Oct 30 11:53:33.313615 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:33.313534 2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-s6k8x" event={"ID":"d659d471-f019-4853-97d0-ee6e948e6c1c","Type":"ContainerDied","Data":"8a6272e6ebb582eb3df7452766fa897f55e19ef3a221218bca60e352ba32c733"} Oct 30 11:53:33.394038 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:33.393783 2545 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-xmrdj" podStartSLOduration=2.706565843 podStartE2EDuration="10.393770249s" totalImagesPullingTime="7.687204406s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 11:53:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:53:33.393144063 +0000 UTC m=+10.833597293" watchObservedRunningTime="2026-10-30 11:53:33.393770249 +0000 UTC m=+10.834223477" Oct 30 11:53:33.394174 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:33.394096 2545 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-tlwp4" podStartSLOduration=3.247116653 podStartE2EDuration="10.394088614s" totalImagesPullingTime="7.146971961s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 11:53:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:53:33.353782375 +0000 UTC m=+10.794235606" watchObservedRunningTime="2026-10-30 11:53:33.394088614 +0000 UTC m=+10.834541847" Oct 30 11:53:34.099689 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:34.099660 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fxjlt" Oct 30 11:53:34.099866 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:34.099766 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fxjlt" podUID="6b350710-c48c-4c7e-bce9-e021099e98ca" Oct 30 11:53:34.318196 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:34.318157 2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h6mlf" event={"ID":"c54f4c1b-f08e-4495-9a14-17469a792ae5","Type":"ContainerStarted","Data":"b3a3dfd3648deb6af439cf4835cac820143db66bba4abd8c51b6290e49d9ad8b"} Oct 30 11:53:34.320705 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:34.320678 2545 generic.go:356] "Generic (PLEG): container finished" podID="d659d471-f019-4853-97d0-ee6e948e6c1c" containerID="c2ed6eb3eaa2e73b85df6a08e944f52c7c943bc396100d50b8408c507567f226" exitCode=0 Oct 30 11:53:34.320797 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:34.320767 2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-s6k8x" event={"ID":"d659d471-f019-4853-97d0-ee6e948e6c1c","Type":"ContainerDied","Data":"c2ed6eb3eaa2e73b85df6a08e944f52c7c943bc396100d50b8408c507567f226"} Oct 30 11:53:34.323239 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:34.323209 2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-8m2w9" event={"ID":"7e8ec7ff-015c-4049-9c5f-aae73dc06e4b","Type":"ContainerStarted","Data":"bf88731227ebeeced7f1fa3be559d1f472a5dc28b232c9571c11d9def11c8ca9"} Oct 30 11:53:34.412440 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:34.412395 2545 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h6mlf" podStartSLOduration=3.82930885 podStartE2EDuration="11.412383481s" totalImagesPullingTime="7.583074631s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 11:53:23 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:53:34.36143265 +0000 UTC m=+11.801885878" watchObservedRunningTime="2026-10-30 11:53:34.412383481 +0000 UTC m=+11.852836710" Oct 30 11:53:34.460598 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:34.460544 2545 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-8m2w9" podStartSLOduration=4.133756127 podStartE2EDuration="11.460528629s" totalImagesPullingTime="7.326772502s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 11:53:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:53:34.455372175 +0000 UTC m=+11.895825415" watchObservedRunningTime="2026-10-30 11:53:34.460528629 +0000 UTC m=+11.900981861" Oct 30 11:53:35.099699 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:35.099611 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dbqhj" Oct 30 11:53:35.099864 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:35.099756 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dbqhj" podUID="1d7f8396-e57c-45e5-ae74-a2244db10364" Oct 30 11:53:35.099864 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:35.099802 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bz65k" Oct 30 11:53:35.099965 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:35.099909 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bz65k" podUID="ff9feaa5-70cc-47fd-a3d1-a9ac6e7eaace" Oct 30 11:53:35.326906 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:35.326875 2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-587rx" event={"ID":"3699031f-ee9e-46cc-b37e-4dfc76b9c0b1","Type":"ContainerStarted","Data":"314c2a0e41f17561b989f4d2cb0478e53b5999dfea8232abd8c28d90d8b7a37a"} Oct 30 11:53:35.329548 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:35.329514 2545 generic.go:356] "Generic (PLEG): container finished" podID="d659d471-f019-4853-97d0-ee6e948e6c1c" containerID="d835da50c2f060635d05553a42b19e8d329fc82dda51e30e72a9c80e6d158927" exitCode=0 Oct 30 11:53:35.329657 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:35.329608 2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-s6k8x" event={"ID":"d659d471-f019-4853-97d0-ee6e948e6c1c","Type":"ContainerDied","Data":"d835da50c2f060635d05553a42b19e8d329fc82dda51e30e72a9c80e6d158927"} Oct 30 11:53:35.556687 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:35.556596 2545 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-587rx" podStartSLOduration=3.155006093 podStartE2EDuration="12.556582281s" totalImagesPullingTime="9.401576188s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 11:53:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:53:35.368742582 +0000 UTC m=+12.809195821" watchObservedRunningTime="2026-10-30 11:53:35.556582281 +0000 UTC m=+12.997035509" Oct 30 11:53:36.100352 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:36.100319 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fxjlt" Oct 30 11:53:36.100525 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:36.100448 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fxjlt" podUID="6b350710-c48c-4c7e-bce9-e021099e98ca" Oct 30 11:53:37.099497 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:37.099463 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dbqhj" Oct 30 11:53:37.099497 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:37.099495 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bz65k" Oct 30 11:53:37.100287 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:37.099597 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dbqhj" podUID="1d7f8396-e57c-45e5-ae74-a2244db10364" Oct 30 11:53:37.100287 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:37.099736 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bz65k" podUID="ff9feaa5-70cc-47fd-a3d1-a9ac6e7eaace" Oct 30 11:53:37.338411 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:37.338208 2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-s6k8x" event={"ID":"d659d471-f019-4853-97d0-ee6e948e6c1c","Type":"ContainerStarted","Data":"754cf3c1b2b2f1490fad873f3d8ed1599e72bf26e97af18acedce0b7c02e2de4"} Oct 30 11:53:38.099330 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:38.099289 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fxjlt" Oct 30 11:53:38.099492 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:38.099424 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fxjlt" podUID="6b350710-c48c-4c7e-bce9-e021099e98ca" Oct 30 11:53:38.343863 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:38.343812 2545 generic.go:356] "Generic (PLEG): container finished" podID="d659d471-f019-4853-97d0-ee6e948e6c1c" containerID="754cf3c1b2b2f1490fad873f3d8ed1599e72bf26e97af18acedce0b7c02e2de4" exitCode=0 Oct 30 11:53:38.344321 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:38.343862 2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-s6k8x" event={"ID":"d659d471-f019-4853-97d0-ee6e948e6c1c","Type":"ContainerDied","Data":"754cf3c1b2b2f1490fad873f3d8ed1599e72bf26e97af18acedce0b7c02e2de4"} Oct 30 11:53:39.099551 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:39.099514 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bz65k" Oct 30 11:53:39.099687 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:39.099632 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bz65k" podUID="ff9feaa5-70cc-47fd-a3d1-a9ac6e7eaace" Oct 30 11:53:39.099687 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:39.099678 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dbqhj" Oct 30 11:53:39.099789 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:39.099773 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dbqhj" podUID="1d7f8396-e57c-45e5-ae74-a2244db10364" Oct 30 11:53:39.349275 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:39.349251 2545 generic.go:356] "Generic (PLEG): container finished" podID="d659d471-f019-4853-97d0-ee6e948e6c1c" containerID="0560e630808cca1e6dc6737abf733037f23162365ee250168087828768b9c1e4" exitCode=0 Oct 30 11:53:39.349595 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:39.349293 2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-s6k8x" event={"ID":"d659d471-f019-4853-97d0-ee6e948e6c1c","Type":"ContainerDied","Data":"0560e630808cca1e6dc6737abf733037f23162365ee250168087828768b9c1e4"} Oct 30 11:53:40.100090 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:40.099922 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fxjlt" Oct 30 11:53:40.100296 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:40.100163 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fxjlt" podUID="6b350710-c48c-4c7e-bce9-e021099e98ca" Oct 30 11:53:40.354413 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:40.354340 2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-s6k8x" event={"ID":"d659d471-f019-4853-97d0-ee6e948e6c1c","Type":"ContainerStarted","Data":"0c6dec04bcd430c2ac27905afc18684d26ec8266bf527cea65d4edee99a406fa"} Oct 30 11:53:40.356913 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:40.356896 2545 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b8bxz_644e19b3-10ad-4145-9ed3-595d19ed9b07/ovn-acl-logging/0.log" Oct 30 11:53:40.357233 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:40.357214 2545 generic.go:356] "Generic (PLEG): container finished" podID="644e19b3-10ad-4145-9ed3-595d19ed9b07" containerID="57452585cea9d773026a664a5799328c1d6d6edbe6d3bfee62b1b81185fb9e49" exitCode=1 Oct 30 11:53:40.357287 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:40.357246 2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b8bxz" event={"ID":"644e19b3-10ad-4145-9ed3-595d19ed9b07","Type":"ContainerStarted","Data":"7004c40f6bc1704229dac4b706a8570216eb2c4f8d981d2391eacd6a48bc349b"} Oct 30 11:53:40.357318 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:40.357287 2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b8bxz" event={"ID":"644e19b3-10ad-4145-9ed3-595d19ed9b07","Type":"ContainerStarted","Data":"13d174e9e2f595602ae4d70c010f129392d637612254cebc57b6fb7c2356d2c3"} Oct 30 11:53:40.357318 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:40.357316 2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b8bxz" event={"ID":"644e19b3-10ad-4145-9ed3-595d19ed9b07","Type":"ContainerStarted","Data":"aef6f20ca0f2e2307e3a9f7b7ff91f7b0ea073ad333082a1579da554e1750e23"} Oct 30 11:53:40.357369 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:40.357342 2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b8bxz" event={"ID":"644e19b3-10ad-4145-9ed3-595d19ed9b07","Type":"ContainerStarted","Data":"3cf7c452862e9e86f86b4ba41c75d7b4023f0d0f1f4cc430aa2d70ecf8c0c6a6"} Oct 30 11:53:40.357395 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:40.357369 2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b8bxz" event={"ID":"644e19b3-10ad-4145-9ed3-595d19ed9b07","Type":"ContainerDied","Data":"57452585cea9d773026a664a5799328c1d6d6edbe6d3bfee62b1b81185fb9e49"} Oct 30 11:53:40.357425 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:40.357395 2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b8bxz" event={"ID":"644e19b3-10ad-4145-9ed3-595d19ed9b07","Type":"ContainerStarted","Data":"ed725bf45359e698cc3be3964ab57391bedf3275bf70c3601cd1903c5e417c9b"} Oct 30 11:53:40.424697 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:40.424647 2545 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-s6k8x" podStartSLOduration=7.430133721 podStartE2EDuration="17.424637054s" totalImagesPullingTime="9.994503333s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 11:53:23 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:53:40.424551232 +0000 UTC m=+17.865004462" watchObservedRunningTime="2026-10-30 11:53:40.424637054 +0000 UTC m=+17.865090276" Oct 30 11:53:40.717533 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:40.717480 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d7f8396-e57c-45e5-ae74-a2244db10364-metrics-certs\") pod \"network-metrics-daemon-dbqhj\" (UID: \"1d7f8396-e57c-45e5-ae74-a2244db10364\") " pod="openshift-multus/network-metrics-daemon-dbqhj" Oct 30 11:53:40.717636 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:40.717579 2545 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 11:53:40.717636 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:40.717619 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d7f8396-e57c-45e5-ae74-a2244db10364-metrics-certs podName:1d7f8396-e57c-45e5-ae74-a2244db10364 nodeName:}" failed. No retries permitted until 2026-10-30 11:53:56.717609065 +0000 UTC m=+34.158062276 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1d7f8396-e57c-45e5-ae74-a2244db10364-metrics-certs") pod "network-metrics-daemon-dbqhj" (UID: "1d7f8396-e57c-45e5-ae74-a2244db10364") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 11:53:40.818703 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:40.818679 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6b350710-c48c-4c7e-bce9-e021099e98ca-original-pull-secret\") pod \"global-pull-secret-syncer-fxjlt\" (UID: \"6b350710-c48c-4c7e-bce9-e021099e98ca\") " pod="kube-system/global-pull-secret-syncer-fxjlt" Oct 30 11:53:40.818792 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:40.818765 2545 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 11:53:40.818873 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:40.818806 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b350710-c48c-4c7e-bce9-e021099e98ca-original-pull-secret podName:6b350710-c48c-4c7e-bce9-e021099e98ca nodeName:}" failed. No retries permitted until 2026-10-30 11:53:56.818795062 +0000 UTC m=+34.259248269 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6b350710-c48c-4c7e-bce9-e021099e98ca-original-pull-secret") pod "global-pull-secret-syncer-fxjlt" (UID: "6b350710-c48c-4c7e-bce9-e021099e98ca") : object "kube-system"/"original-pull-secret" not registered Oct 30 11:53:40.919463 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:40.919429 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zh8h9\" (UniqueName: \"kubernetes.io/projected/ff9feaa5-70cc-47fd-a3d1-a9ac6e7eaace-kube-api-access-zh8h9\") pod \"network-check-target-bz65k\" (UID: \"ff9feaa5-70cc-47fd-a3d1-a9ac6e7eaace\") " pod="openshift-network-diagnostics/network-check-target-bz65k" Oct 30 11:53:40.919592 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:40.919577 2545 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 11:53:40.919641 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:40.919596 2545 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 11:53:40.919641 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:40.919606 2545 projected.go:196] Error preparing data for projected volume kube-api-access-zh8h9 for pod openshift-network-diagnostics/network-check-target-bz65k: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 11:53:40.919694 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:40.919652 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ff9feaa5-70cc-47fd-a3d1-a9ac6e7eaace-kube-api-access-zh8h9 podName:ff9feaa5-70cc-47fd-a3d1-a9ac6e7eaace nodeName:}" failed. No retries permitted until 2026-10-30 11:53:56.919639674 +0000 UTC m=+34.360092881 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-zh8h9" (UniqueName: "kubernetes.io/projected/ff9feaa5-70cc-47fd-a3d1-a9ac6e7eaace-kube-api-access-zh8h9") pod "network-check-target-bz65k" (UID: "ff9feaa5-70cc-47fd-a3d1-a9ac6e7eaace") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 11:53:41.100226 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:41.100201 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bz65k" Oct 30 11:53:41.100324 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:41.100278 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bz65k" podUID="ff9feaa5-70cc-47fd-a3d1-a9ac6e7eaace" Oct 30 11:53:41.100398 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:41.100383 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dbqhj" Oct 30 11:53:41.100503 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:41.100485 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dbqhj" podUID="1d7f8396-e57c-45e5-ae74-a2244db10364" Oct 30 11:53:42.099261 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:42.099235 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fxjlt" Oct 30 11:53:42.099617 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:42.099327 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fxjlt" podUID="6b350710-c48c-4c7e-bce9-e021099e98ca" Oct 30 11:53:42.362937 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:42.362866 2545 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b8bxz_644e19b3-10ad-4145-9ed3-595d19ed9b07/ovn-acl-logging/0.log" Oct 30 11:53:42.363195 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:42.363176 2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b8bxz" event={"ID":"644e19b3-10ad-4145-9ed3-595d19ed9b07","Type":"ContainerStarted","Data":"2f023b0a361a63988647fdf8e1bdbd3ccd1da240537e168b6033a1761d9af71b"} Oct 30 11:53:43.100714 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:43.100680 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dbqhj" Oct 30 11:53:43.101229 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:43.100798 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dbqhj" podUID="1d7f8396-e57c-45e5-ae74-a2244db10364" Oct 30 11:53:43.101229 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:43.100856 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bz65k" Oct 30 11:53:43.101229 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:43.100952 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bz65k" podUID="ff9feaa5-70cc-47fd-a3d1-a9ac6e7eaace" Oct 30 11:53:44.099546 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:44.099475 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fxjlt" Oct 30 11:53:44.099666 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:44.099568 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fxjlt" podUID="6b350710-c48c-4c7e-bce9-e021099e98ca" Oct 30 11:53:44.370878 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:44.370799 2545 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b8bxz_644e19b3-10ad-4145-9ed3-595d19ed9b07/ovn-acl-logging/0.log" Oct 30 11:53:44.371594 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:44.371149 2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b8bxz" event={"ID":"644e19b3-10ad-4145-9ed3-595d19ed9b07","Type":"ContainerStarted","Data":"ca2d2ac6f87a453d5cd9ffc12a5843246e569bb1ac3bbf3af082b80390c38f93"} Oct 30 11:53:44.371594 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:44.371500 2545 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-b8bxz" Oct 30 11:53:44.371594 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:44.371536 2545 scope.go:122] "RemoveContainer" containerID="57452585cea9d773026a664a5799328c1d6d6edbe6d3bfee62b1b81185fb9e49" Oct 30 11:53:44.371594 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:44.371558 2545 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-b8bxz" Oct 30 11:53:44.386493 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:44.386471 2545 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-b8bxz" Oct 30 11:53:45.099812 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:45.099785 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bz65k" Oct 30 11:53:45.099991 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:45.099961 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bz65k" podUID="ff9feaa5-70cc-47fd-a3d1-a9ac6e7eaace" Oct 30 11:53:45.100031 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:45.099997 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dbqhj" Oct 30 11:53:45.100090 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:45.100075 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dbqhj" podUID="1d7f8396-e57c-45e5-ae74-a2244db10364" Oct 30 11:53:45.376284 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:45.376221 2545 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b8bxz_644e19b3-10ad-4145-9ed3-595d19ed9b07/ovn-acl-logging/0.log" Oct 30 11:53:45.376697 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:45.376583 2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b8bxz" event={"ID":"644e19b3-10ad-4145-9ed3-595d19ed9b07","Type":"ContainerStarted","Data":"bc6a09f208f35060e939d7f07fbc65a3e5a963cb561c7c888a6ec26c2e769f64"} Oct 30 11:53:45.376833 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:45.376809 2545 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-b8bxz" Oct 30 11:53:45.390863 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:45.390815 2545 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-b8bxz" Oct 30 11:53:45.466271 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:45.466223 2545 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-b8bxz" podStartSLOduration=8.541137594 podStartE2EDuration="22.466211435s" totalImagesPullingTime="13.925073841s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 11:53:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:53:45.453851269 +0000 UTC m=+22.894304500" watchObservedRunningTime="2026-10-30 11:53:45.466211435 +0000 UTC m=+22.906664664" Oct 30 11:53:45.869500 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:45.869252 2545 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-fxjlt"] Oct 30 11:53:45.869650 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:45.869636 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fxjlt" Oct 30 11:53:45.869740 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:45.869722 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fxjlt" podUID="6b350710-c48c-4c7e-bce9-e021099e98ca" Oct 30 11:53:45.869878 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:45.869859 2545 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-bz65k"] Oct 30 11:53:45.869952 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:45.869941 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bz65k" Oct 30 11:53:45.870030 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:45.870015 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bz65k" podUID="ff9feaa5-70cc-47fd-a3d1-a9ac6e7eaace" Oct 30 11:53:45.871752 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:45.871733 2545 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-dbqhj"] Oct 30 11:53:45.871871 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:45.871849 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dbqhj" Oct 30 11:53:45.871977 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:45.871958 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dbqhj" podUID="1d7f8396-e57c-45e5-ae74-a2244db10364" Oct 30 11:53:47.099881 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:47.099848 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bz65k" Oct 30 11:53:47.100303 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:47.099971 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bz65k" podUID="ff9feaa5-70cc-47fd-a3d1-a9ac6e7eaace" Oct 30 11:53:47.100303 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:47.100019 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dbqhj" Oct 30 11:53:47.100303 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:47.100112 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dbqhj" podUID="1d7f8396-e57c-45e5-ae74-a2244db10364" Oct 30 11:53:48.099960 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:48.099930 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fxjlt" Oct 30 11:53:48.100284 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:48.100018 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fxjlt" podUID="6b350710-c48c-4c7e-bce9-e021099e98ca" Oct 30 11:53:49.099645 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:49.099610 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bz65k" Oct 30 11:53:49.099825 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:49.099609 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dbqhj" Oct 30 11:53:49.099825 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:49.099723 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bz65k" podUID="ff9feaa5-70cc-47fd-a3d1-a9ac6e7eaace" Oct 30 11:53:49.099946 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:49.099841 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dbqhj" podUID="1d7f8396-e57c-45e5-ae74-a2244db10364" Oct 30 11:53:49.175555 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:49.175522 2545 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-tdf8d" Oct 30 11:53:49.175984 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:49.175646 2545 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 11:53:49.176387 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:49.176366 2545 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-tdf8d" Oct 30 11:53:50.099389 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:50.099360 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fxjlt" Oct 30 11:53:50.099608 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:50.099454 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fxjlt" podUID="6b350710-c48c-4c7e-bce9-e021099e98ca" Oct 30 11:53:51.101793 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:51.101765 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dbqhj" Oct 30 11:53:51.102213 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:51.101912 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dbqhj" podUID="1d7f8396-e57c-45e5-ae74-a2244db10364" Oct 30 11:53:51.102213 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:51.101951 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bz65k" Oct 30 11:53:51.102213 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:51.102005 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bz65k" podUID="ff9feaa5-70cc-47fd-a3d1-a9ac6e7eaace" Oct 30 11:53:52.099976 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.099948 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fxjlt" Oct 30 11:53:52.100087 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:52.100053 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fxjlt" podUID="6b350710-c48c-4c7e-bce9-e021099e98ca" Oct 30 11:53:52.348156 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.348136 2545 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-251.ec2.internal" event="NodeReady" Oct 30 11:53:52.348475 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.348218 2545 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 30 11:53:52.409061 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.409034 2545 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-k8ldb"] Oct 30 11:53:52.416569 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.416542 2545 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-kg9td"] Oct 30 11:53:52.416786 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.416761 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-k8ldb" Oct 30 11:53:52.420896 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.420879 2545 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-79fd565486-8fl78"] Oct 30 11:53:52.421005 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.420990 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-kg9td" Oct 30 11:53:52.424663 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.423957 2545 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-ccctl"] Oct 30 11:53:52.424663 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.424628 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-79fd565486-8fl78" Oct 30 11:53:52.426789 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.426772 2545 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-zzxhv"] Oct 30 11:53:52.426892 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.426876 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-ccctl" Oct 30 11:53:52.429329 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.429310 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-zzxhv" Oct 30 11:53:52.429557 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.429542 2545 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-mvhwk"] Oct 30 11:53:52.433016 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.432921 2545 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-xxcbn"] Oct 30 11:53:52.433976 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.433926 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-mvhwk" Oct 30 11:53:52.437402 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.437380 2545 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-2fqgf"] Oct 30 11:53:52.438129 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.438111 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xxcbn" Oct 30 11:53:52.441677 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.441661 2545 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-ksxpw"] Oct 30 11:53:52.442481 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.442460 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-2fqgf" Oct 30 11:53:52.447386 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.447367 2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Oct 30 11:53:52.447647 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.447632 2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Oct 30 11:53:52.449106 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.448728 2545 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-5bfdf864c-tzv67"] Oct 30 11:53:52.449200 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.449138 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-ksxpw" Oct 30 11:53:52.457220 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.457194 2545 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-bs4nn\"" Oct 30 11:53:52.457296 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.457232 2545 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Oct 30 11:53:52.457453 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.457355 2545 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Oct 30 11:53:52.459343 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.459325 2545 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Oct 30 11:53:52.459496 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.459480 2545 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-kv86z\"" Oct 30 11:53:52.460020 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.460000 2545 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Oct 30 11:53:52.460117 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.460032 2545 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Oct 30 11:53:52.460117 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.460047 2545 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-s95lg\"" Oct 30 11:53:52.461180 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.461163 2545 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-k8ldb"] Oct 30 11:53:52.461265 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.461193 2545 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-6l6fr"] Oct 30 11:53:52.461325 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.461283 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5bfdf864c-tzv67" Oct 30 11:53:52.486520 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.486498 2545 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-86p78"] Oct 30 11:53:52.487037 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.487015 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6l6fr" Oct 30 11:53:52.499342 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.499307 2545 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xdxkp"] Oct 30 11:53:52.499947 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.499926 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-86p78" Oct 30 11:53:52.504587 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.504567 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cfaad7d0-d30b-4ddc-81cc-2b603416fc29-bound-sa-token\") pod \"image-registry-79fd565486-8fl78\" (UID: \"cfaad7d0-d30b-4ddc-81cc-2b603416fc29\") " pod="openshift-image-registry/image-registry-79fd565486-8fl78" Oct 30 11:53:52.504682 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.504606 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cfaad7d0-d30b-4ddc-81cc-2b603416fc29-ca-trust-extracted\") pod \"image-registry-79fd565486-8fl78\" (UID: \"cfaad7d0-d30b-4ddc-81cc-2b603416fc29\") " pod="openshift-image-registry/image-registry-79fd565486-8fl78" Oct 30 11:53:52.504682 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.504632 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cfaad7d0-d30b-4ddc-81cc-2b603416fc29-registry-certificates\") pod \"image-registry-79fd565486-8fl78\" (UID: \"cfaad7d0-d30b-4ddc-81cc-2b603416fc29\") " pod="openshift-image-registry/image-registry-79fd565486-8fl78" Oct 30 11:53:52.504682 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.504652 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cfaad7d0-d30b-4ddc-81cc-2b603416fc29-installation-pull-secrets\") pod \"image-registry-79fd565486-8fl78\" (UID: \"cfaad7d0-d30b-4ddc-81cc-2b603416fc29\") " pod="openshift-image-registry/image-registry-79fd565486-8fl78" Oct 30 11:53:52.504682 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.504667 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/221ced85-cc15-4e07-82d9-a1a1f3e958a5-tmp-dir\") pod \"dns-default-xxcbn\" (UID: \"221ced85-cc15-4e07-82d9-a1a1f3e958a5\") " pod="openshift-dns/dns-default-xxcbn" Oct 30 11:53:52.504890 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.504703 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cfaad7d0-d30b-4ddc-81cc-2b603416fc29-registry-tls\") pod \"image-registry-79fd565486-8fl78\" (UID: \"cfaad7d0-d30b-4ddc-81cc-2b603416fc29\") " pod="openshift-image-registry/image-registry-79fd565486-8fl78" Oct 30 11:53:52.504890 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.504723 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tp27x\" (UniqueName: \"kubernetes.io/projected/8e44595b-0bdf-407c-bf22-33eb06272cc1-kube-api-access-tp27x\") pod \"kas-connection-checker-685895c7c9-k8ldb\" (UID: \"8e44595b-0bdf-407c-bf22-33eb06272cc1\") " pod="kube-system/kas-connection-checker-685895c7c9-k8ldb" Oct 30 11:53:52.504890 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.504739 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6d976900-ecd0-469c-a9e2-d60708361f1a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-mvhwk\" (UID: \"6d976900-ecd0-469c-a9e2-d60708361f1a\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-mvhwk" Oct 30 11:53:52.504890 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.504759 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fhb8\" (UniqueName: \"kubernetes.io/projected/6d976900-ecd0-469c-a9e2-d60708361f1a-kube-api-access-5fhb8\") pod \"cluster-monitoring-operator-884576c57-mvhwk\" (UID: \"6d976900-ecd0-469c-a9e2-d60708361f1a\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-mvhwk" Oct 30 11:53:52.504890 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.504778 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1757617-7c51-4029-b329-1b6a4630f860-service-ca-bundle\") pod \"insights-operator-7c4986d449-2fqgf\" (UID: \"b1757617-7c51-4029-b329-1b6a4630f860\") " pod="openshift-insights/insights-operator-7c4986d449-2fqgf" Oct 30 11:53:52.504890 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.504801 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b1757617-7c51-4029-b329-1b6a4630f860-tmp\") pod \"insights-operator-7c4986d449-2fqgf\" (UID: \"b1757617-7c51-4029-b329-1b6a4630f860\") " pod="openshift-insights/insights-operator-7c4986d449-2fqgf" Oct 30 11:53:52.504890 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.504819 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b1757617-7c51-4029-b329-1b6a4630f860-serving-cert\") pod \"insights-operator-7c4986d449-2fqgf\" (UID: \"b1757617-7c51-4029-b329-1b6a4630f860\") " pod="openshift-insights/insights-operator-7c4986d449-2fqgf" Oct 30 11:53:52.504890 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.504872 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cfaad7d0-d30b-4ddc-81cc-2b603416fc29-image-registry-private-configuration\") pod \"image-registry-79fd565486-8fl78\" (UID: \"cfaad7d0-d30b-4ddc-81cc-2b603416fc29\") " pod="openshift-image-registry/image-registry-79fd565486-8fl78" Oct 30 11:53:52.504890 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.504892 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/179d55d3-7958-4928-a98c-68984fca9837-nginx-conf\") pod \"networking-console-plugin-86c478bb88-zzxhv\" (UID: \"179d55d3-7958-4928-a98c-68984fca9837\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-zzxhv" Oct 30 11:53:52.505333 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.504918 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/179d55d3-7958-4928-a98c-68984fca9837-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-zzxhv\" (UID: \"179d55d3-7958-4928-a98c-68984fca9837\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-zzxhv" Oct 30 11:53:52.505333 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.504937 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhxmm\" (UniqueName: \"kubernetes.io/projected/221ced85-cc15-4e07-82d9-a1a1f3e958a5-kube-api-access-bhxmm\") pod \"dns-default-xxcbn\" (UID: \"221ced85-cc15-4e07-82d9-a1a1f3e958a5\") " pod="openshift-dns/dns-default-xxcbn" Oct 30 11:53:52.505333 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.504952 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7mvn\" (UniqueName: \"kubernetes.io/projected/d0e0cec2-f16a-4bae-bc70-6c08da4bda2e-kube-api-access-h7mvn\") pod \"volume-data-source-validator-58c9f5f46d-ksxpw\" (UID: \"d0e0cec2-f16a-4bae-bc70-6c08da4bda2e\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-ksxpw" Oct 30 11:53:52.505333 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.504971 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/6d976900-ecd0-469c-a9e2-d60708361f1a-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-mvhwk\" (UID: \"6d976900-ecd0-469c-a9e2-d60708361f1a\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-mvhwk" Oct 30 11:53:52.505333 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.504989 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/b1757617-7c51-4029-b329-1b6a4630f860-snapshots\") pod \"insights-operator-7c4986d449-2fqgf\" (UID: \"b1757617-7c51-4029-b329-1b6a4630f860\") " pod="openshift-insights/insights-operator-7c4986d449-2fqgf" Oct 30 11:53:52.505333 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.505007 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1757617-7c51-4029-b329-1b6a4630f860-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-2fqgf\" (UID: \"b1757617-7c51-4029-b329-1b6a4630f860\") " pod="openshift-insights/insights-operator-7c4986d449-2fqgf" Oct 30 11:53:52.505333 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.505028 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/221ced85-cc15-4e07-82d9-a1a1f3e958a5-metrics-tls\") pod \"dns-default-xxcbn\" (UID: \"221ced85-cc15-4e07-82d9-a1a1f3e958a5\") " pod="openshift-dns/dns-default-xxcbn" Oct 30 11:53:52.505333 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.505065 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jv6z4\" (UniqueName: \"kubernetes.io/projected/e497352b-2348-4fc7-888f-5b68f116394f-kube-api-access-jv6z4\") pod \"kas-connection-checker-685895c7c9-kg9td\" (UID: \"e497352b-2348-4fc7-888f-5b68f116394f\") " pod="kube-system/kas-connection-checker-685895c7c9-kg9td" Oct 30 11:53:52.505333 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.505094 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cfaad7d0-d30b-4ddc-81cc-2b603416fc29-trusted-ca\") pod \"image-registry-79fd565486-8fl78\" (UID: \"cfaad7d0-d30b-4ddc-81cc-2b603416fc29\") " pod="openshift-image-registry/image-registry-79fd565486-8fl78" Oct 30 11:53:52.505333 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.505144 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/221ced85-cc15-4e07-82d9-a1a1f3e958a5-config-volume\") pod \"dns-default-xxcbn\" (UID: \"221ced85-cc15-4e07-82d9-a1a1f3e958a5\") " pod="openshift-dns/dns-default-xxcbn" Oct 30 11:53:52.505333 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.505178 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75fhw\" (UniqueName: \"kubernetes.io/projected/cfaad7d0-d30b-4ddc-81cc-2b603416fc29-kube-api-access-75fhw\") pod \"image-registry-79fd565486-8fl78\" (UID: \"cfaad7d0-d30b-4ddc-81cc-2b603416fc29\") " pod="openshift-image-registry/image-registry-79fd565486-8fl78" Oct 30 11:53:52.505333 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.505209 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzld2\" (UniqueName: \"kubernetes.io/projected/1f5667a1-a773-4520-b2fb-c69148df632c-kube-api-access-bzld2\") pod \"kas-connection-checker-685895c7c9-ccctl\" (UID: \"1f5667a1-a773-4520-b2fb-c69148df632c\") " pod="kube-system/kas-connection-checker-685895c7c9-ccctl" Oct 30 11:53:52.505333 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.505236 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zscd2\" (UniqueName: \"kubernetes.io/projected/b1757617-7c51-4029-b329-1b6a4630f860-kube-api-access-zscd2\") pod \"insights-operator-7c4986d449-2fqgf\" (UID: \"b1757617-7c51-4029-b329-1b6a4630f860\") " pod="openshift-insights/insights-operator-7c4986d449-2fqgf" Oct 30 11:53:52.513320 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.513301 2545 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-plczt"] Oct 30 11:53:52.513405 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.513390 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xdxkp" Oct 30 11:53:52.527575 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.527556 2545 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-kg9td"] Oct 30 11:53:52.527656 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.527585 2545 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-glvhs"] Oct 30 11:53:52.527695 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.527672 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-plczt" Oct 30 11:53:52.539810 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.539791 2545 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-557c6b9f47-vcr2t"] Oct 30 11:53:52.539914 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.539894 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-glvhs" Oct 30 11:53:52.546515 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.546499 2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Oct 30 11:53:52.546776 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.546760 2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Oct 30 11:53:52.546915 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.546898 2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Oct 30 11:53:52.547006 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.546854 2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Oct 30 11:53:52.547006 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.546765 2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Oct 30 11:53:52.547006 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.546963 2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 11:53:52.547152 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.547032 2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 11:53:52.547152 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.547036 2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 11:53:52.547152 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.547048 2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Oct 30 11:53:52.547152 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.547077 2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Oct 30 11:53:52.547152 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.547086 2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Oct 30 11:53:52.547388 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.547202 2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 30 11:53:52.547388 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.547292 2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 11:53:52.547388 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.547355 2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Oct 30 11:53:52.547679 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.547664 2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Oct 30 11:53:52.547854 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.547821 2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Oct 30 11:53:52.549483 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.549469 2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Oct 30 11:53:52.549544 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.549490 2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 11:53:52.549776 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.549764 2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Oct 30 11:53:52.550128 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.550113 2545 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-gn2z5"] Oct 30 11:53:52.550229 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.550217 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-557c6b9f47-vcr2t" Oct 30 11:53:52.559332 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.559314 2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Oct 30 11:53:52.559418 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.559368 2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 11:53:52.559418 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.559412 2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 11:53:52.559819 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.559797 2545 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Oct 30 11:53:52.560112 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.560099 2545 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Oct 30 11:53:52.560183 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.560149 2545 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Oct 30 11:53:52.560183 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.560166 2545 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Oct 30 11:53:52.560183 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.560172 2545 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Oct 30 11:53:52.560465 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.560446 2545 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Oct 30 11:53:52.560521 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.560465 2545 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-f2lr4\"" Oct 30 11:53:52.560637 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.560624 2545 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-vcl76\"" Oct 30 11:53:52.560704 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.560654 2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Oct 30 11:53:52.560704 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.560693 2545 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Oct 30 11:53:52.560785 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.560706 2545 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 11:53:52.560785 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.560723 2545 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Oct 30 11:53:52.560785 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.560728 2545 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-cng9g\"" Oct 30 11:53:52.560785 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.560733 2545 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-qndch\"" Oct 30 11:53:52.560785 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.560745 2545 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-shg4w\"" Oct 30 11:53:52.560989 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.560804 2545 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-b9dpg\"" Oct 30 11:53:52.560989 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.560861 2545 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-zjl8c\"" Oct 30 11:53:52.560989 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.560891 2545 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-m27bb\"" Oct 30 11:53:52.560989 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.560949 2545 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-96ksc\"" Oct 30 11:53:52.561104 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.560990 2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 30 11:53:52.562051 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.562038 2545 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-79fd565486-8fl78"] Oct 30 11:53:52.562118 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.562061 2545 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-zzxhv"] Oct 30 11:53:52.562118 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.562076 2545 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-ksxpw"] Oct 30 11:53:52.562118 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.562089 2545 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-xxcbn"] Oct 30 11:53:52.562118 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.562104 2545 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-ccctl"] Oct 30 11:53:52.562118 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.562116 2545 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-5bfdf864c-tzv67"] Oct 30 11:53:52.562361 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.562128 2545 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-6l6fr"] Oct 30 11:53:52.562361 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.562141 2545 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-86p78"] Oct 30 11:53:52.562361 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.562153 2545 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-gn2z5"] Oct 30 11:53:52.562361 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.562156 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-gn2z5" Oct 30 11:53:52.562361 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.562165 2545 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-2fqgf"] Oct 30 11:53:52.562361 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.562180 2545 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-plczt"] Oct 30 11:53:52.562361 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.562193 2545 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-mvhwk"] Oct 30 11:53:52.562361 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.562205 2545 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xdxkp"] Oct 30 11:53:52.562361 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.562227 2545 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-glvhs"] Oct 30 11:53:52.562361 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.562245 2545 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-557c6b9f47-vcr2t"] Oct 30 11:53:52.571915 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.571900 2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Oct 30 11:53:52.605590 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.605543 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gr67s\" (UniqueName: \"kubernetes.io/projected/7689cfe5-5a4e-4481-8eb9-107a58ca8bb3-kube-api-access-gr67s\") pod \"cluster-samples-operator-6ddb6c8b94-xdxkp\" (UID: \"7689cfe5-5a4e-4481-8eb9-107a58ca8bb3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xdxkp" Oct 30 11:53:52.605590 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.605573 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cfaad7d0-d30b-4ddc-81cc-2b603416fc29-image-registry-private-configuration\") pod \"image-registry-79fd565486-8fl78\" (UID: \"cfaad7d0-d30b-4ddc-81cc-2b603416fc29\") " pod="openshift-image-registry/image-registry-79fd565486-8fl78" Oct 30 11:53:52.605590 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.605590 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6d976900-ecd0-469c-a9e2-d60708361f1a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-mvhwk\" (UID: \"6d976900-ecd0-469c-a9e2-d60708361f1a\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-mvhwk" Oct 30 11:53:52.605792 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.605607 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1757617-7c51-4029-b329-1b6a4630f860-service-ca-bundle\") pod \"insights-operator-7c4986d449-2fqgf\" (UID: \"b1757617-7c51-4029-b329-1b6a4630f860\") " pod="openshift-insights/insights-operator-7c4986d449-2fqgf" Oct 30 11:53:52.605792 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.605624 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rxc4\" (UniqueName: \"kubernetes.io/projected/6acb13a1-cb2e-45e1-897b-3ef89e6d1b84-kube-api-access-2rxc4\") pod \"router-default-5bfdf864c-tzv67\" (UID: \"6acb13a1-cb2e-45e1-897b-3ef89e6d1b84\") " pod="openshift-ingress/router-default-5bfdf864c-tzv67" Oct 30 11:53:52.605792 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.605681 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/6d976900-ecd0-469c-a9e2-d60708361f1a-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-mvhwk\" (UID: \"6d976900-ecd0-469c-a9e2-d60708361f1a\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-mvhwk" Oct 30 11:53:52.605792 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:52.605740 2545 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 11:53:52.606108 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:52.605985 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d976900-ecd0-469c-a9e2-d60708361f1a-cluster-monitoring-operator-tls podName:6d976900-ecd0-469c-a9e2-d60708361f1a nodeName:}" failed. No retries permitted until 2026-10-30 11:53:53.105963465 +0000 UTC m=+30.546416686 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/6d976900-ecd0-469c-a9e2-d60708361f1a-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-mvhwk" (UID: "6d976900-ecd0-469c-a9e2-d60708361f1a") : secret "cluster-monitoring-operator-tls" not found Oct 30 11:53:52.606291 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.606263 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jv6z4\" (UniqueName: \"kubernetes.io/projected/e497352b-2348-4fc7-888f-5b68f116394f-kube-api-access-jv6z4\") pod \"kas-connection-checker-685895c7c9-kg9td\" (UID: \"e497352b-2348-4fc7-888f-5b68f116394f\") " pod="kube-system/kas-connection-checker-685895c7c9-kg9td" Oct 30 11:53:52.606357 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.606297 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzld2\" (UniqueName: \"kubernetes.io/projected/1f5667a1-a773-4520-b2fb-c69148df632c-kube-api-access-bzld2\") pod \"kas-connection-checker-685895c7c9-ccctl\" (UID: \"1f5667a1-a773-4520-b2fb-c69148df632c\") " pod="kube-system/kas-connection-checker-685895c7c9-ccctl" Oct 30 11:53:52.606357 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.606324 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r226b\" (UniqueName: \"kubernetes.io/projected/b3f20243-155c-4e25-a1ca-6131a141fbd4-kube-api-access-r226b\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-86p78\" (UID: \"b3f20243-155c-4e25-a1ca-6131a141fbd4\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-86p78" Oct 30 11:53:52.606357 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.606349 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a70ee11c-c20e-4a4f-8476-ac5eb02816d0-trusted-ca\") pod \"console-operator-7cd56b8dbc-plczt\" (UID: \"a70ee11c-c20e-4a4f-8476-ac5eb02816d0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-plczt" Oct 30 11:53:52.606503 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.606386 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jf44s\" (UniqueName: \"kubernetes.io/projected/c815b3fc-7f35-4ae6-a015-2e84c6108a64-kube-api-access-jf44s\") pod \"service-ca-operator-544d749c8c-6l6fr\" (UID: \"c815b3fc-7f35-4ae6-a015-2e84c6108a64\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6l6fr" Oct 30 11:53:52.606503 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.606487 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c815b3fc-7f35-4ae6-a015-2e84c6108a64-serving-cert\") pod \"service-ca-operator-544d749c8c-6l6fr\" (UID: \"c815b3fc-7f35-4ae6-a015-2e84c6108a64\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6l6fr" Oct 30 11:53:52.606598 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.606503 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/6d976900-ecd0-469c-a9e2-d60708361f1a-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-mvhwk\" (UID: \"6d976900-ecd0-469c-a9e2-d60708361f1a\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-mvhwk" Oct 30 11:53:52.606598 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.606514 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1757617-7c51-4029-b329-1b6a4630f860-service-ca-bundle\") pod \"insights-operator-7c4986d449-2fqgf\" (UID: \"b1757617-7c51-4029-b329-1b6a4630f860\") " pod="openshift-insights/insights-operator-7c4986d449-2fqgf" Oct 30 11:53:52.606598 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.606553 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/221ced85-cc15-4e07-82d9-a1a1f3e958a5-tmp-dir\") pod \"dns-default-xxcbn\" (UID: \"221ced85-cc15-4e07-82d9-a1a1f3e958a5\") " pod="openshift-dns/dns-default-xxcbn" Oct 30 11:53:52.610489 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.610464 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cfaad7d0-d30b-4ddc-81cc-2b603416fc29-image-registry-private-configuration\") pod \"image-registry-79fd565486-8fl78\" (UID: \"cfaad7d0-d30b-4ddc-81cc-2b603416fc29\") " pod="openshift-image-registry/image-registry-79fd565486-8fl78" Oct 30 11:53:52.610572 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.610536 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/6acb13a1-cb2e-45e1-897b-3ef89e6d1b84-default-certificate\") pod \"router-default-5bfdf864c-tzv67\" (UID: \"6acb13a1-cb2e-45e1-897b-3ef89e6d1b84\") " pod="openshift-ingress/router-default-5bfdf864c-tzv67" Oct 30 11:53:52.610639 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.610588 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bd977255-f9a7-4531-8a9c-b1a3a17e0b79-image-registry-private-configuration\") pod \"image-registry-557c6b9f47-vcr2t\" (UID: \"bd977255-f9a7-4531-8a9c-b1a3a17e0b79\") " pod="openshift-image-registry/image-registry-557c6b9f47-vcr2t" Oct 30 11:53:52.610639 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.610627 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b1757617-7c51-4029-b329-1b6a4630f860-serving-cert\") pod \"insights-operator-7c4986d449-2fqgf\" (UID: \"b1757617-7c51-4029-b329-1b6a4630f860\") " pod="openshift-insights/insights-operator-7c4986d449-2fqgf" Oct 30 11:53:52.610725 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.610652 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7689cfe5-5a4e-4481-8eb9-107a58ca8bb3-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-xdxkp\" (UID: \"7689cfe5-5a4e-4481-8eb9-107a58ca8bb3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xdxkp" Oct 30 11:53:52.610725 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.610686 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tp27x\" (UniqueName: \"kubernetes.io/projected/8e44595b-0bdf-407c-bf22-33eb06272cc1-kube-api-access-tp27x\") pod \"kas-connection-checker-685895c7c9-k8ldb\" (UID: \"8e44595b-0bdf-407c-bf22-33eb06272cc1\") " pod="kube-system/kas-connection-checker-685895c7c9-k8ldb" Oct 30 11:53:52.610725 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.610717 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bd977255-f9a7-4531-8a9c-b1a3a17e0b79-ca-trust-extracted\") pod \"image-registry-557c6b9f47-vcr2t\" (UID: \"bd977255-f9a7-4531-8a9c-b1a3a17e0b79\") " pod="openshift-image-registry/image-registry-557c6b9f47-vcr2t" Oct 30 11:53:52.610889 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.610746 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bd977255-f9a7-4531-8a9c-b1a3a17e0b79-trusted-ca\") pod \"image-registry-557c6b9f47-vcr2t\" (UID: \"bd977255-f9a7-4531-8a9c-b1a3a17e0b79\") " pod="openshift-image-registry/image-registry-557c6b9f47-vcr2t" Oct 30 11:53:52.610889 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.610777 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b3f20243-155c-4e25-a1ca-6131a141fbd4-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-86p78\" (UID: \"b3f20243-155c-4e25-a1ca-6131a141fbd4\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-86p78" Oct 30 11:53:52.610889 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.610805 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a70ee11c-c20e-4a4f-8476-ac5eb02816d0-serving-cert\") pod \"console-operator-7cd56b8dbc-plczt\" (UID: \"a70ee11c-c20e-4a4f-8476-ac5eb02816d0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-plczt" Oct 30 11:53:52.610889 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.610859 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/179d55d3-7958-4928-a98c-68984fca9837-nginx-conf\") pod \"networking-console-plugin-86c478bb88-zzxhv\" (UID: \"179d55d3-7958-4928-a98c-68984fca9837\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-zzxhv" Oct 30 11:53:52.611078 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.610891 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/b1757617-7c51-4029-b329-1b6a4630f860-snapshots\") pod \"insights-operator-7c4986d449-2fqgf\" (UID: \"b1757617-7c51-4029-b329-1b6a4630f860\") " pod="openshift-insights/insights-operator-7c4986d449-2fqgf" Oct 30 11:53:52.611078 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.610921 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fhb8\" (UniqueName: \"kubernetes.io/projected/6d976900-ecd0-469c-a9e2-d60708361f1a-kube-api-access-5fhb8\") pod \"cluster-monitoring-operator-884576c57-mvhwk\" (UID: \"6d976900-ecd0-469c-a9e2-d60708361f1a\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-mvhwk" Oct 30 11:53:52.611078 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.610960 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/179d55d3-7958-4928-a98c-68984fca9837-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-zzxhv\" (UID: \"179d55d3-7958-4928-a98c-68984fca9837\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-zzxhv" Oct 30 11:53:52.611078 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.610983 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhxmm\" (UniqueName: \"kubernetes.io/projected/221ced85-cc15-4e07-82d9-a1a1f3e958a5-kube-api-access-bhxmm\") pod \"dns-default-xxcbn\" (UID: \"221ced85-cc15-4e07-82d9-a1a1f3e958a5\") " pod="openshift-dns/dns-default-xxcbn" Oct 30 11:53:52.611078 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.611012 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7mvn\" (UniqueName: \"kubernetes.io/projected/d0e0cec2-f16a-4bae-bc70-6c08da4bda2e-kube-api-access-h7mvn\") pod \"volume-data-source-validator-58c9f5f46d-ksxpw\" (UID: \"d0e0cec2-f16a-4bae-bc70-6c08da4bda2e\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-ksxpw" Oct 30 11:53:52.611078 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.611042 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1757617-7c51-4029-b329-1b6a4630f860-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-2fqgf\" (UID: \"b1757617-7c51-4029-b329-1b6a4630f860\") " pod="openshift-insights/insights-operator-7c4986d449-2fqgf" Oct 30 11:53:52.611078 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.611071 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hqrf\" (UniqueName: \"kubernetes.io/projected/7ac30952-da5d-42a3-a1aa-bd249bbf6adb-kube-api-access-6hqrf\") pod \"network-check-source-6f78d59f99-glvhs\" (UID: \"7ac30952-da5d-42a3-a1aa-bd249bbf6adb\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-glvhs" Oct 30 11:53:52.611526 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.611099 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vx4mp\" (UniqueName: \"kubernetes.io/projected/a70ee11c-c20e-4a4f-8476-ac5eb02816d0-kube-api-access-vx4mp\") pod \"console-operator-7cd56b8dbc-plczt\" (UID: \"a70ee11c-c20e-4a4f-8476-ac5eb02816d0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-plczt" Oct 30 11:53:52.611526 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.611128 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b3f20243-155c-4e25-a1ca-6131a141fbd4-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-86p78\" (UID: \"b3f20243-155c-4e25-a1ca-6131a141fbd4\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-86p78" Oct 30 11:53:52.611526 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.611157 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/221ced85-cc15-4e07-82d9-a1a1f3e958a5-metrics-tls\") pod \"dns-default-xxcbn\" (UID: \"221ced85-cc15-4e07-82d9-a1a1f3e958a5\") " pod="openshift-dns/dns-default-xxcbn" Oct 30 11:53:52.611526 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.611185 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cfaad7d0-d30b-4ddc-81cc-2b603416fc29-trusted-ca\") pod \"image-registry-79fd565486-8fl78\" (UID: \"cfaad7d0-d30b-4ddc-81cc-2b603416fc29\") " pod="openshift-image-registry/image-registry-79fd565486-8fl78" Oct 30 11:53:52.611526 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.611220 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6acb13a1-cb2e-45e1-897b-3ef89e6d1b84-metrics-certs\") pod \"router-default-5bfdf864c-tzv67\" (UID: \"6acb13a1-cb2e-45e1-897b-3ef89e6d1b84\") " pod="openshift-ingress/router-default-5bfdf864c-tzv67" Oct 30 11:53:52.611526 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.611244 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bd977255-f9a7-4531-8a9c-b1a3a17e0b79-registry-certificates\") pod \"image-registry-557c6b9f47-vcr2t\" (UID: \"bd977255-f9a7-4531-8a9c-b1a3a17e0b79\") " pod="openshift-image-registry/image-registry-557c6b9f47-vcr2t" Oct 30 11:53:52.611526 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.611274 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a70ee11c-c20e-4a4f-8476-ac5eb02816d0-config\") pod \"console-operator-7cd56b8dbc-plczt\" (UID: \"a70ee11c-c20e-4a4f-8476-ac5eb02816d0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-plczt" Oct 30 11:53:52.611526 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.611300 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/6acb13a1-cb2e-45e1-897b-3ef89e6d1b84-stats-auth\") pod \"router-default-5bfdf864c-tzv67\" (UID: \"6acb13a1-cb2e-45e1-897b-3ef89e6d1b84\") " pod="openshift-ingress/router-default-5bfdf864c-tzv67" Oct 30 11:53:52.611526 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.611327 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/7689cfe5-5a4e-4481-8eb9-107a58ca8bb3-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-xdxkp\" (UID: \"7689cfe5-5a4e-4481-8eb9-107a58ca8bb3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xdxkp" Oct 30 11:53:52.611526 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.611358 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bd977255-f9a7-4531-8a9c-b1a3a17e0b79-registry-tls\") pod \"image-registry-557c6b9f47-vcr2t\" (UID: \"bd977255-f9a7-4531-8a9c-b1a3a17e0b79\") " pod="openshift-image-registry/image-registry-557c6b9f47-vcr2t" Oct 30 11:53:52.611526 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.611381 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bd977255-f9a7-4531-8a9c-b1a3a17e0b79-installation-pull-secrets\") pod \"image-registry-557c6b9f47-vcr2t\" (UID: \"bd977255-f9a7-4531-8a9c-b1a3a17e0b79\") " pod="openshift-image-registry/image-registry-557c6b9f47-vcr2t" Oct 30 11:53:52.613086 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:52.612175 2545 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 11:53:52.613086 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:52.612245 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/179d55d3-7958-4928-a98c-68984fca9837-networking-console-plugin-cert podName:179d55d3-7958-4928-a98c-68984fca9837 nodeName:}" failed. No retries permitted until 2026-10-30 11:53:53.112228579 +0000 UTC m=+30.552681786 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/179d55d3-7958-4928-a98c-68984fca9837-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-zzxhv" (UID: "179d55d3-7958-4928-a98c-68984fca9837") : secret "networking-console-plugin-cert" not found Oct 30 11:53:52.613757 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:52.613722 2545 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 11:53:52.613884 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:52.613787 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/221ced85-cc15-4e07-82d9-a1a1f3e958a5-metrics-tls podName:221ced85-cc15-4e07-82d9-a1a1f3e958a5 nodeName:}" failed. No retries permitted until 2026-10-30 11:53:53.11377109 +0000 UTC m=+30.554224312 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/221ced85-cc15-4e07-82d9-a1a1f3e958a5-metrics-tls") pod "dns-default-xxcbn" (UID: "221ced85-cc15-4e07-82d9-a1a1f3e958a5") : secret "dns-default-metrics-tls" not found Oct 30 11:53:52.614786 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.614107 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/221ced85-cc15-4e07-82d9-a1a1f3e958a5-config-volume\") pod \"dns-default-xxcbn\" (UID: \"221ced85-cc15-4e07-82d9-a1a1f3e958a5\") " pod="openshift-dns/dns-default-xxcbn" Oct 30 11:53:52.614786 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.614224 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zscd2\" (UniqueName: \"kubernetes.io/projected/b1757617-7c51-4029-b329-1b6a4630f860-kube-api-access-zscd2\") pod \"insights-operator-7c4986d449-2fqgf\" (UID: \"b1757617-7c51-4029-b329-1b6a4630f860\") " pod="openshift-insights/insights-operator-7c4986d449-2fqgf" Oct 30 11:53:52.614786 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.614264 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bd977255-f9a7-4531-8a9c-b1a3a17e0b79-bound-sa-token\") pod \"image-registry-557c6b9f47-vcr2t\" (UID: \"bd977255-f9a7-4531-8a9c-b1a3a17e0b79\") " pod="openshift-image-registry/image-registry-557c6b9f47-vcr2t" Oct 30 11:53:52.614786 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.614371 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8f7sv\" (UniqueName: \"kubernetes.io/projected/bd977255-f9a7-4531-8a9c-b1a3a17e0b79-kube-api-access-8f7sv\") pod \"image-registry-557c6b9f47-vcr2t\" (UID: \"bd977255-f9a7-4531-8a9c-b1a3a17e0b79\") " pod="openshift-image-registry/image-registry-557c6b9f47-vcr2t" Oct 30 11:53:52.614786 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.614405 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75fhw\" (UniqueName: \"kubernetes.io/projected/cfaad7d0-d30b-4ddc-81cc-2b603416fc29-kube-api-access-75fhw\") pod \"image-registry-79fd565486-8fl78\" (UID: \"cfaad7d0-d30b-4ddc-81cc-2b603416fc29\") " pod="openshift-image-registry/image-registry-79fd565486-8fl78" Oct 30 11:53:52.615434 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.615106 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/221ced85-cc15-4e07-82d9-a1a1f3e958a5-tmp-dir\") pod \"dns-default-xxcbn\" (UID: \"221ced85-cc15-4e07-82d9-a1a1f3e958a5\") " pod="openshift-dns/dns-default-xxcbn" Oct 30 11:53:52.616458 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.616439 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/221ced85-cc15-4e07-82d9-a1a1f3e958a5-config-volume\") pod \"dns-default-xxcbn\" (UID: \"221ced85-cc15-4e07-82d9-a1a1f3e958a5\") " pod="openshift-dns/dns-default-xxcbn" Oct 30 11:53:52.618317 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.618268 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/179d55d3-7958-4928-a98c-68984fca9837-nginx-conf\") pod \"networking-console-plugin-86c478bb88-zzxhv\" (UID: \"179d55d3-7958-4928-a98c-68984fca9837\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-zzxhv" Oct 30 11:53:52.618691 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.618668 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cfaad7d0-d30b-4ddc-81cc-2b603416fc29-bound-sa-token\") pod \"image-registry-79fd565486-8fl78\" (UID: \"cfaad7d0-d30b-4ddc-81cc-2b603416fc29\") " pod="openshift-image-registry/image-registry-79fd565486-8fl78" Oct 30 11:53:52.618781 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.618713 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b1757617-7c51-4029-b329-1b6a4630f860-tmp\") pod \"insights-operator-7c4986d449-2fqgf\" (UID: \"b1757617-7c51-4029-b329-1b6a4630f860\") " pod="openshift-insights/insights-operator-7c4986d449-2fqgf" Oct 30 11:53:52.618781 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.618749 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c815b3fc-7f35-4ae6-a015-2e84c6108a64-config\") pod \"service-ca-operator-544d749c8c-6l6fr\" (UID: \"c815b3fc-7f35-4ae6-a015-2e84c6108a64\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6l6fr" Oct 30 11:53:52.618928 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.618787 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cfaad7d0-d30b-4ddc-81cc-2b603416fc29-ca-trust-extracted\") pod \"image-registry-79fd565486-8fl78\" (UID: \"cfaad7d0-d30b-4ddc-81cc-2b603416fc29\") " pod="openshift-image-registry/image-registry-79fd565486-8fl78" Oct 30 11:53:52.618928 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.618819 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cfaad7d0-d30b-4ddc-81cc-2b603416fc29-registry-certificates\") pod \"image-registry-79fd565486-8fl78\" (UID: \"cfaad7d0-d30b-4ddc-81cc-2b603416fc29\") " pod="openshift-image-registry/image-registry-79fd565486-8fl78" Oct 30 11:53:52.618928 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.618881 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cfaad7d0-d30b-4ddc-81cc-2b603416fc29-trusted-ca\") pod \"image-registry-79fd565486-8fl78\" (UID: \"cfaad7d0-d30b-4ddc-81cc-2b603416fc29\") " pod="openshift-image-registry/image-registry-79fd565486-8fl78" Oct 30 11:53:52.618928 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.618711 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/b1757617-7c51-4029-b329-1b6a4630f860-snapshots\") pod \"insights-operator-7c4986d449-2fqgf\" (UID: \"b1757617-7c51-4029-b329-1b6a4630f860\") " pod="openshift-insights/insights-operator-7c4986d449-2fqgf" Oct 30 11:53:52.619108 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.618937 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cfaad7d0-d30b-4ddc-81cc-2b603416fc29-installation-pull-secrets\") pod \"image-registry-79fd565486-8fl78\" (UID: \"cfaad7d0-d30b-4ddc-81cc-2b603416fc29\") " pod="openshift-image-registry/image-registry-79fd565486-8fl78" Oct 30 11:53:52.619108 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.619013 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cfaad7d0-d30b-4ddc-81cc-2b603416fc29-registry-tls\") pod \"image-registry-79fd565486-8fl78\" (UID: \"cfaad7d0-d30b-4ddc-81cc-2b603416fc29\") " pod="openshift-image-registry/image-registry-79fd565486-8fl78" Oct 30 11:53:52.619108 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.619048 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6acb13a1-cb2e-45e1-897b-3ef89e6d1b84-service-ca-bundle\") pod \"router-default-5bfdf864c-tzv67\" (UID: \"6acb13a1-cb2e-45e1-897b-3ef89e6d1b84\") " pod="openshift-ingress/router-default-5bfdf864c-tzv67" Oct 30 11:53:52.619248 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.619177 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b1757617-7c51-4029-b329-1b6a4630f860-tmp\") pod \"insights-operator-7c4986d449-2fqgf\" (UID: \"b1757617-7c51-4029-b329-1b6a4630f860\") " pod="openshift-insights/insights-operator-7c4986d449-2fqgf" Oct 30 11:53:52.619640 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:52.619541 2545 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 11:53:52.619640 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:52.619567 2545 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-79fd565486-8fl78: secret "image-registry-tls" not found Oct 30 11:53:52.619640 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.619606 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cfaad7d0-d30b-4ddc-81cc-2b603416fc29-ca-trust-extracted\") pod \"image-registry-79fd565486-8fl78\" (UID: \"cfaad7d0-d30b-4ddc-81cc-2b603416fc29\") " pod="openshift-image-registry/image-registry-79fd565486-8fl78" Oct 30 11:53:52.619640 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.619630 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cfaad7d0-d30b-4ddc-81cc-2b603416fc29-registry-certificates\") pod \"image-registry-79fd565486-8fl78\" (UID: \"cfaad7d0-d30b-4ddc-81cc-2b603416fc29\") " pod="openshift-image-registry/image-registry-79fd565486-8fl78" Oct 30 11:53:52.619640 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:52.619617 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cfaad7d0-d30b-4ddc-81cc-2b603416fc29-registry-tls podName:cfaad7d0-d30b-4ddc-81cc-2b603416fc29 nodeName:}" failed. No retries permitted until 2026-10-30 11:53:53.119595332 +0000 UTC m=+30.560048555 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cfaad7d0-d30b-4ddc-81cc-2b603416fc29-registry-tls") pod "image-registry-79fd565486-8fl78" (UID: "cfaad7d0-d30b-4ddc-81cc-2b603416fc29") : secret "image-registry-tls" not found Oct 30 11:53:52.620659 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.620635 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1757617-7c51-4029-b329-1b6a4630f860-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-2fqgf\" (UID: \"b1757617-7c51-4029-b329-1b6a4630f860\") " pod="openshift-insights/insights-operator-7c4986d449-2fqgf" Oct 30 11:53:52.621769 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.621746 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cfaad7d0-d30b-4ddc-81cc-2b603416fc29-installation-pull-secrets\") pod \"image-registry-79fd565486-8fl78\" (UID: \"cfaad7d0-d30b-4ddc-81cc-2b603416fc29\") " pod="openshift-image-registry/image-registry-79fd565486-8fl78" Oct 30 11:53:52.622393 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.622367 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b1757617-7c51-4029-b329-1b6a4630f860-serving-cert\") pod \"insights-operator-7c4986d449-2fqgf\" (UID: \"b1757617-7c51-4029-b329-1b6a4630f860\") " pod="openshift-insights/insights-operator-7c4986d449-2fqgf" Oct 30 11:53:52.632970 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.632947 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jv6z4\" (UniqueName: \"kubernetes.io/projected/e497352b-2348-4fc7-888f-5b68f116394f-kube-api-access-jv6z4\") pod \"kas-connection-checker-685895c7c9-kg9td\" (UID: \"e497352b-2348-4fc7-888f-5b68f116394f\") " pod="kube-system/kas-connection-checker-685895c7c9-kg9td" Oct 30 11:53:52.634281 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.634258 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzld2\" (UniqueName: \"kubernetes.io/projected/1f5667a1-a773-4520-b2fb-c69148df632c-kube-api-access-bzld2\") pod \"kas-connection-checker-685895c7c9-ccctl\" (UID: \"1f5667a1-a773-4520-b2fb-c69148df632c\") " pod="kube-system/kas-connection-checker-685895c7c9-ccctl" Oct 30 11:53:52.645687 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.645669 2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Oct 30 11:53:52.645751 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.645722 2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Oct 30 11:53:52.645801 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.645767 2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 11:53:52.645801 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.645773 2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 11:53:52.645801 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.645769 2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 11:53:52.645919 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.645823 2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Oct 30 11:53:52.645919 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.645857 2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 11:53:52.650675 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.650656 2545 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-tdxpc\"" Oct 30 11:53:52.650768 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.650732 2545 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 11:53:52.652389 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.652371 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fhb8\" (UniqueName: \"kubernetes.io/projected/6d976900-ecd0-469c-a9e2-d60708361f1a-kube-api-access-5fhb8\") pod \"cluster-monitoring-operator-884576c57-mvhwk\" (UID: \"6d976900-ecd0-469c-a9e2-d60708361f1a\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-mvhwk" Oct 30 11:53:52.652849 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.652801 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cfaad7d0-d30b-4ddc-81cc-2b603416fc29-bound-sa-token\") pod \"image-registry-79fd565486-8fl78\" (UID: \"cfaad7d0-d30b-4ddc-81cc-2b603416fc29\") " pod="openshift-image-registry/image-registry-79fd565486-8fl78" Oct 30 11:53:52.652950 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.652936 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7mvn\" (UniqueName: \"kubernetes.io/projected/d0e0cec2-f16a-4bae-bc70-6c08da4bda2e-kube-api-access-h7mvn\") pod \"volume-data-source-validator-58c9f5f46d-ksxpw\" (UID: \"d0e0cec2-f16a-4bae-bc70-6c08da4bda2e\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-ksxpw" Oct 30 11:53:52.652950 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.652943 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhxmm\" (UniqueName: \"kubernetes.io/projected/221ced85-cc15-4e07-82d9-a1a1f3e958a5-kube-api-access-bhxmm\") pod \"dns-default-xxcbn\" (UID: \"221ced85-cc15-4e07-82d9-a1a1f3e958a5\") " pod="openshift-dns/dns-default-xxcbn" Oct 30 11:53:52.653059 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.652935 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zscd2\" (UniqueName: \"kubernetes.io/projected/b1757617-7c51-4029-b329-1b6a4630f860-kube-api-access-zscd2\") pod \"insights-operator-7c4986d449-2fqgf\" (UID: \"b1757617-7c51-4029-b329-1b6a4630f860\") " pod="openshift-insights/insights-operator-7c4986d449-2fqgf" Oct 30 11:53:52.653676 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.653658 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tp27x\" (UniqueName: \"kubernetes.io/projected/8e44595b-0bdf-407c-bf22-33eb06272cc1-kube-api-access-tp27x\") pod \"kas-connection-checker-685895c7c9-k8ldb\" (UID: \"8e44595b-0bdf-407c-bf22-33eb06272cc1\") " pod="kube-system/kas-connection-checker-685895c7c9-k8ldb" Oct 30 11:53:52.653898 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.653881 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-75fhw\" (UniqueName: \"kubernetes.io/projected/cfaad7d0-d30b-4ddc-81cc-2b603416fc29-kube-api-access-75fhw\") pod \"image-registry-79fd565486-8fl78\" (UID: \"cfaad7d0-d30b-4ddc-81cc-2b603416fc29\") " pod="openshift-image-registry/image-registry-79fd565486-8fl78" Oct 30 11:53:52.719765 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.719741 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r226b\" (UniqueName: \"kubernetes.io/projected/b3f20243-155c-4e25-a1ca-6131a141fbd4-kube-api-access-r226b\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-86p78\" (UID: \"b3f20243-155c-4e25-a1ca-6131a141fbd4\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-86p78" Oct 30 11:53:52.719879 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.719769 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a70ee11c-c20e-4a4f-8476-ac5eb02816d0-trusted-ca\") pod \"console-operator-7cd56b8dbc-plczt\" (UID: \"a70ee11c-c20e-4a4f-8476-ac5eb02816d0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-plczt" Oct 30 11:53:52.719879 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.719786 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jf44s\" (UniqueName: \"kubernetes.io/projected/c815b3fc-7f35-4ae6-a015-2e84c6108a64-kube-api-access-jf44s\") pod \"service-ca-operator-544d749c8c-6l6fr\" (UID: \"c815b3fc-7f35-4ae6-a015-2e84c6108a64\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6l6fr" Oct 30 11:53:52.719879 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.719800 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c815b3fc-7f35-4ae6-a015-2e84c6108a64-serving-cert\") pod \"service-ca-operator-544d749c8c-6l6fr\" (UID: \"c815b3fc-7f35-4ae6-a015-2e84c6108a64\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6l6fr" Oct 30 11:53:52.719879 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.719824 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/003c3d18-23f1-45d0-bac4-8c6eea4b7e99-cert\") pod \"ingress-canary-gn2z5\" (UID: \"003c3d18-23f1-45d0-bac4-8c6eea4b7e99\") " pod="openshift-ingress-canary/ingress-canary-gn2z5" Oct 30 11:53:52.719879 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.719868 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/6acb13a1-cb2e-45e1-897b-3ef89e6d1b84-default-certificate\") pod \"router-default-5bfdf864c-tzv67\" (UID: \"6acb13a1-cb2e-45e1-897b-3ef89e6d1b84\") " pod="openshift-ingress/router-default-5bfdf864c-tzv67" Oct 30 11:53:52.720151 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.719937 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bd977255-f9a7-4531-8a9c-b1a3a17e0b79-image-registry-private-configuration\") pod \"image-registry-557c6b9f47-vcr2t\" (UID: \"bd977255-f9a7-4531-8a9c-b1a3a17e0b79\") " pod="openshift-image-registry/image-registry-557c6b9f47-vcr2t" Oct 30 11:53:52.720151 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.719996 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7689cfe5-5a4e-4481-8eb9-107a58ca8bb3-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-xdxkp\" (UID: \"7689cfe5-5a4e-4481-8eb9-107a58ca8bb3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xdxkp" Oct 30 11:53:52.720151 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.720031 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bd977255-f9a7-4531-8a9c-b1a3a17e0b79-ca-trust-extracted\") pod \"image-registry-557c6b9f47-vcr2t\" (UID: \"bd977255-f9a7-4531-8a9c-b1a3a17e0b79\") " pod="openshift-image-registry/image-registry-557c6b9f47-vcr2t" Oct 30 11:53:52.720151 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.720071 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bd977255-f9a7-4531-8a9c-b1a3a17e0b79-trusted-ca\") pod \"image-registry-557c6b9f47-vcr2t\" (UID: \"bd977255-f9a7-4531-8a9c-b1a3a17e0b79\") " pod="openshift-image-registry/image-registry-557c6b9f47-vcr2t" Oct 30 11:53:52.720151 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.720115 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b3f20243-155c-4e25-a1ca-6131a141fbd4-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-86p78\" (UID: \"b3f20243-155c-4e25-a1ca-6131a141fbd4\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-86p78" Oct 30 11:53:52.720391 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.720154 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a70ee11c-c20e-4a4f-8476-ac5eb02816d0-serving-cert\") pod \"console-operator-7cd56b8dbc-plczt\" (UID: \"a70ee11c-c20e-4a4f-8476-ac5eb02816d0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-plczt" Oct 30 11:53:52.720391 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.720185 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jmcf\" (UniqueName: \"kubernetes.io/projected/003c3d18-23f1-45d0-bac4-8c6eea4b7e99-kube-api-access-4jmcf\") pod \"ingress-canary-gn2z5\" (UID: \"003c3d18-23f1-45d0-bac4-8c6eea4b7e99\") " pod="openshift-ingress-canary/ingress-canary-gn2z5" Oct 30 11:53:52.720391 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.720262 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hqrf\" (UniqueName: \"kubernetes.io/projected/7ac30952-da5d-42a3-a1aa-bd249bbf6adb-kube-api-access-6hqrf\") pod \"network-check-source-6f78d59f99-glvhs\" (UID: \"7ac30952-da5d-42a3-a1aa-bd249bbf6adb\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-glvhs" Oct 30 11:53:52.720391 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.720300 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vx4mp\" (UniqueName: \"kubernetes.io/projected/a70ee11c-c20e-4a4f-8476-ac5eb02816d0-kube-api-access-vx4mp\") pod \"console-operator-7cd56b8dbc-plczt\" (UID: \"a70ee11c-c20e-4a4f-8476-ac5eb02816d0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-plczt" Oct 30 11:53:52.720391 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.720325 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b3f20243-155c-4e25-a1ca-6131a141fbd4-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-86p78\" (UID: \"b3f20243-155c-4e25-a1ca-6131a141fbd4\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-86p78" Oct 30 11:53:52.720391 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.720367 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6acb13a1-cb2e-45e1-897b-3ef89e6d1b84-metrics-certs\") pod \"router-default-5bfdf864c-tzv67\" (UID: \"6acb13a1-cb2e-45e1-897b-3ef89e6d1b84\") " pod="openshift-ingress/router-default-5bfdf864c-tzv67" Oct 30 11:53:52.720391 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.720390 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bd977255-f9a7-4531-8a9c-b1a3a17e0b79-registry-certificates\") pod \"image-registry-557c6b9f47-vcr2t\" (UID: \"bd977255-f9a7-4531-8a9c-b1a3a17e0b79\") " pod="openshift-image-registry/image-registry-557c6b9f47-vcr2t" Oct 30 11:53:52.720718 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.720418 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a70ee11c-c20e-4a4f-8476-ac5eb02816d0-config\") pod \"console-operator-7cd56b8dbc-plczt\" (UID: \"a70ee11c-c20e-4a4f-8476-ac5eb02816d0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-plczt" Oct 30 11:53:52.720718 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.720441 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/6acb13a1-cb2e-45e1-897b-3ef89e6d1b84-stats-auth\") pod \"router-default-5bfdf864c-tzv67\" (UID: \"6acb13a1-cb2e-45e1-897b-3ef89e6d1b84\") " pod="openshift-ingress/router-default-5bfdf864c-tzv67" Oct 30 11:53:52.720718 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.720465 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/7689cfe5-5a4e-4481-8eb9-107a58ca8bb3-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-xdxkp\" (UID: \"7689cfe5-5a4e-4481-8eb9-107a58ca8bb3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xdxkp" Oct 30 11:53:52.720718 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.720490 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bd977255-f9a7-4531-8a9c-b1a3a17e0b79-registry-tls\") pod \"image-registry-557c6b9f47-vcr2t\" (UID: \"bd977255-f9a7-4531-8a9c-b1a3a17e0b79\") " pod="openshift-image-registry/image-registry-557c6b9f47-vcr2t" Oct 30 11:53:52.720718 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.720514 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bd977255-f9a7-4531-8a9c-b1a3a17e0b79-installation-pull-secrets\") pod \"image-registry-557c6b9f47-vcr2t\" (UID: \"bd977255-f9a7-4531-8a9c-b1a3a17e0b79\") " pod="openshift-image-registry/image-registry-557c6b9f47-vcr2t" Oct 30 11:53:52.720718 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.720565 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bd977255-f9a7-4531-8a9c-b1a3a17e0b79-bound-sa-token\") pod \"image-registry-557c6b9f47-vcr2t\" (UID: \"bd977255-f9a7-4531-8a9c-b1a3a17e0b79\") " pod="openshift-image-registry/image-registry-557c6b9f47-vcr2t" Oct 30 11:53:52.720718 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.720591 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8f7sv\" (UniqueName: \"kubernetes.io/projected/bd977255-f9a7-4531-8a9c-b1a3a17e0b79-kube-api-access-8f7sv\") pod \"image-registry-557c6b9f47-vcr2t\" (UID: \"bd977255-f9a7-4531-8a9c-b1a3a17e0b79\") " pod="openshift-image-registry/image-registry-557c6b9f47-vcr2t" Oct 30 11:53:52.720718 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.720626 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c815b3fc-7f35-4ae6-a015-2e84c6108a64-config\") pod \"service-ca-operator-544d749c8c-6l6fr\" (UID: \"c815b3fc-7f35-4ae6-a015-2e84c6108a64\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6l6fr" Oct 30 11:53:52.720718 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.720679 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6acb13a1-cb2e-45e1-897b-3ef89e6d1b84-service-ca-bundle\") pod \"router-default-5bfdf864c-tzv67\" (UID: \"6acb13a1-cb2e-45e1-897b-3ef89e6d1b84\") " pod="openshift-ingress/router-default-5bfdf864c-tzv67" Oct 30 11:53:52.721159 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.720724 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gr67s\" (UniqueName: \"kubernetes.io/projected/7689cfe5-5a4e-4481-8eb9-107a58ca8bb3-kube-api-access-gr67s\") pod \"cluster-samples-operator-6ddb6c8b94-xdxkp\" (UID: \"7689cfe5-5a4e-4481-8eb9-107a58ca8bb3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xdxkp" Oct 30 11:53:52.721159 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.720772 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rxc4\" (UniqueName: \"kubernetes.io/projected/6acb13a1-cb2e-45e1-897b-3ef89e6d1b84-kube-api-access-2rxc4\") pod \"router-default-5bfdf864c-tzv67\" (UID: \"6acb13a1-cb2e-45e1-897b-3ef89e6d1b84\") " pod="openshift-ingress/router-default-5bfdf864c-tzv67" Oct 30 11:53:52.721159 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.720887 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a70ee11c-c20e-4a4f-8476-ac5eb02816d0-trusted-ca\") pod \"console-operator-7cd56b8dbc-plczt\" (UID: \"a70ee11c-c20e-4a4f-8476-ac5eb02816d0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-plczt" Oct 30 11:53:52.721159 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.721080 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bd977255-f9a7-4531-8a9c-b1a3a17e0b79-registry-certificates\") pod \"image-registry-557c6b9f47-vcr2t\" (UID: \"bd977255-f9a7-4531-8a9c-b1a3a17e0b79\") " pod="openshift-image-registry/image-registry-557c6b9f47-vcr2t" Oct 30 11:53:52.721344 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.721199 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bd977255-f9a7-4531-8a9c-b1a3a17e0b79-ca-trust-extracted\") pod \"image-registry-557c6b9f47-vcr2t\" (UID: \"bd977255-f9a7-4531-8a9c-b1a3a17e0b79\") " pod="openshift-image-registry/image-registry-557c6b9f47-vcr2t" Oct 30 11:53:52.722754 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:52.721460 2545 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 11:53:52.722754 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:52.721475 2545 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 11:53:52.722754 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:52.721487 2545 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-557c6b9f47-vcr2t: secret "image-registry-tls" not found Oct 30 11:53:52.722754 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:52.721527 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7689cfe5-5a4e-4481-8eb9-107a58ca8bb3-samples-operator-tls podName:7689cfe5-5a4e-4481-8eb9-107a58ca8bb3 nodeName:}" failed. No retries permitted until 2026-10-30 11:53:53.221511305 +0000 UTC m=+30.661964522 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7689cfe5-5a4e-4481-8eb9-107a58ca8bb3-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-xdxkp" (UID: "7689cfe5-5a4e-4481-8eb9-107a58ca8bb3") : secret "samples-operator-tls" not found Oct 30 11:53:52.722754 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:52.721546 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bd977255-f9a7-4531-8a9c-b1a3a17e0b79-registry-tls podName:bd977255-f9a7-4531-8a9c-b1a3a17e0b79 nodeName:}" failed. No retries permitted until 2026-10-30 11:53:53.221537085 +0000 UTC m=+30.661990294 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bd977255-f9a7-4531-8a9c-b1a3a17e0b79-registry-tls") pod "image-registry-557c6b9f47-vcr2t" (UID: "bd977255-f9a7-4531-8a9c-b1a3a17e0b79") : secret "image-registry-tls" not found Oct 30 11:53:52.722754 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.721643 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a70ee11c-c20e-4a4f-8476-ac5eb02816d0-config\") pod \"console-operator-7cd56b8dbc-plczt\" (UID: \"a70ee11c-c20e-4a4f-8476-ac5eb02816d0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-plczt" Oct 30 11:53:52.722754 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.722178 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/7689cfe5-5a4e-4481-8eb9-107a58ca8bb3-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-xdxkp\" (UID: \"7689cfe5-5a4e-4481-8eb9-107a58ca8bb3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xdxkp" Oct 30 11:53:52.722754 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:52.722236 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6acb13a1-cb2e-45e1-897b-3ef89e6d1b84-service-ca-bundle podName:6acb13a1-cb2e-45e1-897b-3ef89e6d1b84 nodeName:}" failed. No retries permitted until 2026-10-30 11:53:53.222221054 +0000 UTC m=+30.662674267 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6acb13a1-cb2e-45e1-897b-3ef89e6d1b84-service-ca-bundle") pod "router-default-5bfdf864c-tzv67" (UID: "6acb13a1-cb2e-45e1-897b-3ef89e6d1b84") : configmap references non-existent config key: service-ca.crt Oct 30 11:53:52.722754 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.722348 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c815b3fc-7f35-4ae6-a015-2e84c6108a64-serving-cert\") pod \"service-ca-operator-544d749c8c-6l6fr\" (UID: \"c815b3fc-7f35-4ae6-a015-2e84c6108a64\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6l6fr" Oct 30 11:53:52.722754 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.722548 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c815b3fc-7f35-4ae6-a015-2e84c6108a64-config\") pod \"service-ca-operator-544d749c8c-6l6fr\" (UID: \"c815b3fc-7f35-4ae6-a015-2e84c6108a64\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6l6fr" Oct 30 11:53:52.722754 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:52.722611 2545 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 11:53:52.722754 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:52.722661 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6acb13a1-cb2e-45e1-897b-3ef89e6d1b84-metrics-certs podName:6acb13a1-cb2e-45e1-897b-3ef89e6d1b84 nodeName:}" failed. No retries permitted until 2026-10-30 11:53:53.222652326 +0000 UTC m=+30.663105533 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6acb13a1-cb2e-45e1-897b-3ef89e6d1b84-metrics-certs") pod "router-default-5bfdf864c-tzv67" (UID: "6acb13a1-cb2e-45e1-897b-3ef89e6d1b84") : secret "router-metrics-certs-default" not found Oct 30 11:53:52.722754 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.722656 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b3f20243-155c-4e25-a1ca-6131a141fbd4-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-86p78\" (UID: \"b3f20243-155c-4e25-a1ca-6131a141fbd4\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-86p78" Oct 30 11:53:52.722754 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.722713 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bd977255-f9a7-4531-8a9c-b1a3a17e0b79-trusted-ca\") pod \"image-registry-557c6b9f47-vcr2t\" (UID: \"bd977255-f9a7-4531-8a9c-b1a3a17e0b79\") " pod="openshift-image-registry/image-registry-557c6b9f47-vcr2t" Oct 30 11:53:52.722754 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.722751 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bd977255-f9a7-4531-8a9c-b1a3a17e0b79-image-registry-private-configuration\") pod \"image-registry-557c6b9f47-vcr2t\" (UID: \"bd977255-f9a7-4531-8a9c-b1a3a17e0b79\") " pod="openshift-image-registry/image-registry-557c6b9f47-vcr2t" Oct 30 11:53:52.723409 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.722877 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b3f20243-155c-4e25-a1ca-6131a141fbd4-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-86p78\" (UID: \"b3f20243-155c-4e25-a1ca-6131a141fbd4\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-86p78" Oct 30 11:53:52.723409 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.723314 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a70ee11c-c20e-4a4f-8476-ac5eb02816d0-serving-cert\") pod \"console-operator-7cd56b8dbc-plczt\" (UID: \"a70ee11c-c20e-4a4f-8476-ac5eb02816d0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-plczt" Oct 30 11:53:52.723802 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.723787 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/6acb13a1-cb2e-45e1-897b-3ef89e6d1b84-stats-auth\") pod \"router-default-5bfdf864c-tzv67\" (UID: \"6acb13a1-cb2e-45e1-897b-3ef89e6d1b84\") " pod="openshift-ingress/router-default-5bfdf864c-tzv67" Oct 30 11:53:52.724329 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.724314 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/6acb13a1-cb2e-45e1-897b-3ef89e6d1b84-default-certificate\") pod \"router-default-5bfdf864c-tzv67\" (UID: \"6acb13a1-cb2e-45e1-897b-3ef89e6d1b84\") " pod="openshift-ingress/router-default-5bfdf864c-tzv67" Oct 30 11:53:52.724629 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.724614 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bd977255-f9a7-4531-8a9c-b1a3a17e0b79-installation-pull-secrets\") pod \"image-registry-557c6b9f47-vcr2t\" (UID: \"bd977255-f9a7-4531-8a9c-b1a3a17e0b79\") " pod="openshift-image-registry/image-registry-557c6b9f47-vcr2t" Oct 30 11:53:52.730576 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.730561 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-k8ldb" Oct 30 11:53:52.741526 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.741508 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-kg9td" Oct 30 11:53:52.753055 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.753026 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bd977255-f9a7-4531-8a9c-b1a3a17e0b79-bound-sa-token\") pod \"image-registry-557c6b9f47-vcr2t\" (UID: \"bd977255-f9a7-4531-8a9c-b1a3a17e0b79\") " pod="openshift-image-registry/image-registry-557c6b9f47-vcr2t" Oct 30 11:53:52.753456 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.753435 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8f7sv\" (UniqueName: \"kubernetes.io/projected/bd977255-f9a7-4531-8a9c-b1a3a17e0b79-kube-api-access-8f7sv\") pod \"image-registry-557c6b9f47-vcr2t\" (UID: \"bd977255-f9a7-4531-8a9c-b1a3a17e0b79\") " pod="openshift-image-registry/image-registry-557c6b9f47-vcr2t" Oct 30 11:53:52.753514 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.753497 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rxc4\" (UniqueName: \"kubernetes.io/projected/6acb13a1-cb2e-45e1-897b-3ef89e6d1b84-kube-api-access-2rxc4\") pod \"router-default-5bfdf864c-tzv67\" (UID: \"6acb13a1-cb2e-45e1-897b-3ef89e6d1b84\") " pod="openshift-ingress/router-default-5bfdf864c-tzv67" Oct 30 11:53:52.753682 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.753667 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r226b\" (UniqueName: \"kubernetes.io/projected/b3f20243-155c-4e25-a1ca-6131a141fbd4-kube-api-access-r226b\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-86p78\" (UID: \"b3f20243-155c-4e25-a1ca-6131a141fbd4\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-86p78" Oct 30 11:53:52.753716 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.753703 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gr67s\" (UniqueName: \"kubernetes.io/projected/7689cfe5-5a4e-4481-8eb9-107a58ca8bb3-kube-api-access-gr67s\") pod \"cluster-samples-operator-6ddb6c8b94-xdxkp\" (UID: \"7689cfe5-5a4e-4481-8eb9-107a58ca8bb3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xdxkp" Oct 30 11:53:52.756322 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.756297 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vx4mp\" (UniqueName: \"kubernetes.io/projected/a70ee11c-c20e-4a4f-8476-ac5eb02816d0-kube-api-access-vx4mp\") pod \"console-operator-7cd56b8dbc-plczt\" (UID: \"a70ee11c-c20e-4a4f-8476-ac5eb02816d0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-plczt" Oct 30 11:53:52.756859 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.756817 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hqrf\" (UniqueName: \"kubernetes.io/projected/7ac30952-da5d-42a3-a1aa-bd249bbf6adb-kube-api-access-6hqrf\") pod \"network-check-source-6f78d59f99-glvhs\" (UID: \"7ac30952-da5d-42a3-a1aa-bd249bbf6adb\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-glvhs" Oct 30 11:53:52.757807 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.757793 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jf44s\" (UniqueName: \"kubernetes.io/projected/c815b3fc-7f35-4ae6-a015-2e84c6108a64-kube-api-access-jf44s\") pod \"service-ca-operator-544d749c8c-6l6fr\" (UID: \"c815b3fc-7f35-4ae6-a015-2e84c6108a64\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6l6fr" Oct 30 11:53:52.775216 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.775201 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-ccctl" Oct 30 11:53:52.808696 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.808666 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-2fqgf" Oct 30 11:53:52.819522 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.819499 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-ksxpw" Oct 30 11:53:52.822813 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.821938 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jmcf\" (UniqueName: \"kubernetes.io/projected/003c3d18-23f1-45d0-bac4-8c6eea4b7e99-kube-api-access-4jmcf\") pod \"ingress-canary-gn2z5\" (UID: \"003c3d18-23f1-45d0-bac4-8c6eea4b7e99\") " pod="openshift-ingress-canary/ingress-canary-gn2z5" Oct 30 11:53:52.822813 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.822092 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/003c3d18-23f1-45d0-bac4-8c6eea4b7e99-cert\") pod \"ingress-canary-gn2z5\" (UID: \"003c3d18-23f1-45d0-bac4-8c6eea4b7e99\") " pod="openshift-ingress-canary/ingress-canary-gn2z5" Oct 30 11:53:52.824809 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.824755 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/003c3d18-23f1-45d0-bac4-8c6eea4b7e99-cert\") pod \"ingress-canary-gn2z5\" (UID: \"003c3d18-23f1-45d0-bac4-8c6eea4b7e99\") " pod="openshift-ingress-canary/ingress-canary-gn2z5" Oct 30 11:53:52.834842 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.834791 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-86p78" Oct 30 11:53:52.845963 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.845944 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jmcf\" (UniqueName: \"kubernetes.io/projected/003c3d18-23f1-45d0-bac4-8c6eea4b7e99-kube-api-access-4jmcf\") pod \"ingress-canary-gn2z5\" (UID: \"003c3d18-23f1-45d0-bac4-8c6eea4b7e99\") " pod="openshift-ingress-canary/ingress-canary-gn2z5" Oct 30 11:53:52.880123 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.880046 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6l6fr" Oct 30 11:53:52.905438 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.905361 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-plczt" Oct 30 11:53:52.947265 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.947241 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-glvhs" Oct 30 11:53:52.963851 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:52.963645 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-gn2z5" Oct 30 11:53:53.106527 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:53.106499 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dbqhj" Oct 30 11:53:53.106655 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:53.106502 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bz65k" Oct 30 11:53:53.113212 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:53.113146 2545 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-kg9td"] Oct 30 11:53:53.113991 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:53.113971 2545 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-k8ldb"] Oct 30 11:53:53.117091 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:53.117071 2545 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode497352b_2348_4fc7_888f_5b68f116394f.slice/crio-419f42abae50a8c097a816062e2e78976ae8ec37f34318f0c1aa15703d4eac60 WatchSource:0}: Error finding container 419f42abae50a8c097a816062e2e78976ae8ec37f34318f0c1aa15703d4eac60: Status 404 returned error can't find the container with id 419f42abae50a8c097a816062e2e78976ae8ec37f34318f0c1aa15703d4eac60 Oct 30 11:53:53.117576 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:53.117558 2545 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e44595b_0bdf_407c_bf22_33eb06272cc1.slice/crio-7e3210ba71b33dbf2744cdcf9f8c0af3b880fefb6715945363dfe73c1c6a67e6 WatchSource:0}: Error finding container 7e3210ba71b33dbf2744cdcf9f8c0af3b880fefb6715945363dfe73c1c6a67e6: Status 404 returned error can't find the container with id 7e3210ba71b33dbf2744cdcf9f8c0af3b880fefb6715945363dfe73c1c6a67e6 Oct 30 11:53:53.117693 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:53.117677 2545 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-ccctl"] Oct 30 11:53:53.121870 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:53.121847 2545 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f5667a1_a773_4520_b2fb_c69148df632c.slice/crio-0ed7babdc2068ee24f20699aa63685bab422bb05562cc3f8e79c91aeb1568a52 WatchSource:0}: Error finding container 0ed7babdc2068ee24f20699aa63685bab422bb05562cc3f8e79c91aeb1568a52: Status 404 returned error can't find the container with id 0ed7babdc2068ee24f20699aa63685bab422bb05562cc3f8e79c91aeb1568a52 Oct 30 11:53:53.125122 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:53.125104 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/179d55d3-7958-4928-a98c-68984fca9837-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-zzxhv\" (UID: \"179d55d3-7958-4928-a98c-68984fca9837\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-zzxhv" Oct 30 11:53:53.125198 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:53.125142 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/221ced85-cc15-4e07-82d9-a1a1f3e958a5-metrics-tls\") pod \"dns-default-xxcbn\" (UID: \"221ced85-cc15-4e07-82d9-a1a1f3e958a5\") " pod="openshift-dns/dns-default-xxcbn" Oct 30 11:53:53.125256 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:53.125228 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cfaad7d0-d30b-4ddc-81cc-2b603416fc29-registry-tls\") pod \"image-registry-79fd565486-8fl78\" (UID: \"cfaad7d0-d30b-4ddc-81cc-2b603416fc29\") " pod="openshift-image-registry/image-registry-79fd565486-8fl78" Oct 30 11:53:53.125256 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:53.125236 2545 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 11:53:53.125350 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:53.125276 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/179d55d3-7958-4928-a98c-68984fca9837-networking-console-plugin-cert podName:179d55d3-7958-4928-a98c-68984fca9837 nodeName:}" failed. No retries permitted until 2026-10-30 11:53:54.125264075 +0000 UTC m=+31.565717299 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/179d55d3-7958-4928-a98c-68984fca9837-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-zzxhv" (UID: "179d55d3-7958-4928-a98c-68984fca9837") : secret "networking-console-plugin-cert" not found Oct 30 11:53:53.125350 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:53.125305 2545 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 11:53:53.125350 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:53.125324 2545 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-79fd565486-8fl78: secret "image-registry-tls" not found Oct 30 11:53:53.125350 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:53.125331 2545 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 11:53:53.125350 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:53.125344 2545 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 11:53:53.125510 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:53.125358 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cfaad7d0-d30b-4ddc-81cc-2b603416fc29-registry-tls podName:cfaad7d0-d30b-4ddc-81cc-2b603416fc29 nodeName:}" failed. No retries permitted until 2026-10-30 11:53:54.125345943 +0000 UTC m=+31.565799180 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cfaad7d0-d30b-4ddc-81cc-2b603416fc29-registry-tls") pod "image-registry-79fd565486-8fl78" (UID: "cfaad7d0-d30b-4ddc-81cc-2b603416fc29") : secret "image-registry-tls" not found Oct 30 11:53:53.125510 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:53.125302 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6d976900-ecd0-469c-a9e2-d60708361f1a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-mvhwk\" (UID: \"6d976900-ecd0-469c-a9e2-d60708361f1a\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-mvhwk" Oct 30 11:53:53.125510 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:53.125370 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/221ced85-cc15-4e07-82d9-a1a1f3e958a5-metrics-tls podName:221ced85-cc15-4e07-82d9-a1a1f3e958a5 nodeName:}" failed. No retries permitted until 2026-10-30 11:53:54.125363803 +0000 UTC m=+31.565817010 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/221ced85-cc15-4e07-82d9-a1a1f3e958a5-metrics-tls") pod "dns-default-xxcbn" (UID: "221ced85-cc15-4e07-82d9-a1a1f3e958a5") : secret "dns-default-metrics-tls" not found Oct 30 11:53:53.125510 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:53.125390 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d976900-ecd0-469c-a9e2-d60708361f1a-cluster-monitoring-operator-tls podName:6d976900-ecd0-469c-a9e2-d60708361f1a nodeName:}" failed. No retries permitted until 2026-10-30 11:53:54.125384349 +0000 UTC m=+31.565837556 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/6d976900-ecd0-469c-a9e2-d60708361f1a-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-mvhwk" (UID: "6d976900-ecd0-469c-a9e2-d60708361f1a") : secret "cluster-monitoring-operator-tls" not found Oct 30 11:53:53.137345 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:53.137328 2545 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-zfblq\"" Oct 30 11:53:53.140246 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:53.140233 2545 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 11:53:53.140473 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:53.140458 2545 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-rpjw7\"" Oct 30 11:53:53.179694 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:53.179672 2545 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-2fqgf"] Oct 30 11:53:53.183694 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:53.183656 2545 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1757617_7c51_4029_b329_1b6a4630f860.slice/crio-bf1bbee327514fab5fac9eb46554e91c28ce5a6387743ae6b1c7d24ba7989f4f WatchSource:0}: Error finding container bf1bbee327514fab5fac9eb46554e91c28ce5a6387743ae6b1c7d24ba7989f4f: Status 404 returned error can't find the container with id bf1bbee327514fab5fac9eb46554e91c28ce5a6387743ae6b1c7d24ba7989f4f Oct 30 11:53:53.208493 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:53.208470 2545 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-ksxpw"] Oct 30 11:53:53.215130 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:53.215025 2545 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-86p78"] Oct 30 11:53:53.226307 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:53.226243 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6acb13a1-cb2e-45e1-897b-3ef89e6d1b84-metrics-certs\") pod \"router-default-5bfdf864c-tzv67\" (UID: \"6acb13a1-cb2e-45e1-897b-3ef89e6d1b84\") " pod="openshift-ingress/router-default-5bfdf864c-tzv67" Oct 30 11:53:53.226307 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:53.226282 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bd977255-f9a7-4531-8a9c-b1a3a17e0b79-registry-tls\") pod \"image-registry-557c6b9f47-vcr2t\" (UID: \"bd977255-f9a7-4531-8a9c-b1a3a17e0b79\") " pod="openshift-image-registry/image-registry-557c6b9f47-vcr2t" Oct 30 11:53:53.226432 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:53.226350 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6acb13a1-cb2e-45e1-897b-3ef89e6d1b84-service-ca-bundle\") pod \"router-default-5bfdf864c-tzv67\" (UID: \"6acb13a1-cb2e-45e1-897b-3ef89e6d1b84\") " pod="openshift-ingress/router-default-5bfdf864c-tzv67" Oct 30 11:53:53.226532 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:53.226460 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7689cfe5-5a4e-4481-8eb9-107a58ca8bb3-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-xdxkp\" (UID: \"7689cfe5-5a4e-4481-8eb9-107a58ca8bb3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xdxkp" Oct 30 11:53:53.226614 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:53.226595 2545 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 11:53:53.226662 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:53.226655 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7689cfe5-5a4e-4481-8eb9-107a58ca8bb3-samples-operator-tls podName:7689cfe5-5a4e-4481-8eb9-107a58ca8bb3 nodeName:}" failed. No retries permitted until 2026-10-30 11:53:54.226637225 +0000 UTC m=+31.667090435 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7689cfe5-5a4e-4481-8eb9-107a58ca8bb3-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-xdxkp" (UID: "7689cfe5-5a4e-4481-8eb9-107a58ca8bb3") : secret "samples-operator-tls" not found Oct 30 11:53:53.226997 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:53.226720 2545 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 11:53:53.226997 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:53.226770 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6acb13a1-cb2e-45e1-897b-3ef89e6d1b84-metrics-certs podName:6acb13a1-cb2e-45e1-897b-3ef89e6d1b84 nodeName:}" failed. No retries permitted until 2026-10-30 11:53:54.226746935 +0000 UTC m=+31.667200143 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6acb13a1-cb2e-45e1-897b-3ef89e6d1b84-metrics-certs") pod "router-default-5bfdf864c-tzv67" (UID: "6acb13a1-cb2e-45e1-897b-3ef89e6d1b84") : secret "router-metrics-certs-default" not found Oct 30 11:53:53.226997 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:53.226849 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6acb13a1-cb2e-45e1-897b-3ef89e6d1b84-service-ca-bundle podName:6acb13a1-cb2e-45e1-897b-3ef89e6d1b84 nodeName:}" failed. No retries permitted until 2026-10-30 11:53:54.22681404 +0000 UTC m=+31.667267262 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6acb13a1-cb2e-45e1-897b-3ef89e6d1b84-service-ca-bundle") pod "router-default-5bfdf864c-tzv67" (UID: "6acb13a1-cb2e-45e1-897b-3ef89e6d1b84") : configmap references non-existent config key: service-ca.crt Oct 30 11:53:53.226997 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:53.226928 2545 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 11:53:53.226997 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:53.226941 2545 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-557c6b9f47-vcr2t: secret "image-registry-tls" not found Oct 30 11:53:53.226997 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:53.226978 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bd977255-f9a7-4531-8a9c-b1a3a17e0b79-registry-tls podName:bd977255-f9a7-4531-8a9c-b1a3a17e0b79 nodeName:}" failed. No retries permitted until 2026-10-30 11:53:54.226966085 +0000 UTC m=+31.667419306 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bd977255-f9a7-4531-8a9c-b1a3a17e0b79-registry-tls") pod "image-registry-557c6b9f47-vcr2t" (UID: "bd977255-f9a7-4531-8a9c-b1a3a17e0b79") : secret "image-registry-tls" not found Oct 30 11:53:53.232688 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:53.232659 2545 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3f20243_155c_4e25_a1ca_6131a141fbd4.slice/crio-36525167111a09951040a77f0c5cf71541a5df0fc8e7bc52add5f57bd46d3612 WatchSource:0}: Error finding container 36525167111a09951040a77f0c5cf71541a5df0fc8e7bc52add5f57bd46d3612: Status 404 returned error can't find the container with id 36525167111a09951040a77f0c5cf71541a5df0fc8e7bc52add5f57bd46d3612 Oct 30 11:53:53.263153 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:53.263132 2545 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-6l6fr"] Oct 30 11:53:53.267967 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:53.267946 2545 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc815b3fc_7f35_4ae6_a015_2e84c6108a64.slice/crio-e03ff409ec2f82625f5cfe631fec11e889aa058ba91dbd505bf4ea6d6b31fec9 WatchSource:0}: Error finding container e03ff409ec2f82625f5cfe631fec11e889aa058ba91dbd505bf4ea6d6b31fec9: Status 404 returned error can't find the container with id e03ff409ec2f82625f5cfe631fec11e889aa058ba91dbd505bf4ea6d6b31fec9 Oct 30 11:53:53.271287 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:53.271084 2545 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-plczt"] Oct 30 11:53:53.273802 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:53.273779 2545 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda70ee11c_c20e_4a4f_8476_ac5eb02816d0.slice/crio-0e385d1aea1378254689b60f24673fa5cb718887b518e18d8f68b64224aa2496 WatchSource:0}: Error finding container 0e385d1aea1378254689b60f24673fa5cb718887b518e18d8f68b64224aa2496: Status 404 returned error can't find the container with id 0e385d1aea1378254689b60f24673fa5cb718887b518e18d8f68b64224aa2496 Oct 30 11:53:53.301773 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:53.301751 2545 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-glvhs"] Oct 30 11:53:53.304457 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:53.304439 2545 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ac30952_da5d_42a3_a1aa_bd249bbf6adb.slice/crio-50aeeaf84669fde1fb553f85753f6f537614a6a045412ab3cf5635b37f049ea8 WatchSource:0}: Error finding container 50aeeaf84669fde1fb553f85753f6f537614a6a045412ab3cf5635b37f049ea8: Status 404 returned error can't find the container with id 50aeeaf84669fde1fb553f85753f6f537614a6a045412ab3cf5635b37f049ea8 Oct 30 11:53:53.320873 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:53.320847 2545 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-gn2z5"] Oct 30 11:53:53.323760 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:53.323742 2545 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod003c3d18_23f1_45d0_bac4_8c6eea4b7e99.slice/crio-4b0f6db58147695aa013d92f1e9abca35d686a1268c30c8b5ffa706aec6e51eb WatchSource:0}: Error finding container 4b0f6db58147695aa013d92f1e9abca35d686a1268c30c8b5ffa706aec6e51eb: Status 404 returned error can't find the container with id 4b0f6db58147695aa013d92f1e9abca35d686a1268c30c8b5ffa706aec6e51eb Oct 30 11:53:53.395012 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:53.394958 2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-2fqgf" event={"ID":"b1757617-7c51-4029-b329-1b6a4630f860","Type":"ContainerStarted","Data":"bf1bbee327514fab5fac9eb46554e91c28ce5a6387743ae6b1c7d24ba7989f4f"} Oct 30 11:53:53.396062 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:53.396039 2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-plczt" event={"ID":"a70ee11c-c20e-4a4f-8476-ac5eb02816d0","Type":"ContainerStarted","Data":"0e385d1aea1378254689b60f24673fa5cb718887b518e18d8f68b64224aa2496"} Oct 30 11:53:53.397378 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:53.397352 2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-ccctl" event={"ID":"1f5667a1-a773-4520-b2fb-c69148df632c","Type":"ContainerStarted","Data":"8d88c7739e06e0595eea72ef7f0a72a16c475057c8aa026082535b89246918e2"} Oct 30 11:53:53.397446 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:53.397388 2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-ccctl" event={"ID":"1f5667a1-a773-4520-b2fb-c69148df632c","Type":"ContainerStarted","Data":"0ed7babdc2068ee24f20699aa63685bab422bb05562cc3f8e79c91aeb1568a52"} Oct 30 11:53:53.398660 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:53.398631 2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-kg9td" event={"ID":"e497352b-2348-4fc7-888f-5b68f116394f","Type":"ContainerStarted","Data":"be0b7a0b06585b6e3fc2fe0db6cecded1dc9562dba2d2c929f8c8fab7cfb8db6"} Oct 30 11:53:53.398734 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:53.398664 2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-kg9td" event={"ID":"e497352b-2348-4fc7-888f-5b68f116394f","Type":"ContainerStarted","Data":"419f42abae50a8c097a816062e2e78976ae8ec37f34318f0c1aa15703d4eac60"} Oct 30 11:53:53.399691 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:53.399665 2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-gn2z5" event={"ID":"003c3d18-23f1-45d0-bac4-8c6eea4b7e99","Type":"ContainerStarted","Data":"4b0f6db58147695aa013d92f1e9abca35d686a1268c30c8b5ffa706aec6e51eb"} Oct 30 11:53:53.400718 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:53.400701 2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6l6fr" event={"ID":"c815b3fc-7f35-4ae6-a015-2e84c6108a64","Type":"ContainerStarted","Data":"e03ff409ec2f82625f5cfe631fec11e889aa058ba91dbd505bf4ea6d6b31fec9"} Oct 30 11:53:53.402013 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:53.401993 2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-k8ldb" event={"ID":"8e44595b-0bdf-407c-bf22-33eb06272cc1","Type":"ContainerStarted","Data":"5831580f18b1df3eeec40c252bcf55ddbf5606c9ba602f11e75696f9dce76c8f"} Oct 30 11:53:53.402069 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:53.402030 2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-k8ldb" event={"ID":"8e44595b-0bdf-407c-bf22-33eb06272cc1","Type":"ContainerStarted","Data":"7e3210ba71b33dbf2744cdcf9f8c0af3b880fefb6715945363dfe73c1c6a67e6"} Oct 30 11:53:53.403098 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:53.403077 2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-ksxpw" event={"ID":"d0e0cec2-f16a-4bae-bc70-6c08da4bda2e","Type":"ContainerStarted","Data":"2c9d279e75511d42a300ea507ae46c045f2f6d7354e80d750835817d62d998b7"} Oct 30 11:53:53.403987 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:53.403970 2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-glvhs" event={"ID":"7ac30952-da5d-42a3-a1aa-bd249bbf6adb","Type":"ContainerStarted","Data":"50aeeaf84669fde1fb553f85753f6f537614a6a045412ab3cf5635b37f049ea8"} Oct 30 11:53:53.404909 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:53.404891 2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-86p78" event={"ID":"b3f20243-155c-4e25-a1ca-6131a141fbd4","Type":"ContainerStarted","Data":"36525167111a09951040a77f0c5cf71541a5df0fc8e7bc52add5f57bd46d3612"} Oct 30 11:53:53.445384 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:53.445343 2545 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-ccctl" podStartSLOduration=419.445330858 podStartE2EDuration="6m59.445330858s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:46:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:53:53.445155638 +0000 UTC m=+30.885608866" watchObservedRunningTime="2026-10-30 11:53:53.445330858 +0000 UTC m=+30.885784087" Oct 30 11:53:53.484508 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:53.484474 2545 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-k8ldb" podStartSLOduration=419.484464234 podStartE2EDuration="6m59.484464234s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:46:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:53:53.483597274 +0000 UTC m=+30.924050503" watchObservedRunningTime="2026-10-30 11:53:53.484464234 +0000 UTC m=+30.924917462" Oct 30 11:53:53.530779 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:53.530732 2545 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-kg9td" podStartSLOduration=419.530723496 podStartE2EDuration="6m59.530723496s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:46:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:53:53.530572884 +0000 UTC m=+30.971026113" watchObservedRunningTime="2026-10-30 11:53:53.530723496 +0000 UTC m=+30.971176724" Oct 30 11:53:54.099884 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:54.099849 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fxjlt" Oct 30 11:53:54.115759 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:54.115642 2545 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-jjcjh\"" Oct 30 11:53:54.115759 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:54.115732 2545 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 11:53:54.136614 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:54.135705 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/179d55d3-7958-4928-a98c-68984fca9837-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-zzxhv\" (UID: \"179d55d3-7958-4928-a98c-68984fca9837\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-zzxhv" Oct 30 11:53:54.136614 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:54.135750 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/221ced85-cc15-4e07-82d9-a1a1f3e958a5-metrics-tls\") pod \"dns-default-xxcbn\" (UID: \"221ced85-cc15-4e07-82d9-a1a1f3e958a5\") " pod="openshift-dns/dns-default-xxcbn" Oct 30 11:53:54.136614 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:54.135848 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cfaad7d0-d30b-4ddc-81cc-2b603416fc29-registry-tls\") pod \"image-registry-79fd565486-8fl78\" (UID: \"cfaad7d0-d30b-4ddc-81cc-2b603416fc29\") " pod="openshift-image-registry/image-registry-79fd565486-8fl78" Oct 30 11:53:54.136614 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:54.135896 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6d976900-ecd0-469c-a9e2-d60708361f1a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-mvhwk\" (UID: \"6d976900-ecd0-469c-a9e2-d60708361f1a\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-mvhwk" Oct 30 11:53:54.136614 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:54.136029 2545 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 11:53:54.136614 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:54.136082 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d976900-ecd0-469c-a9e2-d60708361f1a-cluster-monitoring-operator-tls podName:6d976900-ecd0-469c-a9e2-d60708361f1a nodeName:}" failed. No retries permitted until 2026-10-30 11:53:56.136065504 +0000 UTC m=+33.576518718 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/6d976900-ecd0-469c-a9e2-d60708361f1a-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-mvhwk" (UID: "6d976900-ecd0-469c-a9e2-d60708361f1a") : secret "cluster-monitoring-operator-tls" not found Oct 30 11:53:54.136614 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:54.136385 2545 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 11:53:54.136614 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:54.136435 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/179d55d3-7958-4928-a98c-68984fca9837-networking-console-plugin-cert podName:179d55d3-7958-4928-a98c-68984fca9837 nodeName:}" failed. No retries permitted until 2026-10-30 11:53:56.136421186 +0000 UTC m=+33.576874400 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/179d55d3-7958-4928-a98c-68984fca9837-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-zzxhv" (UID: "179d55d3-7958-4928-a98c-68984fca9837") : secret "networking-console-plugin-cert" not found Oct 30 11:53:54.136614 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:54.136494 2545 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 11:53:54.136614 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:54.136521 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/221ced85-cc15-4e07-82d9-a1a1f3e958a5-metrics-tls podName:221ced85-cc15-4e07-82d9-a1a1f3e958a5 nodeName:}" failed. No retries permitted until 2026-10-30 11:53:56.136511389 +0000 UTC m=+33.576964601 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/221ced85-cc15-4e07-82d9-a1a1f3e958a5-metrics-tls") pod "dns-default-xxcbn" (UID: "221ced85-cc15-4e07-82d9-a1a1f3e958a5") : secret "dns-default-metrics-tls" not found Oct 30 11:53:54.136614 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:54.136568 2545 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 11:53:54.136614 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:54.136575 2545 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-79fd565486-8fl78: secret "image-registry-tls" not found Oct 30 11:53:54.136614 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:54.136594 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cfaad7d0-d30b-4ddc-81cc-2b603416fc29-registry-tls podName:cfaad7d0-d30b-4ddc-81cc-2b603416fc29 nodeName:}" failed. No retries permitted until 2026-10-30 11:53:56.136588733 +0000 UTC m=+33.577041940 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cfaad7d0-d30b-4ddc-81cc-2b603416fc29-registry-tls") pod "image-registry-79fd565486-8fl78" (UID: "cfaad7d0-d30b-4ddc-81cc-2b603416fc29") : secret "image-registry-tls" not found Oct 30 11:53:54.237219 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:54.237187 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6acb13a1-cb2e-45e1-897b-3ef89e6d1b84-metrics-certs\") pod \"router-default-5bfdf864c-tzv67\" (UID: \"6acb13a1-cb2e-45e1-897b-3ef89e6d1b84\") " pod="openshift-ingress/router-default-5bfdf864c-tzv67" Oct 30 11:53:54.237376 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:54.237235 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bd977255-f9a7-4531-8a9c-b1a3a17e0b79-registry-tls\") pod \"image-registry-557c6b9f47-vcr2t\" (UID: \"bd977255-f9a7-4531-8a9c-b1a3a17e0b79\") " pod="openshift-image-registry/image-registry-557c6b9f47-vcr2t" Oct 30 11:53:54.237376 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:54.237298 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6acb13a1-cb2e-45e1-897b-3ef89e6d1b84-service-ca-bundle\") pod \"router-default-5bfdf864c-tzv67\" (UID: \"6acb13a1-cb2e-45e1-897b-3ef89e6d1b84\") " pod="openshift-ingress/router-default-5bfdf864c-tzv67" Oct 30 11:53:54.237471 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:54.237385 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7689cfe5-5a4e-4481-8eb9-107a58ca8bb3-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-xdxkp\" (UID: \"7689cfe5-5a4e-4481-8eb9-107a58ca8bb3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xdxkp" Oct 30 11:53:54.237603 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:54.237585 2545 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 11:53:54.237665 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:54.237650 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7689cfe5-5a4e-4481-8eb9-107a58ca8bb3-samples-operator-tls podName:7689cfe5-5a4e-4481-8eb9-107a58ca8bb3 nodeName:}" failed. No retries permitted until 2026-10-30 11:53:56.237633159 +0000 UTC m=+33.678086379 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7689cfe5-5a4e-4481-8eb9-107a58ca8bb3-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-xdxkp" (UID: "7689cfe5-5a4e-4481-8eb9-107a58ca8bb3") : secret "samples-operator-tls" not found Oct 30 11:53:54.238011 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:54.237991 2545 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 11:53:54.238091 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:54.238041 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6acb13a1-cb2e-45e1-897b-3ef89e6d1b84-metrics-certs podName:6acb13a1-cb2e-45e1-897b-3ef89e6d1b84 nodeName:}" failed. No retries permitted until 2026-10-30 11:53:56.238028082 +0000 UTC m=+33.678481309 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6acb13a1-cb2e-45e1-897b-3ef89e6d1b84-metrics-certs") pod "router-default-5bfdf864c-tzv67" (UID: "6acb13a1-cb2e-45e1-897b-3ef89e6d1b84") : secret "router-metrics-certs-default" not found Oct 30 11:53:54.238584 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:54.238561 2545 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 11:53:54.238667 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:54.238587 2545 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-557c6b9f47-vcr2t: secret "image-registry-tls" not found Oct 30 11:53:54.238667 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:54.238646 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bd977255-f9a7-4531-8a9c-b1a3a17e0b79-registry-tls podName:bd977255-f9a7-4531-8a9c-b1a3a17e0b79 nodeName:}" failed. No retries permitted until 2026-10-30 11:53:56.238630774 +0000 UTC m=+33.679083994 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bd977255-f9a7-4531-8a9c-b1a3a17e0b79-registry-tls") pod "image-registry-557c6b9f47-vcr2t" (UID: "bd977255-f9a7-4531-8a9c-b1a3a17e0b79") : secret "image-registry-tls" not found Oct 30 11:53:54.239927 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:54.239891 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6acb13a1-cb2e-45e1-897b-3ef89e6d1b84-service-ca-bundle podName:6acb13a1-cb2e-45e1-897b-3ef89e6d1b84 nodeName:}" failed. No retries permitted until 2026-10-30 11:53:56.239875765 +0000 UTC m=+33.680328986 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6acb13a1-cb2e-45e1-897b-3ef89e6d1b84-service-ca-bundle") pod "router-default-5bfdf864c-tzv67" (UID: "6acb13a1-cb2e-45e1-897b-3ef89e6d1b84") : configmap references non-existent config key: service-ca.crt Oct 30 11:53:56.158312 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:56.158225 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cfaad7d0-d30b-4ddc-81cc-2b603416fc29-registry-tls\") pod \"image-registry-79fd565486-8fl78\" (UID: \"cfaad7d0-d30b-4ddc-81cc-2b603416fc29\") " pod="openshift-image-registry/image-registry-79fd565486-8fl78" Oct 30 11:53:56.158312 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:56.158299 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6d976900-ecd0-469c-a9e2-d60708361f1a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-mvhwk\" (UID: \"6d976900-ecd0-469c-a9e2-d60708361f1a\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-mvhwk" Oct 30 11:53:56.158797 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:56.158398 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/179d55d3-7958-4928-a98c-68984fca9837-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-zzxhv\" (UID: \"179d55d3-7958-4928-a98c-68984fca9837\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-zzxhv" Oct 30 11:53:56.158797 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:56.158430 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/221ced85-cc15-4e07-82d9-a1a1f3e958a5-metrics-tls\") pod \"dns-default-xxcbn\" (UID: \"221ced85-cc15-4e07-82d9-a1a1f3e958a5\") " pod="openshift-dns/dns-default-xxcbn" Oct 30 11:53:56.158797 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:56.158564 2545 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 11:53:56.158797 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:56.158626 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/221ced85-cc15-4e07-82d9-a1a1f3e958a5-metrics-tls podName:221ced85-cc15-4e07-82d9-a1a1f3e958a5 nodeName:}" failed. No retries permitted until 2026-10-30 11:54:00.158608626 +0000 UTC m=+37.599061855 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/221ced85-cc15-4e07-82d9-a1a1f3e958a5-metrics-tls") pod "dns-default-xxcbn" (UID: "221ced85-cc15-4e07-82d9-a1a1f3e958a5") : secret "dns-default-metrics-tls" not found Oct 30 11:53:56.159313 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:56.159033 2545 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 11:53:56.159313 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:56.159055 2545 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-79fd565486-8fl78: secret "image-registry-tls" not found Oct 30 11:53:56.159313 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:56.159104 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cfaad7d0-d30b-4ddc-81cc-2b603416fc29-registry-tls podName:cfaad7d0-d30b-4ddc-81cc-2b603416fc29 nodeName:}" failed. No retries permitted until 2026-10-30 11:54:00.159090544 +0000 UTC m=+37.599543766 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cfaad7d0-d30b-4ddc-81cc-2b603416fc29-registry-tls") pod "image-registry-79fd565486-8fl78" (UID: "cfaad7d0-d30b-4ddc-81cc-2b603416fc29") : secret "image-registry-tls" not found Oct 30 11:53:56.159313 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:56.159164 2545 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 11:53:56.159313 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:56.159198 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d976900-ecd0-469c-a9e2-d60708361f1a-cluster-monitoring-operator-tls podName:6d976900-ecd0-469c-a9e2-d60708361f1a nodeName:}" failed. No retries permitted until 2026-10-30 11:54:00.159187027 +0000 UTC m=+37.599640249 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/6d976900-ecd0-469c-a9e2-d60708361f1a-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-mvhwk" (UID: "6d976900-ecd0-469c-a9e2-d60708361f1a") : secret "cluster-monitoring-operator-tls" not found Oct 30 11:53:56.159313 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:56.159251 2545 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 11:53:56.159313 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:56.159283 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/179d55d3-7958-4928-a98c-68984fca9837-networking-console-plugin-cert podName:179d55d3-7958-4928-a98c-68984fca9837 nodeName:}" failed. No retries permitted until 2026-10-30 11:54:00.159271518 +0000 UTC m=+37.599724740 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/179d55d3-7958-4928-a98c-68984fca9837-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-zzxhv" (UID: "179d55d3-7958-4928-a98c-68984fca9837") : secret "networking-console-plugin-cert" not found Oct 30 11:53:56.259549 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:56.259510 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6acb13a1-cb2e-45e1-897b-3ef89e6d1b84-service-ca-bundle\") pod \"router-default-5bfdf864c-tzv67\" (UID: \"6acb13a1-cb2e-45e1-897b-3ef89e6d1b84\") " pod="openshift-ingress/router-default-5bfdf864c-tzv67" Oct 30 11:53:56.259679 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:56.259613 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7689cfe5-5a4e-4481-8eb9-107a58ca8bb3-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-xdxkp\" (UID: \"7689cfe5-5a4e-4481-8eb9-107a58ca8bb3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xdxkp" Oct 30 11:53:56.259772 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:56.259689 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6acb13a1-cb2e-45e1-897b-3ef89e6d1b84-metrics-certs\") pod \"router-default-5bfdf864c-tzv67\" (UID: \"6acb13a1-cb2e-45e1-897b-3ef89e6d1b84\") " pod="openshift-ingress/router-default-5bfdf864c-tzv67" Oct 30 11:53:56.259772 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:56.259718 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bd977255-f9a7-4531-8a9c-b1a3a17e0b79-registry-tls\") pod \"image-registry-557c6b9f47-vcr2t\" (UID: \"bd977255-f9a7-4531-8a9c-b1a3a17e0b79\") " pod="openshift-image-registry/image-registry-557c6b9f47-vcr2t" Oct 30 11:53:56.260366 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:56.259856 2545 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 11:53:56.260366 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:56.259875 2545 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-557c6b9f47-vcr2t: secret "image-registry-tls" not found Oct 30 11:53:56.260366 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:56.259926 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bd977255-f9a7-4531-8a9c-b1a3a17e0b79-registry-tls podName:bd977255-f9a7-4531-8a9c-b1a3a17e0b79 nodeName:}" failed. No retries permitted until 2026-10-30 11:54:00.259909361 +0000 UTC m=+37.700362572 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bd977255-f9a7-4531-8a9c-b1a3a17e0b79-registry-tls") pod "image-registry-557c6b9f47-vcr2t" (UID: "bd977255-f9a7-4531-8a9c-b1a3a17e0b79") : secret "image-registry-tls" not found Oct 30 11:53:56.260366 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:56.259968 2545 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 11:53:56.260366 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:56.259988 2545 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 11:53:56.260366 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:56.260018 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6acb13a1-cb2e-45e1-897b-3ef89e6d1b84-metrics-certs podName:6acb13a1-cb2e-45e1-897b-3ef89e6d1b84 nodeName:}" failed. No retries permitted until 2026-10-30 11:54:00.260008299 +0000 UTC m=+37.700461508 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6acb13a1-cb2e-45e1-897b-3ef89e6d1b84-metrics-certs") pod "router-default-5bfdf864c-tzv67" (UID: "6acb13a1-cb2e-45e1-897b-3ef89e6d1b84") : secret "router-metrics-certs-default" not found Oct 30 11:53:56.260366 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:56.260032 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7689cfe5-5a4e-4481-8eb9-107a58ca8bb3-samples-operator-tls podName:7689cfe5-5a4e-4481-8eb9-107a58ca8bb3 nodeName:}" failed. No retries permitted until 2026-10-30 11:54:00.26002428 +0000 UTC m=+37.700477490 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7689cfe5-5a4e-4481-8eb9-107a58ca8bb3-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-xdxkp" (UID: "7689cfe5-5a4e-4481-8eb9-107a58ca8bb3") : secret "samples-operator-tls" not found Oct 30 11:53:56.260366 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:56.260079 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6acb13a1-cb2e-45e1-897b-3ef89e6d1b84-service-ca-bundle podName:6acb13a1-cb2e-45e1-897b-3ef89e6d1b84 nodeName:}" failed. No retries permitted until 2026-10-30 11:54:00.260062284 +0000 UTC m=+37.700515508 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6acb13a1-cb2e-45e1-897b-3ef89e6d1b84-service-ca-bundle") pod "router-default-5bfdf864c-tzv67" (UID: "6acb13a1-cb2e-45e1-897b-3ef89e6d1b84") : configmap references non-existent config key: service-ca.crt Oct 30 11:53:56.420190 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:56.419319 2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-ksxpw" event={"ID":"d0e0cec2-f16a-4bae-bc70-6c08da4bda2e","Type":"ContainerStarted","Data":"1f9683288b42d9080d32fb7d957dab3611a5ca8f9a7b0ab75f199dc198a72bcc"} Oct 30 11:53:56.764572 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:56.764524 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d7f8396-e57c-45e5-ae74-a2244db10364-metrics-certs\") pod \"network-metrics-daemon-dbqhj\" (UID: \"1d7f8396-e57c-45e5-ae74-a2244db10364\") " pod="openshift-multus/network-metrics-daemon-dbqhj" Oct 30 11:53:56.764767 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:56.764717 2545 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 11:53:56.764893 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:56.764797 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d7f8396-e57c-45e5-ae74-a2244db10364-metrics-certs podName:1d7f8396-e57c-45e5-ae74-a2244db10364 nodeName:}" failed. No retries permitted until 2026-10-30 11:54:28.764779486 +0000 UTC m=+66.205232693 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1d7f8396-e57c-45e5-ae74-a2244db10364-metrics-certs") pod "network-metrics-daemon-dbqhj" (UID: "1d7f8396-e57c-45e5-ae74-a2244db10364") : secret "metrics-daemon-secret" not found Oct 30 11:53:56.866095 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:56.866055 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6b350710-c48c-4c7e-bce9-e021099e98ca-original-pull-secret\") pod \"global-pull-secret-syncer-fxjlt\" (UID: \"6b350710-c48c-4c7e-bce9-e021099e98ca\") " pod="kube-system/global-pull-secret-syncer-fxjlt" Oct 30 11:53:56.873448 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:56.873417 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6b350710-c48c-4c7e-bce9-e021099e98ca-original-pull-secret\") pod \"global-pull-secret-syncer-fxjlt\" (UID: \"6b350710-c48c-4c7e-bce9-e021099e98ca\") " pod="kube-system/global-pull-secret-syncer-fxjlt" Oct 30 11:53:56.966986 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:56.966867 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zh8h9\" (UniqueName: \"kubernetes.io/projected/ff9feaa5-70cc-47fd-a3d1-a9ac6e7eaace-kube-api-access-zh8h9\") pod \"network-check-target-bz65k\" (UID: \"ff9feaa5-70cc-47fd-a3d1-a9ac6e7eaace\") " pod="openshift-network-diagnostics/network-check-target-bz65k" Oct 30 11:53:56.971101 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:56.971073 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zh8h9\" (UniqueName: \"kubernetes.io/projected/ff9feaa5-70cc-47fd-a3d1-a9ac6e7eaace-kube-api-access-zh8h9\") pod \"network-check-target-bz65k\" (UID: \"ff9feaa5-70cc-47fd-a3d1-a9ac6e7eaace\") " pod="openshift-network-diagnostics/network-check-target-bz65k" Oct 30 11:53:57.064954 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:57.064878 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bz65k" Oct 30 11:53:57.135442 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:57.135410 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fxjlt" Oct 30 11:53:57.502084 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:57.502037 2545 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-ksxpw" podStartSLOduration=383.087366459 podStartE2EDuration="6m25.502020373s" totalImagesPullingTime="2.414653914s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:47:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:53:56.461314492 +0000 UTC m=+33.901767722" watchObservedRunningTime="2026-10-30 11:53:57.502020373 +0000 UTC m=+34.942473602" Oct 30 11:53:57.502824 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:57.502804 2545 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-fxjlt"] Oct 30 11:53:57.516111 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:57.516084 2545 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-bz65k"] Oct 30 11:53:57.660275 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:57.660238 2545 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b350710_c48c_4c7e_bce9_e021099e98ca.slice/crio-18e782439c70c580ec71ddf416df5349c870d492753c67e14b13bb86f2d87533 WatchSource:0}: Error finding container 18e782439c70c580ec71ddf416df5349c870d492753c67e14b13bb86f2d87533: Status 404 returned error can't find the container with id 18e782439c70c580ec71ddf416df5349c870d492753c67e14b13bb86f2d87533 Oct 30 11:53:57.661517 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:53:57.661450 2545 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff9feaa5_70cc_47fd_a3d1_a9ac6e7eaace.slice/crio-a210218da33eda8d8c8333214533a17c60812fe01e8061c5d1a295774b5006b0 WatchSource:0}: Error finding container a210218da33eda8d8c8333214533a17c60812fe01e8061c5d1a295774b5006b0: Status 404 returned error can't find the container with id a210218da33eda8d8c8333214533a17c60812fe01e8061c5d1a295774b5006b0 Oct 30 11:53:58.426047 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:58.426018 2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-bz65k" event={"ID":"ff9feaa5-70cc-47fd-a3d1-a9ac6e7eaace","Type":"ContainerStarted","Data":"a210218da33eda8d8c8333214533a17c60812fe01e8061c5d1a295774b5006b0"} Oct 30 11:53:58.427360 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:58.427254 2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-fxjlt" event={"ID":"6b350710-c48c-4c7e-bce9-e021099e98ca","Type":"ContainerStarted","Data":"18e782439c70c580ec71ddf416df5349c870d492753c67e14b13bb86f2d87533"} Oct 30 11:53:59.396698 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:59.396665 2545 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-4brks"] Oct 30 11:53:59.415456 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:59.415422 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-4brks" Oct 30 11:53:59.431079 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:59.431052 2545 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-4brks"] Oct 30 11:53:59.441329 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:59.441230 2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Oct 30 11:53:59.441329 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:59.441291 2545 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-sjprl\"" Oct 30 11:53:59.441487 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:59.441354 2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Oct 30 11:53:59.445653 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:59.445606 2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-bz65k" event={"ID":"ff9feaa5-70cc-47fd-a3d1-a9ac6e7eaace","Type":"ContainerStarted","Data":"5bc4259563ea2050edf3d6d371e2a677195e4881f0c212bdc13369bea103f30a"} Oct 30 11:53:59.445749 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:59.445732 2545 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-bz65k" Oct 30 11:53:59.447776 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:59.447427 2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-gn2z5" event={"ID":"003c3d18-23f1-45d0-bac4-8c6eea4b7e99","Type":"ContainerStarted","Data":"a54eae10e602ff2ee8306d0410114f4b6744f3aff5ff5ac8e22b60816d33d5f6"} Oct 30 11:53:59.449042 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:59.449013 2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6l6fr" event={"ID":"c815b3fc-7f35-4ae6-a015-2e84c6108a64","Type":"ContainerStarted","Data":"d2aa6d4f0aa8e185b25bd50d1c110fd6ed07e2f6210f07a050cd7959ded0257d"} Oct 30 11:53:59.450506 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:59.450480 2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-glvhs" event={"ID":"7ac30952-da5d-42a3-a1aa-bd249bbf6adb","Type":"ContainerStarted","Data":"f8472ad3d0907cbe7e90dc00650f1a79e254ac309cb22da5d404c9b004dc54a7"} Oct 30 11:53:59.452130 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:59.452105 2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-86p78" event={"ID":"b3f20243-155c-4e25-a1ca-6131a141fbd4","Type":"ContainerStarted","Data":"7a163cb0cdb6ebf4fc34c853de038086b53b4f3ad84a179b17a8b40ce7b8bad2"} Oct 30 11:53:59.453759 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:59.453737 2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-2fqgf" event={"ID":"b1757617-7c51-4029-b329-1b6a4630f860","Type":"ContainerStarted","Data":"29ce06f60b46d65917881bd1cc093f60b28d3fc22fd8f01633174881ca63e1d9"} Oct 30 11:53:59.456776 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:59.456734 2545 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-plczt_a70ee11c-c20e-4a4f-8476-ac5eb02816d0/console-operator/0.log" Oct 30 11:53:59.456776 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:59.456770 2545 generic.go:356] "Generic (PLEG): container finished" podID="a70ee11c-c20e-4a4f-8476-ac5eb02816d0" containerID="ea90c82d34076ccd2f07ede98d0f2a867f82fa5fc285773bbdd45668e3222a6c" exitCode=255 Oct 30 11:53:59.456970 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:59.456795 2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-plczt" event={"ID":"a70ee11c-c20e-4a4f-8476-ac5eb02816d0","Type":"ContainerDied","Data":"ea90c82d34076ccd2f07ede98d0f2a867f82fa5fc285773bbdd45668e3222a6c"} Oct 30 11:53:59.457047 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:59.457021 2545 scope.go:122] "RemoveContainer" containerID="ea90c82d34076ccd2f07ede98d0f2a867f82fa5fc285773bbdd45668e3222a6c" Oct 30 11:53:59.522711 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:59.521633 2545 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-vn6vv"] Oct 30 11:53:59.525842 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:59.525777 2545 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6l6fr" podStartSLOduration=382.378491927 podStartE2EDuration="6m27.525764638s" totalImagesPullingTime="5.147272711s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:47:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:53:59.52541758 +0000 UTC m=+36.965870821" watchObservedRunningTime="2026-10-30 11:53:59.525764638 +0000 UTC m=+36.966217866" Oct 30 11:53:59.542381 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:59.542336 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-vn6vv" Oct 30 11:53:59.556377 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:59.556300 2545 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-vn6vv"] Oct 30 11:53:59.576298 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:59.575905 2545 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-2fqgf" podStartSLOduration=382.34498878 podStartE2EDuration="6m27.575888313s" totalImagesPullingTime="5.230899533s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:47:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:53:59.574487613 +0000 UTC m=+37.014940838" watchObservedRunningTime="2026-10-30 11:53:59.575888313 +0000 UTC m=+37.016341540" Oct 30 11:53:59.588789 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:59.588759 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zd2h\" (UniqueName: \"kubernetes.io/projected/dab8eab1-3e3f-46b4-b76d-d58a7061996d-kube-api-access-7zd2h\") pod \"migrator-66df5d496b-4brks\" (UID: \"dab8eab1-3e3f-46b4-b76d-d58a7061996d\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-4brks" Oct 30 11:53:59.641583 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:59.641430 2545 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 11:53:59.641583 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:59.641515 2545 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-8pg5x\"" Oct 30 11:53:59.641821 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:59.641793 2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 11:53:59.669522 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:59.669445 2545 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-bz65k" podStartSLOduration=35.666686387 podStartE2EDuration="36.66943041s" totalImagesPullingTime="1.002744023s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:53:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:53:59.668140587 +0000 UTC m=+37.108593831" watchObservedRunningTime="2026-10-30 11:53:59.66943041 +0000 UTC m=+37.109883640" Oct 30 11:53:59.689367 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:59.689341 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zd2h\" (UniqueName: \"kubernetes.io/projected/dab8eab1-3e3f-46b4-b76d-d58a7061996d-kube-api-access-7zd2h\") pod \"migrator-66df5d496b-4brks\" (UID: \"dab8eab1-3e3f-46b4-b76d-d58a7061996d\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-4brks" Oct 30 11:53:59.689454 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:59.689415 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d670b364-53ce-444b-b9c4-ac8ac09cd44f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vn6vv\" (UID: \"d670b364-53ce-444b-b9c4-ac8ac09cd44f\") " pod="openshift-insights/insights-runtime-extractor-vn6vv" Oct 30 11:53:59.689509 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:59.689472 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlp7f\" (UniqueName: \"kubernetes.io/projected/d670b364-53ce-444b-b9c4-ac8ac09cd44f-kube-api-access-rlp7f\") pod \"insights-runtime-extractor-vn6vv\" (UID: \"d670b364-53ce-444b-b9c4-ac8ac09cd44f\") " pod="openshift-insights/insights-runtime-extractor-vn6vv" Oct 30 11:53:59.689509 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:59.689501 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d670b364-53ce-444b-b9c4-ac8ac09cd44f-data-volume\") pod \"insights-runtime-extractor-vn6vv\" (UID: \"d670b364-53ce-444b-b9c4-ac8ac09cd44f\") " pod="openshift-insights/insights-runtime-extractor-vn6vv" Oct 30 11:53:59.689614 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:59.689587 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d670b364-53ce-444b-b9c4-ac8ac09cd44f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vn6vv\" (UID: \"d670b364-53ce-444b-b9c4-ac8ac09cd44f\") " pod="openshift-insights/insights-runtime-extractor-vn6vv" Oct 30 11:53:59.689661 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:59.689620 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d670b364-53ce-444b-b9c4-ac8ac09cd44f-crio-socket\") pod \"insights-runtime-extractor-vn6vv\" (UID: \"d670b364-53ce-444b-b9c4-ac8ac09cd44f\") " pod="openshift-insights/insights-runtime-extractor-vn6vv" Oct 30 11:53:59.720061 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:59.718916 2545 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-86p78" podStartSLOduration=382.546056569 podStartE2EDuration="6m27.718905095s" totalImagesPullingTime="5.172848526s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:47:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:53:59.718746562 +0000 UTC m=+37.159199793" watchObservedRunningTime="2026-10-30 11:53:59.718905095 +0000 UTC m=+37.159358323" Oct 30 11:53:59.729108 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:59.729079 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zd2h\" (UniqueName: \"kubernetes.io/projected/dab8eab1-3e3f-46b4-b76d-d58a7061996d-kube-api-access-7zd2h\") pod \"migrator-66df5d496b-4brks\" (UID: \"dab8eab1-3e3f-46b4-b76d-d58a7061996d\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-4brks" Oct 30 11:53:59.746526 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:59.746497 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-4brks" Oct 30 11:53:59.761526 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:59.761484 2545 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-glvhs" podStartSLOduration=382.156473906 podStartE2EDuration="6m27.761469623s" totalImagesPullingTime="5.604995717s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:47:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:53:59.760076784 +0000 UTC m=+37.200530019" watchObservedRunningTime="2026-10-30 11:53:59.761469623 +0000 UTC m=+37.201922853" Oct 30 11:53:59.790941 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:59.790903 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d670b364-53ce-444b-b9c4-ac8ac09cd44f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vn6vv\" (UID: \"d670b364-53ce-444b-b9c4-ac8ac09cd44f\") " pod="openshift-insights/insights-runtime-extractor-vn6vv" Oct 30 11:53:59.791122 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:59.791093 2545 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 11:53:59.791195 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:59.791060 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d670b364-53ce-444b-b9c4-ac8ac09cd44f-crio-socket\") pod \"insights-runtime-extractor-vn6vv\" (UID: \"d670b364-53ce-444b-b9c4-ac8ac09cd44f\") " pod="openshift-insights/insights-runtime-extractor-vn6vv" Oct 30 11:53:59.791195 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:59.791188 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d670b364-53ce-444b-b9c4-ac8ac09cd44f-insights-runtime-extractor-tls podName:d670b364-53ce-444b-b9c4-ac8ac09cd44f nodeName:}" failed. No retries permitted until 2026-10-30 11:54:00.291167753 +0000 UTC m=+37.731620973 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d670b364-53ce-444b-b9c4-ac8ac09cd44f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vn6vv" (UID: "d670b364-53ce-444b-b9c4-ac8ac09cd44f") : secret "insights-runtime-extractor-tls" not found Oct 30 11:53:59.791306 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:59.791216 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d670b364-53ce-444b-b9c4-ac8ac09cd44f-crio-socket\") pod \"insights-runtime-extractor-vn6vv\" (UID: \"d670b364-53ce-444b-b9c4-ac8ac09cd44f\") " pod="openshift-insights/insights-runtime-extractor-vn6vv" Oct 30 11:53:59.791355 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:59.791323 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d670b364-53ce-444b-b9c4-ac8ac09cd44f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vn6vv\" (UID: \"d670b364-53ce-444b-b9c4-ac8ac09cd44f\") " pod="openshift-insights/insights-runtime-extractor-vn6vv" Oct 30 11:53:59.791632 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:59.791415 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlp7f\" (UniqueName: \"kubernetes.io/projected/d670b364-53ce-444b-b9c4-ac8ac09cd44f-kube-api-access-rlp7f\") pod \"insights-runtime-extractor-vn6vv\" (UID: \"d670b364-53ce-444b-b9c4-ac8ac09cd44f\") " pod="openshift-insights/insights-runtime-extractor-vn6vv" Oct 30 11:53:59.791632 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:59.791480 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d670b364-53ce-444b-b9c4-ac8ac09cd44f-data-volume\") pod \"insights-runtime-extractor-vn6vv\" (UID: \"d670b364-53ce-444b-b9c4-ac8ac09cd44f\") " pod="openshift-insights/insights-runtime-extractor-vn6vv" Oct 30 11:53:59.791632 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:59.791482 2545 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 11:53:59.791632 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:53:59.791542 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d670b364-53ce-444b-b9c4-ac8ac09cd44f-kube-rbac-proxy-cm podName:d670b364-53ce-444b-b9c4-ac8ac09cd44f nodeName:}" failed. No retries permitted until 2026-10-30 11:54:00.291523484 +0000 UTC m=+37.731976708 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d670b364-53ce-444b-b9c4-ac8ac09cd44f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vn6vv" (UID: "d670b364-53ce-444b-b9c4-ac8ac09cd44f") : configmap "kube-rbac-proxy" not found Oct 30 11:53:59.792131 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:59.792083 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d670b364-53ce-444b-b9c4-ac8ac09cd44f-data-volume\") pod \"insights-runtime-extractor-vn6vv\" (UID: \"d670b364-53ce-444b-b9c4-ac8ac09cd44f\") " pod="openshift-insights/insights-runtime-extractor-vn6vv" Oct 30 11:53:59.802392 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:59.802317 2545 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-gn2z5" podStartSLOduration=2.693899063 podStartE2EDuration="7.802301931s" totalImagesPullingTime="5.108402868s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:53:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:53:59.800815902 +0000 UTC m=+37.241269124" watchObservedRunningTime="2026-10-30 11:53:59.802301931 +0000 UTC m=+37.242755161" Oct 30 11:53:59.818669 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:53:59.818637 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlp7f\" (UniqueName: \"kubernetes.io/projected/d670b364-53ce-444b-b9c4-ac8ac09cd44f-kube-api-access-rlp7f\") pod \"insights-runtime-extractor-vn6vv\" (UID: \"d670b364-53ce-444b-b9c4-ac8ac09cd44f\") " pod="openshift-insights/insights-runtime-extractor-vn6vv" Oct 30 11:54:00.070315 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:54:00.070276 2545 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-4brks"] Oct 30 11:54:00.074029 ip-10-0-141-251 kubenswrapper[2545]: W1030 11:54:00.073997 2545 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddab8eab1_3e3f_46b4_b76d_d58a7061996d.slice/crio-50729bdfcb9b631e0bcdd1ee0528e7c3c9dacbac0f5276aa7f47aaca3c6bc046 WatchSource:0}: Error finding container 50729bdfcb9b631e0bcdd1ee0528e7c3c9dacbac0f5276aa7f47aaca3c6bc046: Status 404 returned error can't find the container with id 50729bdfcb9b631e0bcdd1ee0528e7c3c9dacbac0f5276aa7f47aaca3c6bc046 Oct 30 11:54:00.195198 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:54:00.195162 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/179d55d3-7958-4928-a98c-68984fca9837-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-zzxhv\" (UID: \"179d55d3-7958-4928-a98c-68984fca9837\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-zzxhv" Oct 30 11:54:00.195316 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:54:00.195214 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/221ced85-cc15-4e07-82d9-a1a1f3e958a5-metrics-tls\") pod \"dns-default-xxcbn\" (UID: \"221ced85-cc15-4e07-82d9-a1a1f3e958a5\") " pod="openshift-dns/dns-default-xxcbn" Oct 30 11:54:00.195316 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:54:00.195281 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cfaad7d0-d30b-4ddc-81cc-2b603416fc29-registry-tls\") pod \"image-registry-79fd565486-8fl78\" (UID: \"cfaad7d0-d30b-4ddc-81cc-2b603416fc29\") " pod="openshift-image-registry/image-registry-79fd565486-8fl78" Oct 30 11:54:00.195407 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:54:00.195319 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6d976900-ecd0-469c-a9e2-d60708361f1a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-mvhwk\" (UID: \"6d976900-ecd0-469c-a9e2-d60708361f1a\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-mvhwk" Oct 30 11:54:00.195407 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:54:00.195320 2545 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 11:54:00.195407 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:54:00.195381 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/179d55d3-7958-4928-a98c-68984fca9837-networking-console-plugin-cert podName:179d55d3-7958-4928-a98c-68984fca9837 nodeName:}" failed. No retries permitted until 2026-10-30 11:54:08.195361141 +0000 UTC m=+45.635814349 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/179d55d3-7958-4928-a98c-68984fca9837-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-zzxhv" (UID: "179d55d3-7958-4928-a98c-68984fca9837") : secret "networking-console-plugin-cert" not found Oct 30 11:54:00.195585 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:54:00.195431 2545 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 11:54:00.195585 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:54:00.195478 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d976900-ecd0-469c-a9e2-d60708361f1a-cluster-monitoring-operator-tls podName:6d976900-ecd0-469c-a9e2-d60708361f1a nodeName:}" failed. No retries permitted until 2026-10-30 11:54:08.195466926 +0000 UTC m=+45.635920134 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/6d976900-ecd0-469c-a9e2-d60708361f1a-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-mvhwk" (UID: "6d976900-ecd0-469c-a9e2-d60708361f1a") : secret "cluster-monitoring-operator-tls" not found Oct 30 11:54:00.195585 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:54:00.195522 2545 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 11:54:00.195585 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:54:00.195552 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/221ced85-cc15-4e07-82d9-a1a1f3e958a5-metrics-tls podName:221ced85-cc15-4e07-82d9-a1a1f3e958a5 nodeName:}" failed. No retries permitted until 2026-10-30 11:54:08.195542008 +0000 UTC m=+45.635995219 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/221ced85-cc15-4e07-82d9-a1a1f3e958a5-metrics-tls") pod "dns-default-xxcbn" (UID: "221ced85-cc15-4e07-82d9-a1a1f3e958a5") : secret "dns-default-metrics-tls" not found Oct 30 11:54:00.195720 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:54:00.195610 2545 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 11:54:00.195720 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:54:00.195619 2545 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-79fd565486-8fl78: secret "image-registry-tls" not found Oct 30 11:54:00.195720 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:54:00.195648 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cfaad7d0-d30b-4ddc-81cc-2b603416fc29-registry-tls podName:cfaad7d0-d30b-4ddc-81cc-2b603416fc29 nodeName:}" failed. No retries permitted until 2026-10-30 11:54:08.195637947 +0000 UTC m=+45.636091158 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cfaad7d0-d30b-4ddc-81cc-2b603416fc29-registry-tls") pod "image-registry-79fd565486-8fl78" (UID: "cfaad7d0-d30b-4ddc-81cc-2b603416fc29") : secret "image-registry-tls" not found Oct 30 11:54:00.296524 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:54:00.296496 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6acb13a1-cb2e-45e1-897b-3ef89e6d1b84-metrics-certs\") pod \"router-default-5bfdf864c-tzv67\" (UID: \"6acb13a1-cb2e-45e1-897b-3ef89e6d1b84\") " pod="openshift-ingress/router-default-5bfdf864c-tzv67" Oct 30 11:54:00.296654 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:54:00.296531 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bd977255-f9a7-4531-8a9c-b1a3a17e0b79-registry-tls\") pod \"image-registry-557c6b9f47-vcr2t\" (UID: \"bd977255-f9a7-4531-8a9c-b1a3a17e0b79\") " pod="openshift-image-registry/image-registry-557c6b9f47-vcr2t" Oct 30 11:54:00.296654 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:54:00.296552 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d670b364-53ce-444b-b9c4-ac8ac09cd44f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vn6vv\" (UID: \"d670b364-53ce-444b-b9c4-ac8ac09cd44f\") " pod="openshift-insights/insights-runtime-extractor-vn6vv" Oct 30 11:54:00.296654 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:54:00.296578 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6acb13a1-cb2e-45e1-897b-3ef89e6d1b84-service-ca-bundle\") pod \"router-default-5bfdf864c-tzv67\" (UID: \"6acb13a1-cb2e-45e1-897b-3ef89e6d1b84\") " pod="openshift-ingress/router-default-5bfdf864c-tzv67" Oct 30 11:54:00.296654 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:54:00.296636 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7689cfe5-5a4e-4481-8eb9-107a58ca8bb3-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-xdxkp\" (UID: \"7689cfe5-5a4e-4481-8eb9-107a58ca8bb3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xdxkp" Oct 30 11:54:00.296654 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:54:00.296643 2545 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 11:54:00.296815 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:54:00.296660 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d670b364-53ce-444b-b9c4-ac8ac09cd44f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vn6vv\" (UID: \"d670b364-53ce-444b-b9c4-ac8ac09cd44f\") " pod="openshift-insights/insights-runtime-extractor-vn6vv" Oct 30 11:54:00.296815 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:54:00.296642 2545 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 11:54:00.296815 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:54:00.296737 2545 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 11:54:00.296815 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:54:00.296646 2545 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 11:54:00.296815 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:54:00.296765 2545 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 11:54:00.296815 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:54:00.296766 2545 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-557c6b9f47-vcr2t: secret "image-registry-tls" not found Oct 30 11:54:00.296815 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:54:00.296722 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6acb13a1-cb2e-45e1-897b-3ef89e6d1b84-service-ca-bundle podName:6acb13a1-cb2e-45e1-897b-3ef89e6d1b84 nodeName:}" failed. No retries permitted until 2026-10-30 11:54:08.29670793 +0000 UTC m=+45.737161138 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6acb13a1-cb2e-45e1-897b-3ef89e6d1b84-service-ca-bundle") pod "router-default-5bfdf864c-tzv67" (UID: "6acb13a1-cb2e-45e1-897b-3ef89e6d1b84") : configmap references non-existent config key: service-ca.crt Oct 30 11:54:00.297108 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:54:00.296901 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6acb13a1-cb2e-45e1-897b-3ef89e6d1b84-metrics-certs podName:6acb13a1-cb2e-45e1-897b-3ef89e6d1b84 nodeName:}" failed. No retries permitted until 2026-10-30 11:54:08.296880534 +0000 UTC m=+45.737333745 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6acb13a1-cb2e-45e1-897b-3ef89e6d1b84-metrics-certs") pod "router-default-5bfdf864c-tzv67" (UID: "6acb13a1-cb2e-45e1-897b-3ef89e6d1b84") : secret "router-metrics-certs-default" not found Oct 30 11:54:00.297108 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:54:00.296922 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d670b364-53ce-444b-b9c4-ac8ac09cd44f-kube-rbac-proxy-cm podName:d670b364-53ce-444b-b9c4-ac8ac09cd44f nodeName:}" failed. No retries permitted until 2026-10-30 11:54:01.296912357 +0000 UTC m=+38.737365570 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d670b364-53ce-444b-b9c4-ac8ac09cd44f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vn6vv" (UID: "d670b364-53ce-444b-b9c4-ac8ac09cd44f") : configmap "kube-rbac-proxy" not found Oct 30 11:54:00.297108 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:54:00.296940 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7689cfe5-5a4e-4481-8eb9-107a58ca8bb3-samples-operator-tls podName:7689cfe5-5a4e-4481-8eb9-107a58ca8bb3 nodeName:}" failed. No retries permitted until 2026-10-30 11:54:08.296931861 +0000 UTC m=+45.737385068 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7689cfe5-5a4e-4481-8eb9-107a58ca8bb3-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-xdxkp" (UID: "7689cfe5-5a4e-4481-8eb9-107a58ca8bb3") : secret "samples-operator-tls" not found Oct 30 11:54:00.297108 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:54:00.296961 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d670b364-53ce-444b-b9c4-ac8ac09cd44f-insights-runtime-extractor-tls podName:d670b364-53ce-444b-b9c4-ac8ac09cd44f nodeName:}" failed. No retries permitted until 2026-10-30 11:54:01.296952702 +0000 UTC m=+38.737405910 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d670b364-53ce-444b-b9c4-ac8ac09cd44f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vn6vv" (UID: "d670b364-53ce-444b-b9c4-ac8ac09cd44f") : secret "insights-runtime-extractor-tls" not found Oct 30 11:54:00.297108 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:54:00.296982 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bd977255-f9a7-4531-8a9c-b1a3a17e0b79-registry-tls podName:bd977255-f9a7-4531-8a9c-b1a3a17e0b79 nodeName:}" failed. No retries permitted until 2026-10-30 11:54:08.296974605 +0000 UTC m=+45.737427817 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bd977255-f9a7-4531-8a9c-b1a3a17e0b79-registry-tls") pod "image-registry-557c6b9f47-vcr2t" (UID: "bd977255-f9a7-4531-8a9c-b1a3a17e0b79") : secret "image-registry-tls" not found Oct 30 11:54:00.462618 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:54:00.462543 2545 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-plczt_a70ee11c-c20e-4a4f-8476-ac5eb02816d0/console-operator/1.log" Oct 30 11:54:00.463046 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:54:00.463032 2545 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-plczt_a70ee11c-c20e-4a4f-8476-ac5eb02816d0/console-operator/0.log" Oct 30 11:54:00.463121 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:54:00.463063 2545 generic.go:356] "Generic (PLEG): container finished" podID="a70ee11c-c20e-4a4f-8476-ac5eb02816d0" containerID="cc5a67034e1729623ea0114fef781e8204b8eab853c4f28263d4d9d9c45f99fd" exitCode=255 Oct 30 11:54:00.463162 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:54:00.463148 2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-plczt" event={"ID":"a70ee11c-c20e-4a4f-8476-ac5eb02816d0","Type":"ContainerDied","Data":"cc5a67034e1729623ea0114fef781e8204b8eab853c4f28263d4d9d9c45f99fd"} Oct 30 11:54:00.463206 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:54:00.463190 2545 scope.go:122] "RemoveContainer" containerID="ea90c82d34076ccd2f07ede98d0f2a867f82fa5fc285773bbdd45668e3222a6c" Oct 30 11:54:00.463518 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:54:00.463500 2545 scope.go:122] "RemoveContainer" containerID="cc5a67034e1729623ea0114fef781e8204b8eab853c4f28263d4d9d9c45f99fd" Oct 30 11:54:00.463690 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:54:00.463671 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-plczt_openshift-console-operator(a70ee11c-c20e-4a4f-8476-ac5eb02816d0)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-plczt" podUID="a70ee11c-c20e-4a4f-8476-ac5eb02816d0" Oct 30 11:54:00.465417 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:54:00.465393 2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-4brks" event={"ID":"dab8eab1-3e3f-46b4-b76d-d58a7061996d","Type":"ContainerStarted","Data":"50729bdfcb9b631e0bcdd1ee0528e7c3c9dacbac0f5276aa7f47aaca3c6bc046"} Oct 30 11:54:01.308685 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:54:01.308649 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d670b364-53ce-444b-b9c4-ac8ac09cd44f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vn6vv\" (UID: \"d670b364-53ce-444b-b9c4-ac8ac09cd44f\") " pod="openshift-insights/insights-runtime-extractor-vn6vv" Oct 30 11:54:01.308969 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:54:01.308740 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d670b364-53ce-444b-b9c4-ac8ac09cd44f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vn6vv\" (UID: \"d670b364-53ce-444b-b9c4-ac8ac09cd44f\") " pod="openshift-insights/insights-runtime-extractor-vn6vv" Oct 30 11:54:01.308969 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:54:01.308812 2545 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 11:54:01.308969 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:54:01.308817 2545 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 11:54:01.308969 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:54:01.308895 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d670b364-53ce-444b-b9c4-ac8ac09cd44f-kube-rbac-proxy-cm podName:d670b364-53ce-444b-b9c4-ac8ac09cd44f nodeName:}" failed. No retries permitted until 2026-10-30 11:54:03.308876812 +0000 UTC m=+40.749330023 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d670b364-53ce-444b-b9c4-ac8ac09cd44f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vn6vv" (UID: "d670b364-53ce-444b-b9c4-ac8ac09cd44f") : configmap "kube-rbac-proxy" not found Oct 30 11:54:01.308969 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:54:01.308913 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d670b364-53ce-444b-b9c4-ac8ac09cd44f-insights-runtime-extractor-tls podName:d670b364-53ce-444b-b9c4-ac8ac09cd44f nodeName:}" failed. No retries permitted until 2026-10-30 11:54:03.308904005 +0000 UTC m=+40.749357215 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d670b364-53ce-444b-b9c4-ac8ac09cd44f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vn6vv" (UID: "d670b364-53ce-444b-b9c4-ac8ac09cd44f") : secret "insights-runtime-extractor-tls" not found Oct 30 11:54:01.469906 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:54:01.469881 2545 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-plczt_a70ee11c-c20e-4a4f-8476-ac5eb02816d0/console-operator/1.log" Oct 30 11:54:01.470287 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:54:01.470194 2545 scope.go:122] "RemoveContainer" containerID="cc5a67034e1729623ea0114fef781e8204b8eab853c4f28263d4d9d9c45f99fd" Oct 30 11:54:01.470373 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:54:01.470352 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-plczt_openshift-console-operator(a70ee11c-c20e-4a4f-8476-ac5eb02816d0)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-plczt" podUID="a70ee11c-c20e-4a4f-8476-ac5eb02816d0" Oct 30 11:54:02.475110 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:54:02.475073 2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-4brks" event={"ID":"dab8eab1-3e3f-46b4-b76d-d58a7061996d","Type":"ContainerStarted","Data":"1310624df076a702d14105bad70f794ecc8374e61790694264de075f096ff477"} Oct 30 11:54:02.529644 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:54:02.529594 2545 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-4brks" podStartSLOduration=2.171681077 podStartE2EDuration="3.529578557s" totalImagesPullingTime="1.35789748s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:53:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:54:02.528710008 +0000 UTC m=+39.969163238" watchObservedRunningTime="2026-10-30 11:54:02.529578557 +0000 UTC m=+39.970031805" Oct 30 11:54:02.906312 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:54:02.906278 2545 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-plczt" Oct 30 11:54:02.906477 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:54:02.906333 2545 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-plczt" Oct 30 11:54:02.906656 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:54:02.906644 2545 scope.go:122] "RemoveContainer" containerID="cc5a67034e1729623ea0114fef781e8204b8eab853c4f28263d4d9d9c45f99fd" Oct 30 11:54:02.906811 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:54:02.906794 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-plczt_openshift-console-operator(a70ee11c-c20e-4a4f-8476-ac5eb02816d0)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-plczt" podUID="a70ee11c-c20e-4a4f-8476-ac5eb02816d0" Oct 30 11:54:03.325938 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:54:03.325910 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d670b364-53ce-444b-b9c4-ac8ac09cd44f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vn6vv\" (UID: \"d670b364-53ce-444b-b9c4-ac8ac09cd44f\") " pod="openshift-insights/insights-runtime-extractor-vn6vv" Oct 30 11:54:03.326069 ip-10-0-141-251 kubenswrapper[2545]: I1030 11:54:03.325990 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d670b364-53ce-444b-b9c4-ac8ac09cd44f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vn6vv\" (UID: \"d670b364-53ce-444b-b9c4-ac8ac09cd44f\") " pod="openshift-insights/insights-runtime-extractor-vn6vv" Oct 30 11:54:03.326069 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:54:03.326054 2545 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 11:54:03.326143 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:54:03.326068 2545 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 11:54:03.326143 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:54:03.326105 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d670b364-53ce-444b-b9c4-ac8ac09cd44f-kube-rbac-proxy-cm podName:d670b364-53ce-444b-b9c4-ac8ac09cd44f nodeName:}" failed. No retries permitted until 2026-10-30 11:54:07.326090297 +0000 UTC m=+44.766543520 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d670b364-53ce-444b-b9c4-ac8ac09cd44f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vn6vv" (UID: "d670b364-53ce-444b-b9c4-ac8ac09cd44f") : configmap "kube-rbac-proxy" not found Oct 30 11:54:03.326143 ip-10-0-141-251 kubenswrapper[2545]: E1030 11:54:03.326131 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d670b364-53ce-444b-b9c4-ac8ac09cd44f-insights-runtime-extractor-tls podName:d670b364-53ce-444b-b9c4-ac8ac09cd44f nodeName:}" failed. No retries permitted until 2026-10-30 11:54:07.326114394 +0000 UTC m=+44.766567618 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d670b364-53ce-444b-b9c4-ac8ac09cd44f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vn6vv" (UID: "d670b364-53ce-444b-b9c4-ac8ac09cd44f") : secret "insights-runtime-extractor-tls" not found Oct 30 11:54:03.435089 ip-10-0-141-251 systemd[1]: Stopping Kubernetes Kubelet... Oct 30 11:54:03.445177 ip-10-0-141-251 systemd[1]: kubelet.service: Deactivated successfully. Oct 30 11:54:03.445340 ip-10-0-141-251 systemd[1]: Stopped Kubernetes Kubelet. Oct 30 11:54:03.445387 ip-10-0-141-251 systemd[1]: kubelet.service: Consumed 2.820s CPU time. Oct 30 11:54:03.456440 ip-10-0-141-251 systemd[1]: Starting Kubernetes Kubelet... Oct 30 11:54:03.524583 ip-10-0-141-251 kubenswrapper[4908]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 11:54:03.524583 ip-10-0-141-251 kubenswrapper[4908]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 11:54:03.524583 ip-10-0-141-251 kubenswrapper[4908]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 11:54:03.524583 ip-10-0-141-251 kubenswrapper[4908]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 11:54:03.526607 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.526538 4908 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 11:54:03.526607 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.526607 4908 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 11:54:03.526675 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.526611 4908 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 11:54:03.526675 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.526615 4908 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 11:54:03.526675 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.526618 4908 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 11:54:03.526675 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.526621 4908 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 11:54:03.526675 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.526623 4908 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 11:54:03.526675 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.526627 4908 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 11:54:03.526675 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.526629 4908 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 11:54:03.526675 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.526632 4908 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 11:54:03.526675 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.526634 4908 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 11:54:03.526675 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.526637 4908 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 11:54:03.526675 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.526640 4908 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 11:54:03.526675 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.526643 4908 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 11:54:03.526675 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.526649 4908 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 11:54:03.526675 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.526652 4908 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 11:54:03.526675 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.526655 4908 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 11:54:03.526675 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.526657 4908 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 11:54:03.526675 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.526660 4908 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 11:54:03.526675 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.526663 4908 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 11:54:03.526675 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.526665 4908 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 11:54:03.527160 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.526668 4908 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 11:54:03.527160 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.526670 4908 feature_gate.go:386] unrecognized feature gate: Example Oct 30 11:54:03.527160 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.526673 4908 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 11:54:03.527160 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.526675 4908 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 11:54:03.527160 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.526678 4908 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 11:54:03.527160 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.526681 4908 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 11:54:03.527160 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.526684 4908 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 11:54:03.527160 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.526687 4908 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 11:54:03.527160 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.526689 4908 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 11:54:03.527160 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.526692 4908 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 11:54:03.527160 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.526695 4908 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 11:54:03.527160 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.526697 4908 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 11:54:03.527160 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.526700 4908 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 11:54:03.527160 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.526702 4908 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 11:54:03.527160 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.526705 4908 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 11:54:03.527160 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.526707 4908 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 11:54:03.527160 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.526709 4908 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 11:54:03.527160 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.526712 4908 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 11:54:03.527160 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.526715 4908 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 11:54:03.527160 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.526717 4908 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 11:54:03.527662 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.526720 4908 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 11:54:03.527662 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.526722 4908 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 11:54:03.527662 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.526725 4908 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 11:54:03.527662 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.526727 4908 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 11:54:03.527662 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.526730 4908 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 11:54:03.527662 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.526732 4908 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 11:54:03.527662 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.526735 4908 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 11:54:03.527662 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.526738 4908 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 11:54:03.527662 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.526741 4908 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 11:54:03.527662 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.526743 4908 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 11:54:03.527662 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.526746 4908 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 11:54:03.527662 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.526750 4908 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 11:54:03.527662 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.526752 4908 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 11:54:03.527662 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.526755 4908 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 11:54:03.527662 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.526758 4908 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 11:54:03.527662 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.526760 4908 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 11:54:03.527662 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.526763 4908 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 11:54:03.527662 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.526765 4908 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 11:54:03.527662 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.526768 4908 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 11:54:03.527662 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.526771 4908 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 11:54:03.528173 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.526773 4908 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 11:54:03.528173 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.526775 4908 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 11:54:03.528173 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.526778 4908 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 11:54:03.528173 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.526781 4908 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 11:54:03.528173 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.526784 4908 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 11:54:03.528173 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.526786 4908 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 11:54:03.528173 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.526789 4908 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 11:54:03.528173 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.526791 4908 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 11:54:03.528173 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.526794 4908 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 11:54:03.528173 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.526796 4908 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 11:54:03.528173 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.526799 4908 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 11:54:03.528173 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.526802 4908 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 11:54:03.528173 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.526804 4908 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 11:54:03.528173 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.526806 4908 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 11:54:03.528173 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.526809 4908 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 11:54:03.528173 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.526812 4908 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 11:54:03.528173 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.526814 4908 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 11:54:03.528173 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.526816 4908 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 11:54:03.528173 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.526819 4908 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 11:54:03.528639 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.526822 4908 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 11:54:03.528639 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.526837 4908 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 11:54:03.528639 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.526841 4908 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 11:54:03.528639 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.526843 4908 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 11:54:03.528639 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.526846 4908 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 11:54:03.528639 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.526849 4908 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 11:54:03.528639 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.526851 4908 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 11:54:03.528639 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.526854 4908 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 11:54:03.528639 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.526856 4908 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 11:54:03.528639 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.526859 4908 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 11:54:03.528639 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.526861 4908 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 11:54:03.528639 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.526864 4908 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 11:54:03.528639 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.526866 4908 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 11:54:03.528639 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.526869 4908 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 11:54:03.528639 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.526871 4908 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 11:54:03.528639 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.526874 4908 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 11:54:03.528639 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.526878 4908 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 11:54:03.528639 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.526882 4908 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 11:54:03.528639 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.526885 4908 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 11:54:03.529145 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.526888 4908 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 11:54:03.529145 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.526892 4908 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 11:54:03.529145 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.526895 4908 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 11:54:03.529145 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.526897 4908 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 11:54:03.529145 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.526901 4908 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 11:54:03.529145 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.526905 4908 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 11:54:03.529145 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.526908 4908 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 11:54:03.529145 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.527611 4908 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 11:54:03.529145 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.527618 4908 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 11:54:03.529145 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.527620 4908 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 11:54:03.529145 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.527623 4908 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 11:54:03.529145 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.527626 4908 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 11:54:03.529145 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.527628 4908 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 11:54:03.529145 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.527632 4908 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 11:54:03.529145 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.527637 4908 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 11:54:03.529145 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.527640 4908 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 11:54:03.529145 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.527642 4908 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 11:54:03.529145 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.527645 4908 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 11:54:03.529145 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.527648 4908 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 11:54:03.529145 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.527651 4908 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 11:54:03.529637 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.527653 4908 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 11:54:03.529637 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.527656 4908 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 11:54:03.529637 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.527658 4908 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 11:54:03.529637 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.527661 4908 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 11:54:03.529637 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.527663 4908 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 11:54:03.529637 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.527666 4908 feature_gate.go:386] unrecognized feature gate: Example Oct 30 11:54:03.529637 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.527669 4908 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 11:54:03.529637 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.527671 4908 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 11:54:03.529637 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.527674 4908 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 11:54:03.529637 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.527677 4908 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 11:54:03.529637 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.527680 4908 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 11:54:03.529637 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.527682 4908 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 11:54:03.529637 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.527684 4908 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 11:54:03.529637 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.527687 4908 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 11:54:03.529637 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.527690 4908 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 11:54:03.529637 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.527693 4908 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 11:54:03.529637 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.527695 4908 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 11:54:03.529637 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.527697 4908 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 11:54:03.529637 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.527700 4908 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 11:54:03.529637 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.527702 4908 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 11:54:03.530148 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.527705 4908 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 11:54:03.530148 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.527707 4908 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 11:54:03.530148 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.527710 4908 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 11:54:03.530148 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.527712 4908 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 11:54:03.530148 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.527715 4908 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 11:54:03.530148 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.527718 4908 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 11:54:03.530148 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.527720 4908 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 11:54:03.530148 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.527726 4908 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 11:54:03.530148 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.527728 4908 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 11:54:03.530148 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.527731 4908 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 11:54:03.530148 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.527734 4908 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 11:54:03.530148 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.527736 4908 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 11:54:03.530148 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.527738 4908 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 11:54:03.530148 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.527741 4908 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 11:54:03.530148 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.527744 4908 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 11:54:03.530148 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.527746 4908 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 11:54:03.530148 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.527749 4908 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 11:54:03.530148 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.527751 4908 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 11:54:03.530148 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.527754 4908 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 11:54:03.530618 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.527756 4908 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 11:54:03.530618 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.527759 4908 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 11:54:03.530618 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.527761 4908 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 11:54:03.530618 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.527764 4908 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 11:54:03.530618 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.527766 4908 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 11:54:03.530618 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.527769 4908 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 11:54:03.530618 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.527771 4908 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 11:54:03.530618 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.527773 4908 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 11:54:03.530618 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.527776 4908 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 11:54:03.530618 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.527778 4908 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 11:54:03.530618 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.527783 4908 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 11:54:03.530618 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.527787 4908 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 11:54:03.530618 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.527789 4908 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 11:54:03.530618 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.527792 4908 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 11:54:03.530618 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.527794 4908 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 11:54:03.530618 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.527797 4908 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 11:54:03.530618 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.527801 4908 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 11:54:03.530618 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.527805 4908 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 11:54:03.530618 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.527808 4908 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 11:54:03.531093 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.527810 4908 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 11:54:03.531093 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.527813 4908 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 11:54:03.531093 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.527816 4908 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 11:54:03.531093 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.527819 4908 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 11:54:03.531093 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.527821 4908 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 11:54:03.531093 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.527839 4908 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 11:54:03.531093 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.527843 4908 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 11:54:03.531093 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.527847 4908 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 11:54:03.531093 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.527851 4908 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 11:54:03.531093 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.527855 4908 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 11:54:03.531093 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.527859 4908 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 11:54:03.531093 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.527863 4908 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 11:54:03.531093 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.527867 4908 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 11:54:03.531093 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.527871 4908 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 11:54:03.531093 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.527873 4908 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 11:54:03.531093 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.527876 4908 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 11:54:03.531093 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.527879 4908 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 11:54:03.531093 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.527881 4908 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 11:54:03.531093 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.527884 4908 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 11:54:03.531093 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.527886 4908 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 11:54:03.531590 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.527889 4908 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 11:54:03.531590 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.527891 4908 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 11:54:03.531590 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.527894 4908 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 11:54:03.531590 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.527896 4908 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 11:54:03.531590 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.527899 4908 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 11:54:03.531590 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.527902 4908 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 11:54:03.531590 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.527904 4908 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 11:54:03.531590 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.527907 4908 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 11:54:03.531590 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.527913 4908 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 11:54:03.531590 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.527917 4908 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 11:54:03.531590 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.527919 4908 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 11:54:03.531590 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.527922 4908 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 11:54:03.531590 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.527925 4908 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 11:54:03.531590 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.527928 4908 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 11:54:03.531590 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.527931 4908 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 11:54:03.531590 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.528382 4908 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 11:54:03.531590 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.528387 4908 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 11:54:03.531590 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.528390 4908 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 11:54:03.531590 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.528393 4908 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 11:54:03.532118 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.528396 4908 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 11:54:03.532118 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.528399 4908 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 11:54:03.532118 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.528401 4908 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 11:54:03.532118 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.528404 4908 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 11:54:03.532118 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.528407 4908 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 11:54:03.532118 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.528410 4908 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 11:54:03.532118 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.528412 4908 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 11:54:03.532118 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.528415 4908 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 11:54:03.532118 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.528417 4908 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 11:54:03.532118 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.528420 4908 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 11:54:03.532118 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.528422 4908 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 11:54:03.532118 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.528425 4908 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 11:54:03.532118 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.528427 4908 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 11:54:03.532118 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.528430 4908 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 11:54:03.532118 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.528432 4908 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 11:54:03.532118 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.528435 4908 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 11:54:03.532118 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.528437 4908 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 11:54:03.532118 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.528440 4908 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 11:54:03.532118 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.528442 4908 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 11:54:03.532118 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.528445 4908 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 11:54:03.532599 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.528447 4908 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 11:54:03.532599 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.528455 4908 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 11:54:03.532599 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.528457 4908 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 11:54:03.532599 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.528460 4908 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 11:54:03.532599 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.528462 4908 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 11:54:03.532599 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.528465 4908 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 11:54:03.532599 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.528467 4908 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 11:54:03.532599 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.528470 4908 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 11:54:03.532599 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.528474 4908 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 11:54:03.532599 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.528478 4908 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 11:54:03.532599 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.528482 4908 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 11:54:03.532599 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.528485 4908 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 11:54:03.532599 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.528489 4908 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 11:54:03.532599 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.528492 4908 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 11:54:03.532599 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.528495 4908 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 11:54:03.532599 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.528497 4908 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 11:54:03.532599 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.528501 4908 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 11:54:03.532599 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.528503 4908 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 11:54:03.532599 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.528507 4908 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 11:54:03.533089 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.528510 4908 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 11:54:03.533089 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.528513 4908 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 11:54:03.533089 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.528516 4908 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 11:54:03.533089 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.528518 4908 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 11:54:03.533089 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.528521 4908 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 11:54:03.533089 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.528523 4908 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 11:54:03.533089 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.528526 4908 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 11:54:03.533089 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.528529 4908 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 11:54:03.533089 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.528531 4908 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 11:54:03.533089 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.528534 4908 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 11:54:03.533089 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.528536 4908 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 11:54:03.533089 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.528539 4908 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 11:54:03.533089 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.528541 4908 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 11:54:03.533089 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.528544 4908 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 11:54:03.533089 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.528547 4908 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 11:54:03.533089 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.528550 4908 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 11:54:03.533089 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.528553 4908 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 11:54:03.533089 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.528556 4908 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 11:54:03.533089 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.528558 4908 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 11:54:03.533718 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.528561 4908 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 11:54:03.533718 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.528564 4908 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 11:54:03.533718 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.528566 4908 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 11:54:03.533718 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.528569 4908 feature_gate.go:386] unrecognized feature gate: Example Oct 30 11:54:03.533718 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.528572 4908 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 11:54:03.533718 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.528574 4908 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 11:54:03.533718 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.528577 4908 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 11:54:03.533718 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.528579 4908 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 11:54:03.533718 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.528582 4908 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 11:54:03.533718 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.528584 4908 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 11:54:03.533718 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.528587 4908 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 11:54:03.533718 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.528589 4908 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 11:54:03.533718 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.528592 4908 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 11:54:03.533718 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.528594 4908 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 11:54:03.533718 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.528597 4908 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 11:54:03.533718 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.528599 4908 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 11:54:03.533718 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.528602 4908 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 11:54:03.533718 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.528604 4908 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 11:54:03.533718 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.528606 4908 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 11:54:03.533718 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.528609 4908 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 11:54:03.534527 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.528611 4908 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 11:54:03.534527 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.528614 4908 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 11:54:03.534527 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.528616 4908 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 11:54:03.534527 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.528619 4908 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 11:54:03.534527 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.528622 4908 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 11:54:03.534527 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.528624 4908 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 11:54:03.534527 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.528627 4908 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 11:54:03.534527 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.528630 4908 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 11:54:03.534527 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.528633 4908 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 11:54:03.534527 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.528635 4908 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 11:54:03.534527 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.528637 4908 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 11:54:03.534527 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.528640 4908 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 11:54:03.534527 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.528643 4908 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 11:54:03.534527 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.528645 4908 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 11:54:03.534527 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.528648 4908 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 11:54:03.534527 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.528650 4908 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 11:54:03.534527 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.528654 4908 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 11:54:03.534527 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.528656 4908 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 11:54:03.534527 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.528659 4908 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 11:54:03.534527 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.528662 4908 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 11:54:03.535222 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.528665 4908 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 11:54:03.535222 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.528667 4908 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 11:54:03.535222 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.528670 4908 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 11:54:03.535222 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.528672 4908 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 11:54:03.535222 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.528758 4908 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 11:54:03.535222 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.528765 4908 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 11:54:03.535222 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.528772 4908 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 11:54:03.535222 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.528778 4908 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 11:54:03.535222 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.528784 4908 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 11:54:03.535222 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.528789 4908 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 11:54:03.535222 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.528795 4908 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 11:54:03.535222 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.528802 4908 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 11:54:03.535222 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.528805 4908 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 11:54:03.535222 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.528808 4908 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 11:54:03.535222 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.528812 4908 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 11:54:03.535222 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.528815 4908 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 11:54:03.535222 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.528818 4908 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 11:54:03.535222 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.528822 4908 flags.go:64] FLAG: --cgroup-root="" Oct 30 11:54:03.535222 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.528836 4908 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 11:54:03.535222 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.528842 4908 flags.go:64] FLAG: --client-ca-file="" Oct 30 11:54:03.535222 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.528845 4908 flags.go:64] FLAG: --cloud-provider="external" Oct 30 11:54:03.535222 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.528848 4908 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 11:54:03.535222 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.528852 4908 flags.go:64] FLAG: --cluster-domain="" Oct 30 11:54:03.535813 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.528855 4908 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 11:54:03.535813 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.528858 4908 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 11:54:03.535813 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.528861 4908 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 11:54:03.535813 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.528864 4908 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 11:54:03.535813 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.528868 4908 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 11:54:03.535813 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.528871 4908 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 11:54:03.535813 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.528874 4908 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 11:54:03.535813 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.528877 4908 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 11:54:03.535813 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.528880 4908 flags.go:64] FLAG: --contention-profiling="false" Oct 30 11:54:03.535813 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.528885 4908 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 11:54:03.535813 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.528888 4908 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 11:54:03.535813 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.528892 4908 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 11:54:03.535813 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.528895 4908 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 11:54:03.535813 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.528899 4908 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 11:54:03.535813 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.528902 4908 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 11:54:03.535813 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.528905 4908 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 11:54:03.535813 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.528908 4908 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 11:54:03.535813 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.528911 4908 flags.go:64] FLAG: --enable-server="true" Oct 30 11:54:03.535813 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.528913 4908 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 11:54:03.535813 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.528917 4908 flags.go:64] FLAG: --event-burst="100" Oct 30 11:54:03.535813 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.528920 4908 flags.go:64] FLAG: --event-qps="50" Oct 30 11:54:03.535813 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.528924 4908 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 11:54:03.535813 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.528927 4908 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 11:54:03.535813 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.528930 4908 flags.go:64] FLAG: --eviction-hard="" Oct 30 11:54:03.536472 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.528934 4908 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 11:54:03.536472 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.528936 4908 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 11:54:03.536472 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.528939 4908 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 11:54:03.536472 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.528942 4908 flags.go:64] FLAG: --eviction-soft="" Oct 30 11:54:03.536472 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.528947 4908 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 11:54:03.536472 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.528949 4908 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 11:54:03.536472 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.528953 4908 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 11:54:03.536472 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.528955 4908 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 11:54:03.536472 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.528958 4908 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 11:54:03.536472 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.528961 4908 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 11:54:03.536472 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.528964 4908 flags.go:64] FLAG: --feature-gates="" Oct 30 11:54:03.536472 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.528968 4908 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 11:54:03.536472 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.528971 4908 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 11:54:03.536472 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.528974 4908 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 11:54:03.536472 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.528977 4908 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 11:54:03.536472 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.528980 4908 flags.go:64] FLAG: --healthz-port="10248" Oct 30 11:54:03.536472 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.528983 4908 flags.go:64] FLAG: --help="false" Oct 30 11:54:03.536472 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.528988 4908 flags.go:64] FLAG: --hostname-override="ip-10-0-141-251.ec2.internal" Oct 30 11:54:03.536472 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.528991 4908 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 11:54:03.536472 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.528994 4908 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 11:54:03.536472 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.528997 4908 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 11:54:03.536472 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.529000 4908 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 11:54:03.536472 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.529004 4908 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 11:54:03.537051 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.529007 4908 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 11:54:03.537051 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.529010 4908 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 11:54:03.537051 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.529030 4908 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 11:54:03.537051 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.529034 4908 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 11:54:03.537051 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.529037 4908 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 11:54:03.537051 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.529041 4908 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 11:54:03.537051 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.529044 4908 flags.go:64] FLAG: --kube-reserved="" Oct 30 11:54:03.537051 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.529047 4908 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 11:54:03.537051 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.529050 4908 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 11:54:03.537051 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.529053 4908 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 11:54:03.537051 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.529056 4908 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 11:54:03.537051 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.529059 4908 flags.go:64] FLAG: --lock-file="" Oct 30 11:54:03.537051 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.529061 4908 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 11:54:03.537051 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.529066 4908 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 11:54:03.537051 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.529069 4908 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 11:54:03.537051 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.529074 4908 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 11:54:03.537051 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.529077 4908 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 11:54:03.537051 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.529080 4908 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 11:54:03.537051 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.529083 4908 flags.go:64] FLAG: --logging-format="text" Oct 30 11:54:03.537051 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.529085 4908 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 11:54:03.537051 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.529089 4908 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 11:54:03.537051 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.529092 4908 flags.go:64] FLAG: --manifest-url="" Oct 30 11:54:03.537051 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.529094 4908 flags.go:64] FLAG: --manifest-url-header="" Oct 30 11:54:03.537051 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.529099 4908 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 11:54:03.537051 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.529102 4908 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 11:54:03.537639 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.529112 4908 flags.go:64] FLAG: --max-pods="110" Oct 30 11:54:03.537639 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.529117 4908 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 11:54:03.537639 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.529120 4908 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 11:54:03.537639 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.529122 4908 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 11:54:03.537639 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.529125 4908 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 11:54:03.537639 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.529129 4908 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 11:54:03.537639 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.529131 4908 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 11:54:03.537639 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.529134 4908 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 11:54:03.537639 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.529145 4908 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 11:54:03.537639 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.529149 4908 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 11:54:03.537639 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.529152 4908 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 11:54:03.537639 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.529155 4908 flags.go:64] FLAG: --pod-cidr="" Oct 30 11:54:03.537639 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.529158 4908 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 11:54:03.537639 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.529160 4908 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 11:54:03.537639 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.529163 4908 flags.go:64] FLAG: --pods-per-core="0" Oct 30 11:54:03.537639 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.529166 4908 flags.go:64] FLAG: --port="10250" Oct 30 11:54:03.537639 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.529169 4908 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 11:54:03.537639 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.529172 4908 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-01af67c8e3389319a" Oct 30 11:54:03.537639 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.529176 4908 flags.go:64] FLAG: --qos-reserved="" Oct 30 11:54:03.537639 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.529178 4908 flags.go:64] FLAG: --read-only-port="10255" Oct 30 11:54:03.537639 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.529182 4908 flags.go:64] FLAG: --register-node="true" Oct 30 11:54:03.537639 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.529185 4908 flags.go:64] FLAG: --register-with-taints="" Oct 30 11:54:03.537639 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.529189 4908 flags.go:64] FLAG: --registry-burst="10" Oct 30 11:54:03.537639 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.529191 4908 flags.go:64] FLAG: --registry-qps="5" Oct 30 11:54:03.537639 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.529194 4908 flags.go:64] FLAG: --reserved-cpus="" Oct 30 11:54:03.537639 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.529197 4908 flags.go:64] FLAG: --reserved-memory="" Oct 30 11:54:03.538354 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.529201 4908 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 11:54:03.538354 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.529204 4908 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 11:54:03.538354 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.529207 4908 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 11:54:03.538354 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.529210 4908 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 11:54:03.538354 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.529213 4908 flags.go:64] FLAG: --runonce="false" Oct 30 11:54:03.538354 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.529216 4908 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 11:54:03.538354 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.529220 4908 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 11:54:03.538354 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.529224 4908 flags.go:64] FLAG: --seccomp-default="false" Oct 30 11:54:03.538354 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.529227 4908 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 11:54:03.538354 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.529230 4908 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 11:54:03.538354 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.529233 4908 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 11:54:03.538354 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.529236 4908 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 11:54:03.538354 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.529239 4908 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 11:54:03.538354 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.529242 4908 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 11:54:03.538354 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.529245 4908 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 11:54:03.538354 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.529248 4908 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 11:54:03.538354 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.529251 4908 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 11:54:03.538354 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.529254 4908 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 11:54:03.538354 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.529256 4908 flags.go:64] FLAG: --system-cgroups="" Oct 30 11:54:03.538354 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.529259 4908 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 11:54:03.538354 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.529264 4908 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 11:54:03.538354 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.529267 4908 flags.go:64] FLAG: --tls-cert-file="" Oct 30 11:54:03.538354 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.529269 4908 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 11:54:03.538354 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.529273 4908 flags.go:64] FLAG: --tls-min-version="" Oct 30 11:54:03.538354 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.529276 4908 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 11:54:03.539042 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.529278 4908 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 11:54:03.539042 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.529282 4908 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 11:54:03.539042 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.529284 4908 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 11:54:03.539042 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.529287 4908 flags.go:64] FLAG: --v="2" Oct 30 11:54:03.539042 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.529292 4908 flags.go:64] FLAG: --version="false" Oct 30 11:54:03.539042 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.529296 4908 flags.go:64] FLAG: --vmodule="" Oct 30 11:54:03.539042 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.529299 4908 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 11:54:03.539042 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.529302 4908 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 11:54:03.539042 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529425 4908 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 11:54:03.539042 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529428 4908 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 11:54:03.539042 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529430 4908 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 11:54:03.539042 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529433 4908 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 11:54:03.539042 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529436 4908 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 11:54:03.539042 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529439 4908 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 11:54:03.539042 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529443 4908 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 11:54:03.539042 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529447 4908 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 11:54:03.539042 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529450 4908 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 11:54:03.539042 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529453 4908 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 11:54:03.539042 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529456 4908 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 11:54:03.539042 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529458 4908 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 11:54:03.539042 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529461 4908 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 11:54:03.539603 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529463 4908 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 11:54:03.539603 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529466 4908 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 11:54:03.539603 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529468 4908 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 11:54:03.539603 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529471 4908 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 11:54:03.539603 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529474 4908 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 11:54:03.539603 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529476 4908 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 11:54:03.539603 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529479 4908 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 11:54:03.539603 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529481 4908 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 11:54:03.539603 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529484 4908 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 11:54:03.539603 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529487 4908 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 11:54:03.539603 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529490 4908 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 11:54:03.539603 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529492 4908 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 11:54:03.539603 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529495 4908 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 11:54:03.539603 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529497 4908 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 11:54:03.539603 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529500 4908 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 11:54:03.539603 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529502 4908 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 11:54:03.539603 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529505 4908 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 11:54:03.539603 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529508 4908 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 11:54:03.539603 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529510 4908 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 11:54:03.540133 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529513 4908 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 11:54:03.540133 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529515 4908 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 11:54:03.540133 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529517 4908 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 11:54:03.540133 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529520 4908 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 11:54:03.540133 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529523 4908 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 11:54:03.540133 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529525 4908 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 11:54:03.540133 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529531 4908 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 11:54:03.540133 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529533 4908 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 11:54:03.540133 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529536 4908 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 11:54:03.540133 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529539 4908 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 11:54:03.540133 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529542 4908 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 11:54:03.540133 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529544 4908 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 11:54:03.540133 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529547 4908 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 11:54:03.540133 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529550 4908 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 11:54:03.540133 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529552 4908 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 11:54:03.540133 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529555 4908 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 11:54:03.540133 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529558 4908 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 11:54:03.540133 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529560 4908 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 11:54:03.540133 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529563 4908 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 11:54:03.540133 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529565 4908 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 11:54:03.540856 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529568 4908 feature_gate.go:386] unrecognized feature gate: Example Oct 30 11:54:03.540856 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529570 4908 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 11:54:03.540856 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529573 4908 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 11:54:03.540856 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529576 4908 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 11:54:03.540856 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529578 4908 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 11:54:03.540856 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529581 4908 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 11:54:03.540856 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529583 4908 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 11:54:03.540856 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529586 4908 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 11:54:03.540856 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529588 4908 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 11:54:03.540856 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529591 4908 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 11:54:03.540856 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529593 4908 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 11:54:03.540856 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529596 4908 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 11:54:03.540856 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529598 4908 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 11:54:03.540856 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529600 4908 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 11:54:03.540856 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529603 4908 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 11:54:03.540856 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529605 4908 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 11:54:03.540856 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529608 4908 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 11:54:03.540856 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529610 4908 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 11:54:03.540856 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529614 4908 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 11:54:03.540856 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529617 4908 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 11:54:03.541386 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529620 4908 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 11:54:03.541386 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529623 4908 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 11:54:03.541386 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529626 4908 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 11:54:03.541386 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529628 4908 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 11:54:03.541386 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529630 4908 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 11:54:03.541386 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529633 4908 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 11:54:03.541386 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529636 4908 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 11:54:03.541386 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529640 4908 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 11:54:03.541386 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529644 4908 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 11:54:03.541386 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529646 4908 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 11:54:03.541386 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529650 4908 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 11:54:03.541386 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529653 4908 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 11:54:03.541386 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529656 4908 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 11:54:03.541386 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529658 4908 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 11:54:03.541386 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529661 4908 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 11:54:03.541386 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529664 4908 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 11:54:03.541386 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529666 4908 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 11:54:03.541386 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529669 4908 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 11:54:03.541816 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529671 4908 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 11:54:03.541816 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529673 4908 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 11:54:03.541816 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529676 4908 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 11:54:03.541816 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529678 4908 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 11:54:03.541816 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529681 4908 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 11:54:03.541816 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529683 4908 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 11:54:03.541816 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529686 4908 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 11:54:03.541816 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529688 4908 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 11:54:03.541816 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529691 4908 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 11:54:03.541816 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529693 4908 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 11:54:03.541816 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529696 4908 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 11:54:03.541816 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529698 4908 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 11:54:03.541816 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529702 4908 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 11:54:03.541816 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529706 4908 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 11:54:03.541816 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529709 4908 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 11:54:03.541816 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529711 4908 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 11:54:03.542267 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.529737 4908 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 11:54:03.542267 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529770 4908 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 11:54:03.542267 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529774 4908 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 11:54:03.542267 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529777 4908 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 11:54:03.542267 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529780 4908 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 11:54:03.542267 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529783 4908 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 11:54:03.542267 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529785 4908 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 11:54:03.542267 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529788 4908 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 11:54:03.542267 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529791 4908 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 11:54:03.542267 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529793 4908 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 11:54:03.542267 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529796 4908 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 11:54:03.542267 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529798 4908 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 11:54:03.542267 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529801 4908 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 11:54:03.542267 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529803 4908 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 11:54:03.542267 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529807 4908 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 11:54:03.542778 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529811 4908 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 11:54:03.542778 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529815 4908 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 11:54:03.542778 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529818 4908 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 11:54:03.542778 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529821 4908 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 11:54:03.542778 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529835 4908 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 11:54:03.542778 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529841 4908 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 11:54:03.542778 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529845 4908 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 11:54:03.542778 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529850 4908 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 11:54:03.542778 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529854 4908 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 11:54:03.542778 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529858 4908 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 11:54:03.542778 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529861 4908 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 11:54:03.542778 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529864 4908 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 11:54:03.542778 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529866 4908 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 11:54:03.542778 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529869 4908 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 11:54:03.542778 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529874 4908 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 11:54:03.542778 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529877 4908 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 11:54:03.542778 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529879 4908 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 11:54:03.542778 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529882 4908 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 11:54:03.542778 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529885 4908 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 11:54:03.543459 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529889 4908 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 11:54:03.543459 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529891 4908 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 11:54:03.543459 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529894 4908 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 11:54:03.543459 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529896 4908 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 11:54:03.543459 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529899 4908 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 11:54:03.543459 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529902 4908 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 11:54:03.543459 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529904 4908 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 11:54:03.543459 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529907 4908 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 11:54:03.543459 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529909 4908 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 11:54:03.543459 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529912 4908 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 11:54:03.543459 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529914 4908 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 11:54:03.543459 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529917 4908 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 11:54:03.543459 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529919 4908 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 11:54:03.543459 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529922 4908 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 11:54:03.543459 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529927 4908 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 11:54:03.543459 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529930 4908 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 11:54:03.543459 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529933 4908 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 11:54:03.543459 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529935 4908 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 11:54:03.543459 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529938 4908 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 11:54:03.543945 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529941 4908 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 11:54:03.543945 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529943 4908 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 11:54:03.543945 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529945 4908 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 11:54:03.543945 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529948 4908 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 11:54:03.543945 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529950 4908 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 11:54:03.543945 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529953 4908 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 11:54:03.543945 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529955 4908 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 11:54:03.543945 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529958 4908 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 11:54:03.543945 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529963 4908 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 11:54:03.543945 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529966 4908 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 11:54:03.543945 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529968 4908 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 11:54:03.543945 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529971 4908 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 11:54:03.543945 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529973 4908 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 11:54:03.543945 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529976 4908 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 11:54:03.543945 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529979 4908 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 11:54:03.543945 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529981 4908 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 11:54:03.543945 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529984 4908 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 11:54:03.543945 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529987 4908 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 11:54:03.543945 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529989 4908 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 11:54:03.544411 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529991 4908 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 11:54:03.544411 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529994 4908 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 11:54:03.544411 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529997 4908 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 11:54:03.544411 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.529999 4908 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 11:54:03.544411 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.530002 4908 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 11:54:03.544411 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.530004 4908 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 11:54:03.544411 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.530007 4908 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 11:54:03.544411 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.530010 4908 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 11:54:03.544411 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.530012 4908 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 11:54:03.544411 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.530015 4908 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 11:54:03.544411 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.530017 4908 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 11:54:03.544411 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.530020 4908 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 11:54:03.544411 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.530022 4908 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 11:54:03.544411 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.530025 4908 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 11:54:03.544411 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.530027 4908 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 11:54:03.544411 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.530029 4908 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 11:54:03.544411 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.530032 4908 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 11:54:03.544411 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.530034 4908 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 11:54:03.544411 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.530037 4908 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 11:54:03.544411 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.530039 4908 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 11:54:03.544887 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.530042 4908 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 11:54:03.544887 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.530046 4908 feature_gate.go:386] unrecognized feature gate: Example Oct 30 11:54:03.544887 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.530049 4908 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 11:54:03.544887 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.530052 4908 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 11:54:03.544887 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.530055 4908 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 11:54:03.544887 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.530057 4908 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 11:54:03.544887 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.530060 4908 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 11:54:03.544887 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.530062 4908 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 11:54:03.544887 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.530065 4908 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 11:54:03.544887 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.530067 4908 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 11:54:03.544887 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.530070 4908 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 11:54:03.544887 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.530072 4908 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 11:54:03.544887 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.530075 4908 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 11:54:03.544887 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.530077 4908 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 11:54:03.544887 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.530080 4908 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 11:54:03.544887 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.535152 4908 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 11:54:03.544887 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.535169 4908 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 11:54:03.544887 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.535201 4908 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 11:54:03.544887 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.535207 4908 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 11:54:03.545359 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535265 4908 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 11:54:03.545359 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535270 4908 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 11:54:03.545359 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535274 4908 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 11:54:03.545359 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535277 4908 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 11:54:03.545359 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535280 4908 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 11:54:03.545359 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535283 4908 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 11:54:03.545359 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535286 4908 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 11:54:03.545359 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535289 4908 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 11:54:03.545359 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535292 4908 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 11:54:03.545359 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535294 4908 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 11:54:03.545359 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535298 4908 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 11:54:03.545359 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535300 4908 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 11:54:03.545359 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535303 4908 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 11:54:03.545359 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535307 4908 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 11:54:03.545359 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535311 4908 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 11:54:03.545359 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535314 4908 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 11:54:03.545359 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535318 4908 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 11:54:03.545359 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535321 4908 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 11:54:03.545359 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535323 4908 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 11:54:03.545807 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535326 4908 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 11:54:03.545807 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535329 4908 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 11:54:03.545807 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535331 4908 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 11:54:03.545807 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535334 4908 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 11:54:03.545807 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535336 4908 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 11:54:03.545807 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535340 4908 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 11:54:03.545807 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535343 4908 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 11:54:03.545807 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535345 4908 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 11:54:03.545807 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535348 4908 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 11:54:03.545807 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535351 4908 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 11:54:03.545807 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535354 4908 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 11:54:03.545807 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535357 4908 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 11:54:03.545807 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535360 4908 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 11:54:03.545807 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535362 4908 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 11:54:03.545807 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535365 4908 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 11:54:03.545807 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535368 4908 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 11:54:03.545807 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535370 4908 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 11:54:03.545807 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535373 4908 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 11:54:03.545807 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535376 4908 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 11:54:03.545807 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535378 4908 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 11:54:03.546309 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535381 4908 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 11:54:03.546309 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535383 4908 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 11:54:03.546309 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535386 4908 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 11:54:03.546309 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535388 4908 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 11:54:03.546309 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535391 4908 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 11:54:03.546309 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535393 4908 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 11:54:03.546309 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535396 4908 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 11:54:03.546309 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535399 4908 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 11:54:03.546309 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535402 4908 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 11:54:03.546309 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535405 4908 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 11:54:03.546309 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535408 4908 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 11:54:03.546309 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535410 4908 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 11:54:03.546309 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535413 4908 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 11:54:03.546309 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535415 4908 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 11:54:03.546309 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535418 4908 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 11:54:03.546309 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535420 4908 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 11:54:03.546309 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535423 4908 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 11:54:03.546309 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535427 4908 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 11:54:03.546309 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535431 4908 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 11:54:03.546759 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535441 4908 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 11:54:03.546759 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535444 4908 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 11:54:03.546759 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535447 4908 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 11:54:03.546759 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535450 4908 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 11:54:03.546759 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535452 4908 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 11:54:03.546759 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535455 4908 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 11:54:03.546759 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535458 4908 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 11:54:03.546759 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535460 4908 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 11:54:03.546759 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535463 4908 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 11:54:03.546759 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535466 4908 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 11:54:03.546759 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535468 4908 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 11:54:03.546759 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535471 4908 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 11:54:03.546759 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535473 4908 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 11:54:03.546759 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535476 4908 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 11:54:03.546759 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535479 4908 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 11:54:03.546759 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535481 4908 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 11:54:03.546759 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535484 4908 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 11:54:03.546759 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535486 4908 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 11:54:03.546759 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535489 4908 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 11:54:03.547228 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535492 4908 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 11:54:03.547228 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535494 4908 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 11:54:03.547228 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535497 4908 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 11:54:03.547228 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535500 4908 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 11:54:03.547228 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535503 4908 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 11:54:03.547228 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535506 4908 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 11:54:03.547228 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535509 4908 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 11:54:03.547228 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535511 4908 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 11:54:03.547228 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535514 4908 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 11:54:03.547228 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535517 4908 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 11:54:03.547228 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535519 4908 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 11:54:03.547228 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535522 4908 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 11:54:03.547228 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535525 4908 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 11:54:03.547228 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535527 4908 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 11:54:03.547228 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535530 4908 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 11:54:03.547228 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535532 4908 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 11:54:03.547228 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535535 4908 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 11:54:03.547228 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535537 4908 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 11:54:03.547228 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535540 4908 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 11:54:03.547228 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535542 4908 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 11:54:03.547703 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535545 4908 feature_gate.go:386] unrecognized feature gate: Example Oct 30 11:54:03.547703 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535547 4908 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 11:54:03.547703 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535550 4908 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 11:54:03.547703 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535552 4908 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 11:54:03.547703 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535555 4908 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 11:54:03.547703 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535558 4908 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 11:54:03.547703 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535560 4908 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 11:54:03.547703 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535563 4908 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 11:54:03.547703 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535565 4908 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 11:54:03.547703 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.535594 4908 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 11:54:03.547703 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535763 4908 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 11:54:03.547703 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535768 4908 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 11:54:03.547703 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535771 4908 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 11:54:03.547703 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535774 4908 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 11:54:03.547703 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535778 4908 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 11:54:03.547703 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535781 4908 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 11:54:03.548136 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535784 4908 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 11:54:03.548136 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535787 4908 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 11:54:03.548136 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535790 4908 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 11:54:03.548136 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535793 4908 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 11:54:03.548136 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535796 4908 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 11:54:03.548136 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535798 4908 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 11:54:03.548136 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535801 4908 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 11:54:03.548136 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535803 4908 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 11:54:03.548136 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535806 4908 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 11:54:03.548136 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535808 4908 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 11:54:03.548136 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535811 4908 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 11:54:03.548136 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535813 4908 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 11:54:03.548136 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535817 4908 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 11:54:03.548136 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535821 4908 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 11:54:03.548136 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535844 4908 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 11:54:03.548136 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535849 4908 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 11:54:03.548136 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535852 4908 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 11:54:03.548136 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535855 4908 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 11:54:03.548136 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535858 4908 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 11:54:03.548584 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535860 4908 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 11:54:03.548584 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535863 4908 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 11:54:03.548584 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535866 4908 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 11:54:03.548584 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535869 4908 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 11:54:03.548584 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535871 4908 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 11:54:03.548584 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535874 4908 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 11:54:03.548584 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535876 4908 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 11:54:03.548584 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535879 4908 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 11:54:03.548584 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535881 4908 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 11:54:03.548584 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535884 4908 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 11:54:03.548584 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535886 4908 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 11:54:03.548584 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535889 4908 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 11:54:03.548584 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535891 4908 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 11:54:03.548584 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535895 4908 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 11:54:03.548584 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535898 4908 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 11:54:03.548584 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535900 4908 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 11:54:03.548584 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535903 4908 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 11:54:03.548584 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535905 4908 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 11:54:03.548584 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535908 4908 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 11:54:03.549045 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535910 4908 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 11:54:03.549045 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535913 4908 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 11:54:03.549045 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535916 4908 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 11:54:03.549045 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535918 4908 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 11:54:03.549045 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535921 4908 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 11:54:03.549045 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535923 4908 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 11:54:03.549045 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535926 4908 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 11:54:03.549045 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535928 4908 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 11:54:03.549045 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535931 4908 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 11:54:03.549045 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535933 4908 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 11:54:03.549045 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535935 4908 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 11:54:03.549045 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535938 4908 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 11:54:03.549045 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535941 4908 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 11:54:03.549045 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535943 4908 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 11:54:03.549045 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535945 4908 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 11:54:03.549045 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535948 4908 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 11:54:03.549045 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535951 4908 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 11:54:03.549045 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535953 4908 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 11:54:03.549045 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535955 4908 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 11:54:03.549045 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535958 4908 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 11:54:03.549653 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535960 4908 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 11:54:03.549653 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535963 4908 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 11:54:03.549653 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535965 4908 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 11:54:03.549653 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535968 4908 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 11:54:03.549653 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535970 4908 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 11:54:03.549653 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535973 4908 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 11:54:03.549653 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535975 4908 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 11:54:03.549653 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535980 4908 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 11:54:03.549653 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535983 4908 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 11:54:03.549653 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535986 4908 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 11:54:03.549653 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535989 4908 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 11:54:03.549653 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535992 4908 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 11:54:03.549653 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535994 4908 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 11:54:03.549653 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.535997 4908 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 11:54:03.549653 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.536000 4908 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 11:54:03.549653 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.536002 4908 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 11:54:03.549653 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.536005 4908 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 11:54:03.549653 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.536007 4908 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 11:54:03.549653 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.536010 4908 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 11:54:03.550462 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.536012 4908 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 11:54:03.550462 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.536015 4908 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 11:54:03.550462 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.536017 4908 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 11:54:03.550462 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.536020 4908 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 11:54:03.550462 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.536022 4908 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 11:54:03.550462 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.536025 4908 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 11:54:03.550462 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.536027 4908 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 11:54:03.550462 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.536029 4908 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 11:54:03.550462 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.536032 4908 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 11:54:03.550462 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.536034 4908 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 11:54:03.550462 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.536037 4908 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 11:54:03.550462 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.536039 4908 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 11:54:03.550462 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.536041 4908 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 11:54:03.550462 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.536044 4908 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 11:54:03.550462 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.536046 4908 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 11:54:03.550462 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.536049 4908 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 11:54:03.550462 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.536052 4908 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 11:54:03.550462 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.536054 4908 feature_gate.go:386] unrecognized feature gate: Example Oct 30 11:54:03.550462 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.536056 4908 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 11:54:03.550462 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.536059 4908 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 11:54:03.550981 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.536062 4908 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 11:54:03.550981 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.536065 4908 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 11:54:03.550981 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.536067 4908 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 11:54:03.550981 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.536090 4908 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 11:54:03.550981 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.536122 4908 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 11:54:03.550981 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.536126 4908 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 11:54:03.550981 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.536128 4908 feature_gate.go:386] unrecognized feature gate: Example Oct 30 11:54:03.550981 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.536131 4908 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 11:54:03.550981 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.536134 4908 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 11:54:03.550981 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.536136 4908 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 11:54:03.550981 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.536139 4908 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 11:54:03.550981 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.536141 4908 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 11:54:03.550981 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.536144 4908 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 11:54:03.550981 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.536146 4908 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 11:54:03.550981 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.536149 4908 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 11:54:03.550981 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.536151 4908 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 11:54:03.551358 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.536155 4908 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 11:54:03.551358 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.536157 4908 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 11:54:03.551358 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.536160 4908 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 11:54:03.551358 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.536162 4908 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 11:54:03.551358 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.536165 4908 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 11:54:03.551358 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.536168 4908 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 11:54:03.551358 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.536170 4908 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 11:54:03.551358 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.536173 4908 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 11:54:03.551358 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.536175 4908 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 11:54:03.551358 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.536178 4908 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 11:54:03.551358 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.536180 4908 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 11:54:03.551358 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.536183 4908 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 11:54:03.551358 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.536186 4908 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 11:54:03.551358 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.536189 4908 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 11:54:03.551358 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.536193 4908 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 11:54:03.551358 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.536197 4908 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 11:54:03.551358 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.536200 4908 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 11:54:03.551358 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.536203 4908 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 11:54:03.551358 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.536207 4908 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 11:54:03.551358 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.536210 4908 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 11:54:03.551932 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.536212 4908 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 11:54:03.551932 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.536215 4908 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 11:54:03.551932 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.536217 4908 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 11:54:03.551932 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.536220 4908 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 11:54:03.551932 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.536222 4908 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 11:54:03.551932 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.536225 4908 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 11:54:03.551932 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.536227 4908 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 11:54:03.551932 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.536230 4908 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 11:54:03.551932 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.536232 4908 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 11:54:03.551932 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.536234 4908 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 11:54:03.551932 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.536237 4908 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 11:54:03.551932 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.536239 4908 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 11:54:03.551932 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.536242 4908 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 11:54:03.551932 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.536244 4908 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 11:54:03.551932 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.536247 4908 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 11:54:03.551932 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.536249 4908 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 11:54:03.551932 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.536252 4908 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 11:54:03.551932 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.536254 4908 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 11:54:03.551932 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.536257 4908 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 11:54:03.551932 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.536259 4908 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 11:54:03.552441 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.536261 4908 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 11:54:03.552441 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.536265 4908 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 11:54:03.552441 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.536268 4908 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 11:54:03.552441 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.536270 4908 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 11:54:03.552441 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.536273 4908 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 11:54:03.552441 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.536281 4908 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 11:54:03.552441 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.536283 4908 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 11:54:03.552441 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.536286 4908 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 11:54:03.552441 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.536288 4908 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 11:54:03.552441 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.536291 4908 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 11:54:03.552441 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.536294 4908 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 11:54:03.552441 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.536297 4908 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 11:54:03.552441 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.536300 4908 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 11:54:03.552441 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.536302 4908 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 11:54:03.552441 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.536305 4908 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 11:54:03.552441 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.536307 4908 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 11:54:03.552441 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.536310 4908 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 11:54:03.552441 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.536312 4908 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 11:54:03.552441 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.536315 4908 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 11:54:03.552441 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.536317 4908 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 11:54:03.552929 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.536319 4908 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 11:54:03.552929 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.536322 4908 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 11:54:03.552929 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.536324 4908 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 11:54:03.552929 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.536327 4908 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 11:54:03.552929 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.536329 4908 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 11:54:03.552929 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.536332 4908 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 11:54:03.552929 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.536334 4908 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 11:54:03.552929 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.536338 4908 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 11:54:03.552929 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.536342 4908 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 11:54:03.552929 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.536345 4908 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 11:54:03.552929 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.536347 4908 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 11:54:03.552929 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.536350 4908 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 11:54:03.552929 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.536352 4908 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 11:54:03.552929 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.536355 4908 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 11:54:03.552929 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.536358 4908 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 11:54:03.552929 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.536361 4908 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 11:54:03.552929 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.536363 4908 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 11:54:03.552929 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.536367 4908 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 11:54:03.552929 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.536370 4908 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 11:54:03.553414 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.536373 4908 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 11:54:03.553414 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.536375 4908 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 11:54:03.553414 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.536378 4908 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 11:54:03.553414 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.536380 4908 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 11:54:03.553414 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.536383 4908 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 11:54:03.553414 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.536386 4908 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 11:54:03.553414 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.536388 4908 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 11:54:03.553414 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.536391 4908 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 11:54:03.553414 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.536393 4908 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 11:54:03.553414 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.536396 4908 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 11:54:03.553414 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.536398 4908 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 11:54:03.553414 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.536400 4908 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 11:54:03.553414 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.536403 4908 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 11:54:03.553414 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.536405 4908 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 11:54:03.553414 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:03.536408 4908 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 11:54:03.553414 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.536464 4908 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 11:54:03.553414 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.537426 4908 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 30 11:54:03.553414 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.537478 4908 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 30 11:54:03.553414 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.537649 4908 server.go:1014] "Starting client certificate rotation" Oct 30 11:54:03.553878 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.537792 4908 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 11:54:03.553878 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.537865 4908 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 11:48:23 +0000 UTC" deadline="2028-03-30 16:56:01.322221314 +0000 UTC" Oct 30 11:54:03.553878 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.537913 4908 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12413h1m57.784311651s" Oct 30 11:54:03.553878 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.538110 4908 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 11:54:03.553878 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.539233 4908 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 11:54:03.553878 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.540459 4908 log.go:25] "Validated CRI v1 runtime API" Oct 30 11:54:03.553878 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.542104 4908 log.go:25] "Validated CRI v1 image API" Oct 30 11:54:03.553878 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.542869 4908 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 11:54:03.553878 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.544705 4908 fs.go:135] Filesystem UUIDs: map[19a6a40c-0284-4256-b5ae-cc8b2af1b706:/dev/nvme0n1p3 1ae0e2cc-2f11-4a0f-8b2c-453b430f35c7:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2] Oct 30 11:54:03.554115 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.544725 4908 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0e385d1aea1378254689b60f24673fa5cb718887b518e18d8f68b64224aa2496/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0e385d1aea1378254689b60f24673fa5cb718887b518e18d8f68b64224aa2496/userdata/shm major:0 minor:242 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0ed7babdc2068ee24f20699aa63685bab422bb05562cc3f8e79c91aeb1568a52/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0ed7babdc2068ee24f20699aa63685bab422bb05562cc3f8e79c91aeb1568a52/userdata/shm major:0 minor:230 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/18e782439c70c580ec71ddf416df5349c870d492753c67e14b13bb86f2d87533/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/18e782439c70c580ec71ddf416df5349c870d492753c67e14b13bb86f2d87533/userdata/shm major:0 minor:294 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1b6e44060e4775c8234fb5f6b195c11bc0f4beb0b731852b60d9d07f2b70e864/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1b6e44060e4775c8234fb5f6b195c11bc0f4beb0b731852b60d9d07f2b70e864/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1f2563b5c5e8e08274b16bd9bbbccba45366b0aed1b60df792c35b2ac3e13f6f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1f2563b5c5e8e08274b16bd9bbbccba45366b0aed1b60df792c35b2ac3e13f6f/userdata/shm major:0 minor:95 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1f5ac21f834c419c1d2b1b6d14292b5bd03d2741030a178f2a45cdb3262508dd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1f5ac21f834c419c1d2b1b6d14292b5bd03d2741030a178f2a45cdb3262508dd/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2c9d279e75511d42a300ea507ae46c045f2f6d7354e80d750835817d62d998b7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2c9d279e75511d42a300ea507ae46c045f2f6d7354e80d750835817d62d998b7/userdata/shm major:0 minor:236 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/36525167111a09951040a77f0c5cf71541a5df0fc8e7bc52add5f57bd46d3612/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/36525167111a09951040a77f0c5cf71541a5df0fc8e7bc52add5f57bd46d3612/userdata/shm major:0 minor:238 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/37fb31a12739fbc2204e29a20a6a16c2a38ecb9fc54c663aaa811e6d264a8655/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/37fb31a12739fbc2204e29a20a6a16c2a38ecb9fc54c663aaa811e6d264a8655/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/419f42abae50a8c097a816062e2e78976ae8ec37f34318f0c1aa15703d4eac60/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/419f42abae50a8c097a816062e2e78976ae8ec37f34318f0c1aa15703d4eac60/userdata/shm major:0 minor:228 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4b0f6db58147695aa013d92f1e9abca35d686a1268c30c8b5ffa706aec6e51eb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4b0f6db58147695aa013d92f1e9abca35d686a1268c30c8b5ffa706aec6e51eb/userdata/shm major:0 minor:246 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4fe7f906d45dc1117fc963f19feb9094a9508021f874ec361136e810911456ae/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4fe7f906d45dc1117fc963f19feb9094a9508021f874ec361136e810911456ae/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/50729bdfcb9b631e0bcdd1ee0528e7c3c9dacbac0f5276aa7f47aaca3c6bc046/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/50729bdfcb9b631e0bcdd1ee0528e7c3c9dacbac0f5276aa7f47aaca3c6bc046/userdata/shm major:0 minor:315 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/50aeeaf84669fde1fb553f85753f6f537614a6a045412ab3cf5635b37f049ea8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/50aeeaf84669fde1fb553f85753f6f537614a6a045412ab3cf5635b37f049ea8/userdata/shm major:0 minor:244 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/59286100c47dfffc745aeaf1101d1b002609f107813dd3febe7c1a355fbe11db/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/59286100c47dfffc745aeaf1101d1b002609f107813dd3febe7c1a355fbe11db/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7e3210ba71b33dbf2744cdcf9f8c0af3b880fefb6715945363dfe73c1c6a67e6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7e3210ba71b33dbf2744cdcf9f8c0af3b880fefb6715945363dfe73c1c6a67e6/userdata/shm major:0 minor:226 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/81d2c98212fb5e239a5750b14b2032945223d71a039bfef6fd50039af7d88632/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/81d2c98212fb5e239a5750b14b2032945223d71a039bfef6fd50039af7d88632/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a210218da33eda8d8c8333214533a17c60812fe01e8061c5d1a295774b5006b0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a210218da33eda8d8c8333214533a17c60812fe01e8061c5d1a295774b5006b0/userdata/shm major:0 minor:296 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ac1ca6630282423463ddd56cf4dd5bd7e452ffaef60997a055a518a3d193ae1f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ac1ca6630282423463ddd56cf4dd5bd7e452ffaef60997a055a518a3d193ae1f/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bf1bbee327514fab5fac9eb46554e91c28ce5a6387743ae6b1c7d24ba7989f4f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bf1bbee327514fab5fac9eb46554e91c28ce5a6387743ae6b1c7d24ba7989f4f/userdata/shm major:0 minor:233 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c7e539754dfdda2f33506f8d6a7dc8acd027ef96d8d03b975eff48d2c6fc59bd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c7e539754dfdda2f33506f8d6a7dc8acd027ef96d8d03b975eff48d2c6fc59bd/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e03ff409ec2f82625f5cfe631fec11e889aa058ba91dbd505bf4ea6d6b31fec9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e03ff409ec2f82625f5cfe631fec11e889aa058ba91dbd505bf4ea6d6b31fec9/userdata/shm major:0 minor:240 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e1e676a6d3c93a0071e009327272d842160f1207839b7a25a676a8f5e4d83d25/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e1e676a6d3c93a0071e009327272d842160f1207839b7a25a676a8f5e4d83d25/userdata/shm major:0 minor:74 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/003c3d18-23f1-45d0-bac4-8c6eea4b7e99/volumes/kubernetes.io~projected/kube-api-access-4jmcf:{mountpoint:/var/lib/kubelet/pods/003c3d18-23f1-45d0-bac4-8c6eea4b7e99/volumes/kubernetes.io~projected/kube-api-access-4jmcf major:0 minor:234 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/003c3d18-23f1-45d0-bac4-8c6eea4b7e99/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/003c3d18-23f1-45d0-bac4-8c6eea4b7e99/volumes/kubernetes.io~secret/cert major:0 minor:231 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0df1a118-36a5-4c57-8cec-2c8236f1c4de/volumes/kubernetes.io~projected/kube-api-access-r4tdc:{mountpoint:/var/lib/kubelet/pods/0df1a118-36a5-4c57-8cec-2c8236f1c4de/volumes/kubernetes.io~projected/kube-api-access-r4tdc major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1b735438-5947-4374-bd4c-66ac3a299770/volumes/kubernetes.io~projected/kube-api-access-gzc5h:{mountpoint:/var/lib/kubelet/pods/1b735438-5947-4374-bd4c-66ac3a299770/volumes/kubernetes.io~projected/kube-api-access-gzc5h major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1d7f8396-e57c-45e5-ae74-a2244db10364/volumes/kubernetes.io~projected/kube-api-access-bzwcn:{mountpoint:/var/lib/kubelet/pods/1d7f8396-e57c-45e5-ae74-a2244db10364/volumes/kubernetes.io~projected/kube-api-access-bzwcn major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1f5667a1-a773-4520-b2fb-c69148df632c/volumes/kubernetes.io~projected/kube-api-access-bzld2:{mountpoint:/var/lib/kubelet/pods/1f5667a1-a773-4520-b2fb-c69148df632c/volumes/kubernetes.io~projected/kube-api-access-bzld2 major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/221ced85-cc15-4e07-82d9-a1a1f3e958a5/volumes/kubernetes.io~projected/kube-api-access-bhxmm:{mountpoint:/var/lib/kubelet/pods/221ced85-cc15-4e07-82d9-a1a1f3e958a5/volumes/kubernetes.io~projected/kube-api-access-bhxmm major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3699031f-ee9e-46cc-b37e-4dfc76b9c0b1/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/3699031f-ee9e-46cc-b37e-4dfc76b9c0b1/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3699031f-ee9e-46cc-b37e-4dfc76b9c0b1/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/3699031f-ee9e-46cc-b37e-4dfc76b9c0b1/volumes/kubernetes.io~empty-dir/tmp major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3699031f-ee9e-46cc-b37e-4dfc76b9c0b1/volumes/kubernetes.io~projected/kube-api-access-r7vnf:{mountpoint:/var/lib/kubelet/pods/3699031f-ee9e-46cc-b37e-4dfc76b9c0b1/volumes/kubernetes.io~projected/kube-api-access-r7vnf major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4e18f771-273b-49e6-872c-6cbe47ab68c4/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/4e18f771-273b-49e6-872c-6cbe47ab68c4/volumes/kubernetes.io~secret/agent-certs major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5aee536a-8942-4b48-9b2c-3e36089f3ac5/volumes/kubernetes.io~projected/kube-api-access-vmx84:{mountpoint:/var/lib/kubelet/pods/5aee536a-8942-4b48-9b2c-3e36089f3ac5/volumes/kubernetes.io~projected/kube-api-access-vmx84 major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/644e19b3-10ad-4145-9ed3-595d19ed9b07/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/644e19b3-10ad-4145-9ed3-595d19ed9b07/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/644e19b3-10ad-4145-9ed3-595d19ed9b07/volumes/kubernetes.io~projected/kube-api-access-8nzgq:{mountpoint:/var/lib/kubelet/pods/644e19b3-10ad-4145-9ed3-595d19ed9b07/volumes/kubernetes.io~projected/kube-api-access-8nzgq major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/644e19b3-10ad-4145-9ed3-595d19ed9b07/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/644e19b3-10ad-4145-9ed3-595d19ed9b07/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6acb13a1-cb2e-45e1-897b-3ef89e6d1b84/volumes/kubernetes.io~projected/kube-api-access-2rxc4:{mountpoint:/var/lib/kubelet/pods/6acb13a1-cb2e-45e1-897b-3ef89e6d1b84/volumes/kubernetes.io~projected/kube-api-access-2rxc4 major:0 minor:220 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6acb13a1-cb2e-45e1-897b-3ef89e6d1b84/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/6acb13a1-cb2e-45e1-897b-3ef89e6d1b84/volumes/kubernetes.io~secret/default-certificate major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6acb13a1-cb2e-45e1-897b-3ef89e6d1b84/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/6acb13a1-cb2e-45e1-897b-3ef89e6d1b84/volumes/kubernetes.io~secret/stats-auth major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6b350710-c48c-4c7e-bce9-e021099e98ca/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/6b350710-c48c-4c7e-bce9-e021099e98ca/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:288 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6d976900-ecd0-469c-a9e2-d60708361f1a/volumes/kubernetes.io~projected/kube-api-access-5fhb8:{mountpoint:/var/lib/kubelet/pods/6d976900-ecd0-469c-a9e2-d60708361f1a/volumes/kubernetes.io~projected/kube-api-access-5fhb8 major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7689cfe5-5a4e-4481-8eb9-107a58ca8bb3/volumes/kubernetes.io~projected/kube-api-access-gr67s:{mountpoint:/var/lib/kubelet/pods/7689cfe5-5a4e-4481-8eb9-107a58ca8bb3/volumes/kubernetes.io~projected/kube-api-access-gr67s major:0 minor:221 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7ac30952-da5d-42a3-a1aa-bd249bbf6adb/volumes/kubernetes.io~projected/kube-api-access-6hqrf:{mountpoint:/var/lib/kubelet/pods/7ac30952-da5d-42a3-a1aa-bd249bbf6adb/volumes/kubernetes.io~projected/kube-api-access-6hqrf major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7e8ec7ff-015c-4049-9c5f-aae73dc06e4b/volumes/kubernetes.io~projected/kube-api-access-92smt:{mountpoint:/var/lib/kubelet/pods/7e8ec7ff-015c-4049-9c5f-aae73dc06e4b/volumes/kubernetes.io~projected/kube-api-access-92smt major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8e44595b-0bdf-407c-bf22-33eb06272cc1/volumes/kubernetes.io~projected/kube-api-access-tp27x:{mountpoint:/var/lib/kubelet/pods/8e44595b-0bdf-407c-bf22-33eb06272cc1/volumes/kubernetes.io~projected/kube-api-access-tp27x major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a70ee11c-c20e-4a4f-8476-ac5eb02816d0/volumes/kubernetes.io~projected/kube-api-access-vx4mp:{mountpoint:/var/lib/kubelet/pods/a70ee11c-c20e-4a4f-8476-ac5eb02816d0/volumes/kubernetes.io~projected/kube-api-access-vx4mp major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a70ee11c-c20e-4a4f-8476-ac5eb02816d0/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/a70ee11c-c20e-4a4f-8476-ac5eb02816d0/volumes/kubernetes.io~secret/serving-cert major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b1757617-7c51-4029-b329-1b6a4630f860/volumes/kubernetes.io~projected/kube-api-access-zscd2:{mountpoint:/var/lib/kubelet/pods/b1757617-7c51-4029-b329-1b6a4630f860/volumes/kubernetes.io~projected/kube-api-access-zscd2 major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b1757617-7c51-4029-b329-1b6a4630f860/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/b1757617-7c51-4029-b329-1b6a4630f860/volumes/kubernetes.io~secret/serving-cert major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b3f20243-155c-4e25-a1ca-6131a141fbd4/volumes/kubernetes.io~projected/kube-api-access-r226b:{mountpoint:/var/lib/kubelet/pods/b3f20243-155c-4e25-a1ca-6131a141fbd4/volumes/kubernetes.io~projected/kube-api-access-r226b major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b3f20243-155c-4e25-a1ca-6131a141fbd4/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/b3f20243-155c-4e25-a1ca-6131a141fbd4/volumes/kubernetes.io~secret/serving-cert major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bd977255-f9a7-4531-8a9c-b1a3a17e0b79/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/bd977255-f9a7-4531-8a9c-b1a3a17e0b79/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bd977255-f9a7-4531-8a9c-b1a3a17e0b79/volumes/kubernetes.io~projected/kube-api-access-8f7sv:{mountpoint:/var/lib/kubelet/pods/bd977255-f9a7-4531-8a9c-b1a3a17e0b79/volumes/kubernetes.io~projected/kube-api-access-8f7sv major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bd977255-f9a7-4531-8a9c-b1a3a17e0b79/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/bd977255-f9a7-4531-8a9c-b1a3a17e0b79/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bd977255-f9a7-4531-8a9c-b1a3a17e0b79/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/bd977255-f9a7-4531-8a9c-b1a3a17e0b79/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c54f4c1b-f08e-4495-9a14-17469a792ae5/volumes/kubernetes.io~projected/kube-api-access-rh4tb:{mountpoint:/var/lib/kubelet/pods/c54f4c1b-f08e-4495-9a14-17469a792ae5/volumes/kubernetes.io~projected/kube-api-access-rh4tb major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c815b3fc-7f35-4ae6-a015-2e84c6108a64/volumes/kubernetes.io~projected/kube-api-access-jf44s:{mountpoint:/var/lib/kubelet/pods/c815b3fc-7f35-4ae6-a015-2e84c6108a64/volumes/kubernetes.io~projected/kube-api-access-jf44s major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c815b3fc-7f35-4ae6-a015-2e84c6108a64/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/c815b3fc-7f35-4ae6-a015-2e84c6108a64/volumes/kubernetes.io~secret/serving-cert major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cfaad7d0-d30b-4ddc-81cc-2b603416fc29/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/cfaad7d0-d30b-4ddc-81cc-2b603416fc29/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cfaad7d0-d30b-4ddc-81cc-2b603416fc29/volumes/kubernetes.io~projected/kube-api-access-75fhw:{mountpoint:/var/lib/kubelet/pods/cfaad7d0-d30b-4ddc-81cc-2b603416fc29/volumes/kubernetes.io~projected/kube-api-access-75fhw major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cfaad7d0-d30b-4ddc-81cc-2b603416fc29/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/cfaad7d0-d30b-4ddc-81cc-2b603416fc29/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cfaad7d0-d30b-4ddc-81cc-2b603416fc29/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/cfaad7d0-d30b-4ddc-81cc-2b603416fc29/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d0e0cec2-f16a-4bae-bc70-6c08da4bda2e/volumes/kubernetes.io~projected/kube-api-access-h7mvn:{mountpoint:/var/lib/kubelet/pods/d0e0cec2-f16a-4bae-bc70-6c08da4bda2e/volumes/kubernetes.io~projected/kube-api-access-h7mvn major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d659d471-f019-4853-97d0-ee6e948e6c1c/volumes/kubernetes.io~projected/kube-api-access-stff7:{mountpoint:/var/lib/kubelet/pods/d659d471-f019-4853-97d0-ee6e948e6c1c/volumes/kubernetes.io~projected/kube-api-access-stff7 major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d670b364-53ce-444b-b9c4-ac8ac09cd44f/volumes/kubernetes.io~projected/kube-api-access-rlp7f:{mountpoint:/var/lib/kubelet/pods/d670b364-53ce-444b-b9c4-ac8ac09cd44f/volumes/kubernetes.io~projected/kube-api-access-rlp7f major:0 minor:341 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dab8eab1-3e3f-46b4-b76d-d58a7061996d/volumes/kubernetes.io~projected/kube-api-access-7zd2h:{mountpoint:/var/lib/kubelet/pods/dab8eab1-3e3f-46b4-b76d-d58a7061996d/volumes/kubernetes.io~projected/kube-api-access-7zd2h major:0 minor:314 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e497352b-2348-4fc7-888f-5b68f116394f/volumes/kubernetes.io~projected/kube-api-access-jv6z4:{mountpoint:/var/lib/kubelet/pods/e497352b-2348-4fc7-888f-5b68f116394f/volumes/kubernetes.io~projected/kube-api-access-jv6z4 major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ff9feaa5-70cc-47fd-a3d1-a9ac6e7eaace/volumes/kubernetes.io~projected/kube-api-access-zh8h9:{mountpoint:/var/lib/kubelet/pods/ff9feaa5-70cc-47fd-a3d1-a9ac6e7eaace/volumes/kubernetes.io~projected/kube-api-access-zh8h9 major:0 minor:293 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/0e6da7aa9688d71b1bb0751748848d6dfb375b2df540cb19a3d926356160ef20/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/85f0f4034ce672c4b2ef5c992597ff7bdf34f11f7a009cf1683ab153bc296d7f/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/801cf47e3f793d7fc10286a77d5abfb199f5ff499769f29269d1fdbbb8032070/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/f6f22a1b5d9b69c073a896ad75c7c87c9a51246c72963d3384b3168aaf4f29ec/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/039633c70daade3202d5c5fe1d12c36fded6be8e0c75ab84828b2e23bdbd869c/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/bd5554ce1baad83ebadeac3cf6ecc2689d6b9373ba43a067a8bd34de955597f4/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/aea49aaa31459b38a28eb9bb52e721c12a2b9725fdedbb928cbd51c12b1ed420/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/47c87e87c0cdfe8c573967ef08ee2e1c77fdcea39faec384b52b692cc9039e18/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/0d47ee5a20584901344d24f7c209da21a38b9b34b802050e4233a2fd947635f1/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/ee20d181c504b6836c47c68e17531e489a5f1806cb1968741e9c15b9125ae828/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-138:{mountpoint:/var/lib/containers/storage/overlay/1c2d7ef3efa36b877ddaf94ad40b06b04f830d5980b3aaeda42f78ed09853830/merged major:0 minor:138 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/bd03934d7d31071bf902bb28cddf913903b90fe0fa984f15e8a99a694864342c/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/6b749cee5d9c57b8a2ae26e59d9761f0529227f0e16538273dd607917d076cd4/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/0ce5fc4cc3c6139d4f49841c8eb8e5e5dcb507c7c26809a142cb4969972af93c/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-161:{mountpoint:/var/lib/containers/storage/overlay/7fd47f57f7c318c7deceda44a51b144a051c4f5c54a6f3be945b833689a2cb6f/merged major:0 minor:161 fsType:overlay blockSize:0} overlay_0-166:{mountpoint:/var/lib/containers/storage/overlay/0840313b38b301a1a7f9ee6c996a8b4229649459f867ae6f08b21948a53270e8/merged major:0 minor:166 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/606306d5c2b2ba4a421bd37b8998e45bcacf2636aff72ec54029affbf285b2ea/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/f06ab4d6ca7fb331cc51341ec320537e61c6aab8ad3517fb23b5bbb322ccf03f/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/c3238bb03fee9656013d582767f124d406541d0c38f74e73cc8424d66b43f1b2/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/013eab3787ce22335611dbc7ccfaefce90dc5ddc7cff58b069eade4ec4fe03b9/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/adea3c24acfcca611b67dafe16693082499eb39c237c74447dcadf2862d029a6/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/2921b9a3a45444f1497e3028d60f5877341b58779a1cc7cd95be3eb02fcf59c1/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/fbf8dd7414a56e7ae19efd2c22879989a2c725d34f361dc2e68614f72bd8ad79/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/ba919cfbe037acab8da3b1293197e258704e972efef5d5b37c22dfb677108a41/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/277dcb9b4f75dbf94c26ab9c05ca6590d94e03b98056251cbc663fcbca3bd9d0/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/ee548e636b8cba789dbf6e1962403add7358870aa46769eda05827d77ca303a8/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/0ac0654dda622683bed226c49f54c10317179ccad40a2129a7bf72c4bc0149f5/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/92e463ec76f130812f873fc22e0a5beb8bab775d2498fe940db9b95e68838b4f/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/6fffdacb243a34d02234ade72707b0778df6593375d99004291fc4acee5b22de/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-264:{mountpoint:/var/lib/containers/storage/overlay/0eaff1e9a819018ba63141aaf24035a2270f632cc48813b405b0eb5af046aa21/merged major:0 minor:264 fsType:overlay blockSize:0} overlay_0-266:{mountpoint:/var/lib/containers/storage/overlay/a5af5b74c7dd6e586030705f7622004954bdc628afc66ce60f74377ff1075f9f/merged major:0 minor:266 fsType:overlay blockSize:0} overlay_0-268:{mountpoint:/var/lib/containers/storage/overlay/fe3e25ddd537156d7eccd43f6ad1bcea298625745a797ef5a0081d37b7aadae0/merged major:0 minor:268 fsType:overlay blockSize:0} overlay_0-269:{mountpoint:/var/lib/containers/storage/overlay/e593fc34bbe61f9d7670c8681ff038dca6c3e881f0096137f455b38721276e82/merged major:0 minor:269 fsType:overlay blockSize:0} overlay_0-272:{mountpoint:/var/lib/containers/storage/overlay/e86d5c559c910dfe6fd818696f3bba87e62d14d4b5bb05ce3b0e3d04ada16dee/merged major:0 minor:272 fsType:overlay blockSize:0} overlay_0-285:{mountpoint:/var/lib/containers/storage/overlay/5f05cff193880ef2cec579f22fa656a3107828e7248f5365c3969211c5bab2f1/merged major:0 minor:285 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/eff5cd4c7e4b727568c9924408c253ea0000245eb8efd8c4c1b07768b01d4cf3/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-300:{mountpoint:/var/lib/containers/storage/overlay/b985dbecd554f14f2dbf933514de1fa82507242ef81bc80ffbdb89c1a4f6e217/merged major:0 minor:300 fsType:overlay blockSize:0} overlay_0-302:{mountpoint:/var/lib/containers/storage/overlay/476ac2d79fa141d6d3b6dd3de82fc312f9901eaba7bc403af74707b3b9db2170/merged major:0 minor:302 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/09a9725ccdcb202fd3db029188572a5a424d50ca453fddd17fb91155beaaa132/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-306:{mountpoint:/var/lib/containers/storage/overlay/d31191ff7b87d72d23c130e9123296fad17ec0fb5d826e472fad90b17fc00a71/merged major:0 minor:306 fsType:overlay blockSize:0} overlay_0-308:{mountpoint:/var/lib/containers/storage/overlay/4a39be4197fceea0edb1d52d1952ddde0eff7d89483eeb844ad62a3622b7c0c4/merged major:0 minor:308 fsType:overlay blockSize:0} overlay_0-310:{mountpoint:/var/lib/containers/storage/overlay/196068eeb555bcc2232965a17b1b9ead6f36ed4749a9b7b4b2a0c8f02ac805e8/merged major:0 minor:310 fsType:overlay blockSize:0} overlay_0-312:{mountpoint:/var/lib/containers/storage/overlay/36fd0a319bb99fe0228316a1cdb912bb6280ea0a3febded7b25829b6608dd841/merged major:0 minor:312 fsType:overlay blockSize:0} overlay_0-342:{mountpoint:/var/lib/containers/storage/overlay/8bd69e9b93142b2016e289a6278017eddbf4e6b1ce7905396cd44132f05c436b/merged major:0 minor:342 fsType:overlay blockSize:0} overlay_0-344:{mountpoint:/var/lib/containers/storage/overlay/5097e62f12e10600750ccb5d0b45bd322f066e83d25c984aacbbe8deaebf02ef/merged major:0 minor:344 fsType:overlay blockSize:0} overlay_0-346:{mountpoint:/var/lib/containers/storage/overlay/e1b5b5f1d9c7e09c26adcb2c3253f55a795a8c666b9baa5598605c27d7eb366e/merged major:0 minor:346 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/cf725caecf6570c8974b2c098733462ac885497672c3b218090cf20f1efa13d7/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/30d40b0dc4d74ad8488b7cf3bda999ea2eb085baefffcbf8959f74cb673bb3a7/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/06d9afa26ca746f0b00e551214ddd8d91a697108816bcb60cb46adc498945a02/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/92d6c0e78e59a479fd22823a1bcb37e2026b88532569800fa09792d8e0dda2db/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/3466100edd35743d02d808dc2a78ba9dd0647521c6421c183e9df77acc044ffe/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/08737f08c0cd9e646ffe36587b7706ccfbfd90316527106d23b7852c0800ac1d/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/b8d13bfef1bfebc4ec74e1a5c6c3e17159a607f970e5366e433f72fa8ab26ded/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/19ca8bc2a42ce268a558b1d721694340b4755a13e5e2faef4ef4c7e1a2f11344/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/40291e199fa15a59081601a40237cee3805d12e7a5a611c51ca5fc28ff3f18d1/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-98:{mountpoint:/var/lib/containers/storage/overlay/9330a665fafa53b5d0bec3a9b4ce872a1315023ae259dc46843fba28619065da/merged major:0 minor:98 fsType:overlay blockSize:0}] Oct 30 11:54:03.554302 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.553320 4908 manager.go:217] Machine: {Timestamp:2026-10-30 11:54:03.552887089 +0000 UTC m=+0.067315825 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3182733 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2ee154a9a13382148e60952589ab09 SystemUUID:ec2ee154-a9a1-3382-148e-60952589ab09 BootID:8b2fe4e4-2c68-4dc5-aa57-9651d2a3939e Filesystems:[{Device:/var/lib/kubelet/pods/bd977255-f9a7-4531-8a9c-b1a3a17e0b79/volumes/kubernetes.io~projected/kube-api-access-8f7sv DeviceMajor:0 DeviceMinor:219 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4b0f6db58147695aa013d92f1e9abca35d686a1268c30c8b5ffa706aec6e51eb/userdata/shm DeviceMajor:0 DeviceMinor:246 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/3699031f-ee9e-46cc-b37e-4dfc76b9c0b1/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:58 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7e8ec7ff-015c-4049-9c5f-aae73dc06e4b/volumes/kubernetes.io~projected/kube-api-access-92smt DeviceMajor:0 DeviceMinor:65 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b1757617-7c51-4029-b329-1b6a4630f860/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:197 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-308 DeviceMajor:0 DeviceMinor:308 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8e44595b-0bdf-407c-bf22-33eb06272cc1/volumes/kubernetes.io~projected/kube-api-access-tp27x DeviceMajor:0 DeviceMinor:209 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a70ee11c-c20e-4a4f-8476-ac5eb02816d0/volumes/kubernetes.io~projected/kube-api-access-vx4mp DeviceMajor:0 DeviceMinor:223 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bd977255-f9a7-4531-8a9c-b1a3a17e0b79/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:217 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/6acb13a1-cb2e-45e1-897b-3ef89e6d1b84/volumes/kubernetes.io~projected/kube-api-access-2rxc4 DeviceMajor:0 DeviceMinor:220 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-272 DeviceMajor:0 DeviceMinor:272 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/81d2c98212fb5e239a5750b14b2032945223d71a039bfef6fd50039af7d88632/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-344 DeviceMajor:0 DeviceMinor:344 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/644e19b3-10ad-4145-9ed3-595d19ed9b07/volumes/kubernetes.io~projected/kube-api-access-8nzgq DeviceMajor:0 DeviceMinor:66 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-161 DeviceMajor:0 DeviceMinor:161 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cfaad7d0-d30b-4ddc-81cc-2b603416fc29/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:206 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/18e782439c70c580ec71ddf416df5349c870d492753c67e14b13bb86f2d87533/userdata/shm DeviceMajor:0 DeviceMinor:294 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4fe7f906d45dc1117fc963f19feb9094a9508021f874ec361136e810911456ae/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/5aee536a-8942-4b48-9b2c-3e36089f3ac5/volumes/kubernetes.io~projected/kube-api-access-vmx84 DeviceMajor:0 DeviceMinor:63 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c815b3fc-7f35-4ae6-a015-2e84c6108a64/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:212 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/003c3d18-23f1-45d0-bac4-8c6eea4b7e99/volumes/kubernetes.io~projected/kube-api-access-4jmcf DeviceMajor:0 DeviceMinor:234 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-346 DeviceMajor:0 DeviceMinor:346 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ff9feaa5-70cc-47fd-a3d1-a9ac6e7eaace/volumes/kubernetes.io~projected/kube-api-access-zh8h9 DeviceMajor:0 DeviceMinor:293 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-266 DeviceMajor:0 DeviceMinor:266 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cfaad7d0-d30b-4ddc-81cc-2b603416fc29/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:45 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/cfaad7d0-d30b-4ddc-81cc-2b603416fc29/volumes/kubernetes.io~projected/kube-api-access-75fhw DeviceMajor:0 DeviceMinor:210 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/36525167111a09951040a77f0c5cf71541a5df0fc8e7bc52add5f57bd46d3612/userdata/shm DeviceMajor:0 DeviceMinor:238 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4e18f771-273b-49e6-872c-6cbe47ab68c4/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:56 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/bd977255-f9a7-4531-8a9c-b1a3a17e0b79/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:218 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/1b735438-5947-4374-bd4c-66ac3a299770/volumes/kubernetes.io~projected/kube-api-access-gzc5h DeviceMajor:0 DeviceMinor:61 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/419f42abae50a8c097a816062e2e78976ae8ec37f34318f0c1aa15703d4eac60/userdata/shm DeviceMajor:0 DeviceMinor:228 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1f2563b5c5e8e08274b16bd9bbbccba45366b0aed1b60df792c35b2ac3e13f6f/userdata/shm DeviceMajor:0 DeviceMinor:95 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c7e539754dfdda2f33506f8d6a7dc8acd027ef96d8d03b975eff48d2c6fc59bd/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c54f4c1b-f08e-4495-9a14-17469a792ae5/volumes/kubernetes.io~projected/kube-api-access-rh4tb DeviceMajor:0 DeviceMinor:67 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/003c3d18-23f1-45d0-bac4-8c6eea4b7e99/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:231 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/cfaad7d0-d30b-4ddc-81cc-2b603416fc29/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:201 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b1757617-7c51-4029-b329-1b6a4630f860/volumes/kubernetes.io~projected/kube-api-access-zscd2 DeviceMajor:0 DeviceMinor:205 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/221ced85-cc15-4e07-82d9-a1a1f3e958a5/volumes/kubernetes.io~projected/kube-api-access-bhxmm DeviceMajor:0 DeviceMinor:208 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/bd977255-f9a7-4531-8a9c-b1a3a17e0b79/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:211 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b3f20243-155c-4e25-a1ca-6131a141fbd4/volumes/kubernetes.io~projected/kube-api-access-r226b DeviceMajor:0 DeviceMinor:222 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-342 DeviceMajor:0 DeviceMinor:342 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3699031f-ee9e-46cc-b37e-4dfc76b9c0b1/volumes/kubernetes.io~projected/kube-api-access-r7vnf DeviceMajor:0 DeviceMinor:64 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-166 DeviceMajor:0 DeviceMinor:166 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c815b3fc-7f35-4ae6-a015-2e84c6108a64/volumes/kubernetes.io~projected/kube-api-access-jf44s DeviceMajor:0 DeviceMinor:225 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7e3210ba71b33dbf2744cdcf9f8c0af3b880fefb6715945363dfe73c1c6a67e6/userdata/shm DeviceMajor:0 DeviceMinor:226 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ac1ca6630282423463ddd56cf4dd5bd7e452ffaef60997a055a518a3d193ae1f/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d0e0cec2-f16a-4bae-bc70-6c08da4bda2e/volumes/kubernetes.io~projected/kube-api-access-h7mvn DeviceMajor:0 DeviceMinor:207 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7689cfe5-5a4e-4481-8eb9-107a58ca8bb3/volumes/kubernetes.io~projected/kube-api-access-gr67s DeviceMajor:0 DeviceMinor:221 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-310 DeviceMajor:0 DeviceMinor:310 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/dab8eab1-3e3f-46b4-b76d-d58a7061996d/volumes/kubernetes.io~projected/kube-api-access-7zd2h DeviceMajor:0 DeviceMinor:314 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d670b364-53ce-444b-b9c4-ac8ac09cd44f/volumes/kubernetes.io~projected/kube-api-access-rlp7f DeviceMajor:0 DeviceMinor:341 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a70ee11c-c20e-4a4f-8476-ac5eb02816d0/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:214 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bf1bbee327514fab5fac9eb46554e91c28ce5a6387743ae6b1c7d24ba7989f4f/userdata/shm DeviceMajor:0 DeviceMinor:233 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e03ff409ec2f82625f5cfe631fec11e889aa058ba91dbd505bf4ea6d6b31fec9/userdata/shm DeviceMajor:0 DeviceMinor:240 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1f5ac21f834c419c1d2b1b6d14292b5bd03d2741030a178f2a45cdb3262508dd/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-138 DeviceMajor:0 DeviceMinor:138 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6d976900-ecd0-469c-a9e2-d60708361f1a/volumes/kubernetes.io~projected/kube-api-access-5fhb8 DeviceMajor:0 DeviceMinor:204 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7ac30952-da5d-42a3-a1aa-bd249bbf6adb/volumes/kubernetes.io~projected/kube-api-access-6hqrf DeviceMajor:0 DeviceMinor:224 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2c9d279e75511d42a300ea507ae46c045f2f6d7354e80d750835817d62d998b7/userdata/shm DeviceMajor:0 DeviceMinor:236 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a210218da33eda8d8c8333214533a17c60812fe01e8061c5d1a295774b5006b0/userdata/shm DeviceMajor:0 DeviceMinor:296 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-302 DeviceMajor:0 DeviceMinor:302 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/50aeeaf84669fde1fb553f85753f6f537614a6a045412ab3cf5635b37f049ea8/userdata/shm DeviceMajor:0 DeviceMinor:244 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/50729bdfcb9b631e0bcdd1ee0528e7c3c9dacbac0f5276aa7f47aaca3c6bc046/userdata/shm DeviceMajor:0 DeviceMinor:315 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-264 DeviceMajor:0 DeviceMinor:264 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1f5667a1-a773-4520-b2fb-c69148df632c/volumes/kubernetes.io~projected/kube-api-access-bzld2 DeviceMajor:0 DeviceMinor:203 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-268 DeviceMajor:0 DeviceMinor:268 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-306 DeviceMajor:0 DeviceMinor:306 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-312 DeviceMajor:0 DeviceMinor:312 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1d7f8396-e57c-45e5-ae74-a2244db10364/volumes/kubernetes.io~projected/kube-api-access-bzwcn DeviceMajor:0 DeviceMinor:55 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0df1a118-36a5-4c57-8cec-2c8236f1c4de/volumes/kubernetes.io~projected/kube-api-access-r4tdc DeviceMajor:0 DeviceMinor:60 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e497352b-2348-4fc7-888f-5b68f116394f/volumes/kubernetes.io~projected/kube-api-access-jv6z4 DeviceMajor:0 DeviceMinor:202 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/644e19b3-10ad-4145-9ed3-595d19ed9b07/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:57 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b3f20243-155c-4e25-a1ca-6131a141fbd4/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:213 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0e385d1aea1378254689b60f24673fa5cb718887b518e18d8f68b64224aa2496/userdata/shm DeviceMajor:0 DeviceMinor:242 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1b6e44060e4775c8234fb5f6b195c11bc0f4beb0b731852b60d9d07f2b70e864/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/59286100c47dfffc745aeaf1101d1b002609f107813dd3febe7c1a355fbe11db/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-98 DeviceMajor:0 DeviceMinor:98 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/644e19b3-10ad-4145-9ed3-595d19ed9b07/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/6acb13a1-cb2e-45e1-897b-3ef89e6d1b84/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:216 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0ed7babdc2068ee24f20699aa63685bab422bb05562cc3f8e79c91aeb1568a52/userdata/shm DeviceMajor:0 DeviceMinor:230 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-300 DeviceMajor:0 DeviceMinor:300 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d659d471-f019-4853-97d0-ee6e948e6c1c/volumes/kubernetes.io~projected/kube-api-access-stff7 DeviceMajor:0 DeviceMinor:62 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-285 DeviceMajor:0 DeviceMinor:285 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6b350710-c48c-4c7e-bce9-e021099e98ca/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:288 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-269 DeviceMajor:0 DeviceMinor:269 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3699031f-ee9e-46cc-b37e-4dfc76b9c0b1/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:59 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e1e676a6d3c93a0071e009327272d842160f1207839b7a25a676a8f5e4d83d25/userdata/shm DeviceMajor:0 DeviceMinor:74 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/37fb31a12739fbc2204e29a20a6a16c2a38ecb9fc54c663aaa811e6d264a8655/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/6acb13a1-cb2e-45e1-897b-3ef89e6d1b84/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:215 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:0e385d1aea13782 MacAddress:8a:82:b7:fb:c2:2e Speed:10000 Mtu:8901} {Name:0ed7babdc2068ee MacAddress:02:f2:a9:b3:b1:10 Speed:10000 Mtu:8901} {Name:18e782439c70c58 MacAddress:ee:c2:86:a7:70:65 Speed:10000 Mtu:8901} {Name:2c9d279e75511d4 MacAddress:72:26:7b:b8:d1:86 Speed:10000 Mtu:8901} {Name:36525167111a099 MacAddress:8e:2a:84:b7:2a:bd Speed:10000 Mtu:8901} {Name:419f42abae50a8c MacAddress:e2:83:21:65:6b:5d Speed:10000 Mtu:8901} {Name:4b0f6db58147695 MacAddress:96:82:98:ae:c7:b9 Speed:10000 Mtu:8901} {Name:50729bdfcb9b631 MacAddress:ea:ec:2e:c4:bf:b2 Speed:10000 Mtu:8901} {Name:50aeeaf84669fde MacAddress:42:e0:eb:3a:b0:cd Speed:10000 Mtu:8901} {Name:7e3210ba71b33db MacAddress:5a:f1:ed:05:ce:0b Speed:10000 Mtu:8901} {Name:a210218da33eda8 MacAddress:d2:ef:11:5c:ca:0c Speed:10000 Mtu:8901} {Name:bf1bbee327514fa MacAddress:f6:5d:8a:2e:c5:c2 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:f2:db:09:25 Speed:0 Mtu:9001} {Name:br-int MacAddress:d6:b3:c5:c6:a9:3a Speed:0 Mtu:8901} {Name:e03ff409ec2f826 MacAddress:7e:bc:e5:dc:32:ec Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:f2:db:09:25 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:f2:32:0a:e8:4d:92 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:ea:ff:c1:c8:ec:02 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 11:54:03.554302 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.553652 4908 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 11:54:03.554302 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.553692 4908 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 11:54:03.554302 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.554057 4908 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 11:54:03.554302 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.554076 4908 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-251.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 11:54:03.554302 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.554190 4908 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 11:54:03.554302 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.554197 4908 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 11:54:03.554302 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.554209 4908 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 11:54:03.554302 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.554221 4908 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 11:54:03.554302 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.554244 4908 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 11:54:03.554731 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.554382 4908 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 11:54:03.554731 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.554485 4908 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 11:54:03.554731 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.554518 4908 kubelet.go:493] "Attempting to sync node with API server" Oct 30 11:54:03.554731 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.554531 4908 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 11:54:03.554731 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.554540 4908 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 11:54:03.554731 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.554548 4908 kubelet.go:405] "Adding apiserver pod source" Oct 30 11:54:03.554731 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.554556 4908 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 11:54:03.555574 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.555560 4908 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 11:54:03.555612 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.555576 4908 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 11:54:03.556061 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.556047 4908 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 11:54:03.558607 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.558589 4908 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 11:54:03.558607 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.558603 4908 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 11:54:03.558764 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.558752 4908 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 30 11:54:03.558982 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.558968 4908 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 11:54:03.559047 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.559008 4908 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 11:54:03.559125 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.559114 4908 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 11:54:03.559176 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.559133 4908 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 11:54:03.559176 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.559144 4908 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 11:54:03.559176 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.559153 4908 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 11:54:03.559176 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.559162 4908 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 11:54:03.559176 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.559171 4908 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 11:54:03.559409 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.559180 4908 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 11:54:03.559409 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.559189 4908 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 11:54:03.559409 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.559199 4908 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 11:54:03.559409 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.559208 4908 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 11:54:03.559409 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.559236 4908 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 11:54:03.559409 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.559252 4908 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 11:54:03.559409 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.559292 4908 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 11:54:03.559409 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.559320 4908 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 11:54:03.562725 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.562708 4908 server.go:1263] "Started kubelet" Oct 30 11:54:03.562807 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.562765 4908 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 11:54:03.563117 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.562795 4908 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 11:54:03.563117 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.563001 4908 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 11:54:03.563241 ip-10-0-141-251 systemd[1]: Started Kubernetes Kubelet. Oct 30 11:54:03.563875 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.563536 4908 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 11:54:03.568210 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.565678 4908 server.go:317] "Adding debug handlers to kubelet server" Oct 30 11:54:03.568210 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.567327 4908 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 11:54:03.568210 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.567357 4908 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 11:54:03.568210 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.567630 4908 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 11:54:03.568210 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.567641 4908 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 11:54:03.568210 ip-10-0-141-251 kubenswrapper[4908]: E1030 11:54:03.567763 4908 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-251.ec2.internal\" not found" Oct 30 11:54:03.568210 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.567768 4908 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 11:48:25 +0000 UTC" deadline="2028-03-30 19:41:42.7600273 +0000 UTC" Oct 30 11:54:03.568210 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.567802 4908 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12415h47m39.192228816s" Oct 30 11:54:03.568210 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.567881 4908 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 11:54:03.568691 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.568393 4908 factory.go:55] Registering systemd factory Oct 30 11:54:03.568691 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.568408 4908 factory.go:223] Registration of the systemd container factory successfully Oct 30 11:54:03.570424 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.570395 4908 factory.go:153] Registering CRI-O factory Oct 30 11:54:03.570424 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.570415 4908 factory.go:223] Registration of the crio container factory successfully Oct 30 11:54:03.570562 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.570469 4908 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 11:54:03.570562 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.570487 4908 factory.go:103] Registering Raw factory Oct 30 11:54:03.570562 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.570501 4908 manager.go:1199] Started watching for new ooms in manager Oct 30 11:54:03.570814 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.570801 4908 manager.go:319] Starting recovery of all containers Oct 30 11:54:03.571215 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.571179 4908 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="003c3d18-23f1-45d0-bac4-8c6eea4b7e99" volumeName="kubernetes.io/projected/003c3d18-23f1-45d0-bac4-8c6eea4b7e99-kube-api-access-4jmcf" seLinuxMountContext="" Oct 30 11:54:03.571297 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.571219 4908 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="221ced85-cc15-4e07-82d9-a1a1f3e958a5" volumeName="kubernetes.io/projected/221ced85-cc15-4e07-82d9-a1a1f3e958a5-kube-api-access-bhxmm" seLinuxMountContext="" Oct 30 11:54:03.571297 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.571232 4908 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="644e19b3-10ad-4145-9ed3-595d19ed9b07" volumeName="kubernetes.io/configmap/644e19b3-10ad-4145-9ed3-595d19ed9b07-ovnkube-config" seLinuxMountContext="" Oct 30 11:54:03.571297 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.571242 4908 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b1757617-7c51-4029-b329-1b6a4630f860" volumeName="kubernetes.io/projected/b1757617-7c51-4029-b329-1b6a4630f860-kube-api-access-zscd2" seLinuxMountContext="" Oct 30 11:54:03.571297 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.571255 4908 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bd977255-f9a7-4531-8a9c-b1a3a17e0b79" volumeName="kubernetes.io/projected/bd977255-f9a7-4531-8a9c-b1a3a17e0b79-bound-sa-token" seLinuxMountContext="" Oct 30 11:54:03.571297 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.571267 4908 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bd977255-f9a7-4531-8a9c-b1a3a17e0b79" volumeName="kubernetes.io/secret/bd977255-f9a7-4531-8a9c-b1a3a17e0b79-image-registry-private-configuration" seLinuxMountContext="" Oct 30 11:54:03.571297 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.571278 4908 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bd977255-f9a7-4531-8a9c-b1a3a17e0b79" volumeName="kubernetes.io/secret/bd977255-f9a7-4531-8a9c-b1a3a17e0b79-installation-pull-secrets" seLinuxMountContext="" Oct 30 11:54:03.571587 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.571290 4908 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cfaad7d0-d30b-4ddc-81cc-2b603416fc29" volumeName="kubernetes.io/configmap/cfaad7d0-d30b-4ddc-81cc-2b603416fc29-trusted-ca" seLinuxMountContext="" Oct 30 11:54:03.571587 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.571312 4908 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="179d55d3-7958-4928-a98c-68984fca9837" volumeName="kubernetes.io/configmap/179d55d3-7958-4928-a98c-68984fca9837-nginx-conf" seLinuxMountContext="" Oct 30 11:54:03.571587 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.571324 4908 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7ac30952-da5d-42a3-a1aa-bd249bbf6adb" volumeName="kubernetes.io/projected/7ac30952-da5d-42a3-a1aa-bd249bbf6adb-kube-api-access-6hqrf" seLinuxMountContext="" Oct 30 11:54:03.571587 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.571339 4908 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b1757617-7c51-4029-b329-1b6a4630f860" volumeName="kubernetes.io/secret/b1757617-7c51-4029-b329-1b6a4630f860-serving-cert" seLinuxMountContext="" Oct 30 11:54:03.571587 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.571350 4908 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c815b3fc-7f35-4ae6-a015-2e84c6108a64" volumeName="kubernetes.io/projected/c815b3fc-7f35-4ae6-a015-2e84c6108a64-kube-api-access-jf44s" seLinuxMountContext="" Oct 30 11:54:03.571587 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.571361 4908 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d670b364-53ce-444b-b9c4-ac8ac09cd44f" volumeName="kubernetes.io/empty-dir/d670b364-53ce-444b-b9c4-ac8ac09cd44f-data-volume" seLinuxMountContext="" Oct 30 11:54:03.571587 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.571372 4908 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="644e19b3-10ad-4145-9ed3-595d19ed9b07" volumeName="kubernetes.io/projected/644e19b3-10ad-4145-9ed3-595d19ed9b07-kube-api-access-8nzgq" seLinuxMountContext="" Oct 30 11:54:03.571587 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.571386 4908 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6acb13a1-cb2e-45e1-897b-3ef89e6d1b84" volumeName="kubernetes.io/secret/6acb13a1-cb2e-45e1-897b-3ef89e6d1b84-default-certificate" seLinuxMountContext="" Oct 30 11:54:03.571587 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.571398 4908 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6d976900-ecd0-469c-a9e2-d60708361f1a" volumeName="kubernetes.io/configmap/6d976900-ecd0-469c-a9e2-d60708361f1a-telemetry-config" seLinuxMountContext="" Oct 30 11:54:03.571587 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.571411 4908 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a70ee11c-c20e-4a4f-8476-ac5eb02816d0" volumeName="kubernetes.io/configmap/a70ee11c-c20e-4a4f-8476-ac5eb02816d0-config" seLinuxMountContext="" Oct 30 11:54:03.571587 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.571421 4908 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a70ee11c-c20e-4a4f-8476-ac5eb02816d0" volumeName="kubernetes.io/projected/a70ee11c-c20e-4a4f-8476-ac5eb02816d0-kube-api-access-vx4mp" seLinuxMountContext="" Oct 30 11:54:03.571587 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.571432 4908 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bd977255-f9a7-4531-8a9c-b1a3a17e0b79" volumeName="kubernetes.io/configmap/bd977255-f9a7-4531-8a9c-b1a3a17e0b79-registry-certificates" seLinuxMountContext="" Oct 30 11:54:03.571587 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.571459 4908 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cfaad7d0-d30b-4ddc-81cc-2b603416fc29" volumeName="kubernetes.io/projected/cfaad7d0-d30b-4ddc-81cc-2b603416fc29-kube-api-access-75fhw" seLinuxMountContext="" Oct 30 11:54:03.571587 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.571473 4908 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bd977255-f9a7-4531-8a9c-b1a3a17e0b79" volumeName="kubernetes.io/projected/bd977255-f9a7-4531-8a9c-b1a3a17e0b79-kube-api-access-8f7sv" seLinuxMountContext="" Oct 30 11:54:03.571587 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.571511 4908 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="644e19b3-10ad-4145-9ed3-595d19ed9b07" volumeName="kubernetes.io/configmap/644e19b3-10ad-4145-9ed3-595d19ed9b07-ovnkube-script-lib" seLinuxMountContext="" Oct 30 11:54:03.571587 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.571525 4908 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6b350710-c48c-4c7e-bce9-e021099e98ca" volumeName="kubernetes.io/secret/6b350710-c48c-4c7e-bce9-e021099e98ca-original-pull-secret" seLinuxMountContext="" Oct 30 11:54:03.571587 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.571537 4908 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8e44595b-0bdf-407c-bf22-33eb06272cc1" volumeName="kubernetes.io/projected/8e44595b-0bdf-407c-bf22-33eb06272cc1-kube-api-access-tp27x" seLinuxMountContext="" Oct 30 11:54:03.571587 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.571548 4908 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c815b3fc-7f35-4ae6-a015-2e84c6108a64" volumeName="kubernetes.io/configmap/c815b3fc-7f35-4ae6-a015-2e84c6108a64-config" seLinuxMountContext="" Oct 30 11:54:03.571587 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.571560 4908 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cfaad7d0-d30b-4ddc-81cc-2b603416fc29" volumeName="kubernetes.io/empty-dir/cfaad7d0-d30b-4ddc-81cc-2b603416fc29-ca-trust-extracted" seLinuxMountContext="" Oct 30 11:54:03.571587 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.571572 4908 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0df1a118-36a5-4c57-8cec-2c8236f1c4de" volumeName="kubernetes.io/configmap/0df1a118-36a5-4c57-8cec-2c8236f1c4de-multus-daemon-config" seLinuxMountContext="" Oct 30 11:54:03.571587 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.571584 4908 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4e18f771-273b-49e6-872c-6cbe47ab68c4" volumeName="kubernetes.io/configmap/4e18f771-273b-49e6-872c-6cbe47ab68c4-konnectivity-ca" seLinuxMountContext="" Oct 30 11:54:03.572597 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.571609 4908 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="644e19b3-10ad-4145-9ed3-595d19ed9b07" volumeName="kubernetes.io/secret/644e19b3-10ad-4145-9ed3-595d19ed9b07-ovn-node-metrics-cert" seLinuxMountContext="" Oct 30 11:54:03.572597 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.571621 4908 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b3f20243-155c-4e25-a1ca-6131a141fbd4" volumeName="kubernetes.io/projected/b3f20243-155c-4e25-a1ca-6131a141fbd4-kube-api-access-r226b" seLinuxMountContext="" Oct 30 11:54:03.572597 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.571633 4908 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c54f4c1b-f08e-4495-9a14-17469a792ae5" volumeName="kubernetes.io/projected/c54f4c1b-f08e-4495-9a14-17469a792ae5-kube-api-access-rh4tb" seLinuxMountContext="" Oct 30 11:54:03.572597 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.571643 4908 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5aee536a-8942-4b48-9b2c-3e36089f3ac5" volumeName="kubernetes.io/configmap/5aee536a-8942-4b48-9b2c-3e36089f3ac5-serviceca" seLinuxMountContext="" Oct 30 11:54:03.572597 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.571666 4908 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="644e19b3-10ad-4145-9ed3-595d19ed9b07" volumeName="kubernetes.io/configmap/644e19b3-10ad-4145-9ed3-595d19ed9b07-env-overrides" seLinuxMountContext="" Oct 30 11:54:03.572597 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.571676 4908 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a70ee11c-c20e-4a4f-8476-ac5eb02816d0" volumeName="kubernetes.io/secret/a70ee11c-c20e-4a4f-8476-ac5eb02816d0-serving-cert" seLinuxMountContext="" Oct 30 11:54:03.572597 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.571688 4908 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cfaad7d0-d30b-4ddc-81cc-2b603416fc29" volumeName="kubernetes.io/secret/cfaad7d0-d30b-4ddc-81cc-2b603416fc29-image-registry-private-configuration" seLinuxMountContext="" Oct 30 11:54:03.572597 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.571699 4908 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ff9feaa5-70cc-47fd-a3d1-a9ac6e7eaace" volumeName="kubernetes.io/projected/ff9feaa5-70cc-47fd-a3d1-a9ac6e7eaace-kube-api-access-zh8h9" seLinuxMountContext="" Oct 30 11:54:03.572597 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.571709 4908 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1b735438-5947-4374-bd4c-66ac3a299770" volumeName="kubernetes.io/empty-dir/1b735438-5947-4374-bd4c-66ac3a299770-tmp-dir" seLinuxMountContext="" Oct 30 11:54:03.572597 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.571719 4908 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="221ced85-cc15-4e07-82d9-a1a1f3e958a5" volumeName="kubernetes.io/empty-dir/221ced85-cc15-4e07-82d9-a1a1f3e958a5-tmp-dir" seLinuxMountContext="" Oct 30 11:54:03.572597 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.571731 4908 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6acb13a1-cb2e-45e1-897b-3ef89e6d1b84" volumeName="kubernetes.io/secret/6acb13a1-cb2e-45e1-897b-3ef89e6d1b84-stats-auth" seLinuxMountContext="" Oct 30 11:54:03.572597 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.571744 4908 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6d976900-ecd0-469c-a9e2-d60708361f1a" volumeName="kubernetes.io/projected/6d976900-ecd0-469c-a9e2-d60708361f1a-kube-api-access-5fhb8" seLinuxMountContext="" Oct 30 11:54:03.572597 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.571758 4908 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7689cfe5-5a4e-4481-8eb9-107a58ca8bb3" volumeName="kubernetes.io/configmap/7689cfe5-5a4e-4481-8eb9-107a58ca8bb3-samples-operator-config" seLinuxMountContext="" Oct 30 11:54:03.572597 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.571769 4908 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b3f20243-155c-4e25-a1ca-6131a141fbd4" volumeName="kubernetes.io/configmap/b3f20243-155c-4e25-a1ca-6131a141fbd4-config" seLinuxMountContext="" Oct 30 11:54:03.572597 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.571780 4908 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1b735438-5947-4374-bd4c-66ac3a299770" volumeName="kubernetes.io/projected/1b735438-5947-4374-bd4c-66ac3a299770-kube-api-access-gzc5h" seLinuxMountContext="" Oct 30 11:54:03.572597 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.571791 4908 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3699031f-ee9e-46cc-b37e-4dfc76b9c0b1" volumeName="kubernetes.io/empty-dir/3699031f-ee9e-46cc-b37e-4dfc76b9c0b1-etc-tuned" seLinuxMountContext="" Oct 30 11:54:03.572597 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.571802 4908 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4e18f771-273b-49e6-872c-6cbe47ab68c4" volumeName="kubernetes.io/secret/4e18f771-273b-49e6-872c-6cbe47ab68c4-agent-certs" seLinuxMountContext="" Oct 30 11:54:03.572597 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.571816 4908 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c815b3fc-7f35-4ae6-a015-2e84c6108a64" volumeName="kubernetes.io/secret/c815b3fc-7f35-4ae6-a015-2e84c6108a64-serving-cert" seLinuxMountContext="" Oct 30 11:54:03.572597 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.571842 4908 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d659d471-f019-4853-97d0-ee6e948e6c1c" volumeName="kubernetes.io/projected/d659d471-f019-4853-97d0-ee6e948e6c1c-kube-api-access-stff7" seLinuxMountContext="" Oct 30 11:54:03.572597 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.571852 4908 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d670b364-53ce-444b-b9c4-ac8ac09cd44f" volumeName="kubernetes.io/projected/d670b364-53ce-444b-b9c4-ac8ac09cd44f-kube-api-access-rlp7f" seLinuxMountContext="" Oct 30 11:54:03.572597 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.571863 4908 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bd977255-f9a7-4531-8a9c-b1a3a17e0b79" volumeName="kubernetes.io/empty-dir/bd977255-f9a7-4531-8a9c-b1a3a17e0b79-ca-trust-extracted" seLinuxMountContext="" Oct 30 11:54:03.572597 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.571875 4908 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="221ced85-cc15-4e07-82d9-a1a1f3e958a5" volumeName="kubernetes.io/configmap/221ced85-cc15-4e07-82d9-a1a1f3e958a5-config-volume" seLinuxMountContext="" Oct 30 11:54:03.572597 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.571886 4908 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a70ee11c-c20e-4a4f-8476-ac5eb02816d0" volumeName="kubernetes.io/configmap/a70ee11c-c20e-4a4f-8476-ac5eb02816d0-trusted-ca" seLinuxMountContext="" Oct 30 11:54:03.572597 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.571898 4908 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cfaad7d0-d30b-4ddc-81cc-2b603416fc29" volumeName="kubernetes.io/configmap/cfaad7d0-d30b-4ddc-81cc-2b603416fc29-registry-certificates" seLinuxMountContext="" Oct 30 11:54:03.572597 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.571910 4908 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d659d471-f019-4853-97d0-ee6e948e6c1c" volumeName="kubernetes.io/configmap/d659d471-f019-4853-97d0-ee6e948e6c1c-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 30 11:54:03.572597 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.571920 4908 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e497352b-2348-4fc7-888f-5b68f116394f" volumeName="kubernetes.io/projected/e497352b-2348-4fc7-888f-5b68f116394f-kube-api-access-jv6z4" seLinuxMountContext="" Oct 30 11:54:03.572597 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.571933 4908 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0df1a118-36a5-4c57-8cec-2c8236f1c4de" volumeName="kubernetes.io/projected/0df1a118-36a5-4c57-8cec-2c8236f1c4de-kube-api-access-r4tdc" seLinuxMountContext="" Oct 30 11:54:03.572597 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.571945 4908 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5aee536a-8942-4b48-9b2c-3e36089f3ac5" volumeName="kubernetes.io/projected/5aee536a-8942-4b48-9b2c-3e36089f3ac5-kube-api-access-vmx84" seLinuxMountContext="" Oct 30 11:54:03.572597 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.571967 4908 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b1757617-7c51-4029-b329-1b6a4630f860" volumeName="kubernetes.io/configmap/b1757617-7c51-4029-b329-1b6a4630f860-trusted-ca-bundle" seLinuxMountContext="" Oct 30 11:54:03.572597 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.571979 4908 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b1757617-7c51-4029-b329-1b6a4630f860" volumeName="kubernetes.io/empty-dir/b1757617-7c51-4029-b329-1b6a4630f860-tmp" seLinuxMountContext="" Oct 30 11:54:03.572597 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.571990 4908 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1d7f8396-e57c-45e5-ae74-a2244db10364" volumeName="kubernetes.io/projected/1d7f8396-e57c-45e5-ae74-a2244db10364-kube-api-access-bzwcn" seLinuxMountContext="" Oct 30 11:54:03.572597 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.572001 4908 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1f5667a1-a773-4520-b2fb-c69148df632c" volumeName="kubernetes.io/projected/1f5667a1-a773-4520-b2fb-c69148df632c-kube-api-access-bzld2" seLinuxMountContext="" Oct 30 11:54:03.572597 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.572012 4908 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7e8ec7ff-015c-4049-9c5f-aae73dc06e4b" volumeName="kubernetes.io/configmap/7e8ec7ff-015c-4049-9c5f-aae73dc06e4b-iptables-alerter-script" seLinuxMountContext="" Oct 30 11:54:03.572597 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.572024 4908 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b1757617-7c51-4029-b329-1b6a4630f860" volumeName="kubernetes.io/configmap/b1757617-7c51-4029-b329-1b6a4630f860-service-ca-bundle" seLinuxMountContext="" Oct 30 11:54:03.572597 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.572035 4908 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cfaad7d0-d30b-4ddc-81cc-2b603416fc29" volumeName="kubernetes.io/secret/cfaad7d0-d30b-4ddc-81cc-2b603416fc29-installation-pull-secrets" seLinuxMountContext="" Oct 30 11:54:03.572597 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.572045 4908 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0df1a118-36a5-4c57-8cec-2c8236f1c4de" volumeName="kubernetes.io/configmap/0df1a118-36a5-4c57-8cec-2c8236f1c4de-cni-binary-copy" seLinuxMountContext="" Oct 30 11:54:03.572597 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.572056 4908 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3699031f-ee9e-46cc-b37e-4dfc76b9c0b1" volumeName="kubernetes.io/projected/3699031f-ee9e-46cc-b37e-4dfc76b9c0b1-kube-api-access-r7vnf" seLinuxMountContext="" Oct 30 11:54:03.572597 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.572068 4908 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b1757617-7c51-4029-b329-1b6a4630f860" volumeName="kubernetes.io/empty-dir/b1757617-7c51-4029-b329-1b6a4630f860-snapshots" seLinuxMountContext="" Oct 30 11:54:03.572597 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.572079 4908 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bd977255-f9a7-4531-8a9c-b1a3a17e0b79" volumeName="kubernetes.io/configmap/bd977255-f9a7-4531-8a9c-b1a3a17e0b79-trusted-ca" seLinuxMountContext="" Oct 30 11:54:03.572597 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.572091 4908 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="003c3d18-23f1-45d0-bac4-8c6eea4b7e99" volumeName="kubernetes.io/secret/003c3d18-23f1-45d0-bac4-8c6eea4b7e99-cert" seLinuxMountContext="" Oct 30 11:54:03.572597 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.572102 4908 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3699031f-ee9e-46cc-b37e-4dfc76b9c0b1" volumeName="kubernetes.io/empty-dir/3699031f-ee9e-46cc-b37e-4dfc76b9c0b1-tmp" seLinuxMountContext="" Oct 30 11:54:03.572597 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.572113 4908 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7e8ec7ff-015c-4049-9c5f-aae73dc06e4b" volumeName="kubernetes.io/projected/7e8ec7ff-015c-4049-9c5f-aae73dc06e4b-kube-api-access-92smt" seLinuxMountContext="" Oct 30 11:54:03.572597 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.572126 4908 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d0e0cec2-f16a-4bae-bc70-6c08da4bda2e" volumeName="kubernetes.io/projected/d0e0cec2-f16a-4bae-bc70-6c08da4bda2e-kube-api-access-h7mvn" seLinuxMountContext="" Oct 30 11:54:03.572597 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.572137 4908 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d659d471-f019-4853-97d0-ee6e948e6c1c" volumeName="kubernetes.io/configmap/d659d471-f019-4853-97d0-ee6e948e6c1c-cni-binary-copy" seLinuxMountContext="" Oct 30 11:54:03.572597 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.572149 4908 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6acb13a1-cb2e-45e1-897b-3ef89e6d1b84" volumeName="kubernetes.io/projected/6acb13a1-cb2e-45e1-897b-3ef89e6d1b84-kube-api-access-2rxc4" seLinuxMountContext="" Oct 30 11:54:03.572597 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.572160 4908 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b3f20243-155c-4e25-a1ca-6131a141fbd4" volumeName="kubernetes.io/secret/b3f20243-155c-4e25-a1ca-6131a141fbd4-serving-cert" seLinuxMountContext="" Oct 30 11:54:03.572597 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.572171 4908 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cfaad7d0-d30b-4ddc-81cc-2b603416fc29" volumeName="kubernetes.io/projected/cfaad7d0-d30b-4ddc-81cc-2b603416fc29-bound-sa-token" seLinuxMountContext="" Oct 30 11:54:03.572597 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.572183 4908 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dab8eab1-3e3f-46b4-b76d-d58a7061996d" volumeName="kubernetes.io/projected/dab8eab1-3e3f-46b4-b76d-d58a7061996d-kube-api-access-7zd2h" seLinuxMountContext="" Oct 30 11:54:03.572597 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.572196 4908 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7689cfe5-5a4e-4481-8eb9-107a58ca8bb3" volumeName="kubernetes.io/projected/7689cfe5-5a4e-4481-8eb9-107a58ca8bb3-kube-api-access-gr67s" seLinuxMountContext="" Oct 30 11:54:03.572597 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.572206 4908 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d659d471-f019-4853-97d0-ee6e948e6c1c" volumeName="kubernetes.io/configmap/d659d471-f019-4853-97d0-ee6e948e6c1c-cni-sysctl-allowlist" seLinuxMountContext="" Oct 30 11:54:03.572597 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.572218 4908 reconstruct.go:97] "Volume reconstruction finished" Oct 30 11:54:03.572597 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.572225 4908 reconciler.go:29] "Reconciler: start to sync state" Oct 30 11:54:03.586878 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.586859 4908 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-plczt_a70ee11c-c20e-4a4f-8476-ac5eb02816d0/console-operator/1.log" Oct 30 11:54:03.587210 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.587180 4908 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-plczt_a70ee11c-c20e-4a4f-8476-ac5eb02816d0/console-operator/1.log" Oct 30 11:54:03.592341 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.592321 4908 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b8bxz_644e19b3-10ad-4145-9ed3-595d19ed9b07/ovn-acl-logging/0.log" Oct 30 11:54:03.594554 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.594533 4908 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b8bxz_644e19b3-10ad-4145-9ed3-595d19ed9b07/ovn-acl-logging/0.log" Oct 30 11:54:03.595964 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.595929 4908 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 11:54:03.597468 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.597445 4908 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 11:54:03.597468 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.597469 4908 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 11:54:03.597594 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.597488 4908 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 11:54:03.597594 ip-10-0-141-251 kubenswrapper[4908]: E1030 11:54:03.597538 4908 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 30 11:54:03.615853 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.615807 4908 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 11:54:03.616484 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.616468 4908 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 30 11:54:03.620959 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.620939 4908 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b8bxz_644e19b3-10ad-4145-9ed3-595d19ed9b07/ovn-acl-logging/0.log" Oct 30 11:54:03.621345 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.621306 4908 generic.go:356] "Generic (PLEG): container finished" podID="644e19b3-10ad-4145-9ed3-595d19ed9b07" containerID="57452585cea9d773026a664a5799328c1d6d6edbe6d3bfee62b1b81185fb9e49" exitCode=1 Oct 30 11:54:03.627104 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.627075 4908 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 11:54:03.627755 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.627738 4908 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 11:54:03.627874 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.627802 4908 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 11:54:03.641732 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.641711 4908 generic.go:356] "Generic (PLEG): container finished" podID="d659d471-f019-4853-97d0-ee6e948e6c1c" containerID="0560e630808cca1e6dc6737abf733037f23162365ee250168087828768b9c1e4" exitCode=0 Oct 30 11:54:03.641732 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.641730 4908 generic.go:356] "Generic (PLEG): container finished" podID="d659d471-f019-4853-97d0-ee6e948e6c1c" containerID="754cf3c1b2b2f1490fad873f3d8ed1599e72bf26e97af18acedce0b7c02e2de4" exitCode=0 Oct 30 11:54:03.641878 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.641737 4908 generic.go:356] "Generic (PLEG): container finished" podID="d659d471-f019-4853-97d0-ee6e948e6c1c" containerID="d835da50c2f060635d05553a42b19e8d329fc82dda51e30e72a9c80e6d158927" exitCode=0 Oct 30 11:54:03.641878 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.641742 4908 generic.go:356] "Generic (PLEG): container finished" podID="d659d471-f019-4853-97d0-ee6e948e6c1c" containerID="c2ed6eb3eaa2e73b85df6a08e944f52c7c943bc396100d50b8408c507567f226" exitCode=0 Oct 30 11:54:03.641878 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.641747 4908 generic.go:356] "Generic (PLEG): container finished" podID="d659d471-f019-4853-97d0-ee6e948e6c1c" containerID="8a6272e6ebb582eb3df7452766fa897f55e19ef3a221218bca60e352ba32c733" exitCode=0 Oct 30 11:54:03.641878 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.641752 4908 generic.go:356] "Generic (PLEG): container finished" podID="d659d471-f019-4853-97d0-ee6e948e6c1c" containerID="0e465df6f612805073206b30b3905501fa2f2d3a254412f170c0d262bfa82f73" exitCode=0 Oct 30 11:54:03.643623 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.643598 4908 generic.go:356] "Generic (PLEG): container finished" podID="f140bf37195964da2b6ead5845b42a26" containerID="c2e4fd77c6ca53931d448c42822c6f59bf2ff55f29062c62ae4d6869151cbeea" exitCode=0 Oct 30 11:54:03.646505 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.646482 4908 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-plczt_a70ee11c-c20e-4a4f-8476-ac5eb02816d0/console-operator/1.log" Oct 30 11:54:03.646505 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.646501 4908 generic.go:356] "Generic (PLEG): container finished" podID="a70ee11c-c20e-4a4f-8476-ac5eb02816d0" containerID="cc5a67034e1729623ea0114fef781e8204b8eab853c4f28263d4d9d9c45f99fd" exitCode=255 Oct 30 11:54:03.661005 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.660957 4908 manager.go:324] Recovery completed Oct 30 11:54:03.679653 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.679633 4908 cpu_manager.go:226] "Starting" policy="none" Oct 30 11:54:03.679653 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.679644 4908 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 11:54:03.679758 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.679659 4908 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 11:54:03.679789 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.679760 4908 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 30 11:54:03.679789 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.679770 4908 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 30 11:54:03.679789 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.679785 4908 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 30 11:54:03.679900 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.679790 4908 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 30 11:54:03.679900 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.679823 4908 policy_none.go:50] "Start" Oct 30 11:54:03.679900 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.679892 4908 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 11:54:03.679977 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.679901 4908 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 11:54:03.680034 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.680024 4908 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 30 11:54:03.680068 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.680034 4908 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 30 11:54:03.680068 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.680040 4908 policy_none.go:44] "Start" Oct 30 11:54:03.684039 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.684017 4908 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 11:54:03.686628 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.686590 4908 manager.go:341] "Starting Device Plugin manager" Oct 30 11:54:03.687285 ip-10-0-141-251 kubenswrapper[4908]: E1030 11:54:03.686625 4908 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 11:54:03.687327 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.687291 4908 server.go:87] "Starting device plugin registration server" Oct 30 11:54:03.687619 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.687606 4908 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 11:54:03.687659 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.687621 4908 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 11:54:03.687704 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.687688 4908 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 11:54:03.687799 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.687767 4908 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 11:54:03.687872 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.687806 4908 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 11:54:03.687872 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.687813 4908 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 11:54:03.687962 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.687869 4908 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T11:54:03.687796603Z","UUID":"a4cc7404-67cf-419d-9476-0cf35a9629ed","Handler":null,"Name":"","Endpoint":""} Oct 30 11:54:03.689169 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.689149 4908 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 11:54:03.689253 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.689189 4908 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 11:54:03.689253 ip-10-0-141-251 kubenswrapper[4908]: E1030 11:54:03.689237 4908 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 11:54:03.697700 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.697656 4908 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-251.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-141-251.ec2.internal"] Oct 30 11:54:03.698359 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.698342 4908 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-251.ec2.internal" Oct 30 11:54:03.698359 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.698336 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-251.ec2.internal" event={"ID":"a73cae35bcc0a4b145486ce8e9e460c1","Type":"ContainerStarted","Data":"f638e0c45c2e95a85b16efaef0676cb14b6e83ca7bb639fed7a9ea6e571ee59c"} Oct 30 11:54:03.698512 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.698377 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-251.ec2.internal" event={"ID":"a73cae35bcc0a4b145486ce8e9e460c1","Type":"ContainerStarted","Data":"1b6e44060e4775c8234fb5f6b195c11bc0f4beb0b731852b60d9d07f2b70e864"} Oct 30 11:54:03.698512 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.698444 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-251.ec2.internal" event={"ID":"f140bf37195964da2b6ead5845b42a26","Type":"ContainerStarted","Data":"ad4d99f88459a1f4749ea8972b817f7cb94acf7cba423ba8b3e0cf06c3e4176e"} Oct 30 11:54:03.698512 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.698466 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-251.ec2.internal" event={"ID":"f140bf37195964da2b6ead5845b42a26","Type":"ContainerDied","Data":"c2e4fd77c6ca53931d448c42822c6f59bf2ff55f29062c62ae4d6869151cbeea"} Oct 30 11:54:03.698512 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.698482 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-251.ec2.internal" event={"ID":"f140bf37195964da2b6ead5845b42a26","Type":"ContainerStarted","Data":"c7e539754dfdda2f33506f8d6a7dc8acd027ef96d8d03b975eff48d2c6fc59bd"} Oct 30 11:54:03.698638 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.698533 4908 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-251.ec2.internal" Oct 30 11:54:03.723672 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.723654 4908 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 11:54:03.723749 ip-10-0-141-251 kubenswrapper[4908]: E1030 11:54:03.723690 4908 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-141-251.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-251.ec2.internal" Oct 30 11:54:03.724246 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.724225 4908 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 11:54:03.724319 ip-10-0-141-251 kubenswrapper[4908]: E1030 11:54:03.724265 4908 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-141-251.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-251.ec2.internal" Oct 30 11:54:03.788953 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.788936 4908 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 11:54:03.789951 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.789937 4908 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-251.ec2.internal" event="NodeHasSufficientMemory" Oct 30 11:54:03.790018 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.789961 4908 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-251.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 11:54:03.790018 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.789972 4908 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-251.ec2.internal" event="NodeHasSufficientPID" Oct 30 11:54:03.790018 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.790009 4908 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-141-251.ec2.internal" Oct 30 11:54:03.817391 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.817371 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f140bf37195964da2b6ead5845b42a26-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-251.ec2.internal\" (UID: \"f140bf37195964da2b6ead5845b42a26\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-251.ec2.internal" Oct 30 11:54:03.817455 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.817395 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a73cae35bcc0a4b145486ce8e9e460c1-config\") pod \"kube-apiserver-proxy-ip-10-0-141-251.ec2.internal\" (UID: \"a73cae35bcc0a4b145486ce8e9e460c1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-251.ec2.internal" Oct 30 11:54:03.817455 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.817411 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f140bf37195964da2b6ead5845b42a26-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-251.ec2.internal\" (UID: \"f140bf37195964da2b6ead5845b42a26\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-251.ec2.internal" Oct 30 11:54:03.822959 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.822942 4908 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-141-251.ec2.internal" Oct 30 11:54:03.823004 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.822989 4908 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-141-251.ec2.internal" Oct 30 11:54:03.918518 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.918499 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a73cae35bcc0a4b145486ce8e9e460c1-config\") pod \"kube-apiserver-proxy-ip-10-0-141-251.ec2.internal\" (UID: \"a73cae35bcc0a4b145486ce8e9e460c1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-251.ec2.internal" Oct 30 11:54:03.918602 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.918523 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f140bf37195964da2b6ead5845b42a26-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-251.ec2.internal\" (UID: \"f140bf37195964da2b6ead5845b42a26\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-251.ec2.internal" Oct 30 11:54:03.918602 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.918546 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f140bf37195964da2b6ead5845b42a26-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-251.ec2.internal\" (UID: \"f140bf37195964da2b6ead5845b42a26\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-251.ec2.internal" Oct 30 11:54:03.918602 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.918594 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f140bf37195964da2b6ead5845b42a26-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-251.ec2.internal\" (UID: \"f140bf37195964da2b6ead5845b42a26\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-251.ec2.internal" Oct 30 11:54:03.918725 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.918603 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a73cae35bcc0a4b145486ce8e9e460c1-config\") pod \"kube-apiserver-proxy-ip-10-0-141-251.ec2.internal\" (UID: \"a73cae35bcc0a4b145486ce8e9e460c1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-251.ec2.internal" Oct 30 11:54:03.918725 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:03.918626 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f140bf37195964da2b6ead5845b42a26-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-251.ec2.internal\" (UID: \"f140bf37195964da2b6ead5845b42a26\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-251.ec2.internal" Oct 30 11:54:04.555575 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.555553 4908 apiserver.go:52] "Watching apiserver" Oct 30 11:54:04.604664 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.604641 4908 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 11:54:04.605484 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.605464 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-5bfdf864c-tzv67","openshift-service-ca-operator/service-ca-operator-544d749c8c-6l6fr","openshift-dns/dns-default-xxcbn","openshift-image-registry/image-registry-79fd565486-8fl78","openshift-insights/insights-operator-7c4986d449-2fqgf","kube-system/kas-connection-checker-685895c7c9-ccctl","openshift-network-diagnostics/network-check-source-6f78d59f99-glvhs","openshift-cluster-node-tuning-operator/tuned-587rx","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-251.ec2.internal","openshift-monitoring/cluster-monitoring-operator-884576c57-mvhwk","openshift-multus/network-metrics-daemon-dbqhj","kube-system/global-pull-secret-syncer-fxjlt","kube-system/kas-connection-checker-685895c7c9-kg9td","openshift-image-registry/image-registry-557c6b9f47-vcr2t","openshift-insights/insights-runtime-extractor-vn6vv","openshift-network-console/networking-console-plugin-86c478bb88-zzxhv","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-86p78","openshift-network-operator/iptables-alerter-8m2w9","openshift-multus/multus-tlwp4","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-ksxpw","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xdxkp","openshift-console-operator/console-operator-7cd56b8dbc-plczt","openshift-multus/multus-additional-cni-plugins-s6k8x","kube-system/kas-connection-checker-685895c7c9-k8ldb","kube-system/kube-apiserver-proxy-ip-10-0-141-251.ec2.internal","openshift-dns/node-resolver-xmrdj","openshift-network-diagnostics/network-check-target-bz65k","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h6mlf","openshift-kube-storage-version-migrator/migrator-66df5d496b-4brks","openshift-ingress-canary/ingress-canary-gn2z5","kube-system/konnectivity-agent-tdf8d","openshift-image-registry/node-ca-hb8j7","openshift-ovn-kubernetes/ovnkube-node-b8bxz"] Oct 30 11:54:04.610744 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.610720 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xdxkp" Oct 30 11:54:04.611140 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.611124 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5bfdf864c-tzv67" Oct 30 11:54:04.611427 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.611408 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-mvhwk" Oct 30 11:54:04.612546 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.612530 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-557c6b9f47-vcr2t" Oct 30 11:54:04.612861 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.612845 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-zzxhv" Oct 30 11:54:04.613297 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.613279 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dbqhj" Oct 30 11:54:04.617597 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.617581 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-79fd565486-8fl78" Oct 30 11:54:04.619907 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.619885 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xxcbn" Oct 30 11:54:04.620878 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.620861 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-vn6vv" Oct 30 11:54:04.629683 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.629662 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-shg4w\"" Oct 30 11:54:04.629759 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.629669 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-b9dpg\"" Oct 30 11:54:04.637456 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.637439 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Oct 30 11:54:04.638360 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.638344 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Oct 30 11:54:04.638443 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.638405 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-96ksc\"" Oct 30 11:54:04.639257 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.639240 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Oct 30 11:54:04.639366 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.639264 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Oct 30 11:54:04.639366 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.639278 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Oct 30 11:54:04.639366 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.639282 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-qndch\"" Oct 30 11:54:04.639366 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.639292 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 11:54:04.639366 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.639268 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Oct 30 11:54:04.639366 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.639239 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Oct 30 11:54:04.639366 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.639269 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Oct 30 11:54:04.639366 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.639329 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Oct 30 11:54:04.639366 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.639328 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-s95lg\"" Oct 30 11:54:04.639366 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.639287 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Oct 30 11:54:04.639739 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.639418 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 11:54:04.639739 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.639465 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 11:54:04.639739 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.639479 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-6hs8x\"" Oct 30 11:54:04.639739 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.639493 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-jxkrq\"" Oct 30 11:54:04.639739 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.639640 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 11:54:04.639739 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.639639 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 11:54:04.639739 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.639662 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 11:54:04.639739 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.639669 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Oct 30 11:54:04.639739 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.639667 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-m27bb\"" Oct 30 11:54:04.639739 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.639694 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-cng9g\"" Oct 30 11:54:04.640157 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.639757 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-8pg5x\"" Oct 30 11:54:04.640157 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.639874 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Oct 30 11:54:04.640157 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.639890 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 11:54:04.640157 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.639899 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-rpjw7\"" Oct 30 11:54:04.640157 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.639915 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-vcl76\"" Oct 30 11:54:04.640157 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.639894 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-kv86z\"" Oct 30 11:54:04.640157 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.639932 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 11:54:04.640157 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.639928 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-tdxpc\"" Oct 30 11:54:04.640157 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.639944 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-jjcjh\"" Oct 30 11:54:04.640157 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.639945 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-zfblq\"" Oct 30 11:54:04.640157 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.639958 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-8n4dr\"" Oct 30 11:54:04.640157 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.639976 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-llkx2\"" Oct 30 11:54:04.640157 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.639989 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-tz47t\"" Oct 30 11:54:04.640157 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.640023 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-sjprl\"" Oct 30 11:54:04.640157 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.640066 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-7cm7r\"" Oct 30 11:54:04.640157 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.640080 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-sw2hc\"" Oct 30 11:54:04.640157 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.640138 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-zjl8c\"" Oct 30 11:54:04.640651 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.640170 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-lrxbh\"" Oct 30 11:54:04.640651 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.640244 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-f2lr4\"" Oct 30 11:54:04.640651 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.640272 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-bs4nn\"" Oct 30 11:54:04.640651 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.640314 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-pxw6w\"" Oct 30 11:54:04.640765 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.640734 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Oct 30 11:54:04.652171 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.652154 4908 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-251.ec2.internal" Oct 30 11:54:04.652285 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.652274 4908 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-251.ec2.internal" Oct 30 11:54:04.670225 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.670208 4908 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 11:54:04.674299 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.674284 4908 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 11:54:04.674360 ip-10-0-141-251 kubenswrapper[4908]: E1030 11:54:04.674323 4908 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-141-251.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-251.ec2.internal" Oct 30 11:54:04.674360 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.674342 4908 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 11:54:04.674457 ip-10-0-141-251 kubenswrapper[4908]: E1030 11:54:04.674377 4908 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-141-251.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-251.ec2.internal" Oct 30 11:54:04.679728 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.679708 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 11:54:04.683543 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.683524 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Oct 30 11:54:04.683639 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.683607 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Oct 30 11:54:04.683639 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.683607 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 11:54:04.683738 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.683699 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 30 11:54:04.683738 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.683708 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Oct 30 11:54:04.683804 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.683740 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 11:54:04.683943 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.683925 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 11:54:04.684159 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.684148 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 11:54:04.684217 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.684206 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Oct 30 11:54:04.684419 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.684408 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 11:54:04.684419 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.684415 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 11:54:04.684518 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.684448 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 11:54:04.684518 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.684448 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 11:54:04.684518 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.684476 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 11:54:04.684518 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.684487 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 11:54:04.684703 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.684682 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 11:54:04.684794 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.684708 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 11:54:04.684794 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.684737 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 11:54:04.684794 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.684743 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Oct 30 11:54:04.684794 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.684748 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 11:54:04.684794 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.684766 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 11:54:04.684794 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.684772 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Oct 30 11:54:04.684794 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.684745 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 11:54:04.684794 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.684792 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 11:54:04.685224 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.684801 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 11:54:04.685224 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.684810 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 11:54:04.685224 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.684811 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 11:54:04.685224 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.684866 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Oct 30 11:54:04.685224 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.685026 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Oct 30 11:54:04.685224 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.685026 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Oct 30 11:54:04.685224 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.685042 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Oct 30 11:54:04.685224 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.685027 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 11:54:04.685224 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.685051 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 11:54:04.685224 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.685053 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Oct 30 11:54:04.685224 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.685067 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Oct 30 11:54:04.685224 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.685076 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Oct 30 11:54:04.685224 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.685055 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Oct 30 11:54:04.685224 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.685078 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Oct 30 11:54:04.685224 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.685144 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Oct 30 11:54:04.686012 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.685246 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 11:54:04.686012 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.685276 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 11:54:04.686012 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.685285 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 11:54:04.686012 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.685291 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 11:54:04.686012 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.685298 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Oct 30 11:54:04.686012 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.685389 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 11:54:04.686012 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.685398 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Oct 30 11:54:04.686012 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.685404 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Oct 30 11:54:04.686012 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.685402 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 11:54:04.686012 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.685409 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 11:54:04.686012 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.685424 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 11:54:04.686012 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.685430 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 11:54:04.686012 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.685436 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Oct 30 11:54:04.686012 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.685461 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 11:54:04.686012 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.685468 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 11:54:04.686012 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.685440 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 11:54:04.686543 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.686065 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 11:54:04.695555 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.695538 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Oct 30 11:54:04.698843 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.698791 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Oct 30 11:54:04.698843 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.698803 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 30 11:54:04.699181 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.699149 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Oct 30 11:54:04.722486 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.722468 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b1757617-7c51-4029-b329-1b6a4630f860-serving-cert\") pod \"insights-operator-7c4986d449-2fqgf\" (UID: \"b1757617-7c51-4029-b329-1b6a4630f860\") " pod="openshift-insights/insights-operator-7c4986d449-2fqgf" Oct 30 11:54:04.722574 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.722494 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/644e19b3-10ad-4145-9ed3-595d19ed9b07-host-run-ovn-kubernetes\") pod \"ovnkube-node-b8bxz\" (UID: \"644e19b3-10ad-4145-9ed3-595d19ed9b07\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8bxz" Oct 30 11:54:04.722574 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.722511 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rh4tb\" (UniqueName: \"kubernetes.io/projected/c54f4c1b-f08e-4495-9a14-17469a792ae5-kube-api-access-rh4tb\") pod \"aws-ebs-csi-driver-node-h6mlf\" (UID: \"c54f4c1b-f08e-4495-9a14-17469a792ae5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h6mlf" Oct 30 11:54:04.722574 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.722525 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a70ee11c-c20e-4a4f-8476-ac5eb02816d0-serving-cert\") pod \"console-operator-7cd56b8dbc-plczt\" (UID: \"a70ee11c-c20e-4a4f-8476-ac5eb02816d0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-plczt" Oct 30 11:54:04.722574 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.722542 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/644e19b3-10ad-4145-9ed3-595d19ed9b07-var-lib-openvswitch\") pod \"ovnkube-node-b8bxz\" (UID: \"644e19b3-10ad-4145-9ed3-595d19ed9b07\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8bxz" Oct 30 11:54:04.722778 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.722603 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/644e19b3-10ad-4145-9ed3-595d19ed9b07-ovnkube-config\") pod \"ovnkube-node-b8bxz\" (UID: \"644e19b3-10ad-4145-9ed3-595d19ed9b07\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8bxz" Oct 30 11:54:04.722778 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.722630 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c54f4c1b-f08e-4495-9a14-17469a792ae5-etc-selinux\") pod \"aws-ebs-csi-driver-node-h6mlf\" (UID: \"c54f4c1b-f08e-4495-9a14-17469a792ae5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h6mlf" Oct 30 11:54:04.722778 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.722648 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/003c3d18-23f1-45d0-bac4-8c6eea4b7e99-cert\") pod \"ingress-canary-gn2z5\" (UID: \"003c3d18-23f1-45d0-bac4-8c6eea4b7e99\") " pod="openshift-ingress-canary/ingress-canary-gn2z5" Oct 30 11:54:04.722778 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.722664 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cfaad7d0-d30b-4ddc-81cc-2b603416fc29-image-registry-private-configuration\") pod \"image-registry-79fd565486-8fl78\" (UID: \"cfaad7d0-d30b-4ddc-81cc-2b603416fc29\") " pod="openshift-image-registry/image-registry-79fd565486-8fl78" Oct 30 11:54:04.722778 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.722705 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d659d471-f019-4853-97d0-ee6e948e6c1c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-s6k8x\" (UID: \"d659d471-f019-4853-97d0-ee6e948e6c1c\") " pod="openshift-multus/multus-additional-cni-plugins-s6k8x" Oct 30 11:54:04.722778 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.722746 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b1757617-7c51-4029-b329-1b6a4630f860-serving-cert\") pod \"insights-operator-7c4986d449-2fqgf\" (UID: \"b1757617-7c51-4029-b329-1b6a4630f860\") " pod="openshift-insights/insights-operator-7c4986d449-2fqgf" Oct 30 11:54:04.722990 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.722871 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d670b364-53ce-444b-b9c4-ac8ac09cd44f-data-volume\") pod \"insights-runtime-extractor-vn6vv\" (UID: \"d670b364-53ce-444b-b9c4-ac8ac09cd44f\") " pod="openshift-insights/insights-runtime-extractor-vn6vv" Oct 30 11:54:04.722990 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.722906 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jf44s\" (UniqueName: \"kubernetes.io/projected/c815b3fc-7f35-4ae6-a015-2e84c6108a64-kube-api-access-jf44s\") pod \"service-ca-operator-544d749c8c-6l6fr\" (UID: \"c815b3fc-7f35-4ae6-a015-2e84c6108a64\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6l6fr" Oct 30 11:54:04.722990 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.722917 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a70ee11c-c20e-4a4f-8476-ac5eb02816d0-serving-cert\") pod \"console-operator-7cd56b8dbc-plczt\" (UID: \"a70ee11c-c20e-4a4f-8476-ac5eb02816d0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-plczt" Oct 30 11:54:04.722990 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.722932 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fhb8\" (UniqueName: \"kubernetes.io/projected/6d976900-ecd0-469c-a9e2-d60708361f1a-kube-api-access-5fhb8\") pod \"cluster-monitoring-operator-884576c57-mvhwk\" (UID: \"6d976900-ecd0-469c-a9e2-d60708361f1a\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-mvhwk" Oct 30 11:54:04.722990 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.722947 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/003c3d18-23f1-45d0-bac4-8c6eea4b7e99-cert\") pod \"ingress-canary-gn2z5\" (UID: \"003c3d18-23f1-45d0-bac4-8c6eea4b7e99\") " pod="openshift-ingress-canary/ingress-canary-gn2z5" Oct 30 11:54:04.722990 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.722958 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cfaad7d0-d30b-4ddc-81cc-2b603416fc29-image-registry-private-configuration\") pod \"image-registry-79fd565486-8fl78\" (UID: \"cfaad7d0-d30b-4ddc-81cc-2b603416fc29\") " pod="openshift-image-registry/image-registry-79fd565486-8fl78" Oct 30 11:54:04.722990 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.722967 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/b1757617-7c51-4029-b329-1b6a4630f860-snapshots\") pod \"insights-operator-7c4986d449-2fqgf\" (UID: \"b1757617-7c51-4029-b329-1b6a4630f860\") " pod="openshift-insights/insights-operator-7c4986d449-2fqgf" Oct 30 11:54:04.723194 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.722994 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/644e19b3-10ad-4145-9ed3-595d19ed9b07-systemd-units\") pod \"ovnkube-node-b8bxz\" (UID: \"644e19b3-10ad-4145-9ed3-595d19ed9b07\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8bxz" Oct 30 11:54:04.723194 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.723008 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d659d471-f019-4853-97d0-ee6e948e6c1c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-s6k8x\" (UID: \"d659d471-f019-4853-97d0-ee6e948e6c1c\") " pod="openshift-multus/multus-additional-cni-plugins-s6k8x" Oct 30 11:54:04.723194 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.723017 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3699031f-ee9e-46cc-b37e-4dfc76b9c0b1-run\") pod \"tuned-587rx\" (UID: \"3699031f-ee9e-46cc-b37e-4dfc76b9c0b1\") " pod="openshift-cluster-node-tuning-operator/tuned-587rx" Oct 30 11:54:04.723194 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.723024 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d670b364-53ce-444b-b9c4-ac8ac09cd44f-data-volume\") pod \"insights-runtime-extractor-vn6vv\" (UID: \"d670b364-53ce-444b-b9c4-ac8ac09cd44f\") " pod="openshift-insights/insights-runtime-extractor-vn6vv" Oct 30 11:54:04.723194 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.723044 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b3f20243-155c-4e25-a1ca-6131a141fbd4-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-86p78\" (UID: \"b3f20243-155c-4e25-a1ca-6131a141fbd4\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-86p78" Oct 30 11:54:04.723194 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.723017 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/644e19b3-10ad-4145-9ed3-595d19ed9b07-ovnkube-config\") pod \"ovnkube-node-b8bxz\" (UID: \"644e19b3-10ad-4145-9ed3-595d19ed9b07\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8bxz" Oct 30 11:54:04.723194 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.723076 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/b1757617-7c51-4029-b329-1b6a4630f860-snapshots\") pod \"insights-operator-7c4986d449-2fqgf\" (UID: \"b1757617-7c51-4029-b329-1b6a4630f860\") " pod="openshift-insights/insights-operator-7c4986d449-2fqgf" Oct 30 11:54:04.723194 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.723086 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tp27x\" (UniqueName: \"kubernetes.io/projected/8e44595b-0bdf-407c-bf22-33eb06272cc1-kube-api-access-tp27x\") pod \"kas-connection-checker-685895c7c9-k8ldb\" (UID: \"8e44595b-0bdf-407c-bf22-33eb06272cc1\") " pod="kube-system/kas-connection-checker-685895c7c9-k8ldb" Oct 30 11:54:04.723194 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.723178 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6d976900-ecd0-469c-a9e2-d60708361f1a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-mvhwk\" (UID: \"6d976900-ecd0-469c-a9e2-d60708361f1a\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-mvhwk" Oct 30 11:54:04.723483 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.723216 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b3f20243-155c-4e25-a1ca-6131a141fbd4-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-86p78\" (UID: \"b3f20243-155c-4e25-a1ca-6131a141fbd4\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-86p78" Oct 30 11:54:04.723483 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.723241 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/644e19b3-10ad-4145-9ed3-595d19ed9b07-run-ovn\") pod \"ovnkube-node-b8bxz\" (UID: \"644e19b3-10ad-4145-9ed3-595d19ed9b07\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8bxz" Oct 30 11:54:04.723483 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.723270 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4e18f771-273b-49e6-872c-6cbe47ab68c4-agent-certs\") pod \"konnectivity-agent-tdf8d\" (UID: \"4e18f771-273b-49e6-872c-6cbe47ab68c4\") " pod="kube-system/konnectivity-agent-tdf8d" Oct 30 11:54:04.723483 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.723292 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/6acb13a1-cb2e-45e1-897b-3ef89e6d1b84-default-certificate\") pod \"router-default-5bfdf864c-tzv67\" (UID: \"6acb13a1-cb2e-45e1-897b-3ef89e6d1b84\") " pod="openshift-ingress/router-default-5bfdf864c-tzv67" Oct 30 11:54:04.723483 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.723306 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cfaad7d0-d30b-4ddc-81cc-2b603416fc29-registry-certificates\") pod \"image-registry-79fd565486-8fl78\" (UID: \"cfaad7d0-d30b-4ddc-81cc-2b603416fc29\") " pod="openshift-image-registry/image-registry-79fd565486-8fl78" Oct 30 11:54:04.723483 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.723327 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhxmm\" (UniqueName: \"kubernetes.io/projected/221ced85-cc15-4e07-82d9-a1a1f3e958a5-kube-api-access-bhxmm\") pod \"dns-default-xxcbn\" (UID: \"221ced85-cc15-4e07-82d9-a1a1f3e958a5\") " pod="openshift-dns/dns-default-xxcbn" Oct 30 11:54:04.723483 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.723355 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bd977255-f9a7-4531-8a9c-b1a3a17e0b79-trusted-ca\") pod \"image-registry-557c6b9f47-vcr2t\" (UID: \"bd977255-f9a7-4531-8a9c-b1a3a17e0b79\") " pod="openshift-image-registry/image-registry-557c6b9f47-vcr2t" Oct 30 11:54:04.723483 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.723381 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0df1a118-36a5-4c57-8cec-2c8236f1c4de-host-var-lib-kubelet\") pod \"multus-tlwp4\" (UID: \"0df1a118-36a5-4c57-8cec-2c8236f1c4de\") " pod="openshift-multus/multus-tlwp4" Oct 30 11:54:04.723483 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.723402 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0df1a118-36a5-4c57-8cec-2c8236f1c4de-multus-daemon-config\") pod \"multus-tlwp4\" (UID: \"0df1a118-36a5-4c57-8cec-2c8236f1c4de\") " pod="openshift-multus/multus-tlwp4" Oct 30 11:54:04.723483 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.723427 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c54f4c1b-f08e-4495-9a14-17469a792ae5-registration-dir\") pod \"aws-ebs-csi-driver-node-h6mlf\" (UID: \"c54f4c1b-f08e-4495-9a14-17469a792ae5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h6mlf" Oct 30 11:54:04.723483 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.723480 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cfaad7d0-d30b-4ddc-81cc-2b603416fc29-registry-certificates\") pod \"image-registry-79fd565486-8fl78\" (UID: \"cfaad7d0-d30b-4ddc-81cc-2b603416fc29\") " pod="openshift-image-registry/image-registry-79fd565486-8fl78" Oct 30 11:54:04.723975 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.723530 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/6acb13a1-cb2e-45e1-897b-3ef89e6d1b84-default-certificate\") pod \"router-default-5bfdf864c-tzv67\" (UID: \"6acb13a1-cb2e-45e1-897b-3ef89e6d1b84\") " pod="openshift-ingress/router-default-5bfdf864c-tzv67" Oct 30 11:54:04.723975 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.723547 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4e18f771-273b-49e6-872c-6cbe47ab68c4-agent-certs\") pod \"konnectivity-agent-tdf8d\" (UID: \"4e18f771-273b-49e6-872c-6cbe47ab68c4\") " pod="kube-system/konnectivity-agent-tdf8d" Oct 30 11:54:04.723975 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.723563 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zscd2\" (UniqueName: \"kubernetes.io/projected/b1757617-7c51-4029-b329-1b6a4630f860-kube-api-access-zscd2\") pod \"insights-operator-7c4986d449-2fqgf\" (UID: \"b1757617-7c51-4029-b329-1b6a4630f860\") " pod="openshift-insights/insights-operator-7c4986d449-2fqgf" Oct 30 11:54:04.723975 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.723594 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzc5h\" (UniqueName: \"kubernetes.io/projected/1b735438-5947-4374-bd4c-66ac3a299770-kube-api-access-gzc5h\") pod \"node-resolver-xmrdj\" (UID: \"1b735438-5947-4374-bd4c-66ac3a299770\") " pod="openshift-dns/node-resolver-xmrdj" Oct 30 11:54:04.723975 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.723620 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/644e19b3-10ad-4145-9ed3-595d19ed9b07-host-slash\") pod \"ovnkube-node-b8bxz\" (UID: \"644e19b3-10ad-4145-9ed3-595d19ed9b07\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8bxz" Oct 30 11:54:04.723975 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.723644 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c54f4c1b-f08e-4495-9a14-17469a792ae5-socket-dir\") pod \"aws-ebs-csi-driver-node-h6mlf\" (UID: \"c54f4c1b-f08e-4495-9a14-17469a792ae5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h6mlf" Oct 30 11:54:04.723975 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.723677 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bd977255-f9a7-4531-8a9c-b1a3a17e0b79-trusted-ca\") pod \"image-registry-557c6b9f47-vcr2t\" (UID: \"bd977255-f9a7-4531-8a9c-b1a3a17e0b79\") " pod="openshift-image-registry/image-registry-557c6b9f47-vcr2t" Oct 30 11:54:04.723975 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.723689 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bd977255-f9a7-4531-8a9c-b1a3a17e0b79-ca-trust-extracted\") pod \"image-registry-557c6b9f47-vcr2t\" (UID: \"bd977255-f9a7-4531-8a9c-b1a3a17e0b79\") " pod="openshift-image-registry/image-registry-557c6b9f47-vcr2t" Oct 30 11:54:04.723975 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.723735 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0df1a118-36a5-4c57-8cec-2c8236f1c4de-multus-daemon-config\") pod \"multus-tlwp4\" (UID: \"0df1a118-36a5-4c57-8cec-2c8236f1c4de\") " pod="openshift-multus/multus-tlwp4" Oct 30 11:54:04.723975 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.723737 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stff7\" (UniqueName: \"kubernetes.io/projected/d659d471-f019-4853-97d0-ee6e948e6c1c-kube-api-access-stff7\") pod \"multus-additional-cni-plugins-s6k8x\" (UID: \"d659d471-f019-4853-97d0-ee6e948e6c1c\") " pod="openshift-multus/multus-additional-cni-plugins-s6k8x" Oct 30 11:54:04.723975 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.723768 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bd977255-f9a7-4531-8a9c-b1a3a17e0b79-ca-trust-extracted\") pod \"image-registry-557c6b9f47-vcr2t\" (UID: \"bd977255-f9a7-4531-8a9c-b1a3a17e0b79\") " pod="openshift-image-registry/image-registry-557c6b9f47-vcr2t" Oct 30 11:54:04.723975 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.723779 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c54f4c1b-f08e-4495-9a14-17469a792ae5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-h6mlf\" (UID: \"c54f4c1b-f08e-4495-9a14-17469a792ae5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h6mlf" Oct 30 11:54:04.723975 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.723805 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3699031f-ee9e-46cc-b37e-4dfc76b9c0b1-tmp\") pod \"tuned-587rx\" (UID: \"3699031f-ee9e-46cc-b37e-4dfc76b9c0b1\") " pod="openshift-cluster-node-tuning-operator/tuned-587rx" Oct 30 11:54:04.723975 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.723861 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b735438-5947-4374-bd4c-66ac3a299770-tmp-dir\") pod \"node-resolver-xmrdj\" (UID: \"1b735438-5947-4374-bd4c-66ac3a299770\") " pod="openshift-dns/node-resolver-xmrdj" Oct 30 11:54:04.723975 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.723888 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/644e19b3-10ad-4145-9ed3-595d19ed9b07-node-log\") pod \"ovnkube-node-b8bxz\" (UID: \"644e19b3-10ad-4145-9ed3-595d19ed9b07\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8bxz" Oct 30 11:54:04.723975 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.723892 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3699031f-ee9e-46cc-b37e-4dfc76b9c0b1-tmp\") pod \"tuned-587rx\" (UID: \"3699031f-ee9e-46cc-b37e-4dfc76b9c0b1\") " pod="openshift-cluster-node-tuning-operator/tuned-587rx" Oct 30 11:54:04.723975 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.723910 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/221ced85-cc15-4e07-82d9-a1a1f3e958a5-metrics-tls\") pod \"dns-default-xxcbn\" (UID: \"221ced85-cc15-4e07-82d9-a1a1f3e958a5\") " pod="openshift-dns/dns-default-xxcbn" Oct 30 11:54:04.723975 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.723955 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7mvn\" (UniqueName: \"kubernetes.io/projected/d0e0cec2-f16a-4bae-bc70-6c08da4bda2e-kube-api-access-h7mvn\") pod \"volume-data-source-validator-58c9f5f46d-ksxpw\" (UID: \"d0e0cec2-f16a-4bae-bc70-6c08da4bda2e\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-ksxpw" Oct 30 11:54:04.723975 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.723962 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b735438-5947-4374-bd4c-66ac3a299770-tmp-dir\") pod \"node-resolver-xmrdj\" (UID: \"1b735438-5947-4374-bd4c-66ac3a299770\") " pod="openshift-dns/node-resolver-xmrdj" Oct 30 11:54:04.724743 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.724003 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/179d55d3-7958-4928-a98c-68984fca9837-nginx-conf\") pod \"networking-console-plugin-86c478bb88-zzxhv\" (UID: \"179d55d3-7958-4928-a98c-68984fca9837\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-zzxhv" Oct 30 11:54:04.724743 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.724037 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3699031f-ee9e-46cc-b37e-4dfc76b9c0b1-etc-kubernetes\") pod \"tuned-587rx\" (UID: \"3699031f-ee9e-46cc-b37e-4dfc76b9c0b1\") " pod="openshift-cluster-node-tuning-operator/tuned-587rx" Oct 30 11:54:04.724743 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.724060 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4e18f771-273b-49e6-872c-6cbe47ab68c4-konnectivity-ca\") pod \"konnectivity-agent-tdf8d\" (UID: \"4e18f771-273b-49e6-872c-6cbe47ab68c4\") " pod="kube-system/konnectivity-agent-tdf8d" Oct 30 11:54:04.724743 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.724140 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3699031f-ee9e-46cc-b37e-4dfc76b9c0b1-etc-sysctl-d\") pod \"tuned-587rx\" (UID: \"3699031f-ee9e-46cc-b37e-4dfc76b9c0b1\") " pod="openshift-cluster-node-tuning-operator/tuned-587rx" Oct 30 11:54:04.724743 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.724180 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cfaad7d0-d30b-4ddc-81cc-2b603416fc29-trusted-ca\") pod \"image-registry-79fd565486-8fl78\" (UID: \"cfaad7d0-d30b-4ddc-81cc-2b603416fc29\") " pod="openshift-image-registry/image-registry-79fd565486-8fl78" Oct 30 11:54:04.724743 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.724187 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4e18f771-273b-49e6-872c-6cbe47ab68c4-konnectivity-ca\") pod \"konnectivity-agent-tdf8d\" (UID: \"4e18f771-273b-49e6-872c-6cbe47ab68c4\") " pod="kube-system/konnectivity-agent-tdf8d" Oct 30 11:54:04.724743 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.724220 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/644e19b3-10ad-4145-9ed3-595d19ed9b07-host-kubelet\") pod \"ovnkube-node-b8bxz\" (UID: \"644e19b3-10ad-4145-9ed3-595d19ed9b07\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8bxz" Oct 30 11:54:04.724743 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.724260 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/179d55d3-7958-4928-a98c-68984fca9837-nginx-conf\") pod \"networking-console-plugin-86c478bb88-zzxhv\" (UID: \"179d55d3-7958-4928-a98c-68984fca9837\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-zzxhv" Oct 30 11:54:04.724743 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.724278 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bd977255-f9a7-4531-8a9c-b1a3a17e0b79-registry-certificates\") pod \"image-registry-557c6b9f47-vcr2t\" (UID: \"bd977255-f9a7-4531-8a9c-b1a3a17e0b79\") " pod="openshift-image-registry/image-registry-557c6b9f47-vcr2t" Oct 30 11:54:04.724743 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.724313 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6b350710-c48c-4c7e-bce9-e021099e98ca-dbus\") pod \"global-pull-secret-syncer-fxjlt\" (UID: \"6b350710-c48c-4c7e-bce9-e021099e98ca\") " pod="kube-system/global-pull-secret-syncer-fxjlt" Oct 30 11:54:04.724743 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.724355 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c54f4c1b-f08e-4495-9a14-17469a792ae5-device-dir\") pod \"aws-ebs-csi-driver-node-h6mlf\" (UID: \"c54f4c1b-f08e-4495-9a14-17469a792ae5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h6mlf" Oct 30 11:54:04.724743 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.724388 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3699031f-ee9e-46cc-b37e-4dfc76b9c0b1-etc-tuned\") pod \"tuned-587rx\" (UID: \"3699031f-ee9e-46cc-b37e-4dfc76b9c0b1\") " pod="openshift-cluster-node-tuning-operator/tuned-587rx" Oct 30 11:54:04.724743 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.724410 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bd977255-f9a7-4531-8a9c-b1a3a17e0b79-registry-certificates\") pod \"image-registry-557c6b9f47-vcr2t\" (UID: \"bd977255-f9a7-4531-8a9c-b1a3a17e0b79\") " pod="openshift-image-registry/image-registry-557c6b9f47-vcr2t" Oct 30 11:54:04.724743 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.724445 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3699031f-ee9e-46cc-b37e-4dfc76b9c0b1-etc-tuned\") pod \"tuned-587rx\" (UID: \"3699031f-ee9e-46cc-b37e-4dfc76b9c0b1\") " pod="openshift-cluster-node-tuning-operator/tuned-587rx" Oct 30 11:54:04.724743 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.724457 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bd977255-f9a7-4531-8a9c-b1a3a17e0b79-image-registry-private-configuration\") pod \"image-registry-557c6b9f47-vcr2t\" (UID: \"bd977255-f9a7-4531-8a9c-b1a3a17e0b79\") " pod="openshift-image-registry/image-registry-557c6b9f47-vcr2t" Oct 30 11:54:04.724743 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.724480 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d670b364-53ce-444b-b9c4-ac8ac09cd44f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vn6vv\" (UID: \"d670b364-53ce-444b-b9c4-ac8ac09cd44f\") " pod="openshift-insights/insights-runtime-extractor-vn6vv" Oct 30 11:54:04.724743 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.724497 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/644e19b3-10ad-4145-9ed3-595d19ed9b07-log-socket\") pod \"ovnkube-node-b8bxz\" (UID: \"644e19b3-10ad-4145-9ed3-595d19ed9b07\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8bxz" Oct 30 11:54:04.724743 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.724512 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0df1a118-36a5-4c57-8cec-2c8236f1c4de-multus-socket-dir-parent\") pod \"multus-tlwp4\" (UID: \"0df1a118-36a5-4c57-8cec-2c8236f1c4de\") " pod="openshift-multus/multus-tlwp4" Oct 30 11:54:04.724743 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.724539 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7689cfe5-5a4e-4481-8eb9-107a58ca8bb3-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-xdxkp\" (UID: \"7689cfe5-5a4e-4481-8eb9-107a58ca8bb3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xdxkp" Oct 30 11:54:04.724743 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.724542 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cfaad7d0-d30b-4ddc-81cc-2b603416fc29-trusted-ca\") pod \"image-registry-79fd565486-8fl78\" (UID: \"cfaad7d0-d30b-4ddc-81cc-2b603416fc29\") " pod="openshift-image-registry/image-registry-79fd565486-8fl78" Oct 30 11:54:04.724743 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.724561 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92smt\" (UniqueName: \"kubernetes.io/projected/7e8ec7ff-015c-4049-9c5f-aae73dc06e4b-kube-api-access-92smt\") pod \"iptables-alerter-8m2w9\" (UID: \"7e8ec7ff-015c-4049-9c5f-aae73dc06e4b\") " pod="openshift-network-operator/iptables-alerter-8m2w9" Oct 30 11:54:04.724743 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.724631 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r226b\" (UniqueName: \"kubernetes.io/projected/b3f20243-155c-4e25-a1ca-6131a141fbd4-kube-api-access-r226b\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-86p78\" (UID: \"b3f20243-155c-4e25-a1ca-6131a141fbd4\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-86p78" Oct 30 11:54:04.724743 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.724658 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0df1a118-36a5-4c57-8cec-2c8236f1c4de-system-cni-dir\") pod \"multus-tlwp4\" (UID: \"0df1a118-36a5-4c57-8cec-2c8236f1c4de\") " pod="openshift-multus/multus-tlwp4" Oct 30 11:54:04.724743 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.724685 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzld2\" (UniqueName: \"kubernetes.io/projected/1f5667a1-a773-4520-b2fb-c69148df632c-kube-api-access-bzld2\") pod \"kas-connection-checker-685895c7c9-ccctl\" (UID: \"1f5667a1-a773-4520-b2fb-c69148df632c\") " pod="kube-system/kas-connection-checker-685895c7c9-ccctl" Oct 30 11:54:04.724743 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.724735 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vx4mp\" (UniqueName: \"kubernetes.io/projected/a70ee11c-c20e-4a4f-8476-ac5eb02816d0-kube-api-access-vx4mp\") pod \"console-operator-7cd56b8dbc-plczt\" (UID: \"a70ee11c-c20e-4a4f-8476-ac5eb02816d0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-plczt" Oct 30 11:54:04.724743 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.724727 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bd977255-f9a7-4531-8a9c-b1a3a17e0b79-image-registry-private-configuration\") pod \"image-registry-557c6b9f47-vcr2t\" (UID: \"bd977255-f9a7-4531-8a9c-b1a3a17e0b79\") " pod="openshift-image-registry/image-registry-557c6b9f47-vcr2t" Oct 30 11:54:04.725800 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.724860 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75fhw\" (UniqueName: \"kubernetes.io/projected/cfaad7d0-d30b-4ddc-81cc-2b603416fc29-kube-api-access-75fhw\") pod \"image-registry-79fd565486-8fl78\" (UID: \"cfaad7d0-d30b-4ddc-81cc-2b603416fc29\") " pod="openshift-image-registry/image-registry-79fd565486-8fl78" Oct 30 11:54:04.725800 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.724878 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/644e19b3-10ad-4145-9ed3-595d19ed9b07-host-run-netns\") pod \"ovnkube-node-b8bxz\" (UID: \"644e19b3-10ad-4145-9ed3-595d19ed9b07\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8bxz" Oct 30 11:54:04.725800 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.724892 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/644e19b3-10ad-4145-9ed3-595d19ed9b07-etc-openvswitch\") pod \"ovnkube-node-b8bxz\" (UID: \"644e19b3-10ad-4145-9ed3-595d19ed9b07\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8bxz" Oct 30 11:54:04.725800 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.724906 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0df1a118-36a5-4c57-8cec-2c8236f1c4de-host-run-k8s-cni-cncf-io\") pod \"multus-tlwp4\" (UID: \"0df1a118-36a5-4c57-8cec-2c8236f1c4de\") " pod="openshift-multus/multus-tlwp4" Oct 30 11:54:04.725800 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.724931 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7e8ec7ff-015c-4049-9c5f-aae73dc06e4b-iptables-alerter-script\") pod \"iptables-alerter-8m2w9\" (UID: \"7e8ec7ff-015c-4049-9c5f-aae73dc06e4b\") " pod="openshift-network-operator/iptables-alerter-8m2w9" Oct 30 11:54:04.725800 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.724961 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/644e19b3-10ad-4145-9ed3-595d19ed9b07-host-cni-bin\") pod \"ovnkube-node-b8bxz\" (UID: \"644e19b3-10ad-4145-9ed3-595d19ed9b07\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8bxz" Oct 30 11:54:04.725800 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.725055 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0df1a118-36a5-4c57-8cec-2c8236f1c4de-cni-binary-copy\") pod \"multus-tlwp4\" (UID: \"0df1a118-36a5-4c57-8cec-2c8236f1c4de\") " pod="openshift-multus/multus-tlwp4" Oct 30 11:54:04.725800 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.725085 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3699031f-ee9e-46cc-b37e-4dfc76b9c0b1-etc-sysconfig\") pod \"tuned-587rx\" (UID: \"3699031f-ee9e-46cc-b37e-4dfc76b9c0b1\") " pod="openshift-cluster-node-tuning-operator/tuned-587rx" Oct 30 11:54:04.725800 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.725112 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3699031f-ee9e-46cc-b37e-4dfc76b9c0b1-etc-sysctl-conf\") pod \"tuned-587rx\" (UID: \"3699031f-ee9e-46cc-b37e-4dfc76b9c0b1\") " pod="openshift-cluster-node-tuning-operator/tuned-587rx" Oct 30 11:54:04.725800 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.725116 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7e8ec7ff-015c-4049-9c5f-aae73dc06e4b-iptables-alerter-script\") pod \"iptables-alerter-8m2w9\" (UID: \"7e8ec7ff-015c-4049-9c5f-aae73dc06e4b\") " pod="openshift-network-operator/iptables-alerter-8m2w9" Oct 30 11:54:04.725800 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.725139 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/644e19b3-10ad-4145-9ed3-595d19ed9b07-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-b8bxz\" (UID: \"644e19b3-10ad-4145-9ed3-595d19ed9b07\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8bxz" Oct 30 11:54:04.725800 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.725171 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlp7f\" (UniqueName: \"kubernetes.io/projected/d670b364-53ce-444b-b9c4-ac8ac09cd44f-kube-api-access-rlp7f\") pod \"insights-runtime-extractor-vn6vv\" (UID: \"d670b364-53ce-444b-b9c4-ac8ac09cd44f\") " pod="openshift-insights/insights-runtime-extractor-vn6vv" Oct 30 11:54:04.725800 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.725195 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bd977255-f9a7-4531-8a9c-b1a3a17e0b79-registry-tls\") pod \"image-registry-557c6b9f47-vcr2t\" (UID: \"bd977255-f9a7-4531-8a9c-b1a3a17e0b79\") " pod="openshift-image-registry/image-registry-557c6b9f47-vcr2t" Oct 30 11:54:04.725800 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.725219 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0df1a118-36a5-4c57-8cec-2c8236f1c4de-cni-binary-copy\") pod \"multus-tlwp4\" (UID: \"0df1a118-36a5-4c57-8cec-2c8236f1c4de\") " pod="openshift-multus/multus-tlwp4" Oct 30 11:54:04.725800 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.725221 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gr67s\" (UniqueName: \"kubernetes.io/projected/7689cfe5-5a4e-4481-8eb9-107a58ca8bb3-kube-api-access-gr67s\") pod \"cluster-samples-operator-6ddb6c8b94-xdxkp\" (UID: \"7689cfe5-5a4e-4481-8eb9-107a58ca8bb3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xdxkp" Oct 30 11:54:04.725800 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.725246 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a70ee11c-c20e-4a4f-8476-ac5eb02816d0-config\") pod \"console-operator-7cd56b8dbc-plczt\" (UID: \"a70ee11c-c20e-4a4f-8476-ac5eb02816d0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-plczt" Oct 30 11:54:04.725800 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.725272 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zh8h9\" (UniqueName: \"kubernetes.io/projected/ff9feaa5-70cc-47fd-a3d1-a9ac6e7eaace-kube-api-access-zh8h9\") pod \"network-check-target-bz65k\" (UID: \"ff9feaa5-70cc-47fd-a3d1-a9ac6e7eaace\") " pod="openshift-network-diagnostics/network-check-target-bz65k" Oct 30 11:54:04.725800 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.725302 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/644e19b3-10ad-4145-9ed3-595d19ed9b07-run-openvswitch\") pod \"ovnkube-node-b8bxz\" (UID: \"644e19b3-10ad-4145-9ed3-595d19ed9b07\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8bxz" Oct 30 11:54:04.725800 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.725324 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d670b364-53ce-444b-b9c4-ac8ac09cd44f-crio-socket\") pod \"insights-runtime-extractor-vn6vv\" (UID: \"d670b364-53ce-444b-b9c4-ac8ac09cd44f\") " pod="openshift-insights/insights-runtime-extractor-vn6vv" Oct 30 11:54:04.725800 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.725427 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0df1a118-36a5-4c57-8cec-2c8236f1c4de-host-var-lib-cni-multus\") pod \"multus-tlwp4\" (UID: \"0df1a118-36a5-4c57-8cec-2c8236f1c4de\") " pod="openshift-multus/multus-tlwp4" Oct 30 11:54:04.725800 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.725450 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0df1a118-36a5-4c57-8cec-2c8236f1c4de-host-run-multus-certs\") pod \"multus-tlwp4\" (UID: \"0df1a118-36a5-4c57-8cec-2c8236f1c4de\") " pod="openshift-multus/multus-tlwp4" Oct 30 11:54:04.725800 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.725480 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3699031f-ee9e-46cc-b37e-4dfc76b9c0b1-lib-modules\") pod \"tuned-587rx\" (UID: \"3699031f-ee9e-46cc-b37e-4dfc76b9c0b1\") " pod="openshift-cluster-node-tuning-operator/tuned-587rx" Oct 30 11:54:04.725800 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.725494 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a70ee11c-c20e-4a4f-8476-ac5eb02816d0-config\") pod \"console-operator-7cd56b8dbc-plczt\" (UID: \"a70ee11c-c20e-4a4f-8476-ac5eb02816d0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-plczt" Oct 30 11:54:04.725800 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.725529 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/221ced85-cc15-4e07-82d9-a1a1f3e958a5-config-volume\") pod \"dns-default-xxcbn\" (UID: \"221ced85-cc15-4e07-82d9-a1a1f3e958a5\") " pod="openshift-dns/dns-default-xxcbn" Oct 30 11:54:04.725800 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.725548 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c815b3fc-7f35-4ae6-a015-2e84c6108a64-serving-cert\") pod \"service-ca-operator-544d749c8c-6l6fr\" (UID: \"c815b3fc-7f35-4ae6-a015-2e84c6108a64\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6l6fr" Oct 30 11:54:04.725800 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.725567 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0df1a118-36a5-4c57-8cec-2c8236f1c4de-cnibin\") pod \"multus-tlwp4\" (UID: \"0df1a118-36a5-4c57-8cec-2c8236f1c4de\") " pod="openshift-multus/multus-tlwp4" Oct 30 11:54:04.725800 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.725589 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7vnf\" (UniqueName: \"kubernetes.io/projected/3699031f-ee9e-46cc-b37e-4dfc76b9c0b1-kube-api-access-r7vnf\") pod \"tuned-587rx\" (UID: \"3699031f-ee9e-46cc-b37e-4dfc76b9c0b1\") " pod="openshift-cluster-node-tuning-operator/tuned-587rx" Oct 30 11:54:04.725800 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.725617 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rxc4\" (UniqueName: \"kubernetes.io/projected/6acb13a1-cb2e-45e1-897b-3ef89e6d1b84-kube-api-access-2rxc4\") pod \"router-default-5bfdf864c-tzv67\" (UID: \"6acb13a1-cb2e-45e1-897b-3ef89e6d1b84\") " pod="openshift-ingress/router-default-5bfdf864c-tzv67" Oct 30 11:54:04.725800 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.725680 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cfaad7d0-d30b-4ddc-81cc-2b603416fc29-ca-trust-extracted\") pod \"image-registry-79fd565486-8fl78\" (UID: \"cfaad7d0-d30b-4ddc-81cc-2b603416fc29\") " pod="openshift-image-registry/image-registry-79fd565486-8fl78" Oct 30 11:54:04.725800 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.725682 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c815b3fc-7f35-4ae6-a015-2e84c6108a64-serving-cert\") pod \"service-ca-operator-544d749c8c-6l6fr\" (UID: \"c815b3fc-7f35-4ae6-a015-2e84c6108a64\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6l6fr" Oct 30 11:54:04.725800 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.725701 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bd977255-f9a7-4531-8a9c-b1a3a17e0b79-bound-sa-token\") pod \"image-registry-557c6b9f47-vcr2t\" (UID: \"bd977255-f9a7-4531-8a9c-b1a3a17e0b79\") " pod="openshift-image-registry/image-registry-557c6b9f47-vcr2t" Oct 30 11:54:04.725800 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.725727 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0df1a118-36a5-4c57-8cec-2c8236f1c4de-multus-conf-dir\") pod \"multus-tlwp4\" (UID: \"0df1a118-36a5-4c57-8cec-2c8236f1c4de\") " pod="openshift-multus/multus-tlwp4" Oct 30 11:54:04.725800 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.725749 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cfaad7d0-d30b-4ddc-81cc-2b603416fc29-ca-trust-extracted\") pod \"image-registry-79fd565486-8fl78\" (UID: \"cfaad7d0-d30b-4ddc-81cc-2b603416fc29\") " pod="openshift-image-registry/image-registry-79fd565486-8fl78" Oct 30 11:54:04.725800 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.725755 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzwcn\" (UniqueName: \"kubernetes.io/projected/1d7f8396-e57c-45e5-ae74-a2244db10364-kube-api-access-bzwcn\") pod \"network-metrics-daemon-dbqhj\" (UID: \"1d7f8396-e57c-45e5-ae74-a2244db10364\") " pod="openshift-multus/network-metrics-daemon-dbqhj" Oct 30 11:54:04.725800 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.725790 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/179d55d3-7958-4928-a98c-68984fca9837-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-zzxhv\" (UID: \"179d55d3-7958-4928-a98c-68984fca9837\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-zzxhv" Oct 30 11:54:04.725800 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.725806 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/221ced85-cc15-4e07-82d9-a1a1f3e958a5-config-volume\") pod \"dns-default-xxcbn\" (UID: \"221ced85-cc15-4e07-82d9-a1a1f3e958a5\") " pod="openshift-dns/dns-default-xxcbn" Oct 30 11:54:04.725800 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.725810 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3699031f-ee9e-46cc-b37e-4dfc76b9c0b1-etc-modprobe-d\") pod \"tuned-587rx\" (UID: \"3699031f-ee9e-46cc-b37e-4dfc76b9c0b1\") " pod="openshift-cluster-node-tuning-operator/tuned-587rx" Oct 30 11:54:04.727256 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.725868 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/644e19b3-10ad-4145-9ed3-595d19ed9b07-env-overrides\") pod \"ovnkube-node-b8bxz\" (UID: \"644e19b3-10ad-4145-9ed3-595d19ed9b07\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8bxz" Oct 30 11:54:04.727256 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.725895 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1757617-7c51-4029-b329-1b6a4630f860-service-ca-bundle\") pod \"insights-operator-7c4986d449-2fqgf\" (UID: \"b1757617-7c51-4029-b329-1b6a4630f860\") " pod="openshift-insights/insights-operator-7c4986d449-2fqgf" Oct 30 11:54:04.727256 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.725990 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3699031f-ee9e-46cc-b37e-4dfc76b9c0b1-sys\") pod \"tuned-587rx\" (UID: \"3699031f-ee9e-46cc-b37e-4dfc76b9c0b1\") " pod="openshift-cluster-node-tuning-operator/tuned-587rx" Oct 30 11:54:04.727256 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.726009 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/644e19b3-10ad-4145-9ed3-595d19ed9b07-env-overrides\") pod \"ovnkube-node-b8bxz\" (UID: \"644e19b3-10ad-4145-9ed3-595d19ed9b07\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8bxz" Oct 30 11:54:04.727256 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.726010 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1757617-7c51-4029-b329-1b6a4630f860-service-ca-bundle\") pod \"insights-operator-7c4986d449-2fqgf\" (UID: \"b1757617-7c51-4029-b329-1b6a4630f860\") " pod="openshift-insights/insights-operator-7c4986d449-2fqgf" Oct 30 11:54:04.727256 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.726022 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cfaad7d0-d30b-4ddc-81cc-2b603416fc29-bound-sa-token\") pod \"image-registry-79fd565486-8fl78\" (UID: \"cfaad7d0-d30b-4ddc-81cc-2b603416fc29\") " pod="openshift-image-registry/image-registry-79fd565486-8fl78" Oct 30 11:54:04.727256 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.726052 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1757617-7c51-4029-b329-1b6a4630f860-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-2fqgf\" (UID: \"b1757617-7c51-4029-b329-1b6a4630f860\") " pod="openshift-insights/insights-operator-7c4986d449-2fqgf" Oct 30 11:54:04.727256 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.726103 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0df1a118-36a5-4c57-8cec-2c8236f1c4de-host-run-netns\") pod \"multus-tlwp4\" (UID: \"0df1a118-36a5-4c57-8cec-2c8236f1c4de\") " pod="openshift-multus/multus-tlwp4" Oct 30 11:54:04.727256 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.726199 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3699031f-ee9e-46cc-b37e-4dfc76b9c0b1-host\") pod \"tuned-587rx\" (UID: \"3699031f-ee9e-46cc-b37e-4dfc76b9c0b1\") " pod="openshift-cluster-node-tuning-operator/tuned-587rx" Oct 30 11:54:04.727256 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.726221 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/6acb13a1-cb2e-45e1-897b-3ef89e6d1b84-stats-auth\") pod \"router-default-5bfdf864c-tzv67\" (UID: \"6acb13a1-cb2e-45e1-897b-3ef89e6d1b84\") " pod="openshift-ingress/router-default-5bfdf864c-tzv67" Oct 30 11:54:04.727256 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.726237 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/6d976900-ecd0-469c-a9e2-d60708361f1a-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-mvhwk\" (UID: \"6d976900-ecd0-469c-a9e2-d60708361f1a\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-mvhwk" Oct 30 11:54:04.727256 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.726276 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/221ced85-cc15-4e07-82d9-a1a1f3e958a5-tmp-dir\") pod \"dns-default-xxcbn\" (UID: \"221ced85-cc15-4e07-82d9-a1a1f3e958a5\") " pod="openshift-dns/dns-default-xxcbn" Oct 30 11:54:04.727256 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.726293 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jmcf\" (UniqueName: \"kubernetes.io/projected/003c3d18-23f1-45d0-bac4-8c6eea4b7e99-kube-api-access-4jmcf\") pod \"ingress-canary-gn2z5\" (UID: \"003c3d18-23f1-45d0-bac4-8c6eea4b7e99\") " pod="openshift-ingress-canary/ingress-canary-gn2z5" Oct 30 11:54:04.727256 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.726309 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jv6z4\" (UniqueName: \"kubernetes.io/projected/e497352b-2348-4fc7-888f-5b68f116394f-kube-api-access-jv6z4\") pod \"kas-connection-checker-685895c7c9-kg9td\" (UID: \"e497352b-2348-4fc7-888f-5b68f116394f\") " pod="kube-system/kas-connection-checker-685895c7c9-kg9td" Oct 30 11:54:04.727256 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.726326 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0df1a118-36a5-4c57-8cec-2c8236f1c4de-host-var-lib-cni-bin\") pod \"multus-tlwp4\" (UID: \"0df1a118-36a5-4c57-8cec-2c8236f1c4de\") " pod="openshift-multus/multus-tlwp4" Oct 30 11:54:04.727256 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.726340 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d659d471-f019-4853-97d0-ee6e948e6c1c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-s6k8x\" (UID: \"d659d471-f019-4853-97d0-ee6e948e6c1c\") " pod="openshift-multus/multus-additional-cni-plugins-s6k8x" Oct 30 11:54:04.727256 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.726358 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5aee536a-8942-4b48-9b2c-3e36089f3ac5-serviceca\") pod \"node-ca-hb8j7\" (UID: \"5aee536a-8942-4b48-9b2c-3e36089f3ac5\") " pod="openshift-image-registry/node-ca-hb8j7" Oct 30 11:54:04.727256 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.726383 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cfaad7d0-d30b-4ddc-81cc-2b603416fc29-registry-tls\") pod \"image-registry-79fd565486-8fl78\" (UID: \"cfaad7d0-d30b-4ddc-81cc-2b603416fc29\") " pod="openshift-image-registry/image-registry-79fd565486-8fl78" Oct 30 11:54:04.727256 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.726397 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/644e19b3-10ad-4145-9ed3-595d19ed9b07-run-systemd\") pod \"ovnkube-node-b8bxz\" (UID: \"644e19b3-10ad-4145-9ed3-595d19ed9b07\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8bxz" Oct 30 11:54:04.727256 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.726412 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b3f20243-155c-4e25-a1ca-6131a141fbd4-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-86p78\" (UID: \"b3f20243-155c-4e25-a1ca-6131a141fbd4\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-86p78" Oct 30 11:54:04.727256 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.726430 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/644e19b3-10ad-4145-9ed3-595d19ed9b07-host-cni-netd\") pod \"ovnkube-node-b8bxz\" (UID: \"644e19b3-10ad-4145-9ed3-595d19ed9b07\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8bxz" Oct 30 11:54:04.727256 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.726445 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c815b3fc-7f35-4ae6-a015-2e84c6108a64-config\") pod \"service-ca-operator-544d749c8c-6l6fr\" (UID: \"c815b3fc-7f35-4ae6-a015-2e84c6108a64\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6l6fr" Oct 30 11:54:04.727256 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.726465 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bd977255-f9a7-4531-8a9c-b1a3a17e0b79-installation-pull-secrets\") pod \"image-registry-557c6b9f47-vcr2t\" (UID: \"bd977255-f9a7-4531-8a9c-b1a3a17e0b79\") " pod="openshift-image-registry/image-registry-557c6b9f47-vcr2t" Oct 30 11:54:04.727256 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.726479 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0df1a118-36a5-4c57-8cec-2c8236f1c4de-hostroot\") pod \"multus-tlwp4\" (UID: \"0df1a118-36a5-4c57-8cec-2c8236f1c4de\") " pod="openshift-multus/multus-tlwp4" Oct 30 11:54:04.727256 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.726481 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/6acb13a1-cb2e-45e1-897b-3ef89e6d1b84-stats-auth\") pod \"router-default-5bfdf864c-tzv67\" (UID: \"6acb13a1-cb2e-45e1-897b-3ef89e6d1b84\") " pod="openshift-ingress/router-default-5bfdf864c-tzv67" Oct 30 11:54:04.727256 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.726488 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/6d976900-ecd0-469c-a9e2-d60708361f1a-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-mvhwk\" (UID: \"6d976900-ecd0-469c-a9e2-d60708361f1a\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-mvhwk" Oct 30 11:54:04.727256 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.726494 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6b350710-c48c-4c7e-bce9-e021099e98ca-original-pull-secret\") pod \"global-pull-secret-syncer-fxjlt\" (UID: \"6b350710-c48c-4c7e-bce9-e021099e98ca\") " pod="kube-system/global-pull-secret-syncer-fxjlt" Oct 30 11:54:04.727256 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.726540 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmx84\" (UniqueName: \"kubernetes.io/projected/5aee536a-8942-4b48-9b2c-3e36089f3ac5-kube-api-access-vmx84\") pod \"node-ca-hb8j7\" (UID: \"5aee536a-8942-4b48-9b2c-3e36089f3ac5\") " pod="openshift-image-registry/node-ca-hb8j7" Oct 30 11:54:04.727256 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.726579 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/221ced85-cc15-4e07-82d9-a1a1f3e958a5-tmp-dir\") pod \"dns-default-xxcbn\" (UID: \"221ced85-cc15-4e07-82d9-a1a1f3e958a5\") " pod="openshift-dns/dns-default-xxcbn" Oct 30 11:54:04.727256 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.726632 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b3f20243-155c-4e25-a1ca-6131a141fbd4-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-86p78\" (UID: \"b3f20243-155c-4e25-a1ca-6131a141fbd4\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-86p78" Oct 30 11:54:04.727256 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.726684 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bd977255-f9a7-4531-8a9c-b1a3a17e0b79-installation-pull-secrets\") pod \"image-registry-557c6b9f47-vcr2t\" (UID: \"bd977255-f9a7-4531-8a9c-b1a3a17e0b79\") " pod="openshift-image-registry/image-registry-557c6b9f47-vcr2t" Oct 30 11:54:04.727256 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.726728 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1757617-7c51-4029-b329-1b6a4630f860-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-2fqgf\" (UID: \"b1757617-7c51-4029-b329-1b6a4630f860\") " pod="openshift-insights/insights-operator-7c4986d449-2fqgf" Oct 30 11:54:04.727256 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.726728 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c54f4c1b-f08e-4495-9a14-17469a792ae5-sys-fs\") pod \"aws-ebs-csi-driver-node-h6mlf\" (UID: \"c54f4c1b-f08e-4495-9a14-17469a792ae5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h6mlf" Oct 30 11:54:04.727256 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.726777 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a70ee11c-c20e-4a4f-8476-ac5eb02816d0-trusted-ca\") pod \"console-operator-7cd56b8dbc-plczt\" (UID: \"a70ee11c-c20e-4a4f-8476-ac5eb02816d0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-plczt" Oct 30 11:54:04.727256 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.726815 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1b735438-5947-4374-bd4c-66ac3a299770-hosts-file\") pod \"node-resolver-xmrdj\" (UID: \"1b735438-5947-4374-bd4c-66ac3a299770\") " pod="openshift-dns/node-resolver-xmrdj" Oct 30 11:54:04.727256 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.726843 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6b350710-c48c-4c7e-bce9-e021099e98ca-original-pull-secret\") pod \"global-pull-secret-syncer-fxjlt\" (UID: \"6b350710-c48c-4c7e-bce9-e021099e98ca\") " pod="kube-system/global-pull-secret-syncer-fxjlt" Oct 30 11:54:04.727256 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.726820 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5aee536a-8942-4b48-9b2c-3e36089f3ac5-serviceca\") pod \"node-ca-hb8j7\" (UID: \"5aee536a-8942-4b48-9b2c-3e36089f3ac5\") " pod="openshift-image-registry/node-ca-hb8j7" Oct 30 11:54:04.727256 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.726863 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c815b3fc-7f35-4ae6-a015-2e84c6108a64-config\") pod \"service-ca-operator-544d749c8c-6l6fr\" (UID: \"c815b3fc-7f35-4ae6-a015-2e84c6108a64\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6l6fr" Oct 30 11:54:04.727256 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.727000 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/644e19b3-10ad-4145-9ed3-595d19ed9b07-ovnkube-script-lib\") pod \"ovnkube-node-b8bxz\" (UID: \"644e19b3-10ad-4145-9ed3-595d19ed9b07\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8bxz" Oct 30 11:54:04.727256 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.727033 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6acb13a1-cb2e-45e1-897b-3ef89e6d1b84-metrics-certs\") pod \"router-default-5bfdf864c-tzv67\" (UID: \"6acb13a1-cb2e-45e1-897b-3ef89e6d1b84\") " pod="openshift-ingress/router-default-5bfdf864c-tzv67" Oct 30 11:54:04.727256 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.727062 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a70ee11c-c20e-4a4f-8476-ac5eb02816d0-trusted-ca\") pod \"console-operator-7cd56b8dbc-plczt\" (UID: \"a70ee11c-c20e-4a4f-8476-ac5eb02816d0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-plczt" Oct 30 11:54:04.727256 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.727062 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b1757617-7c51-4029-b329-1b6a4630f860-tmp\") pod \"insights-operator-7c4986d449-2fqgf\" (UID: \"b1757617-7c51-4029-b329-1b6a4630f860\") " pod="openshift-insights/insights-operator-7c4986d449-2fqgf" Oct 30 11:54:04.727256 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.727104 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d7f8396-e57c-45e5-ae74-a2244db10364-metrics-certs\") pod \"network-metrics-daemon-dbqhj\" (UID: \"1d7f8396-e57c-45e5-ae74-a2244db10364\") " pod="openshift-multus/network-metrics-daemon-dbqhj" Oct 30 11:54:04.727256 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.727133 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b1757617-7c51-4029-b329-1b6a4630f860-tmp\") pod \"insights-operator-7c4986d449-2fqgf\" (UID: \"b1757617-7c51-4029-b329-1b6a4630f860\") " pod="openshift-insights/insights-operator-7c4986d449-2fqgf" Oct 30 11:54:04.727256 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.727155 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6b350710-c48c-4c7e-bce9-e021099e98ca-kubelet-config\") pod \"global-pull-secret-syncer-fxjlt\" (UID: \"6b350710-c48c-4c7e-bce9-e021099e98ca\") " pod="kube-system/global-pull-secret-syncer-fxjlt" Oct 30 11:54:04.727256 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.727189 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/7689cfe5-5a4e-4481-8eb9-107a58ca8bb3-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-xdxkp\" (UID: \"7689cfe5-5a4e-4481-8eb9-107a58ca8bb3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xdxkp" Oct 30 11:54:04.727256 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.727216 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5aee536a-8942-4b48-9b2c-3e36089f3ac5-host\") pod \"node-ca-hb8j7\" (UID: \"5aee536a-8942-4b48-9b2c-3e36089f3ac5\") " pod="openshift-image-registry/node-ca-hb8j7" Oct 30 11:54:04.727256 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.727248 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6acb13a1-cb2e-45e1-897b-3ef89e6d1b84-service-ca-bundle\") pod \"router-default-5bfdf864c-tzv67\" (UID: \"6acb13a1-cb2e-45e1-897b-3ef89e6d1b84\") " pod="openshift-ingress/router-default-5bfdf864c-tzv67" Oct 30 11:54:04.727256 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.727251 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/644e19b3-10ad-4145-9ed3-595d19ed9b07-ovnkube-script-lib\") pod \"ovnkube-node-b8bxz\" (UID: \"644e19b3-10ad-4145-9ed3-595d19ed9b07\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8bxz" Oct 30 11:54:04.728938 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.727321 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0df1a118-36a5-4c57-8cec-2c8236f1c4de-etc-kubernetes\") pod \"multus-tlwp4\" (UID: \"0df1a118-36a5-4c57-8cec-2c8236f1c4de\") " pod="openshift-multus/multus-tlwp4" Oct 30 11:54:04.728938 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.727346 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4tdc\" (UniqueName: \"kubernetes.io/projected/0df1a118-36a5-4c57-8cec-2c8236f1c4de-kube-api-access-r4tdc\") pod \"multus-tlwp4\" (UID: \"0df1a118-36a5-4c57-8cec-2c8236f1c4de\") " pod="openshift-multus/multus-tlwp4" Oct 30 11:54:04.728938 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.727362 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/7689cfe5-5a4e-4481-8eb9-107a58ca8bb3-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-xdxkp\" (UID: \"7689cfe5-5a4e-4481-8eb9-107a58ca8bb3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xdxkp" Oct 30 11:54:04.728938 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.727388 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d659d471-f019-4853-97d0-ee6e948e6c1c-cnibin\") pod \"multus-additional-cni-plugins-s6k8x\" (UID: \"d659d471-f019-4853-97d0-ee6e948e6c1c\") " pod="openshift-multus/multus-additional-cni-plugins-s6k8x" Oct 30 11:54:04.728938 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.727430 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7e8ec7ff-015c-4049-9c5f-aae73dc06e4b-host-slash\") pod \"iptables-alerter-8m2w9\" (UID: \"7e8ec7ff-015c-4049-9c5f-aae73dc06e4b\") " pod="openshift-network-operator/iptables-alerter-8m2w9" Oct 30 11:54:04.728938 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.727475 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d659d471-f019-4853-97d0-ee6e948e6c1c-cni-binary-copy\") pod \"multus-additional-cni-plugins-s6k8x\" (UID: \"d659d471-f019-4853-97d0-ee6e948e6c1c\") " pod="openshift-multus/multus-additional-cni-plugins-s6k8x" Oct 30 11:54:04.728938 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.727523 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d659d471-f019-4853-97d0-ee6e948e6c1c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-s6k8x\" (UID: \"d659d471-f019-4853-97d0-ee6e948e6c1c\") " pod="openshift-multus/multus-additional-cni-plugins-s6k8x" Oct 30 11:54:04.728938 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.727576 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zd2h\" (UniqueName: \"kubernetes.io/projected/dab8eab1-3e3f-46b4-b76d-d58a7061996d-kube-api-access-7zd2h\") pod \"migrator-66df5d496b-4brks\" (UID: \"dab8eab1-3e3f-46b4-b76d-d58a7061996d\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-4brks" Oct 30 11:54:04.728938 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.727592 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3699031f-ee9e-46cc-b37e-4dfc76b9c0b1-var-lib-kubelet\") pod \"tuned-587rx\" (UID: \"3699031f-ee9e-46cc-b37e-4dfc76b9c0b1\") " pod="openshift-cluster-node-tuning-operator/tuned-587rx" Oct 30 11:54:04.728938 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.727619 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cfaad7d0-d30b-4ddc-81cc-2b603416fc29-installation-pull-secrets\") pod \"image-registry-79fd565486-8fl78\" (UID: \"cfaad7d0-d30b-4ddc-81cc-2b603416fc29\") " pod="openshift-image-registry/image-registry-79fd565486-8fl78" Oct 30 11:54:04.728938 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.727623 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d659d471-f019-4853-97d0-ee6e948e6c1c-cni-binary-copy\") pod \"multus-additional-cni-plugins-s6k8x\" (UID: \"d659d471-f019-4853-97d0-ee6e948e6c1c\") " pod="openshift-multus/multus-additional-cni-plugins-s6k8x" Oct 30 11:54:04.728938 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.727696 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/644e19b3-10ad-4145-9ed3-595d19ed9b07-ovn-node-metrics-cert\") pod \"ovnkube-node-b8bxz\" (UID: \"644e19b3-10ad-4145-9ed3-595d19ed9b07\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8bxz" Oct 30 11:54:04.728938 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.727765 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d659d471-f019-4853-97d0-ee6e948e6c1c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-s6k8x\" (UID: \"d659d471-f019-4853-97d0-ee6e948e6c1c\") " pod="openshift-multus/multus-additional-cni-plugins-s6k8x" Oct 30 11:54:04.728938 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.727791 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d670b364-53ce-444b-b9c4-ac8ac09cd44f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vn6vv\" (UID: \"d670b364-53ce-444b-b9c4-ac8ac09cd44f\") " pod="openshift-insights/insights-runtime-extractor-vn6vv" Oct 30 11:54:04.728938 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.727852 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cfaad7d0-d30b-4ddc-81cc-2b603416fc29-installation-pull-secrets\") pod \"image-registry-79fd565486-8fl78\" (UID: \"cfaad7d0-d30b-4ddc-81cc-2b603416fc29\") " pod="openshift-image-registry/image-registry-79fd565486-8fl78" Oct 30 11:54:04.728938 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.727861 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8f7sv\" (UniqueName: \"kubernetes.io/projected/bd977255-f9a7-4531-8a9c-b1a3a17e0b79-kube-api-access-8f7sv\") pod \"image-registry-557c6b9f47-vcr2t\" (UID: \"bd977255-f9a7-4531-8a9c-b1a3a17e0b79\") " pod="openshift-image-registry/image-registry-557c6b9f47-vcr2t" Oct 30 11:54:04.728938 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.727886 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0df1a118-36a5-4c57-8cec-2c8236f1c4de-multus-cni-dir\") pod \"multus-tlwp4\" (UID: \"0df1a118-36a5-4c57-8cec-2c8236f1c4de\") " pod="openshift-multus/multus-tlwp4" Oct 30 11:54:04.728938 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.727937 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/644e19b3-10ad-4145-9ed3-595d19ed9b07-ovn-node-metrics-cert\") pod \"ovnkube-node-b8bxz\" (UID: \"644e19b3-10ad-4145-9ed3-595d19ed9b07\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8bxz" Oct 30 11:54:04.728938 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.727972 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d659d471-f019-4853-97d0-ee6e948e6c1c-system-cni-dir\") pod \"multus-additional-cni-plugins-s6k8x\" (UID: \"d659d471-f019-4853-97d0-ee6e948e6c1c\") " pod="openshift-multus/multus-additional-cni-plugins-s6k8x" Oct 30 11:54:04.728938 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.728009 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3699031f-ee9e-46cc-b37e-4dfc76b9c0b1-etc-systemd\") pod \"tuned-587rx\" (UID: \"3699031f-ee9e-46cc-b37e-4dfc76b9c0b1\") " pod="openshift-cluster-node-tuning-operator/tuned-587rx" Oct 30 11:54:04.728938 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.728046 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hqrf\" (UniqueName: \"kubernetes.io/projected/7ac30952-da5d-42a3-a1aa-bd249bbf6adb-kube-api-access-6hqrf\") pod \"network-check-source-6f78d59f99-glvhs\" (UID: \"7ac30952-da5d-42a3-a1aa-bd249bbf6adb\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-glvhs" Oct 30 11:54:04.728938 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.728070 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nzgq\" (UniqueName: \"kubernetes.io/projected/644e19b3-10ad-4145-9ed3-595d19ed9b07-kube-api-access-8nzgq\") pod \"ovnkube-node-b8bxz\" (UID: \"644e19b3-10ad-4145-9ed3-595d19ed9b07\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8bxz" Oct 30 11:54:04.746980 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.746949 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tp27x\" (UniqueName: \"kubernetes.io/projected/8e44595b-0bdf-407c-bf22-33eb06272cc1-kube-api-access-tp27x\") pod \"kas-connection-checker-685895c7c9-k8ldb\" (UID: \"8e44595b-0bdf-407c-bf22-33eb06272cc1\") " pod="kube-system/kas-connection-checker-685895c7c9-k8ldb" Oct 30 11:54:04.746980 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.746977 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92smt\" (UniqueName: \"kubernetes.io/projected/7e8ec7ff-015c-4049-9c5f-aae73dc06e4b-kube-api-access-92smt\") pod \"iptables-alerter-8m2w9\" (UID: \"7e8ec7ff-015c-4049-9c5f-aae73dc06e4b\") " pod="openshift-network-operator/iptables-alerter-8m2w9" Oct 30 11:54:04.747121 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.747080 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fhb8\" (UniqueName: \"kubernetes.io/projected/6d976900-ecd0-469c-a9e2-d60708361f1a-kube-api-access-5fhb8\") pod \"cluster-monitoring-operator-884576c57-mvhwk\" (UID: \"6d976900-ecd0-469c-a9e2-d60708361f1a\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-mvhwk" Oct 30 11:54:04.747397 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.747368 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jf44s\" (UniqueName: \"kubernetes.io/projected/c815b3fc-7f35-4ae6-a015-2e84c6108a64-kube-api-access-jf44s\") pod \"service-ca-operator-544d749c8c-6l6fr\" (UID: \"c815b3fc-7f35-4ae6-a015-2e84c6108a64\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6l6fr" Oct 30 11:54:04.749181 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.749161 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rh4tb\" (UniqueName: \"kubernetes.io/projected/c54f4c1b-f08e-4495-9a14-17469a792ae5-kube-api-access-rh4tb\") pod \"aws-ebs-csi-driver-node-h6mlf\" (UID: \"c54f4c1b-f08e-4495-9a14-17469a792ae5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h6mlf" Oct 30 11:54:04.749242 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.749212 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7vnf\" (UniqueName: \"kubernetes.io/projected/3699031f-ee9e-46cc-b37e-4dfc76b9c0b1-kube-api-access-r7vnf\") pod \"tuned-587rx\" (UID: \"3699031f-ee9e-46cc-b37e-4dfc76b9c0b1\") " pod="openshift-cluster-node-tuning-operator/tuned-587rx" Oct 30 11:54:04.751891 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.751865 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r226b\" (UniqueName: \"kubernetes.io/projected/b3f20243-155c-4e25-a1ca-6131a141fbd4-kube-api-access-r226b\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-86p78\" (UID: \"b3f20243-155c-4e25-a1ca-6131a141fbd4\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-86p78" Oct 30 11:54:04.751891 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.751880 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bd977255-f9a7-4531-8a9c-b1a3a17e0b79-bound-sa-token\") pod \"image-registry-557c6b9f47-vcr2t\" (UID: \"bd977255-f9a7-4531-8a9c-b1a3a17e0b79\") " pod="openshift-image-registry/image-registry-557c6b9f47-vcr2t" Oct 30 11:54:04.752015 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.751900 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7mvn\" (UniqueName: \"kubernetes.io/projected/d0e0cec2-f16a-4bae-bc70-6c08da4bda2e-kube-api-access-h7mvn\") pod \"volume-data-source-validator-58c9f5f46d-ksxpw\" (UID: \"d0e0cec2-f16a-4bae-bc70-6c08da4bda2e\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-ksxpw" Oct 30 11:54:04.752069 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.752014 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vx4mp\" (UniqueName: \"kubernetes.io/projected/a70ee11c-c20e-4a4f-8476-ac5eb02816d0-kube-api-access-vx4mp\") pod \"console-operator-7cd56b8dbc-plczt\" (UID: \"a70ee11c-c20e-4a4f-8476-ac5eb02816d0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-plczt" Oct 30 11:54:04.752392 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.752375 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-75fhw\" (UniqueName: \"kubernetes.io/projected/cfaad7d0-d30b-4ddc-81cc-2b603416fc29-kube-api-access-75fhw\") pod \"image-registry-79fd565486-8fl78\" (UID: \"cfaad7d0-d30b-4ddc-81cc-2b603416fc29\") " pod="openshift-image-registry/image-registry-79fd565486-8fl78" Oct 30 11:54:04.752562 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.752547 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zscd2\" (UniqueName: \"kubernetes.io/projected/b1757617-7c51-4029-b329-1b6a4630f860-kube-api-access-zscd2\") pod \"insights-operator-7c4986d449-2fqgf\" (UID: \"b1757617-7c51-4029-b329-1b6a4630f860\") " pod="openshift-insights/insights-operator-7c4986d449-2fqgf" Oct 30 11:54:04.754460 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.754444 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zh8h9\" (UniqueName: \"kubernetes.io/projected/ff9feaa5-70cc-47fd-a3d1-a9ac6e7eaace-kube-api-access-zh8h9\") pod \"network-check-target-bz65k\" (UID: \"ff9feaa5-70cc-47fd-a3d1-a9ac6e7eaace\") " pod="openshift-network-diagnostics/network-check-target-bz65k" Oct 30 11:54:04.755722 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.755705 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzc5h\" (UniqueName: \"kubernetes.io/projected/1b735438-5947-4374-bd4c-66ac3a299770-kube-api-access-gzc5h\") pod \"node-resolver-xmrdj\" (UID: \"1b735438-5947-4374-bd4c-66ac3a299770\") " pod="openshift-dns/node-resolver-xmrdj" Oct 30 11:54:04.756668 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.756651 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhxmm\" (UniqueName: \"kubernetes.io/projected/221ced85-cc15-4e07-82d9-a1a1f3e958a5-kube-api-access-bhxmm\") pod \"dns-default-xxcbn\" (UID: \"221ced85-cc15-4e07-82d9-a1a1f3e958a5\") " pod="openshift-dns/dns-default-xxcbn" Oct 30 11:54:04.757152 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.757135 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-stff7\" (UniqueName: \"kubernetes.io/projected/d659d471-f019-4853-97d0-ee6e948e6c1c-kube-api-access-stff7\") pod \"multus-additional-cni-plugins-s6k8x\" (UID: \"d659d471-f019-4853-97d0-ee6e948e6c1c\") " pod="openshift-multus/multus-additional-cni-plugins-s6k8x" Oct 30 11:54:04.762598 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.762574 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gr67s\" (UniqueName: \"kubernetes.io/projected/7689cfe5-5a4e-4481-8eb9-107a58ca8bb3-kube-api-access-gr67s\") pod \"cluster-samples-operator-6ddb6c8b94-xdxkp\" (UID: \"7689cfe5-5a4e-4481-8eb9-107a58ca8bb3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xdxkp" Oct 30 11:54:04.762815 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.762794 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzld2\" (UniqueName: \"kubernetes.io/projected/1f5667a1-a773-4520-b2fb-c69148df632c-kube-api-access-bzld2\") pod \"kas-connection-checker-685895c7c9-ccctl\" (UID: \"1f5667a1-a773-4520-b2fb-c69148df632c\") " pod="kube-system/kas-connection-checker-685895c7c9-ccctl" Oct 30 11:54:04.763036 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.763010 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4tdc\" (UniqueName: \"kubernetes.io/projected/0df1a118-36a5-4c57-8cec-2c8236f1c4de-kube-api-access-r4tdc\") pod \"multus-tlwp4\" (UID: \"0df1a118-36a5-4c57-8cec-2c8236f1c4de\") " pod="openshift-multus/multus-tlwp4" Oct 30 11:54:04.763124 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.763046 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8f7sv\" (UniqueName: \"kubernetes.io/projected/bd977255-f9a7-4531-8a9c-b1a3a17e0b79-kube-api-access-8f7sv\") pod \"image-registry-557c6b9f47-vcr2t\" (UID: \"bd977255-f9a7-4531-8a9c-b1a3a17e0b79\") " pod="openshift-image-registry/image-registry-557c6b9f47-vcr2t" Oct 30 11:54:04.763124 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.763060 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlp7f\" (UniqueName: \"kubernetes.io/projected/d670b364-53ce-444b-b9c4-ac8ac09cd44f-kube-api-access-rlp7f\") pod \"insights-runtime-extractor-vn6vv\" (UID: \"d670b364-53ce-444b-b9c4-ac8ac09cd44f\") " pod="openshift-insights/insights-runtime-extractor-vn6vv" Oct 30 11:54:04.763124 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.763073 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jv6z4\" (UniqueName: \"kubernetes.io/projected/e497352b-2348-4fc7-888f-5b68f116394f-kube-api-access-jv6z4\") pod \"kas-connection-checker-685895c7c9-kg9td\" (UID: \"e497352b-2348-4fc7-888f-5b68f116394f\") " pod="kube-system/kas-connection-checker-685895c7c9-kg9td" Oct 30 11:54:04.763226 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.763120 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hqrf\" (UniqueName: \"kubernetes.io/projected/7ac30952-da5d-42a3-a1aa-bd249bbf6adb-kube-api-access-6hqrf\") pod \"network-check-source-6f78d59f99-glvhs\" (UID: \"7ac30952-da5d-42a3-a1aa-bd249bbf6adb\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-glvhs" Oct 30 11:54:04.763526 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.763511 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cfaad7d0-d30b-4ddc-81cc-2b603416fc29-bound-sa-token\") pod \"image-registry-79fd565486-8fl78\" (UID: \"cfaad7d0-d30b-4ddc-81cc-2b603416fc29\") " pod="openshift-image-registry/image-registry-79fd565486-8fl78" Oct 30 11:54:04.763564 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.763523 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzwcn\" (UniqueName: \"kubernetes.io/projected/1d7f8396-e57c-45e5-ae74-a2244db10364-kube-api-access-bzwcn\") pod \"network-metrics-daemon-dbqhj\" (UID: \"1d7f8396-e57c-45e5-ae74-a2244db10364\") " pod="openshift-multus/network-metrics-daemon-dbqhj" Oct 30 11:54:04.764205 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.764190 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jmcf\" (UniqueName: \"kubernetes.io/projected/003c3d18-23f1-45d0-bac4-8c6eea4b7e99-kube-api-access-4jmcf\") pod \"ingress-canary-gn2z5\" (UID: \"003c3d18-23f1-45d0-bac4-8c6eea4b7e99\") " pod="openshift-ingress-canary/ingress-canary-gn2z5" Oct 30 11:54:04.764563 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.764548 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nzgq\" (UniqueName: \"kubernetes.io/projected/644e19b3-10ad-4145-9ed3-595d19ed9b07-kube-api-access-8nzgq\") pod \"ovnkube-node-b8bxz\" (UID: \"644e19b3-10ad-4145-9ed3-595d19ed9b07\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8bxz" Oct 30 11:54:04.765249 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.765234 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rxc4\" (UniqueName: \"kubernetes.io/projected/6acb13a1-cb2e-45e1-897b-3ef89e6d1b84-kube-api-access-2rxc4\") pod \"router-default-5bfdf864c-tzv67\" (UID: \"6acb13a1-cb2e-45e1-897b-3ef89e6d1b84\") " pod="openshift-ingress/router-default-5bfdf864c-tzv67" Oct 30 11:54:04.765997 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.765981 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmx84\" (UniqueName: \"kubernetes.io/projected/5aee536a-8942-4b48-9b2c-3e36089f3ac5-kube-api-access-vmx84\") pod \"node-ca-hb8j7\" (UID: \"5aee536a-8942-4b48-9b2c-3e36089f3ac5\") " pod="openshift-image-registry/node-ca-hb8j7" Oct 30 11:54:04.766517 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.766502 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zd2h\" (UniqueName: \"kubernetes.io/projected/dab8eab1-3e3f-46b4-b76d-d58a7061996d-kube-api-access-7zd2h\") pod \"migrator-66df5d496b-4brks\" (UID: \"dab8eab1-3e3f-46b4-b76d-d58a7061996d\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-4brks" Oct 30 11:54:04.828516 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.828469 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3699031f-ee9e-46cc-b37e-4dfc76b9c0b1-etc-sysctl-conf\") pod \"tuned-587rx\" (UID: \"3699031f-ee9e-46cc-b37e-4dfc76b9c0b1\") " pod="openshift-cluster-node-tuning-operator/tuned-587rx" Oct 30 11:54:04.828516 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.828493 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/644e19b3-10ad-4145-9ed3-595d19ed9b07-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-b8bxz\" (UID: \"644e19b3-10ad-4145-9ed3-595d19ed9b07\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8bxz" Oct 30 11:54:04.828516 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.828508 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bd977255-f9a7-4531-8a9c-b1a3a17e0b79-registry-tls\") pod \"image-registry-557c6b9f47-vcr2t\" (UID: \"bd977255-f9a7-4531-8a9c-b1a3a17e0b79\") " pod="openshift-image-registry/image-registry-557c6b9f47-vcr2t" Oct 30 11:54:04.828634 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.828523 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/644e19b3-10ad-4145-9ed3-595d19ed9b07-run-openvswitch\") pod \"ovnkube-node-b8bxz\" (UID: \"644e19b3-10ad-4145-9ed3-595d19ed9b07\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8bxz" Oct 30 11:54:04.828634 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.828572 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/644e19b3-10ad-4145-9ed3-595d19ed9b07-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-b8bxz\" (UID: \"644e19b3-10ad-4145-9ed3-595d19ed9b07\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8bxz" Oct 30 11:54:04.828634 ip-10-0-141-251 kubenswrapper[4908]: E1030 11:54:04.828586 4908 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 11:54:04.828634 ip-10-0-141-251 kubenswrapper[4908]: E1030 11:54:04.828597 4908 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-557c6b9f47-vcr2t: secret "image-registry-tls" not found Oct 30 11:54:04.828754 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.828638 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d670b364-53ce-444b-b9c4-ac8ac09cd44f-crio-socket\") pod \"insights-runtime-extractor-vn6vv\" (UID: \"d670b364-53ce-444b-b9c4-ac8ac09cd44f\") " pod="openshift-insights/insights-runtime-extractor-vn6vv" Oct 30 11:54:04.828754 ip-10-0-141-251 kubenswrapper[4908]: E1030 11:54:04.828653 4908 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bd977255-f9a7-4531-8a9c-b1a3a17e0b79-registry-tls podName:bd977255-f9a7-4531-8a9c-b1a3a17e0b79 nodeName:}" failed. No retries permitted until 2026-10-30 11:54:05.32863454 +0000 UTC m=+1.843063277 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bd977255-f9a7-4531-8a9c-b1a3a17e0b79-registry-tls") pod "image-registry-557c6b9f47-vcr2t" (UID: "bd977255-f9a7-4531-8a9c-b1a3a17e0b79") : secret "image-registry-tls" not found Oct 30 11:54:04.828754 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.828651 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3699031f-ee9e-46cc-b37e-4dfc76b9c0b1-etc-sysctl-conf\") pod \"tuned-587rx\" (UID: \"3699031f-ee9e-46cc-b37e-4dfc76b9c0b1\") " pod="openshift-cluster-node-tuning-operator/tuned-587rx" Oct 30 11:54:04.828754 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.828682 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/644e19b3-10ad-4145-9ed3-595d19ed9b07-run-openvswitch\") pod \"ovnkube-node-b8bxz\" (UID: \"644e19b3-10ad-4145-9ed3-595d19ed9b07\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8bxz" Oct 30 11:54:04.828754 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.828679 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0df1a118-36a5-4c57-8cec-2c8236f1c4de-host-var-lib-cni-multus\") pod \"multus-tlwp4\" (UID: \"0df1a118-36a5-4c57-8cec-2c8236f1c4de\") " pod="openshift-multus/multus-tlwp4" Oct 30 11:54:04.828754 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.828689 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d670b364-53ce-444b-b9c4-ac8ac09cd44f-crio-socket\") pod \"insights-runtime-extractor-vn6vv\" (UID: \"d670b364-53ce-444b-b9c4-ac8ac09cd44f\") " pod="openshift-insights/insights-runtime-extractor-vn6vv" Oct 30 11:54:04.828754 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.828709 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0df1a118-36a5-4c57-8cec-2c8236f1c4de-host-var-lib-cni-multus\") pod \"multus-tlwp4\" (UID: \"0df1a118-36a5-4c57-8cec-2c8236f1c4de\") " pod="openshift-multus/multus-tlwp4" Oct 30 11:54:04.828754 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.828719 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0df1a118-36a5-4c57-8cec-2c8236f1c4de-host-run-multus-certs\") pod \"multus-tlwp4\" (UID: \"0df1a118-36a5-4c57-8cec-2c8236f1c4de\") " pod="openshift-multus/multus-tlwp4" Oct 30 11:54:04.828754 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.828743 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3699031f-ee9e-46cc-b37e-4dfc76b9c0b1-lib-modules\") pod \"tuned-587rx\" (UID: \"3699031f-ee9e-46cc-b37e-4dfc76b9c0b1\") " pod="openshift-cluster-node-tuning-operator/tuned-587rx" Oct 30 11:54:04.828754 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.828750 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0df1a118-36a5-4c57-8cec-2c8236f1c4de-host-run-multus-certs\") pod \"multus-tlwp4\" (UID: \"0df1a118-36a5-4c57-8cec-2c8236f1c4de\") " pod="openshift-multus/multus-tlwp4" Oct 30 11:54:04.829080 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.828767 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0df1a118-36a5-4c57-8cec-2c8236f1c4de-cnibin\") pod \"multus-tlwp4\" (UID: \"0df1a118-36a5-4c57-8cec-2c8236f1c4de\") " pod="openshift-multus/multus-tlwp4" Oct 30 11:54:04.829080 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.828794 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0df1a118-36a5-4c57-8cec-2c8236f1c4de-multus-conf-dir\") pod \"multus-tlwp4\" (UID: \"0df1a118-36a5-4c57-8cec-2c8236f1c4de\") " pod="openshift-multus/multus-tlwp4" Oct 30 11:54:04.829080 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.828808 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/179d55d3-7958-4928-a98c-68984fca9837-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-zzxhv\" (UID: \"179d55d3-7958-4928-a98c-68984fca9837\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-zzxhv" Oct 30 11:54:04.829080 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.828843 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0df1a118-36a5-4c57-8cec-2c8236f1c4de-cnibin\") pod \"multus-tlwp4\" (UID: \"0df1a118-36a5-4c57-8cec-2c8236f1c4de\") " pod="openshift-multus/multus-tlwp4" Oct 30 11:54:04.829080 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.828844 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3699031f-ee9e-46cc-b37e-4dfc76b9c0b1-etc-modprobe-d\") pod \"tuned-587rx\" (UID: \"3699031f-ee9e-46cc-b37e-4dfc76b9c0b1\") " pod="openshift-cluster-node-tuning-operator/tuned-587rx" Oct 30 11:54:04.829080 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.828862 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0df1a118-36a5-4c57-8cec-2c8236f1c4de-multus-conf-dir\") pod \"multus-tlwp4\" (UID: \"0df1a118-36a5-4c57-8cec-2c8236f1c4de\") " pod="openshift-multus/multus-tlwp4" Oct 30 11:54:04.829080 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.828852 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3699031f-ee9e-46cc-b37e-4dfc76b9c0b1-lib-modules\") pod \"tuned-587rx\" (UID: \"3699031f-ee9e-46cc-b37e-4dfc76b9c0b1\") " pod="openshift-cluster-node-tuning-operator/tuned-587rx" Oct 30 11:54:04.829080 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.828874 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3699031f-ee9e-46cc-b37e-4dfc76b9c0b1-sys\") pod \"tuned-587rx\" (UID: \"3699031f-ee9e-46cc-b37e-4dfc76b9c0b1\") " pod="openshift-cluster-node-tuning-operator/tuned-587rx" Oct 30 11:54:04.829080 ip-10-0-141-251 kubenswrapper[4908]: E1030 11:54:04.828896 4908 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 11:54:04.829080 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.828902 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0df1a118-36a5-4c57-8cec-2c8236f1c4de-host-run-netns\") pod \"multus-tlwp4\" (UID: \"0df1a118-36a5-4c57-8cec-2c8236f1c4de\") " pod="openshift-multus/multus-tlwp4" Oct 30 11:54:04.829080 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.828922 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3699031f-ee9e-46cc-b37e-4dfc76b9c0b1-etc-modprobe-d\") pod \"tuned-587rx\" (UID: \"3699031f-ee9e-46cc-b37e-4dfc76b9c0b1\") " pod="openshift-cluster-node-tuning-operator/tuned-587rx" Oct 30 11:54:04.829080 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.828923 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3699031f-ee9e-46cc-b37e-4dfc76b9c0b1-host\") pod \"tuned-587rx\" (UID: \"3699031f-ee9e-46cc-b37e-4dfc76b9c0b1\") " pod="openshift-cluster-node-tuning-operator/tuned-587rx" Oct 30 11:54:04.829080 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.828930 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3699031f-ee9e-46cc-b37e-4dfc76b9c0b1-sys\") pod \"tuned-587rx\" (UID: \"3699031f-ee9e-46cc-b37e-4dfc76b9c0b1\") " pod="openshift-cluster-node-tuning-operator/tuned-587rx" Oct 30 11:54:04.829080 ip-10-0-141-251 kubenswrapper[4908]: E1030 11:54:04.828952 4908 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/179d55d3-7958-4928-a98c-68984fca9837-networking-console-plugin-cert podName:179d55d3-7958-4928-a98c-68984fca9837 nodeName:}" failed. No retries permitted until 2026-10-30 11:54:05.328936054 +0000 UTC m=+1.843364792 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/179d55d3-7958-4928-a98c-68984fca9837-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-zzxhv" (UID: "179d55d3-7958-4928-a98c-68984fca9837") : secret "networking-console-plugin-cert" not found Oct 30 11:54:04.829080 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.828959 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0df1a118-36a5-4c57-8cec-2c8236f1c4de-host-run-netns\") pod \"multus-tlwp4\" (UID: \"0df1a118-36a5-4c57-8cec-2c8236f1c4de\") " pod="openshift-multus/multus-tlwp4" Oct 30 11:54:04.829080 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.828965 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3699031f-ee9e-46cc-b37e-4dfc76b9c0b1-host\") pod \"tuned-587rx\" (UID: \"3699031f-ee9e-46cc-b37e-4dfc76b9c0b1\") " pod="openshift-cluster-node-tuning-operator/tuned-587rx" Oct 30 11:54:04.829080 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.828995 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0df1a118-36a5-4c57-8cec-2c8236f1c4de-host-var-lib-cni-bin\") pod \"multus-tlwp4\" (UID: \"0df1a118-36a5-4c57-8cec-2c8236f1c4de\") " pod="openshift-multus/multus-tlwp4" Oct 30 11:54:04.829080 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.829013 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d659d471-f019-4853-97d0-ee6e948e6c1c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-s6k8x\" (UID: \"d659d471-f019-4853-97d0-ee6e948e6c1c\") " pod="openshift-multus/multus-additional-cni-plugins-s6k8x" Oct 30 11:54:04.829080 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.829027 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cfaad7d0-d30b-4ddc-81cc-2b603416fc29-registry-tls\") pod \"image-registry-79fd565486-8fl78\" (UID: \"cfaad7d0-d30b-4ddc-81cc-2b603416fc29\") " pod="openshift-image-registry/image-registry-79fd565486-8fl78" Oct 30 11:54:04.829080 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.829041 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/644e19b3-10ad-4145-9ed3-595d19ed9b07-run-systemd\") pod \"ovnkube-node-b8bxz\" (UID: \"644e19b3-10ad-4145-9ed3-595d19ed9b07\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8bxz" Oct 30 11:54:04.829080 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.829055 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/644e19b3-10ad-4145-9ed3-595d19ed9b07-host-cni-netd\") pod \"ovnkube-node-b8bxz\" (UID: \"644e19b3-10ad-4145-9ed3-595d19ed9b07\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8bxz" Oct 30 11:54:04.829080 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.829068 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0df1a118-36a5-4c57-8cec-2c8236f1c4de-hostroot\") pod \"multus-tlwp4\" (UID: \"0df1a118-36a5-4c57-8cec-2c8236f1c4de\") " pod="openshift-multus/multus-tlwp4" Oct 30 11:54:04.829080 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.829083 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c54f4c1b-f08e-4495-9a14-17469a792ae5-sys-fs\") pod \"aws-ebs-csi-driver-node-h6mlf\" (UID: \"c54f4c1b-f08e-4495-9a14-17469a792ae5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h6mlf" Oct 30 11:54:04.829080 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.829028 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0df1a118-36a5-4c57-8cec-2c8236f1c4de-host-var-lib-cni-bin\") pod \"multus-tlwp4\" (UID: \"0df1a118-36a5-4c57-8cec-2c8236f1c4de\") " pod="openshift-multus/multus-tlwp4" Oct 30 11:54:04.830045 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.829099 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1b735438-5947-4374-bd4c-66ac3a299770-hosts-file\") pod \"node-resolver-xmrdj\" (UID: \"1b735438-5947-4374-bd4c-66ac3a299770\") " pod="openshift-dns/node-resolver-xmrdj" Oct 30 11:54:04.830045 ip-10-0-141-251 kubenswrapper[4908]: E1030 11:54:04.829109 4908 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 11:54:04.830045 ip-10-0-141-251 kubenswrapper[4908]: E1030 11:54:04.829121 4908 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-79fd565486-8fl78: secret "image-registry-tls" not found Oct 30 11:54:04.830045 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.829120 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/644e19b3-10ad-4145-9ed3-595d19ed9b07-run-systemd\") pod \"ovnkube-node-b8bxz\" (UID: \"644e19b3-10ad-4145-9ed3-595d19ed9b07\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8bxz" Oct 30 11:54:04.830045 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.829137 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0df1a118-36a5-4c57-8cec-2c8236f1c4de-hostroot\") pod \"multus-tlwp4\" (UID: \"0df1a118-36a5-4c57-8cec-2c8236f1c4de\") " pod="openshift-multus/multus-tlwp4" Oct 30 11:54:04.830045 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.829123 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6acb13a1-cb2e-45e1-897b-3ef89e6d1b84-metrics-certs\") pod \"router-default-5bfdf864c-tzv67\" (UID: \"6acb13a1-cb2e-45e1-897b-3ef89e6d1b84\") " pod="openshift-ingress/router-default-5bfdf864c-tzv67" Oct 30 11:54:04.830045 ip-10-0-141-251 kubenswrapper[4908]: E1030 11:54:04.829150 4908 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cfaad7d0-d30b-4ddc-81cc-2b603416fc29-registry-tls podName:cfaad7d0-d30b-4ddc-81cc-2b603416fc29 nodeName:}" failed. No retries permitted until 2026-10-30 11:54:05.329141539 +0000 UTC m=+1.843570262 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cfaad7d0-d30b-4ddc-81cc-2b603416fc29-registry-tls") pod "image-registry-79fd565486-8fl78" (UID: "cfaad7d0-d30b-4ddc-81cc-2b603416fc29") : secret "image-registry-tls" not found Oct 30 11:54:04.830045 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.829148 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c54f4c1b-f08e-4495-9a14-17469a792ae5-sys-fs\") pod \"aws-ebs-csi-driver-node-h6mlf\" (UID: \"c54f4c1b-f08e-4495-9a14-17469a792ae5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h6mlf" Oct 30 11:54:04.830045 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.829159 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1b735438-5947-4374-bd4c-66ac3a299770-hosts-file\") pod \"node-resolver-xmrdj\" (UID: \"1b735438-5947-4374-bd4c-66ac3a299770\") " pod="openshift-dns/node-resolver-xmrdj" Oct 30 11:54:04.830045 ip-10-0-141-251 kubenswrapper[4908]: E1030 11:54:04.829179 4908 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 11:54:04.830045 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.829184 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d7f8396-e57c-45e5-ae74-a2244db10364-metrics-certs\") pod \"network-metrics-daemon-dbqhj\" (UID: \"1d7f8396-e57c-45e5-ae74-a2244db10364\") " pod="openshift-multus/network-metrics-daemon-dbqhj" Oct 30 11:54:04.830045 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.829184 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d659d471-f019-4853-97d0-ee6e948e6c1c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-s6k8x\" (UID: \"d659d471-f019-4853-97d0-ee6e948e6c1c\") " pod="openshift-multus/multus-additional-cni-plugins-s6k8x" Oct 30 11:54:04.830045 ip-10-0-141-251 kubenswrapper[4908]: E1030 11:54:04.829211 4908 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6acb13a1-cb2e-45e1-897b-3ef89e6d1b84-metrics-certs podName:6acb13a1-cb2e-45e1-897b-3ef89e6d1b84 nodeName:}" failed. No retries permitted until 2026-10-30 11:54:05.329202159 +0000 UTC m=+1.843630895 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6acb13a1-cb2e-45e1-897b-3ef89e6d1b84-metrics-certs") pod "router-default-5bfdf864c-tzv67" (UID: "6acb13a1-cb2e-45e1-897b-3ef89e6d1b84") : secret "router-metrics-certs-default" not found Oct 30 11:54:04.830045 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.829223 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6b350710-c48c-4c7e-bce9-e021099e98ca-kubelet-config\") pod \"global-pull-secret-syncer-fxjlt\" (UID: \"6b350710-c48c-4c7e-bce9-e021099e98ca\") " pod="kube-system/global-pull-secret-syncer-fxjlt" Oct 30 11:54:04.830045 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.829239 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5aee536a-8942-4b48-9b2c-3e36089f3ac5-host\") pod \"node-ca-hb8j7\" (UID: \"5aee536a-8942-4b48-9b2c-3e36089f3ac5\") " pod="openshift-image-registry/node-ca-hb8j7" Oct 30 11:54:04.830045 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.829258 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6acb13a1-cb2e-45e1-897b-3ef89e6d1b84-service-ca-bundle\") pod \"router-default-5bfdf864c-tzv67\" (UID: \"6acb13a1-cb2e-45e1-897b-3ef89e6d1b84\") " pod="openshift-ingress/router-default-5bfdf864c-tzv67" Oct 30 11:54:04.830045 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.829267 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6b350710-c48c-4c7e-bce9-e021099e98ca-kubelet-config\") pod \"global-pull-secret-syncer-fxjlt\" (UID: \"6b350710-c48c-4c7e-bce9-e021099e98ca\") " pod="kube-system/global-pull-secret-syncer-fxjlt" Oct 30 11:54:04.830045 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.829281 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0df1a118-36a5-4c57-8cec-2c8236f1c4de-etc-kubernetes\") pod \"multus-tlwp4\" (UID: \"0df1a118-36a5-4c57-8cec-2c8236f1c4de\") " pod="openshift-multus/multus-tlwp4" Oct 30 11:54:04.830045 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.829306 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d659d471-f019-4853-97d0-ee6e948e6c1c-cnibin\") pod \"multus-additional-cni-plugins-s6k8x\" (UID: \"d659d471-f019-4853-97d0-ee6e948e6c1c\") " pod="openshift-multus/multus-additional-cni-plugins-s6k8x" Oct 30 11:54:04.830045 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.829283 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5aee536a-8942-4b48-9b2c-3e36089f3ac5-host\") pod \"node-ca-hb8j7\" (UID: \"5aee536a-8942-4b48-9b2c-3e36089f3ac5\") " pod="openshift-image-registry/node-ca-hb8j7" Oct 30 11:54:04.830045 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.829319 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0df1a118-36a5-4c57-8cec-2c8236f1c4de-etc-kubernetes\") pod \"multus-tlwp4\" (UID: \"0df1a118-36a5-4c57-8cec-2c8236f1c4de\") " pod="openshift-multus/multus-tlwp4" Oct 30 11:54:04.830045 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.829328 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7e8ec7ff-015c-4049-9c5f-aae73dc06e4b-host-slash\") pod \"iptables-alerter-8m2w9\" (UID: \"7e8ec7ff-015c-4049-9c5f-aae73dc06e4b\") " pod="openshift-network-operator/iptables-alerter-8m2w9" Oct 30 11:54:04.830045 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.829352 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3699031f-ee9e-46cc-b37e-4dfc76b9c0b1-var-lib-kubelet\") pod \"tuned-587rx\" (UID: \"3699031f-ee9e-46cc-b37e-4dfc76b9c0b1\") " pod="openshift-cluster-node-tuning-operator/tuned-587rx" Oct 30 11:54:04.830045 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.829367 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d659d471-f019-4853-97d0-ee6e948e6c1c-cnibin\") pod \"multus-additional-cni-plugins-s6k8x\" (UID: \"d659d471-f019-4853-97d0-ee6e948e6c1c\") " pod="openshift-multus/multus-additional-cni-plugins-s6k8x" Oct 30 11:54:04.830045 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.829368 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d670b364-53ce-444b-b9c4-ac8ac09cd44f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vn6vv\" (UID: \"d670b364-53ce-444b-b9c4-ac8ac09cd44f\") " pod="openshift-insights/insights-runtime-extractor-vn6vv" Oct 30 11:54:04.830045 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.829405 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0df1a118-36a5-4c57-8cec-2c8236f1c4de-multus-cni-dir\") pod \"multus-tlwp4\" (UID: \"0df1a118-36a5-4c57-8cec-2c8236f1c4de\") " pod="openshift-multus/multus-tlwp4" Oct 30 11:54:04.830045 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.829428 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d659d471-f019-4853-97d0-ee6e948e6c1c-system-cni-dir\") pod \"multus-additional-cni-plugins-s6k8x\" (UID: \"d659d471-f019-4853-97d0-ee6e948e6c1c\") " pod="openshift-multus/multus-additional-cni-plugins-s6k8x" Oct 30 11:54:04.830045 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.829427 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7e8ec7ff-015c-4049-9c5f-aae73dc06e4b-host-slash\") pod \"iptables-alerter-8m2w9\" (UID: \"7e8ec7ff-015c-4049-9c5f-aae73dc06e4b\") " pod="openshift-network-operator/iptables-alerter-8m2w9" Oct 30 11:54:04.830045 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.829447 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3699031f-ee9e-46cc-b37e-4dfc76b9c0b1-var-lib-kubelet\") pod \"tuned-587rx\" (UID: \"3699031f-ee9e-46cc-b37e-4dfc76b9c0b1\") " pod="openshift-cluster-node-tuning-operator/tuned-587rx" Oct 30 11:54:04.830045 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.829480 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d659d471-f019-4853-97d0-ee6e948e6c1c-system-cni-dir\") pod \"multus-additional-cni-plugins-s6k8x\" (UID: \"d659d471-f019-4853-97d0-ee6e948e6c1c\") " pod="openshift-multus/multus-additional-cni-plugins-s6k8x" Oct 30 11:54:04.830045 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.829483 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0df1a118-36a5-4c57-8cec-2c8236f1c4de-multus-cni-dir\") pod \"multus-tlwp4\" (UID: \"0df1a118-36a5-4c57-8cec-2c8236f1c4de\") " pod="openshift-multus/multus-tlwp4" Oct 30 11:54:04.830045 ip-10-0-141-251 kubenswrapper[4908]: E1030 11:54:04.829500 4908 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 11:54:04.830045 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.829502 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3699031f-ee9e-46cc-b37e-4dfc76b9c0b1-etc-systemd\") pod \"tuned-587rx\" (UID: \"3699031f-ee9e-46cc-b37e-4dfc76b9c0b1\") " pod="openshift-cluster-node-tuning-operator/tuned-587rx" Oct 30 11:54:04.830045 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.829548 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/644e19b3-10ad-4145-9ed3-595d19ed9b07-host-run-ovn-kubernetes\") pod \"ovnkube-node-b8bxz\" (UID: \"644e19b3-10ad-4145-9ed3-595d19ed9b07\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8bxz" Oct 30 11:54:04.830045 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.829570 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/644e19b3-10ad-4145-9ed3-595d19ed9b07-host-run-ovn-kubernetes\") pod \"ovnkube-node-b8bxz\" (UID: \"644e19b3-10ad-4145-9ed3-595d19ed9b07\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8bxz" Oct 30 11:54:04.830045 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.829558 4908 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 11:54:04.830045 ip-10-0-141-251 kubenswrapper[4908]: E1030 11:54:04.829586 4908 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d670b364-53ce-444b-b9c4-ac8ac09cd44f-insights-runtime-extractor-tls podName:d670b364-53ce-444b-b9c4-ac8ac09cd44f nodeName:}" failed. No retries permitted until 2026-10-30 11:54:05.329553112 +0000 UTC m=+1.843981861 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d670b364-53ce-444b-b9c4-ac8ac09cd44f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vn6vv" (UID: "d670b364-53ce-444b-b9c4-ac8ac09cd44f") : secret "insights-runtime-extractor-tls" not found Oct 30 11:54:04.830045 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.829617 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3699031f-ee9e-46cc-b37e-4dfc76b9c0b1-etc-systemd\") pod \"tuned-587rx\" (UID: \"3699031f-ee9e-46cc-b37e-4dfc76b9c0b1\") " pod="openshift-cluster-node-tuning-operator/tuned-587rx" Oct 30 11:54:04.830045 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.829619 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/644e19b3-10ad-4145-9ed3-595d19ed9b07-var-lib-openvswitch\") pod \"ovnkube-node-b8bxz\" (UID: \"644e19b3-10ad-4145-9ed3-595d19ed9b07\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8bxz" Oct 30 11:54:04.830045 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.829654 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c54f4c1b-f08e-4495-9a14-17469a792ae5-etc-selinux\") pod \"aws-ebs-csi-driver-node-h6mlf\" (UID: \"c54f4c1b-f08e-4495-9a14-17469a792ae5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h6mlf" Oct 30 11:54:04.830045 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.829657 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/644e19b3-10ad-4145-9ed3-595d19ed9b07-var-lib-openvswitch\") pod \"ovnkube-node-b8bxz\" (UID: \"644e19b3-10ad-4145-9ed3-595d19ed9b07\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8bxz" Oct 30 11:54:04.830045 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.829179 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/644e19b3-10ad-4145-9ed3-595d19ed9b07-host-cni-netd\") pod \"ovnkube-node-b8bxz\" (UID: \"644e19b3-10ad-4145-9ed3-595d19ed9b07\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8bxz" Oct 30 11:54:04.830045 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.829693 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/644e19b3-10ad-4145-9ed3-595d19ed9b07-systemd-units\") pod \"ovnkube-node-b8bxz\" (UID: \"644e19b3-10ad-4145-9ed3-595d19ed9b07\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8bxz" Oct 30 11:54:04.830045 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.829709 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c54f4c1b-f08e-4495-9a14-17469a792ae5-etc-selinux\") pod \"aws-ebs-csi-driver-node-h6mlf\" (UID: \"c54f4c1b-f08e-4495-9a14-17469a792ae5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h6mlf" Oct 30 11:54:04.830045 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.829716 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3699031f-ee9e-46cc-b37e-4dfc76b9c0b1-run\") pod \"tuned-587rx\" (UID: \"3699031f-ee9e-46cc-b37e-4dfc76b9c0b1\") " pod="openshift-cluster-node-tuning-operator/tuned-587rx" Oct 30 11:54:04.830045 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.829740 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6d976900-ecd0-469c-a9e2-d60708361f1a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-mvhwk\" (UID: \"6d976900-ecd0-469c-a9e2-d60708361f1a\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-mvhwk" Oct 30 11:54:04.830045 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.829756 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/644e19b3-10ad-4145-9ed3-595d19ed9b07-systemd-units\") pod \"ovnkube-node-b8bxz\" (UID: \"644e19b3-10ad-4145-9ed3-595d19ed9b07\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8bxz" Oct 30 11:54:04.830045 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.829765 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3699031f-ee9e-46cc-b37e-4dfc76b9c0b1-run\") pod \"tuned-587rx\" (UID: \"3699031f-ee9e-46cc-b37e-4dfc76b9c0b1\") " pod="openshift-cluster-node-tuning-operator/tuned-587rx" Oct 30 11:54:04.830045 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.829761 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/644e19b3-10ad-4145-9ed3-595d19ed9b07-run-ovn\") pod \"ovnkube-node-b8bxz\" (UID: \"644e19b3-10ad-4145-9ed3-595d19ed9b07\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8bxz" Oct 30 11:54:04.830045 ip-10-0-141-251 kubenswrapper[4908]: E1030 11:54:04.829804 4908 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 11:54:04.830045 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.829813 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0df1a118-36a5-4c57-8cec-2c8236f1c4de-host-var-lib-kubelet\") pod \"multus-tlwp4\" (UID: \"0df1a118-36a5-4c57-8cec-2c8236f1c4de\") " pod="openshift-multus/multus-tlwp4" Oct 30 11:54:04.830045 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.829863 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/644e19b3-10ad-4145-9ed3-595d19ed9b07-run-ovn\") pod \"ovnkube-node-b8bxz\" (UID: \"644e19b3-10ad-4145-9ed3-595d19ed9b07\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8bxz" Oct 30 11:54:04.830045 ip-10-0-141-251 kubenswrapper[4908]: E1030 11:54:04.829855 4908 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d976900-ecd0-469c-a9e2-d60708361f1a-cluster-monitoring-operator-tls podName:6d976900-ecd0-469c-a9e2-d60708361f1a nodeName:}" failed. No retries permitted until 2026-10-30 11:54:05.329824049 +0000 UTC m=+1.844252812 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/6d976900-ecd0-469c-a9e2-d60708361f1a-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-mvhwk" (UID: "6d976900-ecd0-469c-a9e2-d60708361f1a") : secret "cluster-monitoring-operator-tls" not found Oct 30 11:54:04.830045 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.829868 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0df1a118-36a5-4c57-8cec-2c8236f1c4de-host-var-lib-kubelet\") pod \"multus-tlwp4\" (UID: \"0df1a118-36a5-4c57-8cec-2c8236f1c4de\") " pod="openshift-multus/multus-tlwp4" Oct 30 11:54:04.830045 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.829895 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6acb13a1-cb2e-45e1-897b-3ef89e6d1b84-service-ca-bundle\") pod \"router-default-5bfdf864c-tzv67\" (UID: \"6acb13a1-cb2e-45e1-897b-3ef89e6d1b84\") " pod="openshift-ingress/router-default-5bfdf864c-tzv67" Oct 30 11:54:04.830045 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.829908 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c54f4c1b-f08e-4495-9a14-17469a792ae5-registration-dir\") pod \"aws-ebs-csi-driver-node-h6mlf\" (UID: \"c54f4c1b-f08e-4495-9a14-17469a792ae5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h6mlf" Oct 30 11:54:04.830045 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.829925 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/644e19b3-10ad-4145-9ed3-595d19ed9b07-host-slash\") pod \"ovnkube-node-b8bxz\" (UID: \"644e19b3-10ad-4145-9ed3-595d19ed9b07\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8bxz" Oct 30 11:54:04.830045 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.829948 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c54f4c1b-f08e-4495-9a14-17469a792ae5-socket-dir\") pod \"aws-ebs-csi-driver-node-h6mlf\" (UID: \"c54f4c1b-f08e-4495-9a14-17469a792ae5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h6mlf" Oct 30 11:54:04.830045 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.829964 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c54f4c1b-f08e-4495-9a14-17469a792ae5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-h6mlf\" (UID: \"c54f4c1b-f08e-4495-9a14-17469a792ae5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h6mlf" Oct 30 11:54:04.830045 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.829967 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c54f4c1b-f08e-4495-9a14-17469a792ae5-registration-dir\") pod \"aws-ebs-csi-driver-node-h6mlf\" (UID: \"c54f4c1b-f08e-4495-9a14-17469a792ae5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h6mlf" Oct 30 11:54:04.830045 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.829988 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/644e19b3-10ad-4145-9ed3-595d19ed9b07-host-slash\") pod \"ovnkube-node-b8bxz\" (UID: \"644e19b3-10ad-4145-9ed3-595d19ed9b07\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8bxz" Oct 30 11:54:04.830045 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.829985 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/644e19b3-10ad-4145-9ed3-595d19ed9b07-node-log\") pod \"ovnkube-node-b8bxz\" (UID: \"644e19b3-10ad-4145-9ed3-595d19ed9b07\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8bxz" Oct 30 11:54:04.830045 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.830020 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c54f4c1b-f08e-4495-9a14-17469a792ae5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-h6mlf\" (UID: \"c54f4c1b-f08e-4495-9a14-17469a792ae5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h6mlf" Oct 30 11:54:04.830045 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.830031 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/221ced85-cc15-4e07-82d9-a1a1f3e958a5-metrics-tls\") pod \"dns-default-xxcbn\" (UID: \"221ced85-cc15-4e07-82d9-a1a1f3e958a5\") " pod="openshift-dns/dns-default-xxcbn" Oct 30 11:54:04.830045 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.830058 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/644e19b3-10ad-4145-9ed3-595d19ed9b07-node-log\") pod \"ovnkube-node-b8bxz\" (UID: \"644e19b3-10ad-4145-9ed3-595d19ed9b07\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8bxz" Oct 30 11:54:04.830045 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.830026 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c54f4c1b-f08e-4495-9a14-17469a792ae5-socket-dir\") pod \"aws-ebs-csi-driver-node-h6mlf\" (UID: \"c54f4c1b-f08e-4495-9a14-17469a792ae5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h6mlf" Oct 30 11:54:04.830045 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.830059 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3699031f-ee9e-46cc-b37e-4dfc76b9c0b1-etc-kubernetes\") pod \"tuned-587rx\" (UID: \"3699031f-ee9e-46cc-b37e-4dfc76b9c0b1\") " pod="openshift-cluster-node-tuning-operator/tuned-587rx" Oct 30 11:54:04.832362 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.830103 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3699031f-ee9e-46cc-b37e-4dfc76b9c0b1-etc-kubernetes\") pod \"tuned-587rx\" (UID: \"3699031f-ee9e-46cc-b37e-4dfc76b9c0b1\") " pod="openshift-cluster-node-tuning-operator/tuned-587rx" Oct 30 11:54:04.832362 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.830116 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3699031f-ee9e-46cc-b37e-4dfc76b9c0b1-etc-sysctl-d\") pod \"tuned-587rx\" (UID: \"3699031f-ee9e-46cc-b37e-4dfc76b9c0b1\") " pod="openshift-cluster-node-tuning-operator/tuned-587rx" Oct 30 11:54:04.832362 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.830134 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/644e19b3-10ad-4145-9ed3-595d19ed9b07-host-kubelet\") pod \"ovnkube-node-b8bxz\" (UID: \"644e19b3-10ad-4145-9ed3-595d19ed9b07\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8bxz" Oct 30 11:54:04.832362 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.830148 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6b350710-c48c-4c7e-bce9-e021099e98ca-dbus\") pod \"global-pull-secret-syncer-fxjlt\" (UID: \"6b350710-c48c-4c7e-bce9-e021099e98ca\") " pod="kube-system/global-pull-secret-syncer-fxjlt" Oct 30 11:54:04.832362 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.830160 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c54f4c1b-f08e-4495-9a14-17469a792ae5-device-dir\") pod \"aws-ebs-csi-driver-node-h6mlf\" (UID: \"c54f4c1b-f08e-4495-9a14-17469a792ae5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h6mlf" Oct 30 11:54:04.832362 ip-10-0-141-251 kubenswrapper[4908]: E1030 11:54:04.830118 4908 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 11:54:04.832362 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.830200 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/644e19b3-10ad-4145-9ed3-595d19ed9b07-host-kubelet\") pod \"ovnkube-node-b8bxz\" (UID: \"644e19b3-10ad-4145-9ed3-595d19ed9b07\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8bxz" Oct 30 11:54:04.832362 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.830207 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c54f4c1b-f08e-4495-9a14-17469a792ae5-device-dir\") pod \"aws-ebs-csi-driver-node-h6mlf\" (UID: \"c54f4c1b-f08e-4495-9a14-17469a792ae5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h6mlf" Oct 30 11:54:04.832362 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.830209 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6b350710-c48c-4c7e-bce9-e021099e98ca-dbus\") pod \"global-pull-secret-syncer-fxjlt\" (UID: \"6b350710-c48c-4c7e-bce9-e021099e98ca\") " pod="kube-system/global-pull-secret-syncer-fxjlt" Oct 30 11:54:04.832362 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.830225 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3699031f-ee9e-46cc-b37e-4dfc76b9c0b1-etc-sysctl-d\") pod \"tuned-587rx\" (UID: \"3699031f-ee9e-46cc-b37e-4dfc76b9c0b1\") " pod="openshift-cluster-node-tuning-operator/tuned-587rx" Oct 30 11:54:04.832362 ip-10-0-141-251 kubenswrapper[4908]: E1030 11:54:04.830235 4908 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/221ced85-cc15-4e07-82d9-a1a1f3e958a5-metrics-tls podName:221ced85-cc15-4e07-82d9-a1a1f3e958a5 nodeName:}" failed. No retries permitted until 2026-10-30 11:54:05.33022112 +0000 UTC m=+1.844649848 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/221ced85-cc15-4e07-82d9-a1a1f3e958a5-metrics-tls") pod "dns-default-xxcbn" (UID: "221ced85-cc15-4e07-82d9-a1a1f3e958a5") : secret "dns-default-metrics-tls" not found Oct 30 11:54:04.832362 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.830228 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d670b364-53ce-444b-b9c4-ac8ac09cd44f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vn6vv\" (UID: \"d670b364-53ce-444b-b9c4-ac8ac09cd44f\") " pod="openshift-insights/insights-runtime-extractor-vn6vv" Oct 30 11:54:04.832362 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.830270 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/644e19b3-10ad-4145-9ed3-595d19ed9b07-log-socket\") pod \"ovnkube-node-b8bxz\" (UID: \"644e19b3-10ad-4145-9ed3-595d19ed9b07\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8bxz" Oct 30 11:54:04.832362 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.830297 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0df1a118-36a5-4c57-8cec-2c8236f1c4de-multus-socket-dir-parent\") pod \"multus-tlwp4\" (UID: \"0df1a118-36a5-4c57-8cec-2c8236f1c4de\") " pod="openshift-multus/multus-tlwp4" Oct 30 11:54:04.832362 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.830319 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/644e19b3-10ad-4145-9ed3-595d19ed9b07-log-socket\") pod \"ovnkube-node-b8bxz\" (UID: \"644e19b3-10ad-4145-9ed3-595d19ed9b07\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8bxz" Oct 30 11:54:04.832362 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.830321 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7689cfe5-5a4e-4481-8eb9-107a58ca8bb3-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-xdxkp\" (UID: \"7689cfe5-5a4e-4481-8eb9-107a58ca8bb3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xdxkp" Oct 30 11:54:04.832362 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.830354 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0df1a118-36a5-4c57-8cec-2c8236f1c4de-system-cni-dir\") pod \"multus-tlwp4\" (UID: \"0df1a118-36a5-4c57-8cec-2c8236f1c4de\") " pod="openshift-multus/multus-tlwp4" Oct 30 11:54:04.832362 ip-10-0-141-251 kubenswrapper[4908]: E1030 11:54:04.830380 4908 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 11:54:04.832362 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.830381 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/644e19b3-10ad-4145-9ed3-595d19ed9b07-host-run-netns\") pod \"ovnkube-node-b8bxz\" (UID: \"644e19b3-10ad-4145-9ed3-595d19ed9b07\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8bxz" Oct 30 11:54:04.832362 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.830398 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0df1a118-36a5-4c57-8cec-2c8236f1c4de-multus-socket-dir-parent\") pod \"multus-tlwp4\" (UID: \"0df1a118-36a5-4c57-8cec-2c8236f1c4de\") " pod="openshift-multus/multus-tlwp4" Oct 30 11:54:04.832362 ip-10-0-141-251 kubenswrapper[4908]: E1030 11:54:04.830425 4908 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7689cfe5-5a4e-4481-8eb9-107a58ca8bb3-samples-operator-tls podName:7689cfe5-5a4e-4481-8eb9-107a58ca8bb3 nodeName:}" failed. No retries permitted until 2026-10-30 11:54:05.330411054 +0000 UTC m=+1.844839783 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7689cfe5-5a4e-4481-8eb9-107a58ca8bb3-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-xdxkp" (UID: "7689cfe5-5a4e-4481-8eb9-107a58ca8bb3") : secret "samples-operator-tls" not found Oct 30 11:54:04.832362 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.830432 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/644e19b3-10ad-4145-9ed3-595d19ed9b07-host-run-netns\") pod \"ovnkube-node-b8bxz\" (UID: \"644e19b3-10ad-4145-9ed3-595d19ed9b07\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8bxz" Oct 30 11:54:04.832362 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.830439 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0df1a118-36a5-4c57-8cec-2c8236f1c4de-system-cni-dir\") pod \"multus-tlwp4\" (UID: \"0df1a118-36a5-4c57-8cec-2c8236f1c4de\") " pod="openshift-multus/multus-tlwp4" Oct 30 11:54:04.832362 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.830445 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/644e19b3-10ad-4145-9ed3-595d19ed9b07-etc-openvswitch\") pod \"ovnkube-node-b8bxz\" (UID: \"644e19b3-10ad-4145-9ed3-595d19ed9b07\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8bxz" Oct 30 11:54:04.832362 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.830469 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0df1a118-36a5-4c57-8cec-2c8236f1c4de-host-run-k8s-cni-cncf-io\") pod \"multus-tlwp4\" (UID: \"0df1a118-36a5-4c57-8cec-2c8236f1c4de\") " pod="openshift-multus/multus-tlwp4" Oct 30 11:54:04.832362 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.830476 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/644e19b3-10ad-4145-9ed3-595d19ed9b07-etc-openvswitch\") pod \"ovnkube-node-b8bxz\" (UID: \"644e19b3-10ad-4145-9ed3-595d19ed9b07\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8bxz" Oct 30 11:54:04.832362 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.830502 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/644e19b3-10ad-4145-9ed3-595d19ed9b07-host-cni-bin\") pod \"ovnkube-node-b8bxz\" (UID: \"644e19b3-10ad-4145-9ed3-595d19ed9b07\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8bxz" Oct 30 11:54:04.832362 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.830526 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3699031f-ee9e-46cc-b37e-4dfc76b9c0b1-etc-sysconfig\") pod \"tuned-587rx\" (UID: \"3699031f-ee9e-46cc-b37e-4dfc76b9c0b1\") " pod="openshift-cluster-node-tuning-operator/tuned-587rx" Oct 30 11:54:04.832362 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.830536 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0df1a118-36a5-4c57-8cec-2c8236f1c4de-host-run-k8s-cni-cncf-io\") pod \"multus-tlwp4\" (UID: \"0df1a118-36a5-4c57-8cec-2c8236f1c4de\") " pod="openshift-multus/multus-tlwp4" Oct 30 11:54:04.832362 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.830559 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/644e19b3-10ad-4145-9ed3-595d19ed9b07-host-cni-bin\") pod \"ovnkube-node-b8bxz\" (UID: \"644e19b3-10ad-4145-9ed3-595d19ed9b07\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8bxz" Oct 30 11:54:04.832362 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.830597 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3699031f-ee9e-46cc-b37e-4dfc76b9c0b1-etc-sysconfig\") pod \"tuned-587rx\" (UID: \"3699031f-ee9e-46cc-b37e-4dfc76b9c0b1\") " pod="openshift-cluster-node-tuning-operator/tuned-587rx" Oct 30 11:54:04.832362 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.830684 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d670b364-53ce-444b-b9c4-ac8ac09cd44f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vn6vv\" (UID: \"d670b364-53ce-444b-b9c4-ac8ac09cd44f\") " pod="openshift-insights/insights-runtime-extractor-vn6vv" Oct 30 11:54:04.832362 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.831726 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d7f8396-e57c-45e5-ae74-a2244db10364-metrics-certs\") pod \"network-metrics-daemon-dbqhj\" (UID: \"1d7f8396-e57c-45e5-ae74-a2244db10364\") " pod="openshift-multus/network-metrics-daemon-dbqhj" Oct 30 11:54:04.910704 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.910681 4908 scope.go:122] "RemoveContainer" containerID="cc5a67034e1729623ea0114fef781e8204b8eab853c4f28263d4d9d9c45f99fd" Oct 30 11:54:04.913745 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:04.913731 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dbqhj" Oct 30 11:54:05.162732 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:05.162684 4908 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-plczt" Oct 30 11:54:05.240194 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:05.240173 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-dbqhj"] Oct 30 11:54:05.245010 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:05.244975 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d7f8396_e57c_45e5_ae74_a2244db10364.slice/crio-4ae279c0aa251c704e02f973ab0c4298197a0da18902a5a4e8f51e259ce90630 WatchSource:0}: Error finding container 4ae279c0aa251c704e02f973ab0c4298197a0da18902a5a4e8f51e259ce90630: Status 404 returned error can't find the container with id 4ae279c0aa251c704e02f973ab0c4298197a0da18902a5a4e8f51e259ce90630 Oct 30 11:54:05.246949 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:05.246932 4908 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 11:54:05.334700 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:05.334678 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/179d55d3-7958-4928-a98c-68984fca9837-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-zzxhv\" (UID: \"179d55d3-7958-4928-a98c-68984fca9837\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-zzxhv" Oct 30 11:54:05.334804 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:05.334713 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cfaad7d0-d30b-4ddc-81cc-2b603416fc29-registry-tls\") pod \"image-registry-79fd565486-8fl78\" (UID: \"cfaad7d0-d30b-4ddc-81cc-2b603416fc29\") " pod="openshift-image-registry/image-registry-79fd565486-8fl78" Oct 30 11:54:05.334804 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:05.334736 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6acb13a1-cb2e-45e1-897b-3ef89e6d1b84-metrics-certs\") pod \"router-default-5bfdf864c-tzv67\" (UID: \"6acb13a1-cb2e-45e1-897b-3ef89e6d1b84\") " pod="openshift-ingress/router-default-5bfdf864c-tzv67" Oct 30 11:54:05.334804 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:05.334757 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d670b364-53ce-444b-b9c4-ac8ac09cd44f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vn6vv\" (UID: \"d670b364-53ce-444b-b9c4-ac8ac09cd44f\") " pod="openshift-insights/insights-runtime-extractor-vn6vv" Oct 30 11:54:05.334804 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:05.334790 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6d976900-ecd0-469c-a9e2-d60708361f1a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-mvhwk\" (UID: \"6d976900-ecd0-469c-a9e2-d60708361f1a\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-mvhwk" Oct 30 11:54:05.335021 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:05.334815 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/221ced85-cc15-4e07-82d9-a1a1f3e958a5-metrics-tls\") pod \"dns-default-xxcbn\" (UID: \"221ced85-cc15-4e07-82d9-a1a1f3e958a5\") " pod="openshift-dns/dns-default-xxcbn" Oct 30 11:54:05.335021 ip-10-0-141-251 kubenswrapper[4908]: E1030 11:54:05.334849 4908 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 11:54:05.335021 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:05.334858 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7689cfe5-5a4e-4481-8eb9-107a58ca8bb3-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-xdxkp\" (UID: \"7689cfe5-5a4e-4481-8eb9-107a58ca8bb3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xdxkp" Oct 30 11:54:05.335021 ip-10-0-141-251 kubenswrapper[4908]: E1030 11:54:05.334909 4908 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6acb13a1-cb2e-45e1-897b-3ef89e6d1b84-metrics-certs podName:6acb13a1-cb2e-45e1-897b-3ef89e6d1b84 nodeName:}" failed. No retries permitted until 2026-10-30 11:54:06.334890554 +0000 UTC m=+2.849319277 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6acb13a1-cb2e-45e1-897b-3ef89e6d1b84-metrics-certs") pod "router-default-5bfdf864c-tzv67" (UID: "6acb13a1-cb2e-45e1-897b-3ef89e6d1b84") : secret "router-metrics-certs-default" not found Oct 30 11:54:05.335021 ip-10-0-141-251 kubenswrapper[4908]: E1030 11:54:05.334928 4908 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 11:54:05.335021 ip-10-0-141-251 kubenswrapper[4908]: E1030 11:54:05.334947 4908 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 11:54:05.335021 ip-10-0-141-251 kubenswrapper[4908]: E1030 11:54:05.334956 4908 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 11:54:05.335021 ip-10-0-141-251 kubenswrapper[4908]: E1030 11:54:05.334973 4908 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7689cfe5-5a4e-4481-8eb9-107a58ca8bb3-samples-operator-tls podName:7689cfe5-5a4e-4481-8eb9-107a58ca8bb3 nodeName:}" failed. No retries permitted until 2026-10-30 11:54:06.334959738 +0000 UTC m=+2.849388465 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7689cfe5-5a4e-4481-8eb9-107a58ca8bb3-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-xdxkp" (UID: "7689cfe5-5a4e-4481-8eb9-107a58ca8bb3") : secret "samples-operator-tls" not found Oct 30 11:54:05.335021 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:05.334956 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bd977255-f9a7-4531-8a9c-b1a3a17e0b79-registry-tls\") pod \"image-registry-557c6b9f47-vcr2t\" (UID: \"bd977255-f9a7-4531-8a9c-b1a3a17e0b79\") " pod="openshift-image-registry/image-registry-557c6b9f47-vcr2t" Oct 30 11:54:05.335021 ip-10-0-141-251 kubenswrapper[4908]: E1030 11:54:05.335011 4908 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d976900-ecd0-469c-a9e2-d60708361f1a-cluster-monitoring-operator-tls podName:6d976900-ecd0-469c-a9e2-d60708361f1a nodeName:}" failed. No retries permitted until 2026-10-30 11:54:06.335000256 +0000 UTC m=+2.849428996 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/6d976900-ecd0-469c-a9e2-d60708361f1a-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-mvhwk" (UID: "6d976900-ecd0-469c-a9e2-d60708361f1a") : secret "cluster-monitoring-operator-tls" not found Oct 30 11:54:05.335021 ip-10-0-141-251 kubenswrapper[4908]: E1030 11:54:05.335029 4908 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d670b364-53ce-444b-b9c4-ac8ac09cd44f-insights-runtime-extractor-tls podName:d670b364-53ce-444b-b9c4-ac8ac09cd44f nodeName:}" failed. No retries permitted until 2026-10-30 11:54:06.335023336 +0000 UTC m=+2.849452059 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d670b364-53ce-444b-b9c4-ac8ac09cd44f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vn6vv" (UID: "d670b364-53ce-444b-b9c4-ac8ac09cd44f") : secret "insights-runtime-extractor-tls" not found Oct 30 11:54:05.335393 ip-10-0-141-251 kubenswrapper[4908]: E1030 11:54:05.335292 4908 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 11:54:05.335393 ip-10-0-141-251 kubenswrapper[4908]: E1030 11:54:05.335350 4908 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/221ced85-cc15-4e07-82d9-a1a1f3e958a5-metrics-tls podName:221ced85-cc15-4e07-82d9-a1a1f3e958a5 nodeName:}" failed. No retries permitted until 2026-10-30 11:54:06.335336225 +0000 UTC m=+2.849764963 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/221ced85-cc15-4e07-82d9-a1a1f3e958a5-metrics-tls") pod "dns-default-xxcbn" (UID: "221ced85-cc15-4e07-82d9-a1a1f3e958a5") : secret "dns-default-metrics-tls" not found Oct 30 11:54:05.336945 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:05.336926 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/179d55d3-7958-4928-a98c-68984fca9837-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-zzxhv\" (UID: \"179d55d3-7958-4928-a98c-68984fca9837\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-zzxhv" Oct 30 11:54:05.337379 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:05.337361 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cfaad7d0-d30b-4ddc-81cc-2b603416fc29-registry-tls\") pod \"image-registry-79fd565486-8fl78\" (UID: \"cfaad7d0-d30b-4ddc-81cc-2b603416fc29\") " pod="openshift-image-registry/image-registry-79fd565486-8fl78" Oct 30 11:54:05.337425 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:05.337389 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bd977255-f9a7-4531-8a9c-b1a3a17e0b79-registry-tls\") pod \"image-registry-557c6b9f47-vcr2t\" (UID: \"bd977255-f9a7-4531-8a9c-b1a3a17e0b79\") " pod="openshift-image-registry/image-registry-557c6b9f47-vcr2t" Oct 30 11:54:05.513422 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:05.513397 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-zzxhv" Oct 30 11:54:05.513517 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:05.513436 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-557c6b9f47-vcr2t" Oct 30 11:54:05.518518 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:05.518495 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-79fd565486-8fl78" Oct 30 11:54:05.655656 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:05.655630 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-dbqhj" event={"ID":"1d7f8396-e57c-45e5-ae74-a2244db10364","Type":"ContainerStarted","Data":"4ae279c0aa251c704e02f973ab0c4298197a0da18902a5a4e8f51e259ce90630"} Oct 30 11:54:05.657173 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:05.657156 4908 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-plczt_a70ee11c-c20e-4a4f-8476-ac5eb02816d0/console-operator/2.log" Oct 30 11:54:05.657580 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:05.657563 4908 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-plczt_a70ee11c-c20e-4a4f-8476-ac5eb02816d0/console-operator/1.log" Oct 30 11:54:05.657652 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:05.657600 4908 generic.go:356] "Generic (PLEG): container finished" podID="a70ee11c-c20e-4a4f-8476-ac5eb02816d0" containerID="e269d4e97647eeac46f1b7b961248f6a6b0c439df2aa6aaed35717b2e3781f6e" exitCode=255 Oct 30 11:54:05.657702 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:05.657682 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-plczt" event={"ID":"a70ee11c-c20e-4a4f-8476-ac5eb02816d0","Type":"ContainerDied","Data":"e269d4e97647eeac46f1b7b961248f6a6b0c439df2aa6aaed35717b2e3781f6e"} Oct 30 11:54:05.657749 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:05.657735 4908 scope.go:122] "RemoveContainer" containerID="cc5a67034e1729623ea0114fef781e8204b8eab853c4f28263d4d9d9c45f99fd" Oct 30 11:54:05.657971 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:05.657951 4908 scope.go:122] "RemoveContainer" containerID="e269d4e97647eeac46f1b7b961248f6a6b0c439df2aa6aaed35717b2e3781f6e" Oct 30 11:54:05.658226 ip-10-0-141-251 kubenswrapper[4908]: E1030 11:54:05.658203 4908 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-plczt_openshift-console-operator(a70ee11c-c20e-4a4f-8476-ac5eb02816d0)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-plczt" podUID="a70ee11c-c20e-4a4f-8476-ac5eb02816d0" Oct 30 11:54:05.809387 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:05.809329 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-zzxhv"] Oct 30 11:54:05.814729 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:05.814709 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod179d55d3_7958_4928_a98c_68984fca9837.slice/crio-869479b4bf6c180a3fb8621a3ade7bf33bfa97f7a84e478f7caa2969dcdf2298 WatchSource:0}: Error finding container 869479b4bf6c180a3fb8621a3ade7bf33bfa97f7a84e478f7caa2969dcdf2298: Status 404 returned error can't find the container with id 869479b4bf6c180a3fb8621a3ade7bf33bfa97f7a84e478f7caa2969dcdf2298 Oct 30 11:54:05.828577 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:05.828555 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-557c6b9f47-vcr2t"] Oct 30 11:54:05.830984 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:05.830959 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-79fd565486-8fl78"] Oct 30 11:54:05.832888 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:05.832801 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd977255_f9a7_4531_8a9c_b1a3a17e0b79.slice/crio-914e9bb474175e0e9380bc28a8141b3a8b4db1b61e5beede916579ec3f61c73f WatchSource:0}: Error finding container 914e9bb474175e0e9380bc28a8141b3a8b4db1b61e5beede916579ec3f61c73f: Status 404 returned error can't find the container with id 914e9bb474175e0e9380bc28a8141b3a8b4db1b61e5beede916579ec3f61c73f Oct 30 11:54:05.836114 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:05.835952 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfaad7d0_d30b_4ddc_81cc_2b603416fc29.slice/crio-8ff668624ba16b6f4bf01b1c8f33218ee6244ab3f8dae1e74ae504963d7b4ebe WatchSource:0}: Error finding container 8ff668624ba16b6f4bf01b1c8f33218ee6244ab3f8dae1e74ae504963d7b4ebe: Status 404 returned error can't find the container with id 8ff668624ba16b6f4bf01b1c8f33218ee6244ab3f8dae1e74ae504963d7b4ebe Oct 30 11:54:06.266310 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:06.266283 4908 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-xmrdj_1b735438-5947-4374-bd4c-66ac3a299770/dns-node-resolver/0.log" Oct 30 11:54:06.342803 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:06.342770 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6acb13a1-cb2e-45e1-897b-3ef89e6d1b84-metrics-certs\") pod \"router-default-5bfdf864c-tzv67\" (UID: \"6acb13a1-cb2e-45e1-897b-3ef89e6d1b84\") " pod="openshift-ingress/router-default-5bfdf864c-tzv67" Oct 30 11:54:06.342998 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:06.342835 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d670b364-53ce-444b-b9c4-ac8ac09cd44f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vn6vv\" (UID: \"d670b364-53ce-444b-b9c4-ac8ac09cd44f\") " pod="openshift-insights/insights-runtime-extractor-vn6vv" Oct 30 11:54:06.342998 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:06.342881 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6d976900-ecd0-469c-a9e2-d60708361f1a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-mvhwk\" (UID: \"6d976900-ecd0-469c-a9e2-d60708361f1a\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-mvhwk" Oct 30 11:54:06.342998 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:06.342919 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/221ced85-cc15-4e07-82d9-a1a1f3e958a5-metrics-tls\") pod \"dns-default-xxcbn\" (UID: \"221ced85-cc15-4e07-82d9-a1a1f3e958a5\") " pod="openshift-dns/dns-default-xxcbn" Oct 30 11:54:06.342998 ip-10-0-141-251 kubenswrapper[4908]: E1030 11:54:06.342974 4908 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 11:54:06.343201 ip-10-0-141-251 kubenswrapper[4908]: E1030 11:54:06.343006 4908 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 11:54:06.343201 ip-10-0-141-251 kubenswrapper[4908]: E1030 11:54:06.343061 4908 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/221ced85-cc15-4e07-82d9-a1a1f3e958a5-metrics-tls podName:221ced85-cc15-4e07-82d9-a1a1f3e958a5 nodeName:}" failed. No retries permitted until 2026-10-30 11:54:08.343043323 +0000 UTC m=+4.857472061 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/221ced85-cc15-4e07-82d9-a1a1f3e958a5-metrics-tls") pod "dns-default-xxcbn" (UID: "221ced85-cc15-4e07-82d9-a1a1f3e958a5") : secret "dns-default-metrics-tls" not found Oct 30 11:54:06.343201 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:06.343081 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7689cfe5-5a4e-4481-8eb9-107a58ca8bb3-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-xdxkp\" (UID: \"7689cfe5-5a4e-4481-8eb9-107a58ca8bb3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xdxkp" Oct 30 11:54:06.343201 ip-10-0-141-251 kubenswrapper[4908]: E1030 11:54:06.343118 4908 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d670b364-53ce-444b-b9c4-ac8ac09cd44f-insights-runtime-extractor-tls podName:d670b364-53ce-444b-b9c4-ac8ac09cd44f nodeName:}" failed. No retries permitted until 2026-10-30 11:54:08.343097878 +0000 UTC m=+4.857526614 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d670b364-53ce-444b-b9c4-ac8ac09cd44f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vn6vv" (UID: "d670b364-53ce-444b-b9c4-ac8ac09cd44f") : secret "insights-runtime-extractor-tls" not found Oct 30 11:54:06.343201 ip-10-0-141-251 kubenswrapper[4908]: E1030 11:54:06.342986 4908 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 11:54:06.343201 ip-10-0-141-251 kubenswrapper[4908]: E1030 11:54:06.343149 4908 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 11:54:06.343201 ip-10-0-141-251 kubenswrapper[4908]: E1030 11:54:06.343185 4908 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6acb13a1-cb2e-45e1-897b-3ef89e6d1b84-metrics-certs podName:6acb13a1-cb2e-45e1-897b-3ef89e6d1b84 nodeName:}" failed. No retries permitted until 2026-10-30 11:54:08.343169001 +0000 UTC m=+4.857597748 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6acb13a1-cb2e-45e1-897b-3ef89e6d1b84-metrics-certs") pod "router-default-5bfdf864c-tzv67" (UID: "6acb13a1-cb2e-45e1-897b-3ef89e6d1b84") : secret "router-metrics-certs-default" not found Oct 30 11:54:06.343201 ip-10-0-141-251 kubenswrapper[4908]: E1030 11:54:06.343205 4908 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7689cfe5-5a4e-4481-8eb9-107a58ca8bb3-samples-operator-tls podName:7689cfe5-5a4e-4481-8eb9-107a58ca8bb3 nodeName:}" failed. No retries permitted until 2026-10-30 11:54:08.343195742 +0000 UTC m=+4.857624465 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7689cfe5-5a4e-4481-8eb9-107a58ca8bb3-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-xdxkp" (UID: "7689cfe5-5a4e-4481-8eb9-107a58ca8bb3") : secret "samples-operator-tls" not found Oct 30 11:54:06.345391 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:06.345358 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6d976900-ecd0-469c-a9e2-d60708361f1a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-mvhwk\" (UID: \"6d976900-ecd0-469c-a9e2-d60708361f1a\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-mvhwk" Oct 30 11:54:06.411688 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:06.411663 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-mvhwk" Oct 30 11:54:06.592429 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:06.592404 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-bz65k" Oct 30 11:54:06.664776 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:06.664745 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-557c6b9f47-vcr2t" event={"ID":"bd977255-f9a7-4531-8a9c-b1a3a17e0b79","Type":"ContainerStarted","Data":"4889b4249d2a5758089345f3d4b30fb5921f82dba9f2b0887738eafc5cb54dce"} Oct 30 11:54:06.665153 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:06.664794 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-557c6b9f47-vcr2t" event={"ID":"bd977255-f9a7-4531-8a9c-b1a3a17e0b79","Type":"ContainerStarted","Data":"914e9bb474175e0e9380bc28a8141b3a8b4db1b61e5beede916579ec3f61c73f"} Oct 30 11:54:06.667591 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:06.667565 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-dbqhj" event={"ID":"1d7f8396-e57c-45e5-ae74-a2244db10364","Type":"ContainerStarted","Data":"0fb688f1ee5b09e8f12e69dc74325e9fe14d643d0ae046779d7d7eceac0d467f"} Oct 30 11:54:06.667721 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:06.667611 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-dbqhj" event={"ID":"1d7f8396-e57c-45e5-ae74-a2244db10364","Type":"ContainerStarted","Data":"f797f8c1308696d7accd690734bf285d5b630bacd893e14d0053e652748a2035"} Oct 30 11:54:06.669269 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:06.669245 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-79fd565486-8fl78" event={"ID":"cfaad7d0-d30b-4ddc-81cc-2b603416fc29","Type":"ContainerStarted","Data":"a699cee80ab4f93e500af119a3a188a6166fe8fd4625cdf1345dda0daf8ed3b6"} Oct 30 11:54:06.669364 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:06.669289 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-79fd565486-8fl78" event={"ID":"cfaad7d0-d30b-4ddc-81cc-2b603416fc29","Type":"ContainerStarted","Data":"8ff668624ba16b6f4bf01b1c8f33218ee6244ab3f8dae1e74ae504963d7b4ebe"} Oct 30 11:54:06.670565 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:06.670536 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-zzxhv" event={"ID":"179d55d3-7958-4928-a98c-68984fca9837","Type":"ContainerStarted","Data":"869479b4bf6c180a3fb8621a3ade7bf33bfa97f7a84e478f7caa2969dcdf2298"} Oct 30 11:54:06.672166 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:06.672149 4908 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-plczt_a70ee11c-c20e-4a4f-8476-ac5eb02816d0/console-operator/2.log" Oct 30 11:54:06.672483 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:06.672466 4908 scope.go:122] "RemoveContainer" containerID="e269d4e97647eeac46f1b7b961248f6a6b0c439df2aa6aaed35717b2e3781f6e" Oct 30 11:54:06.672663 ip-10-0-141-251 kubenswrapper[4908]: E1030 11:54:06.672642 4908 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-plczt_openshift-console-operator(a70ee11c-c20e-4a4f-8476-ac5eb02816d0)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-plczt" podUID="a70ee11c-c20e-4a4f-8476-ac5eb02816d0" Oct 30 11:54:06.706552 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:06.706525 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-mvhwk"] Oct 30 11:54:06.713062 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:06.713040 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d976900_ecd0_469c_a9e2_d60708361f1a.slice/crio-3883dd26d29340b47e5b13441c2c8f84b3bac043d0c91f4e3692ab3dc2b494eb WatchSource:0}: Error finding container 3883dd26d29340b47e5b13441c2c8f84b3bac043d0c91f4e3692ab3dc2b494eb: Status 404 returned error can't find the container with id 3883dd26d29340b47e5b13441c2c8f84b3bac043d0c91f4e3692ab3dc2b494eb Oct 30 11:54:07.265631 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:07.265608 4908 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-hb8j7_5aee536a-8942-4b48-9b2c-3e36089f3ac5/node-ca/0.log" Oct 30 11:54:07.665664 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:07.665633 4908 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-gn2z5_003c3d18-23f1-45d0-bac4-8c6eea4b7e99/serve-healthcheck-canary/0.log" Oct 30 11:54:07.676626 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:07.676588 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-mvhwk" event={"ID":"6d976900-ecd0-469c-a9e2-d60708361f1a","Type":"ContainerStarted","Data":"3883dd26d29340b47e5b13441c2c8f84b3bac043d0c91f4e3692ab3dc2b494eb"} Oct 30 11:54:07.678601 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:07.678460 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-zzxhv" event={"ID":"179d55d3-7958-4928-a98c-68984fca9837","Type":"ContainerStarted","Data":"47fb51b3525f9d1abd257a15306b324cdbfc2afbc1ae6313345cdd02b8b315c8"} Oct 30 11:54:07.679178 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:07.679141 4908 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 11:54:07.679293 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:07.679207 4908 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 11:54:07.852623 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:07.852596 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-b8bxz" Oct 30 11:54:08.266458 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:08.266432 4908 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-4brks_dab8eab1-3e3f-46b4-b76d-d58a7061996d/migrator/0.log" Oct 30 11:54:08.361658 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:08.361635 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6acb13a1-cb2e-45e1-897b-3ef89e6d1b84-metrics-certs\") pod \"router-default-5bfdf864c-tzv67\" (UID: \"6acb13a1-cb2e-45e1-897b-3ef89e6d1b84\") " pod="openshift-ingress/router-default-5bfdf864c-tzv67" Oct 30 11:54:08.361777 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:08.361683 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d670b364-53ce-444b-b9c4-ac8ac09cd44f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vn6vv\" (UID: \"d670b364-53ce-444b-b9c4-ac8ac09cd44f\") " pod="openshift-insights/insights-runtime-extractor-vn6vv" Oct 30 11:54:08.361777 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:08.361734 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/221ced85-cc15-4e07-82d9-a1a1f3e958a5-metrics-tls\") pod \"dns-default-xxcbn\" (UID: \"221ced85-cc15-4e07-82d9-a1a1f3e958a5\") " pod="openshift-dns/dns-default-xxcbn" Oct 30 11:54:08.361777 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:08.361767 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7689cfe5-5a4e-4481-8eb9-107a58ca8bb3-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-xdxkp\" (UID: \"7689cfe5-5a4e-4481-8eb9-107a58ca8bb3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xdxkp" Oct 30 11:54:08.361949 ip-10-0-141-251 kubenswrapper[4908]: E1030 11:54:08.361865 4908 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 11:54:08.361949 ip-10-0-141-251 kubenswrapper[4908]: E1030 11:54:08.361873 4908 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 11:54:08.361949 ip-10-0-141-251 kubenswrapper[4908]: E1030 11:54:08.361900 4908 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 11:54:08.361949 ip-10-0-141-251 kubenswrapper[4908]: E1030 11:54:08.361933 4908 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/221ced85-cc15-4e07-82d9-a1a1f3e958a5-metrics-tls podName:221ced85-cc15-4e07-82d9-a1a1f3e958a5 nodeName:}" failed. No retries permitted until 2026-10-30 11:54:12.361913638 +0000 UTC m=+8.876342380 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/221ced85-cc15-4e07-82d9-a1a1f3e958a5-metrics-tls") pod "dns-default-xxcbn" (UID: "221ced85-cc15-4e07-82d9-a1a1f3e958a5") : secret "dns-default-metrics-tls" not found Oct 30 11:54:08.362071 ip-10-0-141-251 kubenswrapper[4908]: E1030 11:54:08.361955 4908 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d670b364-53ce-444b-b9c4-ac8ac09cd44f-insights-runtime-extractor-tls podName:d670b364-53ce-444b-b9c4-ac8ac09cd44f nodeName:}" failed. No retries permitted until 2026-10-30 11:54:12.361944535 +0000 UTC m=+8.876373259 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d670b364-53ce-444b-b9c4-ac8ac09cd44f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vn6vv" (UID: "d670b364-53ce-444b-b9c4-ac8ac09cd44f") : secret "insights-runtime-extractor-tls" not found Oct 30 11:54:08.362071 ip-10-0-141-251 kubenswrapper[4908]: E1030 11:54:08.361970 4908 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7689cfe5-5a4e-4481-8eb9-107a58ca8bb3-samples-operator-tls podName:7689cfe5-5a4e-4481-8eb9-107a58ca8bb3 nodeName:}" failed. No retries permitted until 2026-10-30 11:54:12.361961774 +0000 UTC m=+8.876390498 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7689cfe5-5a4e-4481-8eb9-107a58ca8bb3-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-xdxkp" (UID: "7689cfe5-5a4e-4481-8eb9-107a58ca8bb3") : secret "samples-operator-tls" not found Oct 30 11:54:08.363842 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:08.363814 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6acb13a1-cb2e-45e1-897b-3ef89e6d1b84-metrics-certs\") pod \"router-default-5bfdf864c-tzv67\" (UID: \"6acb13a1-cb2e-45e1-897b-3ef89e6d1b84\") " pod="openshift-ingress/router-default-5bfdf864c-tzv67" Oct 30 11:54:08.511976 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:08.511947 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5bfdf864c-tzv67" Oct 30 11:54:08.666759 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:08.666731 4908 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-86p78_b3f20243-155c-4e25-a1ca-6131a141fbd4/kube-storage-version-migrator-operator/0.log" Oct 30 11:54:08.685096 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:08.685063 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-mvhwk" event={"ID":"6d976900-ecd0-469c-a9e2-d60708361f1a","Type":"ContainerStarted","Data":"f7466fa09561d3809beca560d0ff2bc1cd068ef4807900ad203b97f834bcd91d"} Oct 30 11:54:08.850865 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:08.850789 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-5bfdf864c-tzv67"] Oct 30 11:54:08.855062 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:08.855034 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6acb13a1_cb2e_45e1_897b_3ef89e6d1b84.slice/crio-9c715b7b4266b0851b091cd67d570e0fa1dc7442d5e43872fdbf18d8b18bc745 WatchSource:0}: Error finding container 9c715b7b4266b0851b091cd67d570e0fa1dc7442d5e43872fdbf18d8b18bc745: Status 404 returned error can't find the container with id 9c715b7b4266b0851b091cd67d570e0fa1dc7442d5e43872fdbf18d8b18bc745 Oct 30 11:54:09.690814 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:09.690769 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5bfdf864c-tzv67" event={"ID":"6acb13a1-cb2e-45e1-897b-3ef89e6d1b84","Type":"ContainerStarted","Data":"0745f2d526859085016a8eb06ed68a6d26f35f0ce830742df0f8534f6a7159ec"} Oct 30 11:54:09.691341 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:09.690849 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5bfdf864c-tzv67" event={"ID":"6acb13a1-cb2e-45e1-897b-3ef89e6d1b84","Type":"ContainerStarted","Data":"9c715b7b4266b0851b091cd67d570e0fa1dc7442d5e43872fdbf18d8b18bc745"} Oct 30 11:54:09.829744 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:09.829697 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-plczt" Oct 30 11:54:09.830147 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:09.830128 4908 scope.go:122] "RemoveContainer" containerID="e269d4e97647eeac46f1b7b961248f6a6b0c439df2aa6aaed35717b2e3781f6e" Oct 30 11:54:09.830319 ip-10-0-141-251 kubenswrapper[4908]: E1030 11:54:09.830296 4908 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-plczt_openshift-console-operator(a70ee11c-c20e-4a4f-8476-ac5eb02816d0)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-plczt" podUID="a70ee11c-c20e-4a4f-8476-ac5eb02816d0" Oct 30 11:54:10.512541 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:10.512510 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-5bfdf864c-tzv67" Oct 30 11:54:10.514963 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:10.514943 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-5bfdf864c-tzv67" Oct 30 11:54:10.724862 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:10.724824 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-b8bxz" Oct 30 11:54:10.725286 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:10.725028 4908 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 11:54:11.696222 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:11.696196 4908 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 11:54:12.392342 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:12.392305 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/221ced85-cc15-4e07-82d9-a1a1f3e958a5-metrics-tls\") pod \"dns-default-xxcbn\" (UID: \"221ced85-cc15-4e07-82d9-a1a1f3e958a5\") " pod="openshift-dns/dns-default-xxcbn" Oct 30 11:54:12.392909 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:12.392352 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7689cfe5-5a4e-4481-8eb9-107a58ca8bb3-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-xdxkp\" (UID: \"7689cfe5-5a4e-4481-8eb9-107a58ca8bb3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xdxkp" Oct 30 11:54:12.392909 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:12.392495 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d670b364-53ce-444b-b9c4-ac8ac09cd44f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vn6vv\" (UID: \"d670b364-53ce-444b-b9c4-ac8ac09cd44f\") " pod="openshift-insights/insights-runtime-extractor-vn6vv" Oct 30 11:54:12.392909 ip-10-0-141-251 kubenswrapper[4908]: E1030 11:54:12.392625 4908 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 11:54:12.392909 ip-10-0-141-251 kubenswrapper[4908]: E1030 11:54:12.392719 4908 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d670b364-53ce-444b-b9c4-ac8ac09cd44f-insights-runtime-extractor-tls podName:d670b364-53ce-444b-b9c4-ac8ac09cd44f nodeName:}" failed. No retries permitted until 2026-10-30 11:54:20.392700394 +0000 UTC m=+16.907129126 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d670b364-53ce-444b-b9c4-ac8ac09cd44f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vn6vv" (UID: "d670b364-53ce-444b-b9c4-ac8ac09cd44f") : secret "insights-runtime-extractor-tls" not found Oct 30 11:54:12.395048 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:12.395022 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/221ced85-cc15-4e07-82d9-a1a1f3e958a5-metrics-tls\") pod \"dns-default-xxcbn\" (UID: \"221ced85-cc15-4e07-82d9-a1a1f3e958a5\") " pod="openshift-dns/dns-default-xxcbn" Oct 30 11:54:12.395177 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:12.395127 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7689cfe5-5a4e-4481-8eb9-107a58ca8bb3-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-xdxkp\" (UID: \"7689cfe5-5a4e-4481-8eb9-107a58ca8bb3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xdxkp" Oct 30 11:54:12.411843 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:12.411813 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xdxkp" Oct 30 11:54:12.421089 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:12.421068 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xxcbn" Oct 30 11:54:12.714748 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:12.714682 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-79fd565486-8fl78" Oct 30 11:54:12.722571 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:12.722550 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xdxkp"] Oct 30 11:54:12.734012 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:12.733916 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-xxcbn"] Oct 30 11:54:12.744909 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:12.744880 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod221ced85_cc15_4e07_82d9_a1a1f3e958a5.slice/crio-3f0ebefe14a7db325cbde34e0059b040502791fb9e85c24048efc861a9447a15 WatchSource:0}: Error finding container 3f0ebefe14a7db325cbde34e0059b040502791fb9e85c24048efc861a9447a15: Status 404 returned error can't find the container with id 3f0ebefe14a7db325cbde34e0059b040502791fb9e85c24048efc861a9447a15 Oct 30 11:54:13.500769 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:13.500739 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-b8bxz" Oct 30 11:54:13.703725 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:13.703680 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xdxkp" event={"ID":"7689cfe5-5a4e-4481-8eb9-107a58ca8bb3","Type":"ContainerStarted","Data":"036d3be49f024d590d4a438ec2bc9e5b6ad6adbecec4b1b2465bf076ab9d50b7"} Oct 30 11:54:13.704858 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:13.704811 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xxcbn" event={"ID":"221ced85-cc15-4e07-82d9-a1a1f3e958a5","Type":"ContainerStarted","Data":"3f0ebefe14a7db325cbde34e0059b040502791fb9e85c24048efc861a9447a15"} Oct 30 11:54:14.343518 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:14.342907 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-557c6b9f47-vcr2t" Oct 30 11:54:14.710235 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:14.710200 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xdxkp" event={"ID":"7689cfe5-5a4e-4481-8eb9-107a58ca8bb3","Type":"ContainerStarted","Data":"1200bc39485cc25bb340cfc17b08606868e38f7c2759fbcba9bf40401d96597d"} Oct 30 11:54:14.712047 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:14.712023 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xxcbn" event={"ID":"221ced85-cc15-4e07-82d9-a1a1f3e958a5","Type":"ContainerStarted","Data":"872dad574a1dcf81aee3490e518b9ae49e22b9855074ededefb6f5969b1f0ed6"} Oct 30 11:54:14.712135 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:14.712072 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xxcbn" event={"ID":"221ced85-cc15-4e07-82d9-a1a1f3e958a5","Type":"ContainerStarted","Data":"579ec2f86cbbbc1d80498ee4d901b7a9998a82a638074bdb9764373831e7647d"} Oct 30 11:54:14.712194 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:14.712169 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-xxcbn" Oct 30 11:54:15.163372 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:15.163332 4908 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-plczt" Oct 30 11:54:15.163684 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:15.163670 4908 scope.go:122] "RemoveContainer" containerID="e269d4e97647eeac46f1b7b961248f6a6b0c439df2aa6aaed35717b2e3781f6e" Oct 30 11:54:15.717316 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:15.717283 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xdxkp" event={"ID":"7689cfe5-5a4e-4481-8eb9-107a58ca8bb3","Type":"ContainerStarted","Data":"d7e6826931bff329d16a1517388fa0393a381d396196bc9ad6a48a452047d0f4"} Oct 30 11:54:15.718979 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:15.718957 4908 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-plczt_a70ee11c-c20e-4a4f-8476-ac5eb02816d0/console-operator/3.log" Oct 30 11:54:15.719360 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:15.719346 4908 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-plczt_a70ee11c-c20e-4a4f-8476-ac5eb02816d0/console-operator/2.log" Oct 30 11:54:15.719413 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:15.719377 4908 generic.go:356] "Generic (PLEG): container finished" podID="a70ee11c-c20e-4a4f-8476-ac5eb02816d0" containerID="5796becf35193ac11b8559dbac1500897ca9b5287402da754e4c31769adc1b83" exitCode=255 Oct 30 11:54:15.719479 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:15.719456 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-plczt" event={"ID":"a70ee11c-c20e-4a4f-8476-ac5eb02816d0","Type":"ContainerDied","Data":"5796becf35193ac11b8559dbac1500897ca9b5287402da754e4c31769adc1b83"} Oct 30 11:54:15.719520 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:15.719508 4908 scope.go:122] "RemoveContainer" containerID="e269d4e97647eeac46f1b7b961248f6a6b0c439df2aa6aaed35717b2e3781f6e" Oct 30 11:54:15.719901 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:15.719876 4908 scope.go:122] "RemoveContainer" containerID="5796becf35193ac11b8559dbac1500897ca9b5287402da754e4c31769adc1b83" Oct 30 11:54:15.720071 ip-10-0-141-251 kubenswrapper[4908]: E1030 11:54:15.720049 4908 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-plczt_openshift-console-operator(a70ee11c-c20e-4a4f-8476-ac5eb02816d0)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-plczt" podUID="a70ee11c-c20e-4a4f-8476-ac5eb02816d0" Oct 30 11:54:16.317257 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:16.317223 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-5bfdf864c-tzv67" Oct 30 11:54:16.317404 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:16.317370 4908 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 11:54:16.318538 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:16.318521 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-5bfdf864c-tzv67" Oct 30 11:54:16.724300 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:16.724242 4908 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-plczt_a70ee11c-c20e-4a4f-8476-ac5eb02816d0/console-operator/3.log" Oct 30 11:54:19.287323 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:19.287283 4908 patch_prober.go:29] interesting pod/image-registry-79fd565486-8fl78 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Oct 30 11:54:19.287898 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:19.287344 4908 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-79fd565486-8fl78" podUID="cfaad7d0-d30b-4ddc-81cc-2b603416fc29" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 11:54:19.829772 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:19.829744 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-plczt" Oct 30 11:54:19.830142 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:19.830115 4908 scope.go:122] "RemoveContainer" containerID="5796becf35193ac11b8559dbac1500897ca9b5287402da754e4c31769adc1b83" Oct 30 11:54:19.830308 ip-10-0-141-251 kubenswrapper[4908]: E1030 11:54:19.830287 4908 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-plczt_openshift-console-operator(a70ee11c-c20e-4a4f-8476-ac5eb02816d0)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-plczt" podUID="a70ee11c-c20e-4a4f-8476-ac5eb02816d0" Oct 30 11:54:20.451880 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:20.451821 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d670b364-53ce-444b-b9c4-ac8ac09cd44f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vn6vv\" (UID: \"d670b364-53ce-444b-b9c4-ac8ac09cd44f\") " pod="openshift-insights/insights-runtime-extractor-vn6vv" Oct 30 11:54:20.454383 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:20.454359 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d670b364-53ce-444b-b9c4-ac8ac09cd44f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vn6vv\" (UID: \"d670b364-53ce-444b-b9c4-ac8ac09cd44f\") " pod="openshift-insights/insights-runtime-extractor-vn6vv" Oct 30 11:54:20.521120 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:20.521097 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-vn6vv" Oct 30 11:54:20.823624 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:20.823589 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-vn6vv"] Oct 30 11:54:20.828516 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:20.828487 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd670b364_53ce_444b_b9c4_ac8ac09cd44f.slice/crio-4c933f687d336f13510981dd8d4aa3e39127e7b48677c0f9eb380f7e80ff90ef WatchSource:0}: Error finding container 4c933f687d336f13510981dd8d4aa3e39127e7b48677c0f9eb380f7e80ff90ef: Status 404 returned error can't find the container with id 4c933f687d336f13510981dd8d4aa3e39127e7b48677c0f9eb380f7e80ff90ef Oct 30 11:54:21.150848 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:21.150772 4908 patch_prober.go:29] interesting pod/image-registry-557c6b9f47-vcr2t container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Oct 30 11:54:21.150848 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:21.150816 4908 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-557c6b9f47-vcr2t" podUID="bd977255-f9a7-4531-8a9c-b1a3a17e0b79" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 11:54:21.744533 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:21.744499 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vn6vv" event={"ID":"d670b364-53ce-444b-b9c4-ac8ac09cd44f","Type":"ContainerStarted","Data":"bad4b332e45062f203b23b64a06b5109d2a70b074edcf06d51fb76bae73fccb7"} Oct 30 11:54:21.744906 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:21.744568 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vn6vv" event={"ID":"d670b364-53ce-444b-b9c4-ac8ac09cd44f","Type":"ContainerStarted","Data":"6d2d6de34c0bf78ef6514ddee3bb28f4ea3ddd5bb8c4d0cb102e6f9ee4538b1a"} Oct 30 11:54:21.744906 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:21.744600 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vn6vv" event={"ID":"d670b364-53ce-444b-b9c4-ac8ac09cd44f","Type":"ContainerStarted","Data":"4c933f687d336f13510981dd8d4aa3e39127e7b48677c0f9eb380f7e80ff90ef"} Oct 30 11:54:22.719368 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:22.719345 4908 patch_prober.go:29] interesting pod/image-registry-79fd565486-8fl78 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Oct 30 11:54:22.719482 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:22.719382 4908 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-79fd565486-8fl78" podUID="cfaad7d0-d30b-4ddc-81cc-2b603416fc29" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 11:54:22.750217 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:22.750195 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vn6vv" event={"ID":"d670b364-53ce-444b-b9c4-ac8ac09cd44f","Type":"ContainerStarted","Data":"0f9016fdaf6fd7cf60bbaf2a6666585565a973a0bcf01d6a5d8a932432567f24"} Oct 30 11:54:22.891089 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:22.891023 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-557c6b9f47-vcr2t"] Oct 30 11:54:22.895291 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:22.895264 4908 patch_prober.go:29] interesting pod/image-registry-557c6b9f47-vcr2t container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Oct 30 11:54:22.895430 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:22.895305 4908 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-557c6b9f47-vcr2t" podUID="bd977255-f9a7-4531-8a9c-b1a3a17e0b79" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 11:54:24.722236 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:24.722201 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-xxcbn" Oct 30 11:54:25.162979 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:25.162937 4908 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-plczt" Oct 30 11:54:25.163320 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:25.163304 4908 scope.go:122] "RemoveContainer" containerID="5796becf35193ac11b8559dbac1500897ca9b5287402da754e4c31769adc1b83" Oct 30 11:54:25.163496 ip-10-0-141-251 kubenswrapper[4908]: E1030 11:54:25.163466 4908 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-plczt_openshift-console-operator(a70ee11c-c20e-4a4f-8476-ac5eb02816d0)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-plczt" podUID="a70ee11c-c20e-4a4f-8476-ac5eb02816d0" Oct 30 11:54:30.360820 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:30.360787 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-h7xsz"] Oct 30 11:54:30.401330 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:30.401305 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-h7xsz" Oct 30 11:54:30.416434 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:30.416413 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 11:54:30.416773 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:30.416756 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 11:54:30.427670 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:30.427653 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-xwvts\"" Oct 30 11:54:30.430065 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:30.430047 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c22cb98a-c283-4b39-af4d-4a4edf388b16-node-exporter-wtmp\") pod \"node-exporter-h7xsz\" (UID: \"c22cb98a-c283-4b39-af4d-4a4edf388b16\") " pod="openshift-monitoring/node-exporter-h7xsz" Oct 30 11:54:30.430125 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:30.430075 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c22cb98a-c283-4b39-af4d-4a4edf388b16-sys\") pod \"node-exporter-h7xsz\" (UID: \"c22cb98a-c283-4b39-af4d-4a4edf388b16\") " pod="openshift-monitoring/node-exporter-h7xsz" Oct 30 11:54:30.430125 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:30.430092 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c22cb98a-c283-4b39-af4d-4a4edf388b16-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-h7xsz\" (UID: \"c22cb98a-c283-4b39-af4d-4a4edf388b16\") " pod="openshift-monitoring/node-exporter-h7xsz" Oct 30 11:54:30.430187 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:30.430158 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c22cb98a-c283-4b39-af4d-4a4edf388b16-node-exporter-accelerators-collector-config\") pod \"node-exporter-h7xsz\" (UID: \"c22cb98a-c283-4b39-af4d-4a4edf388b16\") " pod="openshift-monitoring/node-exporter-h7xsz" Oct 30 11:54:30.430223 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:30.430194 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lds8z\" (UniqueName: \"kubernetes.io/projected/c22cb98a-c283-4b39-af4d-4a4edf388b16-kube-api-access-lds8z\") pod \"node-exporter-h7xsz\" (UID: \"c22cb98a-c283-4b39-af4d-4a4edf388b16\") " pod="openshift-monitoring/node-exporter-h7xsz" Oct 30 11:54:30.430265 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:30.430251 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c22cb98a-c283-4b39-af4d-4a4edf388b16-node-exporter-tls\") pod \"node-exporter-h7xsz\" (UID: \"c22cb98a-c283-4b39-af4d-4a4edf388b16\") " pod="openshift-monitoring/node-exporter-h7xsz" Oct 30 11:54:30.430301 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:30.430275 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c22cb98a-c283-4b39-af4d-4a4edf388b16-node-exporter-textfile\") pod \"node-exporter-h7xsz\" (UID: \"c22cb98a-c283-4b39-af4d-4a4edf388b16\") " pod="openshift-monitoring/node-exporter-h7xsz" Oct 30 11:54:30.430330 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:30.430300 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c22cb98a-c283-4b39-af4d-4a4edf388b16-root\") pod \"node-exporter-h7xsz\" (UID: \"c22cb98a-c283-4b39-af4d-4a4edf388b16\") " pod="openshift-monitoring/node-exporter-h7xsz" Oct 30 11:54:30.430330 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:30.430318 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c22cb98a-c283-4b39-af4d-4a4edf388b16-metrics-client-ca\") pod \"node-exporter-h7xsz\" (UID: \"c22cb98a-c283-4b39-af4d-4a4edf388b16\") " pod="openshift-monitoring/node-exporter-h7xsz" Oct 30 11:54:30.450811 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:30.450796 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 11:54:30.450911 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:30.450886 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 11:54:30.530926 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:30.530909 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lds8z\" (UniqueName: \"kubernetes.io/projected/c22cb98a-c283-4b39-af4d-4a4edf388b16-kube-api-access-lds8z\") pod \"node-exporter-h7xsz\" (UID: \"c22cb98a-c283-4b39-af4d-4a4edf388b16\") " pod="openshift-monitoring/node-exporter-h7xsz" Oct 30 11:54:30.531025 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:30.530933 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c22cb98a-c283-4b39-af4d-4a4edf388b16-node-exporter-tls\") pod \"node-exporter-h7xsz\" (UID: \"c22cb98a-c283-4b39-af4d-4a4edf388b16\") " pod="openshift-monitoring/node-exporter-h7xsz" Oct 30 11:54:30.531025 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:30.530951 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c22cb98a-c283-4b39-af4d-4a4edf388b16-node-exporter-textfile\") pod \"node-exporter-h7xsz\" (UID: \"c22cb98a-c283-4b39-af4d-4a4edf388b16\") " pod="openshift-monitoring/node-exporter-h7xsz" Oct 30 11:54:30.531025 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:30.530971 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c22cb98a-c283-4b39-af4d-4a4edf388b16-root\") pod \"node-exporter-h7xsz\" (UID: \"c22cb98a-c283-4b39-af4d-4a4edf388b16\") " pod="openshift-monitoring/node-exporter-h7xsz" Oct 30 11:54:30.531025 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:30.530988 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c22cb98a-c283-4b39-af4d-4a4edf388b16-metrics-client-ca\") pod \"node-exporter-h7xsz\" (UID: \"c22cb98a-c283-4b39-af4d-4a4edf388b16\") " pod="openshift-monitoring/node-exporter-h7xsz" Oct 30 11:54:30.531025 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:30.531012 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c22cb98a-c283-4b39-af4d-4a4edf388b16-node-exporter-wtmp\") pod \"node-exporter-h7xsz\" (UID: \"c22cb98a-c283-4b39-af4d-4a4edf388b16\") " pod="openshift-monitoring/node-exporter-h7xsz" Oct 30 11:54:30.531260 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:30.531075 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c22cb98a-c283-4b39-af4d-4a4edf388b16-root\") pod \"node-exporter-h7xsz\" (UID: \"c22cb98a-c283-4b39-af4d-4a4edf388b16\") " pod="openshift-monitoring/node-exporter-h7xsz" Oct 30 11:54:30.531260 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:30.531125 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c22cb98a-c283-4b39-af4d-4a4edf388b16-sys\") pod \"node-exporter-h7xsz\" (UID: \"c22cb98a-c283-4b39-af4d-4a4edf388b16\") " pod="openshift-monitoring/node-exporter-h7xsz" Oct 30 11:54:30.531260 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:30.531142 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c22cb98a-c283-4b39-af4d-4a4edf388b16-node-exporter-wtmp\") pod \"node-exporter-h7xsz\" (UID: \"c22cb98a-c283-4b39-af4d-4a4edf388b16\") " pod="openshift-monitoring/node-exporter-h7xsz" Oct 30 11:54:30.531260 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:30.531158 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c22cb98a-c283-4b39-af4d-4a4edf388b16-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-h7xsz\" (UID: \"c22cb98a-c283-4b39-af4d-4a4edf388b16\") " pod="openshift-monitoring/node-exporter-h7xsz" Oct 30 11:54:30.531260 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:30.531188 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c22cb98a-c283-4b39-af4d-4a4edf388b16-sys\") pod \"node-exporter-h7xsz\" (UID: \"c22cb98a-c283-4b39-af4d-4a4edf388b16\") " pod="openshift-monitoring/node-exporter-h7xsz" Oct 30 11:54:30.531260 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:30.531226 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c22cb98a-c283-4b39-af4d-4a4edf388b16-node-exporter-accelerators-collector-config\") pod \"node-exporter-h7xsz\" (UID: \"c22cb98a-c283-4b39-af4d-4a4edf388b16\") " pod="openshift-monitoring/node-exporter-h7xsz" Oct 30 11:54:30.531525 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:30.531339 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c22cb98a-c283-4b39-af4d-4a4edf388b16-node-exporter-textfile\") pod \"node-exporter-h7xsz\" (UID: \"c22cb98a-c283-4b39-af4d-4a4edf388b16\") " pod="openshift-monitoring/node-exporter-h7xsz" Oct 30 11:54:30.531590 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:30.531571 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c22cb98a-c283-4b39-af4d-4a4edf388b16-metrics-client-ca\") pod \"node-exporter-h7xsz\" (UID: \"c22cb98a-c283-4b39-af4d-4a4edf388b16\") " pod="openshift-monitoring/node-exporter-h7xsz" Oct 30 11:54:30.531690 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:30.531674 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c22cb98a-c283-4b39-af4d-4a4edf388b16-node-exporter-accelerators-collector-config\") pod \"node-exporter-h7xsz\" (UID: \"c22cb98a-c283-4b39-af4d-4a4edf388b16\") " pod="openshift-monitoring/node-exporter-h7xsz" Oct 30 11:54:30.533270 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:30.533241 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c22cb98a-c283-4b39-af4d-4a4edf388b16-node-exporter-tls\") pod \"node-exporter-h7xsz\" (UID: \"c22cb98a-c283-4b39-af4d-4a4edf388b16\") " pod="openshift-monitoring/node-exporter-h7xsz" Oct 30 11:54:30.533343 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:30.533282 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c22cb98a-c283-4b39-af4d-4a4edf388b16-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-h7xsz\" (UID: \"c22cb98a-c283-4b39-af4d-4a4edf388b16\") " pod="openshift-monitoring/node-exporter-h7xsz" Oct 30 11:54:30.561456 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:30.561438 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lds8z\" (UniqueName: \"kubernetes.io/projected/c22cb98a-c283-4b39-af4d-4a4edf388b16-kube-api-access-lds8z\") pod \"node-exporter-h7xsz\" (UID: \"c22cb98a-c283-4b39-af4d-4a4edf388b16\") " pod="openshift-monitoring/node-exporter-h7xsz" Oct 30 11:54:30.752003 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:30.751972 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-h7xsz" Oct 30 11:54:30.794247 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:54:30.794225 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc22cb98a_c283_4b39_af4d_4a4edf388b16.slice/crio-a6e1ce8a117cabfb8e1fd78b724626b7d210893a5a494e62b720689e7cde51f3 WatchSource:0}: Error finding container a6e1ce8a117cabfb8e1fd78b724626b7d210893a5a494e62b720689e7cde51f3: Status 404 returned error can't find the container with id a6e1ce8a117cabfb8e1fd78b724626b7d210893a5a494e62b720689e7cde51f3 Oct 30 11:54:31.424795 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:31.424764 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-tdf8d" Oct 30 11:54:31.781992 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:31.781956 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-h7xsz" event={"ID":"c22cb98a-c283-4b39-af4d-4a4edf388b16","Type":"ContainerStarted","Data":"a6e1ce8a117cabfb8e1fd78b724626b7d210893a5a494e62b720689e7cde51f3"} Oct 30 11:54:32.719229 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:32.719205 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-79fd565486-8fl78" Oct 30 11:54:32.787770 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:32.787738 4908 generic.go:356] "Generic (PLEG): container finished" podID="c22cb98a-c283-4b39-af4d-4a4edf388b16" containerID="10b3d99d1c4550953b732e920a292aa041b219facbe233c63ed5ae2de410f7d0" exitCode=0 Oct 30 11:54:32.787910 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:32.787815 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-h7xsz" event={"ID":"c22cb98a-c283-4b39-af4d-4a4edf388b16","Type":"ContainerDied","Data":"10b3d99d1c4550953b732e920a292aa041b219facbe233c63ed5ae2de410f7d0"} Oct 30 11:54:32.895652 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:32.895630 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-557c6b9f47-vcr2t" Oct 30 11:54:33.792872 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:33.792815 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-h7xsz" event={"ID":"c22cb98a-c283-4b39-af4d-4a4edf388b16","Type":"ContainerStarted","Data":"00f3281d1960dfb70bc51e733322b0dcf78bcacaf0f4d8c2be7b5fd380b5bacb"} Oct 30 11:54:33.793239 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:33.792897 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-h7xsz" event={"ID":"c22cb98a-c283-4b39-af4d-4a4edf388b16","Type":"ContainerStarted","Data":"322882e36e6ca406a8e1cefdebb127460b8363c7aa04f3a909a0fc5452f40361"} Oct 30 11:54:33.835525 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:33.835463 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-h7xsz" podStartSLOduration=2.3809203979999998 podStartE2EDuration="3.835451597s" totalImagesPullingTime="1.37620409s" totalInitContainerRuntime="78.327109ms" isStatefulPod=true podCreationTimestamp="2026-10-30 11:54:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:54:33.834211892 +0000 UTC m=+30.348640637" watchObservedRunningTime="2026-10-30 11:54:33.835451597 +0000 UTC m=+30.349880338" Oct 30 11:54:36.595563 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:36.595498 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-bz65k" Oct 30 11:54:38.598807 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:38.598773 4908 scope.go:122] "RemoveContainer" containerID="5796becf35193ac11b8559dbac1500897ca9b5287402da754e4c31769adc1b83" Oct 30 11:54:38.813722 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:38.813696 4908 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-plczt_a70ee11c-c20e-4a4f-8476-ac5eb02816d0/console-operator/3.log" Oct 30 11:54:38.813878 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:38.813767 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-plczt" event={"ID":"a70ee11c-c20e-4a4f-8476-ac5eb02816d0","Type":"ContainerStarted","Data":"c379318f86d856c5afa2946a3c4710bbb92e9d319d4d6af2943754c4349c5739"} Oct 30 11:54:38.814070 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:38.814044 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-plczt" Oct 30 11:54:39.157402 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:39.157378 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-plczt" Oct 30 11:54:45.301310 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:45.301275 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-79fd565486-8fl78"] Oct 30 11:54:47.910542 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:47.910428 4908 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-557c6b9f47-vcr2t" podUID="bd977255-f9a7-4531-8a9c-b1a3a17e0b79" containerName="registry" containerID="cri-o://4889b4249d2a5758089345f3d4b30fb5921f82dba9f2b0887738eafc5cb54dce" gracePeriod=30 Oct 30 11:54:48.154148 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:48.154127 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-557c6b9f47-vcr2t" Oct 30 11:54:48.255226 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:48.255199 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bd977255-f9a7-4531-8a9c-b1a3a17e0b79-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bd977255-f9a7-4531-8a9c-b1a3a17e0b79-image-registry-private-configuration\") pod \"bd977255-f9a7-4531-8a9c-b1a3a17e0b79\" (UID: \"bd977255-f9a7-4531-8a9c-b1a3a17e0b79\") " Oct 30 11:54:48.255342 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:48.255245 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bd977255-f9a7-4531-8a9c-b1a3a17e0b79-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bd977255-f9a7-4531-8a9c-b1a3a17e0b79-installation-pull-secrets\") pod \"bd977255-f9a7-4531-8a9c-b1a3a17e0b79\" (UID: \"bd977255-f9a7-4531-8a9c-b1a3a17e0b79\") " Oct 30 11:54:48.255342 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:48.255270 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd977255-f9a7-4531-8a9c-b1a3a17e0b79-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bd977255-f9a7-4531-8a9c-b1a3a17e0b79-bound-sa-token\") pod \"bd977255-f9a7-4531-8a9c-b1a3a17e0b79\" (UID: \"bd977255-f9a7-4531-8a9c-b1a3a17e0b79\") " Oct 30 11:54:48.255451 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:48.255433 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd977255-f9a7-4531-8a9c-b1a3a17e0b79-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bd977255-f9a7-4531-8a9c-b1a3a17e0b79-trusted-ca\") pod \"bd977255-f9a7-4531-8a9c-b1a3a17e0b79\" (UID: \"bd977255-f9a7-4531-8a9c-b1a3a17e0b79\") " Oct 30 11:54:48.255498 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:48.255475 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd977255-f9a7-4531-8a9c-b1a3a17e0b79-kube-api-access-8f7sv\" (UniqueName: \"kubernetes.io/projected/bd977255-f9a7-4531-8a9c-b1a3a17e0b79-kube-api-access-8f7sv\") pod \"bd977255-f9a7-4531-8a9c-b1a3a17e0b79\" (UID: \"bd977255-f9a7-4531-8a9c-b1a3a17e0b79\") " Oct 30 11:54:48.255549 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:48.255518 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd977255-f9a7-4531-8a9c-b1a3a17e0b79-registry-tls\" (UniqueName: \"kubernetes.io/projected/bd977255-f9a7-4531-8a9c-b1a3a17e0b79-registry-tls\") pod \"bd977255-f9a7-4531-8a9c-b1a3a17e0b79\" (UID: \"bd977255-f9a7-4531-8a9c-b1a3a17e0b79\") " Oct 30 11:54:48.255603 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:48.255571 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd977255-f9a7-4531-8a9c-b1a3a17e0b79-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bd977255-f9a7-4531-8a9c-b1a3a17e0b79-registry-certificates\") pod \"bd977255-f9a7-4531-8a9c-b1a3a17e0b79\" (UID: \"bd977255-f9a7-4531-8a9c-b1a3a17e0b79\") " Oct 30 11:54:48.255664 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:48.255625 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd977255-f9a7-4531-8a9c-b1a3a17e0b79-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bd977255-f9a7-4531-8a9c-b1a3a17e0b79-ca-trust-extracted\") pod \"bd977255-f9a7-4531-8a9c-b1a3a17e0b79\" (UID: \"bd977255-f9a7-4531-8a9c-b1a3a17e0b79\") " Oct 30 11:54:48.256070 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:48.255951 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd977255-f9a7-4531-8a9c-b1a3a17e0b79-trusted-ca" pod "bd977255-f9a7-4531-8a9c-b1a3a17e0b79" (UID: "bd977255-f9a7-4531-8a9c-b1a3a17e0b79"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:54:48.256400 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:48.256293 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd977255-f9a7-4531-8a9c-b1a3a17e0b79-registry-certificates" pod "bd977255-f9a7-4531-8a9c-b1a3a17e0b79" (UID: "bd977255-f9a7-4531-8a9c-b1a3a17e0b79"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:54:48.257864 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:48.257820 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd977255-f9a7-4531-8a9c-b1a3a17e0b79-bound-sa-token" pod "bd977255-f9a7-4531-8a9c-b1a3a17e0b79" (UID: "bd977255-f9a7-4531-8a9c-b1a3a17e0b79"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:54:48.257971 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:48.257897 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd977255-f9a7-4531-8a9c-b1a3a17e0b79-installation-pull-secrets" pod "bd977255-f9a7-4531-8a9c-b1a3a17e0b79" (UID: "bd977255-f9a7-4531-8a9c-b1a3a17e0b79"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:54:48.258021 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:48.257981 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd977255-f9a7-4531-8a9c-b1a3a17e0b79-kube-api-access-8f7sv" pod "bd977255-f9a7-4531-8a9c-b1a3a17e0b79" (UID: "bd977255-f9a7-4531-8a9c-b1a3a17e0b79"). InnerVolumeSpecName "kube-api-access-8f7sv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:54:48.258064 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:48.258015 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd977255-f9a7-4531-8a9c-b1a3a17e0b79-image-registry-private-configuration" pod "bd977255-f9a7-4531-8a9c-b1a3a17e0b79" (UID: "bd977255-f9a7-4531-8a9c-b1a3a17e0b79"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:54:48.258139 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:48.258118 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd977255-f9a7-4531-8a9c-b1a3a17e0b79-registry-tls" pod "bd977255-f9a7-4531-8a9c-b1a3a17e0b79" (UID: "bd977255-f9a7-4531-8a9c-b1a3a17e0b79"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:54:48.262417 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:48.262394 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd977255-f9a7-4531-8a9c-b1a3a17e0b79-ca-trust-extracted" pod "bd977255-f9a7-4531-8a9c-b1a3a17e0b79" (UID: "bd977255-f9a7-4531-8a9c-b1a3a17e0b79"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:54:48.356286 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:48.356259 4908 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bd977255-f9a7-4531-8a9c-b1a3a17e0b79-installation-pull-secrets\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 11:54:48.356286 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:48.356284 4908 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bd977255-f9a7-4531-8a9c-b1a3a17e0b79-bound-sa-token\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 11:54:48.356435 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:48.356298 4908 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bd977255-f9a7-4531-8a9c-b1a3a17e0b79-trusted-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 11:54:48.356435 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:48.356310 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8f7sv\" (UniqueName: \"kubernetes.io/projected/bd977255-f9a7-4531-8a9c-b1a3a17e0b79-kube-api-access-8f7sv\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 11:54:48.356435 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:48.356322 4908 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bd977255-f9a7-4531-8a9c-b1a3a17e0b79-registry-tls\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 11:54:48.356435 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:48.356335 4908 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bd977255-f9a7-4531-8a9c-b1a3a17e0b79-registry-certificates\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 11:54:48.356435 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:48.356346 4908 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bd977255-f9a7-4531-8a9c-b1a3a17e0b79-ca-trust-extracted\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 11:54:48.356435 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:48.356361 4908 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bd977255-f9a7-4531-8a9c-b1a3a17e0b79-image-registry-private-configuration\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 11:54:48.849562 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:48.849530 4908 generic.go:356] "Generic (PLEG): container finished" podID="bd977255-f9a7-4531-8a9c-b1a3a17e0b79" containerID="4889b4249d2a5758089345f3d4b30fb5921f82dba9f2b0887738eafc5cb54dce" exitCode=0 Oct 30 11:54:48.849700 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:48.849570 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-557c6b9f47-vcr2t" Oct 30 11:54:48.849700 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:48.849618 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-557c6b9f47-vcr2t" event={"ID":"bd977255-f9a7-4531-8a9c-b1a3a17e0b79","Type":"ContainerDied","Data":"4889b4249d2a5758089345f3d4b30fb5921f82dba9f2b0887738eafc5cb54dce"} Oct 30 11:54:48.849700 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:48.849667 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-557c6b9f47-vcr2t" event={"ID":"bd977255-f9a7-4531-8a9c-b1a3a17e0b79","Type":"ContainerDied","Data":"914e9bb474175e0e9380bc28a8141b3a8b4db1b61e5beede916579ec3f61c73f"} Oct 30 11:54:48.849700 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:48.849691 4908 scope.go:122] "RemoveContainer" containerID="4889b4249d2a5758089345f3d4b30fb5921f82dba9f2b0887738eafc5cb54dce" Oct 30 11:54:48.858223 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:48.858208 4908 scope.go:122] "RemoveContainer" containerID="4889b4249d2a5758089345f3d4b30fb5921f82dba9f2b0887738eafc5cb54dce" Oct 30 11:54:48.858466 ip-10-0-141-251 kubenswrapper[4908]: E1030 11:54:48.858445 4908 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4889b4249d2a5758089345f3d4b30fb5921f82dba9f2b0887738eafc5cb54dce\": container with ID starting with 4889b4249d2a5758089345f3d4b30fb5921f82dba9f2b0887738eafc5cb54dce not found: ID does not exist" containerID="4889b4249d2a5758089345f3d4b30fb5921f82dba9f2b0887738eafc5cb54dce" Oct 30 11:54:48.858525 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:48.858474 4908 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4889b4249d2a5758089345f3d4b30fb5921f82dba9f2b0887738eafc5cb54dce"} err="failed to get container status \"4889b4249d2a5758089345f3d4b30fb5921f82dba9f2b0887738eafc5cb54dce\": rpc error: code = NotFound desc = could not find container \"4889b4249d2a5758089345f3d4b30fb5921f82dba9f2b0887738eafc5cb54dce\": container with ID starting with 4889b4249d2a5758089345f3d4b30fb5921f82dba9f2b0887738eafc5cb54dce not found: ID does not exist" Oct 30 11:54:48.918010 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:48.917988 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-557c6b9f47-vcr2t"] Oct 30 11:54:48.929322 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:48.929300 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-557c6b9f47-vcr2t"] Oct 30 11:54:49.603360 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:54:49.603316 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd977255-f9a7-4531-8a9c-b1a3a17e0b79" path="/var/lib/kubelet/pods/bd977255-f9a7-4531-8a9c-b1a3a17e0b79/volumes" Oct 30 11:54:49.684664 ip-10-0-141-251 kubenswrapper[4908]: E1030 11:54:49.684632 4908 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vn6vv" containerName="extractor" Oct 30 11:55:10.320460 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:55:10.320417 4908 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-79fd565486-8fl78" podUID="cfaad7d0-d30b-4ddc-81cc-2b603416fc29" containerName="registry" containerID="cri-o://a699cee80ab4f93e500af119a3a188a6166fe8fd4625cdf1345dda0daf8ed3b6" gracePeriod=30 Oct 30 11:55:10.567769 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:55:10.567742 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-79fd565486-8fl78" Oct 30 11:55:10.614154 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:55:10.614092 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfaad7d0-d30b-4ddc-81cc-2b603416fc29-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cfaad7d0-d30b-4ddc-81cc-2b603416fc29-bound-sa-token\") pod \"cfaad7d0-d30b-4ddc-81cc-2b603416fc29\" (UID: \"cfaad7d0-d30b-4ddc-81cc-2b603416fc29\") " Oct 30 11:55:10.614270 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:55:10.614153 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cfaad7d0-d30b-4ddc-81cc-2b603416fc29-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cfaad7d0-d30b-4ddc-81cc-2b603416fc29-image-registry-private-configuration\") pod \"cfaad7d0-d30b-4ddc-81cc-2b603416fc29\" (UID: \"cfaad7d0-d30b-4ddc-81cc-2b603416fc29\") " Oct 30 11:55:10.614270 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:55:10.614198 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cfaad7d0-d30b-4ddc-81cc-2b603416fc29-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cfaad7d0-d30b-4ddc-81cc-2b603416fc29-registry-certificates\") pod \"cfaad7d0-d30b-4ddc-81cc-2b603416fc29\" (UID: \"cfaad7d0-d30b-4ddc-81cc-2b603416fc29\") " Oct 30 11:55:10.614270 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:55:10.614226 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfaad7d0-d30b-4ddc-81cc-2b603416fc29-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cfaad7d0-d30b-4ddc-81cc-2b603416fc29-ca-trust-extracted\") pod \"cfaad7d0-d30b-4ddc-81cc-2b603416fc29\" (UID: \"cfaad7d0-d30b-4ddc-81cc-2b603416fc29\") " Oct 30 11:55:10.614424 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:55:10.614284 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfaad7d0-d30b-4ddc-81cc-2b603416fc29-kube-api-access-75fhw\" (UniqueName: \"kubernetes.io/projected/cfaad7d0-d30b-4ddc-81cc-2b603416fc29-kube-api-access-75fhw\") pod \"cfaad7d0-d30b-4ddc-81cc-2b603416fc29\" (UID: \"cfaad7d0-d30b-4ddc-81cc-2b603416fc29\") " Oct 30 11:55:10.614424 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:55:10.614322 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cfaad7d0-d30b-4ddc-81cc-2b603416fc29-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cfaad7d0-d30b-4ddc-81cc-2b603416fc29-trusted-ca\") pod \"cfaad7d0-d30b-4ddc-81cc-2b603416fc29\" (UID: \"cfaad7d0-d30b-4ddc-81cc-2b603416fc29\") " Oct 30 11:55:10.614424 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:55:10.614364 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cfaad7d0-d30b-4ddc-81cc-2b603416fc29-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cfaad7d0-d30b-4ddc-81cc-2b603416fc29-installation-pull-secrets\") pod \"cfaad7d0-d30b-4ddc-81cc-2b603416fc29\" (UID: \"cfaad7d0-d30b-4ddc-81cc-2b603416fc29\") " Oct 30 11:55:10.614424 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:55:10.614393 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfaad7d0-d30b-4ddc-81cc-2b603416fc29-registry-tls\" (UniqueName: \"kubernetes.io/projected/cfaad7d0-d30b-4ddc-81cc-2b603416fc29-registry-tls\") pod \"cfaad7d0-d30b-4ddc-81cc-2b603416fc29\" (UID: \"cfaad7d0-d30b-4ddc-81cc-2b603416fc29\") " Oct 30 11:55:10.614748 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:55:10.614697 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfaad7d0-d30b-4ddc-81cc-2b603416fc29-registry-certificates" pod "cfaad7d0-d30b-4ddc-81cc-2b603416fc29" (UID: "cfaad7d0-d30b-4ddc-81cc-2b603416fc29"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:55:10.614940 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:55:10.614743 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfaad7d0-d30b-4ddc-81cc-2b603416fc29-trusted-ca" pod "cfaad7d0-d30b-4ddc-81cc-2b603416fc29" (UID: "cfaad7d0-d30b-4ddc-81cc-2b603416fc29"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:55:10.616387 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:55:10.616364 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfaad7d0-d30b-4ddc-81cc-2b603416fc29-kube-api-access-75fhw" pod "cfaad7d0-d30b-4ddc-81cc-2b603416fc29" (UID: "cfaad7d0-d30b-4ddc-81cc-2b603416fc29"). InnerVolumeSpecName "kube-api-access-75fhw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:55:10.616667 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:55:10.616630 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfaad7d0-d30b-4ddc-81cc-2b603416fc29-bound-sa-token" pod "cfaad7d0-d30b-4ddc-81cc-2b603416fc29" (UID: "cfaad7d0-d30b-4ddc-81cc-2b603416fc29"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:55:10.617034 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:55:10.617007 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfaad7d0-d30b-4ddc-81cc-2b603416fc29-registry-tls" pod "cfaad7d0-d30b-4ddc-81cc-2b603416fc29" (UID: "cfaad7d0-d30b-4ddc-81cc-2b603416fc29"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:55:10.617139 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:55:10.617090 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cfaad7d0-d30b-4ddc-81cc-2b603416fc29-installation-pull-secrets" pod "cfaad7d0-d30b-4ddc-81cc-2b603416fc29" (UID: "cfaad7d0-d30b-4ddc-81cc-2b603416fc29"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:55:10.617139 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:55:10.617128 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cfaad7d0-d30b-4ddc-81cc-2b603416fc29-image-registry-private-configuration" pod "cfaad7d0-d30b-4ddc-81cc-2b603416fc29" (UID: "cfaad7d0-d30b-4ddc-81cc-2b603416fc29"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:55:10.621133 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:55:10.621102 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfaad7d0-d30b-4ddc-81cc-2b603416fc29-ca-trust-extracted" pod "cfaad7d0-d30b-4ddc-81cc-2b603416fc29" (UID: "cfaad7d0-d30b-4ddc-81cc-2b603416fc29"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:55:10.718218 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:55:10.715203 4908 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cfaad7d0-d30b-4ddc-81cc-2b603416fc29-registry-certificates\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 11:55:10.718218 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:55:10.715244 4908 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cfaad7d0-d30b-4ddc-81cc-2b603416fc29-ca-trust-extracted\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 11:55:10.718218 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:55:10.715259 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75fhw\" (UniqueName: \"kubernetes.io/projected/cfaad7d0-d30b-4ddc-81cc-2b603416fc29-kube-api-access-75fhw\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 11:55:10.718218 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:55:10.715273 4908 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cfaad7d0-d30b-4ddc-81cc-2b603416fc29-trusted-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 11:55:10.718218 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:55:10.715288 4908 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cfaad7d0-d30b-4ddc-81cc-2b603416fc29-installation-pull-secrets\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 11:55:10.718218 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:55:10.715306 4908 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cfaad7d0-d30b-4ddc-81cc-2b603416fc29-registry-tls\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 11:55:10.718218 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:55:10.715319 4908 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cfaad7d0-d30b-4ddc-81cc-2b603416fc29-bound-sa-token\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 11:55:10.718218 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:55:10.715333 4908 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cfaad7d0-d30b-4ddc-81cc-2b603416fc29-image-registry-private-configuration\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 11:55:10.934003 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:55:10.933935 4908 generic.go:356] "Generic (PLEG): container finished" podID="cfaad7d0-d30b-4ddc-81cc-2b603416fc29" containerID="a699cee80ab4f93e500af119a3a188a6166fe8fd4625cdf1345dda0daf8ed3b6" exitCode=0 Oct 30 11:55:10.934003 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:55:10.933971 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-79fd565486-8fl78" event={"ID":"cfaad7d0-d30b-4ddc-81cc-2b603416fc29","Type":"ContainerDied","Data":"a699cee80ab4f93e500af119a3a188a6166fe8fd4625cdf1345dda0daf8ed3b6"} Oct 30 11:55:10.934003 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:55:10.933993 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-79fd565486-8fl78" Oct 30 11:55:10.934178 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:55:10.934038 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-79fd565486-8fl78" event={"ID":"cfaad7d0-d30b-4ddc-81cc-2b603416fc29","Type":"ContainerDied","Data":"8ff668624ba16b6f4bf01b1c8f33218ee6244ab3f8dae1e74ae504963d7b4ebe"} Oct 30 11:55:10.934178 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:55:10.934057 4908 scope.go:122] "RemoveContainer" containerID="a699cee80ab4f93e500af119a3a188a6166fe8fd4625cdf1345dda0daf8ed3b6" Oct 30 11:55:10.943001 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:55:10.942982 4908 scope.go:122] "RemoveContainer" containerID="a699cee80ab4f93e500af119a3a188a6166fe8fd4625cdf1345dda0daf8ed3b6" Oct 30 11:55:10.943251 ip-10-0-141-251 kubenswrapper[4908]: E1030 11:55:10.943229 4908 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a699cee80ab4f93e500af119a3a188a6166fe8fd4625cdf1345dda0daf8ed3b6\": container with ID starting with a699cee80ab4f93e500af119a3a188a6166fe8fd4625cdf1345dda0daf8ed3b6 not found: ID does not exist" containerID="a699cee80ab4f93e500af119a3a188a6166fe8fd4625cdf1345dda0daf8ed3b6" Oct 30 11:55:10.943314 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:55:10.943264 4908 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a699cee80ab4f93e500af119a3a188a6166fe8fd4625cdf1345dda0daf8ed3b6"} err="failed to get container status \"a699cee80ab4f93e500af119a3a188a6166fe8fd4625cdf1345dda0daf8ed3b6\": rpc error: code = NotFound desc = could not find container \"a699cee80ab4f93e500af119a3a188a6166fe8fd4625cdf1345dda0daf8ed3b6\": container with ID starting with a699cee80ab4f93e500af119a3a188a6166fe8fd4625cdf1345dda0daf8ed3b6 not found: ID does not exist" Oct 30 11:55:10.992025 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:55:10.992003 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-79fd565486-8fl78"] Oct 30 11:55:11.003403 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:55:11.003382 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-79fd565486-8fl78"] Oct 30 11:55:11.602823 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:55:11.602793 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cfaad7d0-d30b-4ddc-81cc-2b603416fc29" path="/var/lib/kubelet/pods/cfaad7d0-d30b-4ddc-81cc-2b603416fc29/volumes" Oct 30 11:55:14.950372 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:55:14.950334 4908 generic.go:356] "Generic (PLEG): container finished" podID="b1757617-7c51-4029-b329-1b6a4630f860" containerID="29ce06f60b46d65917881bd1cc093f60b28d3fc22fd8f01633174881ca63e1d9" exitCode=0 Oct 30 11:55:14.950750 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:55:14.950413 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-2fqgf" event={"ID":"b1757617-7c51-4029-b329-1b6a4630f860","Type":"ContainerDied","Data":"29ce06f60b46d65917881bd1cc093f60b28d3fc22fd8f01633174881ca63e1d9"} Oct 30 11:55:14.950793 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:55:14.950753 4908 scope.go:122] "RemoveContainer" containerID="29ce06f60b46d65917881bd1cc093f60b28d3fc22fd8f01633174881ca63e1d9" Oct 30 11:55:15.955758 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:55:15.955723 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-2fqgf" event={"ID":"b1757617-7c51-4029-b329-1b6a4630f860","Type":"ContainerStarted","Data":"4084eebf0506b660800129f80f946f9abe509550beb45a97d0674eafa8964161"} Oct 30 11:55:19.680883 ip-10-0-141-251 kubenswrapper[4908]: E1030 11:55:19.680848 4908 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vn6vv" containerName="extractor" Oct 30 11:55:29.009572 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:55:29.009542 4908 generic.go:356] "Generic (PLEG): container finished" podID="c815b3fc-7f35-4ae6-a015-2e84c6108a64" containerID="d2aa6d4f0aa8e185b25bd50d1c110fd6ed07e2f6210f07a050cd7959ded0257d" exitCode=0 Oct 30 11:55:29.009990 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:55:29.009614 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6l6fr" event={"ID":"c815b3fc-7f35-4ae6-a015-2e84c6108a64","Type":"ContainerDied","Data":"d2aa6d4f0aa8e185b25bd50d1c110fd6ed07e2f6210f07a050cd7959ded0257d"} Oct 30 11:55:29.009990 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:55:29.009963 4908 scope.go:122] "RemoveContainer" containerID="d2aa6d4f0aa8e185b25bd50d1c110fd6ed07e2f6210f07a050cd7959ded0257d" Oct 30 11:55:29.011137 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:55:29.011115 4908 generic.go:356] "Generic (PLEG): container finished" podID="b3f20243-155c-4e25-a1ca-6131a141fbd4" containerID="7a163cb0cdb6ebf4fc34c853de038086b53b4f3ad84a179b17a8b40ce7b8bad2" exitCode=0 Oct 30 11:55:29.011243 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:55:29.011185 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-86p78" event={"ID":"b3f20243-155c-4e25-a1ca-6131a141fbd4","Type":"ContainerDied","Data":"7a163cb0cdb6ebf4fc34c853de038086b53b4f3ad84a179b17a8b40ce7b8bad2"} Oct 30 11:55:29.011445 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:55:29.011430 4908 scope.go:122] "RemoveContainer" containerID="7a163cb0cdb6ebf4fc34c853de038086b53b4f3ad84a179b17a8b40ce7b8bad2" Oct 30 11:55:30.016683 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:55:30.016644 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6l6fr" event={"ID":"c815b3fc-7f35-4ae6-a015-2e84c6108a64","Type":"ContainerStarted","Data":"7bbd1a3b843b2526aed546b51521265735374d3ca371d2b2cba508080c44806e"} Oct 30 11:55:30.018352 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:55:30.018328 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-86p78" event={"ID":"b3f20243-155c-4e25-a1ca-6131a141fbd4","Type":"ContainerStarted","Data":"53367e8d18d162bba345e4a2fe12bb180c6762b98f3dd3ef769e96cf48e74f82"} Oct 30 11:55:49.681685 ip-10-0-141-251 kubenswrapper[4908]: E1030 11:55:49.681647 4908 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vn6vv" containerName="extractor" Oct 30 11:56:19.681094 ip-10-0-141-251 kubenswrapper[4908]: E1030 11:56:19.681016 4908 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vn6vv" containerName="extractor" Oct 30 11:56:49.681531 ip-10-0-141-251 kubenswrapper[4908]: E1030 11:56:49.681496 4908 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vn6vv" containerName="extractor" Oct 30 11:57:19.681521 ip-10-0-141-251 kubenswrapper[4908]: E1030 11:57:19.681484 4908 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vn6vv" containerName="extractor" Oct 30 11:57:32.256029 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:32.255992 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71m7gfr"] Oct 30 11:57:32.256620 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:32.256325 4908 cpu_manager.go:414] "containerMap: removing container" podUID="bd977255-f9a7-4531-8a9c-b1a3a17e0b79" containerName="registry" Oct 30 11:57:32.256620 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:32.256337 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd977255-f9a7-4531-8a9c-b1a3a17e0b79" containerName="registry" Oct 30 11:57:32.256620 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:32.256349 4908 cpu_manager.go:414] "containerMap: removing container" podUID="cfaad7d0-d30b-4ddc-81cc-2b603416fc29" containerName="registry" Oct 30 11:57:32.256620 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:32.256354 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfaad7d0-d30b-4ddc-81cc-2b603416fc29" containerName="registry" Oct 30 11:57:32.256620 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:32.256412 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd977255-f9a7-4531-8a9c-b1a3a17e0b79" containerName="registry" Oct 30 11:57:32.256620 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:32.256422 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfaad7d0-d30b-4ddc-81cc-2b603416fc29" containerName="registry" Oct 30 11:57:32.259478 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:32.259461 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71m7gfr" Oct 30 11:57:32.284603 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:32.284574 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-2cx6f\"" Oct 30 11:57:32.284603 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:32.284584 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Oct 30 11:57:32.284773 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:32.284585 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Oct 30 11:57:32.285978 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:32.285953 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71m7gfr"] Oct 30 11:57:32.385728 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:32.385702 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvkfpf"] Oct 30 11:57:32.389042 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:32.389024 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvkfpf" Oct 30 11:57:32.411279 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:32.411255 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvkfpf"] Oct 30 11:57:32.418788 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:32.418768 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/baabd525-2d60-468e-a946-61cdb004deef-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71m7gfr\" (UID: \"baabd525-2d60-468e-a946-61cdb004deef\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71m7gfr" Oct 30 11:57:32.418885 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:32.418822 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/baabd525-2d60-468e-a946-61cdb004deef-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71m7gfr\" (UID: \"baabd525-2d60-468e-a946-61cdb004deef\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71m7gfr" Oct 30 11:57:32.418934 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:32.418915 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rw8hf\" (UniqueName: \"kubernetes.io/projected/baabd525-2d60-468e-a946-61cdb004deef-kube-api-access-rw8hf\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71m7gfr\" (UID: \"baabd525-2d60-468e-a946-61cdb004deef\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71m7gfr" Oct 30 11:57:32.473612 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:32.473588 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-2wfgv\"" Oct 30 11:57:32.520086 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:32.520036 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6r8tt\" (UniqueName: \"kubernetes.io/projected/b4b41920-9ac9-412b-9afb-6c1d1ac23595-kube-api-access-6r8tt\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvkfpf\" (UID: \"b4b41920-9ac9-412b-9afb-6c1d1ac23595\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvkfpf" Oct 30 11:57:32.520086 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:32.520072 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b4b41920-9ac9-412b-9afb-6c1d1ac23595-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvkfpf\" (UID: \"b4b41920-9ac9-412b-9afb-6c1d1ac23595\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvkfpf" Oct 30 11:57:32.520200 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:32.520091 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/baabd525-2d60-468e-a946-61cdb004deef-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71m7gfr\" (UID: \"baabd525-2d60-468e-a946-61cdb004deef\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71m7gfr" Oct 30 11:57:32.520200 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:32.520193 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rw8hf\" (UniqueName: \"kubernetes.io/projected/baabd525-2d60-468e-a946-61cdb004deef-kube-api-access-rw8hf\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71m7gfr\" (UID: \"baabd525-2d60-468e-a946-61cdb004deef\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71m7gfr" Oct 30 11:57:32.520265 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:32.520226 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/baabd525-2d60-468e-a946-61cdb004deef-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71m7gfr\" (UID: \"baabd525-2d60-468e-a946-61cdb004deef\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71m7gfr" Oct 30 11:57:32.520265 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:32.520259 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b4b41920-9ac9-412b-9afb-6c1d1ac23595-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvkfpf\" (UID: \"b4b41920-9ac9-412b-9afb-6c1d1ac23595\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvkfpf" Oct 30 11:57:32.520451 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:32.520433 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/baabd525-2d60-468e-a946-61cdb004deef-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71m7gfr\" (UID: \"baabd525-2d60-468e-a946-61cdb004deef\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71m7gfr" Oct 30 11:57:32.520499 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:32.520484 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/baabd525-2d60-468e-a946-61cdb004deef-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71m7gfr\" (UID: \"baabd525-2d60-468e-a946-61cdb004deef\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71m7gfr" Oct 30 11:57:32.542817 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:32.542794 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rw8hf\" (UniqueName: \"kubernetes.io/projected/baabd525-2d60-468e-a946-61cdb004deef-kube-api-access-rw8hf\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71m7gfr\" (UID: \"baabd525-2d60-468e-a946-61cdb004deef\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71m7gfr" Oct 30 11:57:32.574546 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:32.574524 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71m7gfr" Oct 30 11:57:32.621581 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:32.621548 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b4b41920-9ac9-412b-9afb-6c1d1ac23595-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvkfpf\" (UID: \"b4b41920-9ac9-412b-9afb-6c1d1ac23595\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvkfpf" Oct 30 11:57:32.621766 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:32.621606 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6r8tt\" (UniqueName: \"kubernetes.io/projected/b4b41920-9ac9-412b-9afb-6c1d1ac23595-kube-api-access-6r8tt\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvkfpf\" (UID: \"b4b41920-9ac9-412b-9afb-6c1d1ac23595\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvkfpf" Oct 30 11:57:32.621766 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:32.621648 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b4b41920-9ac9-412b-9afb-6c1d1ac23595-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvkfpf\" (UID: \"b4b41920-9ac9-412b-9afb-6c1d1ac23595\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvkfpf" Oct 30 11:57:32.622027 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:32.622008 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b4b41920-9ac9-412b-9afb-6c1d1ac23595-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvkfpf\" (UID: \"b4b41920-9ac9-412b-9afb-6c1d1ac23595\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvkfpf" Oct 30 11:57:32.622732 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:32.622378 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b4b41920-9ac9-412b-9afb-6c1d1ac23595-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvkfpf\" (UID: \"b4b41920-9ac9-412b-9afb-6c1d1ac23595\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvkfpf" Oct 30 11:57:32.648639 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:32.648615 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6r8tt\" (UniqueName: \"kubernetes.io/projected/b4b41920-9ac9-412b-9afb-6c1d1ac23595-kube-api-access-6r8tt\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvkfpf\" (UID: \"b4b41920-9ac9-412b-9afb-6c1d1ac23595\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvkfpf" Oct 30 11:57:32.703670 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:32.703643 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvkfpf" Oct 30 11:57:32.872726 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:32.872655 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71m7gfr"] Oct 30 11:57:32.875619 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:57:32.875593 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbaabd525_2d60_468e_a946_61cdb004deef.slice/crio-fa1ecd4a745dccd2c08a84592cc2e8e444fd9a6927563c8b75fabef44205247e WatchSource:0}: Error finding container fa1ecd4a745dccd2c08a84592cc2e8e444fd9a6927563c8b75fabef44205247e: Status 404 returned error can't find the container with id fa1ecd4a745dccd2c08a84592cc2e8e444fd9a6927563c8b75fabef44205247e Oct 30 11:57:33.002584 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:33.002515 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvkfpf"] Oct 30 11:57:33.007212 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:57:33.007190 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4b41920_9ac9_412b_9afb_6c1d1ac23595.slice/crio-a3fcd5723d53461f9e01fe859e77f2642336d829619b3872a448da9057b00a69 WatchSource:0}: Error finding container a3fcd5723d53461f9e01fe859e77f2642336d829619b3872a448da9057b00a69: Status 404 returned error can't find the container with id a3fcd5723d53461f9e01fe859e77f2642336d829619b3872a448da9057b00a69 Oct 30 11:57:33.443592 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:33.443545 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71m7gfr" event={"ID":"baabd525-2d60-468e-a946-61cdb004deef","Type":"ContainerStarted","Data":"fa1ecd4a745dccd2c08a84592cc2e8e444fd9a6927563c8b75fabef44205247e"} Oct 30 11:57:33.445020 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:33.444988 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvkfpf" event={"ID":"b4b41920-9ac9-412b-9afb-6c1d1ac23595","Type":"ContainerStarted","Data":"a3fcd5723d53461f9e01fe859e77f2642336d829619b3872a448da9057b00a69"} Oct 30 11:57:33.483696 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:33.483668 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644clgth"] Oct 30 11:57:33.487879 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:33.487307 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644clgth" Oct 30 11:57:33.501654 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:33.501631 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-stp2d\"" Oct 30 11:57:33.506703 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:33.506663 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644clgth"] Oct 30 11:57:33.629622 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:33.629594 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2d7e3b95-ffe2-497f-baf4-86d71d7f3abe-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644clgth\" (UID: \"2d7e3b95-ffe2-497f-baf4-86d71d7f3abe\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644clgth" Oct 30 11:57:33.629761 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:33.629721 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmccn\" (UniqueName: \"kubernetes.io/projected/2d7e3b95-ffe2-497f-baf4-86d71d7f3abe-kube-api-access-fmccn\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644clgth\" (UID: \"2d7e3b95-ffe2-497f-baf4-86d71d7f3abe\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644clgth" Oct 30 11:57:33.629823 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:33.629770 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2d7e3b95-ffe2-497f-baf4-86d71d7f3abe-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644clgth\" (UID: \"2d7e3b95-ffe2-497f-baf4-86d71d7f3abe\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644clgth" Oct 30 11:57:33.730245 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:33.730215 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmccn\" (UniqueName: \"kubernetes.io/projected/2d7e3b95-ffe2-497f-baf4-86d71d7f3abe-kube-api-access-fmccn\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644clgth\" (UID: \"2d7e3b95-ffe2-497f-baf4-86d71d7f3abe\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644clgth" Oct 30 11:57:33.730375 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:33.730266 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2d7e3b95-ffe2-497f-baf4-86d71d7f3abe-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644clgth\" (UID: \"2d7e3b95-ffe2-497f-baf4-86d71d7f3abe\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644clgth" Oct 30 11:57:33.730375 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:33.730316 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2d7e3b95-ffe2-497f-baf4-86d71d7f3abe-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644clgth\" (UID: \"2d7e3b95-ffe2-497f-baf4-86d71d7f3abe\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644clgth" Oct 30 11:57:33.730667 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:33.730627 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2d7e3b95-ffe2-497f-baf4-86d71d7f3abe-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644clgth\" (UID: \"2d7e3b95-ffe2-497f-baf4-86d71d7f3abe\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644clgth" Oct 30 11:57:33.730667 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:33.730666 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2d7e3b95-ffe2-497f-baf4-86d71d7f3abe-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644clgth\" (UID: \"2d7e3b95-ffe2-497f-baf4-86d71d7f3abe\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644clgth" Oct 30 11:57:33.753156 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:33.753132 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmccn\" (UniqueName: \"kubernetes.io/projected/2d7e3b95-ffe2-497f-baf4-86d71d7f3abe-kube-api-access-fmccn\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644clgth\" (UID: \"2d7e3b95-ffe2-497f-baf4-86d71d7f3abe\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644clgth" Oct 30 11:57:33.806397 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:33.806374 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644clgth" Oct 30 11:57:34.114086 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:34.114008 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644clgth"] Oct 30 11:57:34.120465 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:57:34.120432 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d7e3b95_ffe2_497f_baf4_86d71d7f3abe.slice/crio-6597b86a4f7ae8b4f84349f9c41969ab61e35dc2baacdb403fefe9abb209a020 WatchSource:0}: Error finding container 6597b86a4f7ae8b4f84349f9c41969ab61e35dc2baacdb403fefe9abb209a020: Status 404 returned error can't find the container with id 6597b86a4f7ae8b4f84349f9c41969ab61e35dc2baacdb403fefe9abb209a020 Oct 30 11:57:34.451974 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:34.451891 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644clgth" event={"ID":"2d7e3b95-ffe2-497f-baf4-86d71d7f3abe","Type":"ContainerStarted","Data":"6597b86a4f7ae8b4f84349f9c41969ab61e35dc2baacdb403fefe9abb209a020"} Oct 30 11:57:36.140961 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:36.140931 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czcpfc"] Oct 30 11:57:36.145540 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:36.144820 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czcpfc" Oct 30 11:57:36.158872 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:36.158845 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-wg8tk\"" Oct 30 11:57:36.172812 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:36.172788 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czcpfc"] Oct 30 11:57:36.242634 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:36.242548 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w8g9t"] Oct 30 11:57:36.246910 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:36.246412 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w8g9t" Oct 30 11:57:36.249944 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:36.249916 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/338c3005-c916-496a-b7af-6ea1b829b0fd-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czcpfc\" (UID: \"338c3005-c916-496a-b7af-6ea1b829b0fd\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czcpfc" Oct 30 11:57:36.250064 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:36.249960 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/338c3005-c916-496a-b7af-6ea1b829b0fd-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czcpfc\" (UID: \"338c3005-c916-496a-b7af-6ea1b829b0fd\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czcpfc" Oct 30 11:57:36.250130 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:36.250108 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7cwb\" (UniqueName: \"kubernetes.io/projected/338c3005-c916-496a-b7af-6ea1b829b0fd-kube-api-access-p7cwb\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czcpfc\" (UID: \"338c3005-c916-496a-b7af-6ea1b829b0fd\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czcpfc" Oct 30 11:57:36.260498 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:36.260460 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-87d6h\"" Oct 30 11:57:36.268445 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:36.268424 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w8g9t"] Oct 30 11:57:36.347862 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:36.347823 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ab4ddk"] Oct 30 11:57:36.351274 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:36.351239 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7cwb\" (UniqueName: \"kubernetes.io/projected/338c3005-c916-496a-b7af-6ea1b829b0fd-kube-api-access-p7cwb\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czcpfc\" (UID: \"338c3005-c916-496a-b7af-6ea1b829b0fd\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czcpfc" Oct 30 11:57:36.351388 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:36.351299 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aa0ab887-afc9-4b8b-a76f-f43606327180-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w8g9t\" (UID: \"aa0ab887-afc9-4b8b-a76f-f43606327180\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w8g9t" Oct 30 11:57:36.351388 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:36.351377 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/338c3005-c916-496a-b7af-6ea1b829b0fd-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czcpfc\" (UID: \"338c3005-c916-496a-b7af-6ea1b829b0fd\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czcpfc" Oct 30 11:57:36.351477 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:36.351414 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/338c3005-c916-496a-b7af-6ea1b829b0fd-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czcpfc\" (UID: \"338c3005-c916-496a-b7af-6ea1b829b0fd\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czcpfc" Oct 30 11:57:36.351477 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:36.351471 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aa0ab887-afc9-4b8b-a76f-f43606327180-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w8g9t\" (UID: \"aa0ab887-afc9-4b8b-a76f-f43606327180\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w8g9t" Oct 30 11:57:36.351563 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:36.351495 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbfht\" (UniqueName: \"kubernetes.io/projected/aa0ab887-afc9-4b8b-a76f-f43606327180-kube-api-access-tbfht\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w8g9t\" (UID: \"aa0ab887-afc9-4b8b-a76f-f43606327180\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w8g9t" Oct 30 11:57:36.352496 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:36.351877 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ab4ddk" Oct 30 11:57:36.352496 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:36.352271 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/338c3005-c916-496a-b7af-6ea1b829b0fd-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czcpfc\" (UID: \"338c3005-c916-496a-b7af-6ea1b829b0fd\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czcpfc" Oct 30 11:57:36.352496 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:36.352298 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/338c3005-c916-496a-b7af-6ea1b829b0fd-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czcpfc\" (UID: \"338c3005-c916-496a-b7af-6ea1b829b0fd\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czcpfc" Oct 30 11:57:36.366476 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:36.366453 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-sg7cb\"" Oct 30 11:57:36.371615 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:36.371590 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ab4ddk"] Oct 30 11:57:36.378384 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:36.378341 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7cwb\" (UniqueName: \"kubernetes.io/projected/338c3005-c916-496a-b7af-6ea1b829b0fd-kube-api-access-p7cwb\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czcpfc\" (UID: \"338c3005-c916-496a-b7af-6ea1b829b0fd\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czcpfc" Oct 30 11:57:36.452679 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:36.452610 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aa0ab887-afc9-4b8b-a76f-f43606327180-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w8g9t\" (UID: \"aa0ab887-afc9-4b8b-a76f-f43606327180\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w8g9t" Oct 30 11:57:36.452679 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:36.452654 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbfht\" (UniqueName: \"kubernetes.io/projected/aa0ab887-afc9-4b8b-a76f-f43606327180-kube-api-access-tbfht\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w8g9t\" (UID: \"aa0ab887-afc9-4b8b-a76f-f43606327180\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w8g9t" Oct 30 11:57:36.452866 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:36.452688 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bf5b2c54-0d4a-401f-b012-b3438ca0e40d-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ab4ddk\" (UID: \"bf5b2c54-0d4a-401f-b012-b3438ca0e40d\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ab4ddk" Oct 30 11:57:36.452925 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:36.452906 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bf5b2c54-0d4a-401f-b012-b3438ca0e40d-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ab4ddk\" (UID: \"bf5b2c54-0d4a-401f-b012-b3438ca0e40d\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ab4ddk" Oct 30 11:57:36.452972 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:36.452954 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8vzc\" (UniqueName: \"kubernetes.io/projected/bf5b2c54-0d4a-401f-b012-b3438ca0e40d-kube-api-access-l8vzc\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ab4ddk\" (UID: \"bf5b2c54-0d4a-401f-b012-b3438ca0e40d\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ab4ddk" Oct 30 11:57:36.453062 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:36.453032 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aa0ab887-afc9-4b8b-a76f-f43606327180-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w8g9t\" (UID: \"aa0ab887-afc9-4b8b-a76f-f43606327180\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w8g9t" Oct 30 11:57:36.453110 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:36.453090 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aa0ab887-afc9-4b8b-a76f-f43606327180-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w8g9t\" (UID: \"aa0ab887-afc9-4b8b-a76f-f43606327180\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w8g9t" Oct 30 11:57:36.453390 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:36.453352 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aa0ab887-afc9-4b8b-a76f-f43606327180-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w8g9t\" (UID: \"aa0ab887-afc9-4b8b-a76f-f43606327180\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w8g9t" Oct 30 11:57:36.458128 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:36.458100 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69rqj8"] Oct 30 11:57:36.462694 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:36.462164 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69rqj8" Oct 30 11:57:36.465037 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:36.465016 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czcpfc" Oct 30 11:57:36.476744 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:36.476714 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-c9tz9\"" Oct 30 11:57:36.481994 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:36.481949 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbfht\" (UniqueName: \"kubernetes.io/projected/aa0ab887-afc9-4b8b-a76f-f43606327180-kube-api-access-tbfht\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w8g9t\" (UID: \"aa0ab887-afc9-4b8b-a76f-f43606327180\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w8g9t" Oct 30 11:57:36.484121 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:36.484061 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69rqj8"] Oct 30 11:57:36.556307 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:36.554925 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8vzc\" (UniqueName: \"kubernetes.io/projected/bf5b2c54-0d4a-401f-b012-b3438ca0e40d-kube-api-access-l8vzc\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ab4ddk\" (UID: \"bf5b2c54-0d4a-401f-b012-b3438ca0e40d\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ab4ddk" Oct 30 11:57:36.556307 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:36.554998 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stmzr\" (UniqueName: \"kubernetes.io/projected/ced4bb34-b6ca-4e9a-990b-2aa6d0ee3843-kube-api-access-stmzr\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69rqj8\" (UID: \"ced4bb34-b6ca-4e9a-990b-2aa6d0ee3843\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69rqj8" Oct 30 11:57:36.556307 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:36.555068 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ced4bb34-b6ca-4e9a-990b-2aa6d0ee3843-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69rqj8\" (UID: \"ced4bb34-b6ca-4e9a-990b-2aa6d0ee3843\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69rqj8" Oct 30 11:57:36.556307 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:36.555107 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ced4bb34-b6ca-4e9a-990b-2aa6d0ee3843-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69rqj8\" (UID: \"ced4bb34-b6ca-4e9a-990b-2aa6d0ee3843\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69rqj8" Oct 30 11:57:36.556307 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:36.555146 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bf5b2c54-0d4a-401f-b012-b3438ca0e40d-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ab4ddk\" (UID: \"bf5b2c54-0d4a-401f-b012-b3438ca0e40d\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ab4ddk" Oct 30 11:57:36.556307 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:36.555179 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bf5b2c54-0d4a-401f-b012-b3438ca0e40d-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ab4ddk\" (UID: \"bf5b2c54-0d4a-401f-b012-b3438ca0e40d\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ab4ddk" Oct 30 11:57:36.556307 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:36.555590 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bf5b2c54-0d4a-401f-b012-b3438ca0e40d-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ab4ddk\" (UID: \"bf5b2c54-0d4a-401f-b012-b3438ca0e40d\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ab4ddk" Oct 30 11:57:36.556307 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:36.556256 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bf5b2c54-0d4a-401f-b012-b3438ca0e40d-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ab4ddk\" (UID: \"bf5b2c54-0d4a-401f-b012-b3438ca0e40d\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ab4ddk" Oct 30 11:57:36.567062 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:36.566843 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w8g9t" Oct 30 11:57:36.596951 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:36.596924 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8vzc\" (UniqueName: \"kubernetes.io/projected/bf5b2c54-0d4a-401f-b012-b3438ca0e40d-kube-api-access-l8vzc\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ab4ddk\" (UID: \"bf5b2c54-0d4a-401f-b012-b3438ca0e40d\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ab4ddk" Oct 30 11:57:36.656813 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:36.655667 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stmzr\" (UniqueName: \"kubernetes.io/projected/ced4bb34-b6ca-4e9a-990b-2aa6d0ee3843-kube-api-access-stmzr\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69rqj8\" (UID: \"ced4bb34-b6ca-4e9a-990b-2aa6d0ee3843\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69rqj8" Oct 30 11:57:36.656813 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:36.655745 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ced4bb34-b6ca-4e9a-990b-2aa6d0ee3843-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69rqj8\" (UID: \"ced4bb34-b6ca-4e9a-990b-2aa6d0ee3843\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69rqj8" Oct 30 11:57:36.656813 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:36.655782 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ced4bb34-b6ca-4e9a-990b-2aa6d0ee3843-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69rqj8\" (UID: \"ced4bb34-b6ca-4e9a-990b-2aa6d0ee3843\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69rqj8" Oct 30 11:57:36.656813 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:36.656220 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ced4bb34-b6ca-4e9a-990b-2aa6d0ee3843-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69rqj8\" (UID: \"ced4bb34-b6ca-4e9a-990b-2aa6d0ee3843\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69rqj8" Oct 30 11:57:36.656813 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:36.656767 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ced4bb34-b6ca-4e9a-990b-2aa6d0ee3843-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69rqj8\" (UID: \"ced4bb34-b6ca-4e9a-990b-2aa6d0ee3843\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69rqj8" Oct 30 11:57:36.673780 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:36.673757 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ab4ddk" Oct 30 11:57:36.680669 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:36.680638 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-stmzr\" (UniqueName: \"kubernetes.io/projected/ced4bb34-b6ca-4e9a-990b-2aa6d0ee3843-kube-api-access-stmzr\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69rqj8\" (UID: \"ced4bb34-b6ca-4e9a-990b-2aa6d0ee3843\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69rqj8" Oct 30 11:57:36.781283 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:36.781254 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czcpfc"] Oct 30 11:57:36.781500 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:36.781394 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69rqj8" Oct 30 11:57:36.795222 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:57:36.795192 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod338c3005_c916_496a_b7af_6ea1b829b0fd.slice/crio-516126b009cf777057ba3a054881f8a6c292bd78cdba1625cce1083cf49ef308 WatchSource:0}: Error finding container 516126b009cf777057ba3a054881f8a6c292bd78cdba1625cce1083cf49ef308: Status 404 returned error can't find the container with id 516126b009cf777057ba3a054881f8a6c292bd78cdba1625cce1083cf49ef308 Oct 30 11:57:36.880414 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:36.880039 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w8g9t"] Oct 30 11:57:36.884235 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:57:36.884206 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa0ab887_afc9_4b8b_a76f_f43606327180.slice/crio-5b46ee25438148c64af80534168a3bcbf21fd086c60a56e373e3f717778e625e WatchSource:0}: Error finding container 5b46ee25438148c64af80534168a3bcbf21fd086c60a56e373e3f717778e625e: Status 404 returned error can't find the container with id 5b46ee25438148c64af80534168a3bcbf21fd086c60a56e373e3f717778e625e Oct 30 11:57:36.998978 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:36.998899 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ab4ddk"] Oct 30 11:57:37.005144 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:57:37.005094 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf5b2c54_0d4a_401f_b012_b3438ca0e40d.slice/crio-b37657b5c0659914e9fe5373cd3040d3fac60108764c80e1f60a258a05c5dea3 WatchSource:0}: Error finding container b37657b5c0659914e9fe5373cd3040d3fac60108764c80e1f60a258a05c5dea3: Status 404 returned error can't find the container with id b37657b5c0659914e9fe5373cd3040d3fac60108764c80e1f60a258a05c5dea3 Oct 30 11:57:37.103820 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:37.103766 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69rqj8"] Oct 30 11:57:37.111081 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:57:37.111051 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podced4bb34_b6ca_4e9a_990b_2aa6d0ee3843.slice/crio-63319d181b29ec1a7a8e471f64b8cfc90e845be3ccbf47d7c75526d5c60dc469 WatchSource:0}: Error finding container 63319d181b29ec1a7a8e471f64b8cfc90e845be3ccbf47d7c75526d5c60dc469: Status 404 returned error can't find the container with id 63319d181b29ec1a7a8e471f64b8cfc90e845be3ccbf47d7c75526d5c60dc469 Oct 30 11:57:37.470078 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:37.469964 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w8g9t" event={"ID":"aa0ab887-afc9-4b8b-a76f-f43606327180","Type":"ContainerStarted","Data":"5b46ee25438148c64af80534168a3bcbf21fd086c60a56e373e3f717778e625e"} Oct 30 11:57:37.472381 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:37.472343 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69rqj8" event={"ID":"ced4bb34-b6ca-4e9a-990b-2aa6d0ee3843","Type":"ContainerStarted","Data":"63319d181b29ec1a7a8e471f64b8cfc90e845be3ccbf47d7c75526d5c60dc469"} Oct 30 11:57:37.474734 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:37.474700 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czcpfc" event={"ID":"338c3005-c916-496a-b7af-6ea1b829b0fd","Type":"ContainerStarted","Data":"516126b009cf777057ba3a054881f8a6c292bd78cdba1625cce1083cf49ef308"} Oct 30 11:57:37.476985 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:37.476933 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ab4ddk" event={"ID":"bf5b2c54-0d4a-401f-b012-b3438ca0e40d","Type":"ContainerStarted","Data":"b37657b5c0659914e9fe5373cd3040d3fac60108764c80e1f60a258a05c5dea3"} Oct 30 11:57:37.661011 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:37.660971 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7155tk9"] Oct 30 11:57:37.664921 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:37.664896 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7155tk9" Oct 30 11:57:37.692146 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:37.692118 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7155tk9"] Oct 30 11:57:37.769941 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:37.769906 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4ltg\" (UniqueName: \"kubernetes.io/projected/5eb05b86-ebf0-487d-b535-43f910fa5ff1-kube-api-access-r4ltg\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7155tk9\" (UID: \"5eb05b86-ebf0-487d-b535-43f910fa5ff1\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7155tk9" Oct 30 11:57:37.770198 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:37.769994 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5eb05b86-ebf0-487d-b535-43f910fa5ff1-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7155tk9\" (UID: \"5eb05b86-ebf0-487d-b535-43f910fa5ff1\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7155tk9" Oct 30 11:57:37.770198 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:37.770048 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5eb05b86-ebf0-487d-b535-43f910fa5ff1-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7155tk9\" (UID: \"5eb05b86-ebf0-487d-b535-43f910fa5ff1\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7155tk9" Oct 30 11:57:37.813925 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:37.813889 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-d9b94"] Oct 30 11:57:37.817792 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:37.817769 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-d9b94" Oct 30 11:57:37.832392 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:37.832294 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 30 11:57:37.832538 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:37.832445 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-lkcrv\"" Oct 30 11:57:37.843799 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:37.843725 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-d9b94"] Oct 30 11:57:37.870678 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:37.870644 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5eb05b86-ebf0-487d-b535-43f910fa5ff1-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7155tk9\" (UID: \"5eb05b86-ebf0-487d-b535-43f910fa5ff1\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7155tk9" Oct 30 11:57:37.870884 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:37.870692 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4ltg\" (UniqueName: \"kubernetes.io/projected/5eb05b86-ebf0-487d-b535-43f910fa5ff1-kube-api-access-r4ltg\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7155tk9\" (UID: \"5eb05b86-ebf0-487d-b535-43f910fa5ff1\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7155tk9" Oct 30 11:57:37.870884 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:37.870770 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5eb05b86-ebf0-487d-b535-43f910fa5ff1-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7155tk9\" (UID: \"5eb05b86-ebf0-487d-b535-43f910fa5ff1\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7155tk9" Oct 30 11:57:37.871181 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:37.871156 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5eb05b86-ebf0-487d-b535-43f910fa5ff1-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7155tk9\" (UID: \"5eb05b86-ebf0-487d-b535-43f910fa5ff1\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7155tk9" Oct 30 11:57:37.871264 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:37.871204 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5eb05b86-ebf0-487d-b535-43f910fa5ff1-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7155tk9\" (UID: \"5eb05b86-ebf0-487d-b535-43f910fa5ff1\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7155tk9" Oct 30 11:57:37.902637 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:37.902602 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4ltg\" (UniqueName: \"kubernetes.io/projected/5eb05b86-ebf0-487d-b535-43f910fa5ff1-kube-api-access-r4ltg\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7155tk9\" (UID: \"5eb05b86-ebf0-487d-b535-43f910fa5ff1\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7155tk9" Oct 30 11:57:37.946906 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:37.946871 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-bt2fx"] Oct 30 11:57:37.952152 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:37.951176 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-bt2fx" Oct 30 11:57:37.969237 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:37.969214 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-vf9rm\"" Oct 30 11:57:37.978517 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:37.978488 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5b1685b-e185-4b12-9c20-3c87e9837fd7-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-d9b94\" (UID: \"a5b1685b-e185-4b12-9c20-3c87e9837fd7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-d9b94" Oct 30 11:57:37.978648 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:37.978549 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5b1685b-e185-4b12-9c20-3c87e9837fd7-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-d9b94\" (UID: \"a5b1685b-e185-4b12-9c20-3c87e9837fd7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-d9b94" Oct 30 11:57:37.978648 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:37.978586 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5b1685b-e185-4b12-9c20-3c87e9837fd7-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-d9b94\" (UID: \"a5b1685b-e185-4b12-9c20-3c87e9837fd7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-d9b94" Oct 30 11:57:37.978648 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:37.978633 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5b1685b-e185-4b12-9c20-3c87e9837fd7-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-d9b94\" (UID: \"a5b1685b-e185-4b12-9c20-3c87e9837fd7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-d9b94" Oct 30 11:57:37.978810 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:37.978664 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5b1685b-e185-4b12-9c20-3c87e9837fd7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-d9b94\" (UID: \"a5b1685b-e185-4b12-9c20-3c87e9837fd7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-d9b94" Oct 30 11:57:37.978810 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:37.978691 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5b1685b-e185-4b12-9c20-3c87e9837fd7-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-d9b94\" (UID: \"a5b1685b-e185-4b12-9c20-3c87e9837fd7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-d9b94" Oct 30 11:57:37.978810 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:37.978722 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5b1685b-e185-4b12-9c20-3c87e9837fd7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-d9b94\" (UID: \"a5b1685b-e185-4b12-9c20-3c87e9837fd7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-d9b94" Oct 30 11:57:37.978810 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:37.978764 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5b1685b-e185-4b12-9c20-3c87e9837fd7-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-d9b94\" (UID: \"a5b1685b-e185-4b12-9c20-3c87e9837fd7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-d9b94" Oct 30 11:57:37.978810 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:37.978793 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5zfh\" (UniqueName: \"kubernetes.io/projected/a5b1685b-e185-4b12-9c20-3c87e9837fd7-kube-api-access-l5zfh\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-d9b94\" (UID: \"a5b1685b-e185-4b12-9c20-3c87e9837fd7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-d9b94" Oct 30 11:57:37.981966 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:37.981942 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-bt2fx"] Oct 30 11:57:37.986635 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:37.986555 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7155tk9" Oct 30 11:57:38.080897 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:38.079401 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5b1685b-e185-4b12-9c20-3c87e9837fd7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-d9b94\" (UID: \"a5b1685b-e185-4b12-9c20-3c87e9837fd7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-d9b94" Oct 30 11:57:38.080897 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:38.079456 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3c15d0a-6612-444e-9e5f-af10d8bc367d-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-bt2fx\" (UID: \"b3c15d0a-6612-444e-9e5f-af10d8bc367d\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-bt2fx" Oct 30 11:57:38.080897 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:38.079502 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5b1685b-e185-4b12-9c20-3c87e9837fd7-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-d9b94\" (UID: \"a5b1685b-e185-4b12-9c20-3c87e9837fd7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-d9b94" Oct 30 11:57:38.080897 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:38.079527 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3c15d0a-6612-444e-9e5f-af10d8bc367d-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-bt2fx\" (UID: \"b3c15d0a-6612-444e-9e5f-af10d8bc367d\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-bt2fx" Oct 30 11:57:38.080897 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:38.079566 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5zfh\" (UniqueName: \"kubernetes.io/projected/a5b1685b-e185-4b12-9c20-3c87e9837fd7-kube-api-access-l5zfh\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-d9b94\" (UID: \"a5b1685b-e185-4b12-9c20-3c87e9837fd7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-d9b94" Oct 30 11:57:38.080897 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:38.079590 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hmbj\" (UniqueName: \"kubernetes.io/projected/b3c15d0a-6612-444e-9e5f-af10d8bc367d-kube-api-access-4hmbj\") pod \"openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-bt2fx\" (UID: \"b3c15d0a-6612-444e-9e5f-af10d8bc367d\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-bt2fx" Oct 30 11:57:38.080897 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:38.079616 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3c15d0a-6612-444e-9e5f-af10d8bc367d-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-bt2fx\" (UID: \"b3c15d0a-6612-444e-9e5f-af10d8bc367d\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-bt2fx" Oct 30 11:57:38.080897 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:38.079655 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3c15d0a-6612-444e-9e5f-af10d8bc367d-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-bt2fx\" (UID: \"b3c15d0a-6612-444e-9e5f-af10d8bc367d\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-bt2fx" Oct 30 11:57:38.080897 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:38.079689 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3c15d0a-6612-444e-9e5f-af10d8bc367d-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-bt2fx\" (UID: \"b3c15d0a-6612-444e-9e5f-af10d8bc367d\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-bt2fx" Oct 30 11:57:38.080897 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:38.079724 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5b1685b-e185-4b12-9c20-3c87e9837fd7-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-d9b94\" (UID: \"a5b1685b-e185-4b12-9c20-3c87e9837fd7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-d9b94" Oct 30 11:57:38.080897 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:38.079771 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5b1685b-e185-4b12-9c20-3c87e9837fd7-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-d9b94\" (UID: \"a5b1685b-e185-4b12-9c20-3c87e9837fd7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-d9b94" Oct 30 11:57:38.080897 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:38.079801 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5b1685b-e185-4b12-9c20-3c87e9837fd7-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-d9b94\" (UID: \"a5b1685b-e185-4b12-9c20-3c87e9837fd7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-d9b94" Oct 30 11:57:38.080897 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:38.079844 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3c15d0a-6612-444e-9e5f-af10d8bc367d-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-bt2fx\" (UID: \"b3c15d0a-6612-444e-9e5f-af10d8bc367d\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-bt2fx" Oct 30 11:57:38.080897 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:38.079887 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3c15d0a-6612-444e-9e5f-af10d8bc367d-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-bt2fx\" (UID: \"b3c15d0a-6612-444e-9e5f-af10d8bc367d\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-bt2fx" Oct 30 11:57:38.080897 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:38.079919 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5b1685b-e185-4b12-9c20-3c87e9837fd7-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-d9b94\" (UID: \"a5b1685b-e185-4b12-9c20-3c87e9837fd7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-d9b94" Oct 30 11:57:38.080897 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:38.079947 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3c15d0a-6612-444e-9e5f-af10d8bc367d-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-bt2fx\" (UID: \"b3c15d0a-6612-444e-9e5f-af10d8bc367d\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-bt2fx" Oct 30 11:57:38.080897 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:38.079975 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5b1685b-e185-4b12-9c20-3c87e9837fd7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-d9b94\" (UID: \"a5b1685b-e185-4b12-9c20-3c87e9837fd7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-d9b94" Oct 30 11:57:38.080897 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:38.080003 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5b1685b-e185-4b12-9c20-3c87e9837fd7-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-d9b94\" (UID: \"a5b1685b-e185-4b12-9c20-3c87e9837fd7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-d9b94" Oct 30 11:57:38.082036 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:38.081112 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5b1685b-e185-4b12-9c20-3c87e9837fd7-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-d9b94\" (UID: \"a5b1685b-e185-4b12-9c20-3c87e9837fd7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-d9b94" Oct 30 11:57:38.082036 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:38.081355 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5b1685b-e185-4b12-9c20-3c87e9837fd7-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-d9b94\" (UID: \"a5b1685b-e185-4b12-9c20-3c87e9837fd7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-d9b94" Oct 30 11:57:38.082036 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:38.081550 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5b1685b-e185-4b12-9c20-3c87e9837fd7-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-d9b94\" (UID: \"a5b1685b-e185-4b12-9c20-3c87e9837fd7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-d9b94" Oct 30 11:57:38.082036 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:38.081808 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5b1685b-e185-4b12-9c20-3c87e9837fd7-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-d9b94\" (UID: \"a5b1685b-e185-4b12-9c20-3c87e9837fd7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-d9b94" Oct 30 11:57:38.086188 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:38.082250 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5b1685b-e185-4b12-9c20-3c87e9837fd7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-d9b94\" (UID: \"a5b1685b-e185-4b12-9c20-3c87e9837fd7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-d9b94" Oct 30 11:57:38.087060 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:38.086780 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5b1685b-e185-4b12-9c20-3c87e9837fd7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-d9b94\" (UID: \"a5b1685b-e185-4b12-9c20-3c87e9837fd7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-d9b94" Oct 30 11:57:38.087060 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:38.086997 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5b1685b-e185-4b12-9c20-3c87e9837fd7-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-d9b94\" (UID: \"a5b1685b-e185-4b12-9c20-3c87e9837fd7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-d9b94" Oct 30 11:57:38.104072 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:38.104043 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5zfh\" (UniqueName: \"kubernetes.io/projected/a5b1685b-e185-4b12-9c20-3c87e9837fd7-kube-api-access-l5zfh\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-d9b94\" (UID: \"a5b1685b-e185-4b12-9c20-3c87e9837fd7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-d9b94" Oct 30 11:57:38.117639 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:38.117596 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5b1685b-e185-4b12-9c20-3c87e9837fd7-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-d9b94\" (UID: \"a5b1685b-e185-4b12-9c20-3c87e9837fd7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-d9b94" Oct 30 11:57:38.135507 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:38.135485 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-d9b94" Oct 30 11:57:38.182363 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:38.181240 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3c15d0a-6612-444e-9e5f-af10d8bc367d-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-bt2fx\" (UID: \"b3c15d0a-6612-444e-9e5f-af10d8bc367d\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-bt2fx" Oct 30 11:57:38.182363 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:38.181297 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3c15d0a-6612-444e-9e5f-af10d8bc367d-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-bt2fx\" (UID: \"b3c15d0a-6612-444e-9e5f-af10d8bc367d\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-bt2fx" Oct 30 11:57:38.182363 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:38.181330 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3c15d0a-6612-444e-9e5f-af10d8bc367d-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-bt2fx\" (UID: \"b3c15d0a-6612-444e-9e5f-af10d8bc367d\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-bt2fx" Oct 30 11:57:38.182363 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:38.181374 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3c15d0a-6612-444e-9e5f-af10d8bc367d-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-bt2fx\" (UID: \"b3c15d0a-6612-444e-9e5f-af10d8bc367d\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-bt2fx" Oct 30 11:57:38.182363 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:38.181421 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3c15d0a-6612-444e-9e5f-af10d8bc367d-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-bt2fx\" (UID: \"b3c15d0a-6612-444e-9e5f-af10d8bc367d\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-bt2fx" Oct 30 11:57:38.182363 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:38.181453 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hmbj\" (UniqueName: \"kubernetes.io/projected/b3c15d0a-6612-444e-9e5f-af10d8bc367d-kube-api-access-4hmbj\") pod \"openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-bt2fx\" (UID: \"b3c15d0a-6612-444e-9e5f-af10d8bc367d\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-bt2fx" Oct 30 11:57:38.182363 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:38.181483 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3c15d0a-6612-444e-9e5f-af10d8bc367d-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-bt2fx\" (UID: \"b3c15d0a-6612-444e-9e5f-af10d8bc367d\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-bt2fx" Oct 30 11:57:38.182363 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:38.181516 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3c15d0a-6612-444e-9e5f-af10d8bc367d-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-bt2fx\" (UID: \"b3c15d0a-6612-444e-9e5f-af10d8bc367d\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-bt2fx" Oct 30 11:57:38.182363 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:38.181548 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3c15d0a-6612-444e-9e5f-af10d8bc367d-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-bt2fx\" (UID: \"b3c15d0a-6612-444e-9e5f-af10d8bc367d\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-bt2fx" Oct 30 11:57:38.183488 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:38.183444 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3c15d0a-6612-444e-9e5f-af10d8bc367d-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-bt2fx\" (UID: \"b3c15d0a-6612-444e-9e5f-af10d8bc367d\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-bt2fx" Oct 30 11:57:38.184359 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:38.183794 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3c15d0a-6612-444e-9e5f-af10d8bc367d-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-bt2fx\" (UID: \"b3c15d0a-6612-444e-9e5f-af10d8bc367d\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-bt2fx" Oct 30 11:57:38.184359 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:38.184317 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3c15d0a-6612-444e-9e5f-af10d8bc367d-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-bt2fx\" (UID: \"b3c15d0a-6612-444e-9e5f-af10d8bc367d\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-bt2fx" Oct 30 11:57:38.184513 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:38.184465 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3c15d0a-6612-444e-9e5f-af10d8bc367d-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-bt2fx\" (UID: \"b3c15d0a-6612-444e-9e5f-af10d8bc367d\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-bt2fx" Oct 30 11:57:38.184947 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:38.184903 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3c15d0a-6612-444e-9e5f-af10d8bc367d-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-bt2fx\" (UID: \"b3c15d0a-6612-444e-9e5f-af10d8bc367d\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-bt2fx" Oct 30 11:57:38.187907 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:38.187850 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3c15d0a-6612-444e-9e5f-af10d8bc367d-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-bt2fx\" (UID: \"b3c15d0a-6612-444e-9e5f-af10d8bc367d\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-bt2fx" Oct 30 11:57:38.189761 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:38.189741 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3c15d0a-6612-444e-9e5f-af10d8bc367d-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-bt2fx\" (UID: \"b3c15d0a-6612-444e-9e5f-af10d8bc367d\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-bt2fx" Oct 30 11:57:38.209169 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:38.209107 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3c15d0a-6612-444e-9e5f-af10d8bc367d-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-bt2fx\" (UID: \"b3c15d0a-6612-444e-9e5f-af10d8bc367d\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-bt2fx" Oct 30 11:57:38.211902 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:38.211860 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hmbj\" (UniqueName: \"kubernetes.io/projected/b3c15d0a-6612-444e-9e5f-af10d8bc367d-kube-api-access-4hmbj\") pod \"openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-bt2fx\" (UID: \"b3c15d0a-6612-444e-9e5f-af10d8bc367d\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-bt2fx" Oct 30 11:57:38.271876 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:38.269309 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-bt2fx" Oct 30 11:57:38.309364 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:38.309131 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvnsnt"] Oct 30 11:57:38.320736 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:38.320710 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7155tk9"] Oct 30 11:57:38.321000 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:38.320845 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvnsnt" Oct 30 11:57:38.325479 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:57:38.325411 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5eb05b86_ebf0_487d_b535_43f910fa5ff1.slice/crio-271f244be9d1e68f0ec194362fa0850feb47abe6f6c65725eb03a88254d81b62 WatchSource:0}: Error finding container 271f244be9d1e68f0ec194362fa0850feb47abe6f6c65725eb03a88254d81b62: Status 404 returned error can't find the container with id 271f244be9d1e68f0ec194362fa0850feb47abe6f6c65725eb03a88254d81b62 Oct 30 11:57:38.337717 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:38.337666 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvnsnt"] Oct 30 11:57:38.383243 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:38.383193 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6d7843ed-9ada-42f3-954e-170c578d1115-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvnsnt\" (UID: \"6d7843ed-9ada-42f3-954e-170c578d1115\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvnsnt" Oct 30 11:57:38.383364 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:38.383294 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6d7843ed-9ada-42f3-954e-170c578d1115-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvnsnt\" (UID: \"6d7843ed-9ada-42f3-954e-170c578d1115\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvnsnt" Oct 30 11:57:38.383364 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:38.383349 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-224jb\" (UniqueName: \"kubernetes.io/projected/6d7843ed-9ada-42f3-954e-170c578d1115-kube-api-access-224jb\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvnsnt\" (UID: \"6d7843ed-9ada-42f3-954e-170c578d1115\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvnsnt" Oct 30 11:57:38.486015 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:38.484630 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-224jb\" (UniqueName: \"kubernetes.io/projected/6d7843ed-9ada-42f3-954e-170c578d1115-kube-api-access-224jb\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvnsnt\" (UID: \"6d7843ed-9ada-42f3-954e-170c578d1115\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvnsnt" Oct 30 11:57:38.486015 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:38.484685 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6d7843ed-9ada-42f3-954e-170c578d1115-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvnsnt\" (UID: \"6d7843ed-9ada-42f3-954e-170c578d1115\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvnsnt" Oct 30 11:57:38.486015 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:38.484750 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6d7843ed-9ada-42f3-954e-170c578d1115-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvnsnt\" (UID: \"6d7843ed-9ada-42f3-954e-170c578d1115\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvnsnt" Oct 30 11:57:38.486015 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:38.485168 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6d7843ed-9ada-42f3-954e-170c578d1115-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvnsnt\" (UID: \"6d7843ed-9ada-42f3-954e-170c578d1115\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvnsnt" Oct 30 11:57:38.488630 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:38.486915 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6d7843ed-9ada-42f3-954e-170c578d1115-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvnsnt\" (UID: \"6d7843ed-9ada-42f3-954e-170c578d1115\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvnsnt" Oct 30 11:57:38.500693 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:38.498656 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7155tk9" event={"ID":"5eb05b86-ebf0-487d-b535-43f910fa5ff1","Type":"ContainerStarted","Data":"271f244be9d1e68f0ec194362fa0850feb47abe6f6c65725eb03a88254d81b62"} Oct 30 11:57:38.507619 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:38.507553 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-d9b94"] Oct 30 11:57:38.511759 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:38.511669 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-224jb\" (UniqueName: \"kubernetes.io/projected/6d7843ed-9ada-42f3-954e-170c578d1115-kube-api-access-224jb\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvnsnt\" (UID: \"6d7843ed-9ada-42f3-954e-170c578d1115\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvnsnt" Oct 30 11:57:38.515748 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:57:38.515710 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5b1685b_e185_4b12_9c20_3c87e9837fd7.slice/crio-c33eae1ff5a10b776e2333b31251d70d4d881e0a6f6be490896eca127a32e637 WatchSource:0}: Error finding container c33eae1ff5a10b776e2333b31251d70d4d881e0a6f6be490896eca127a32e637: Status 404 returned error can't find the container with id c33eae1ff5a10b776e2333b31251d70d4d881e0a6f6be490896eca127a32e637 Oct 30 11:57:38.618717 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:38.618561 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-bt2fx"] Oct 30 11:57:38.647154 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:38.646972 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvnsnt" Oct 30 11:57:39.054980 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:39.054204 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvnsnt"] Oct 30 11:57:39.080581 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:57:39.080549 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d7843ed_9ada_42f3_954e_170c578d1115.slice/crio-548045554b901c67556835cf8f4a99933954aea99349bdca23c491cb97b588ac WatchSource:0}: Error finding container 548045554b901c67556835cf8f4a99933954aea99349bdca23c491cb97b588ac: Status 404 returned error can't find the container with id 548045554b901c67556835cf8f4a99933954aea99349bdca23c491cb97b588ac Oct 30 11:57:39.513140 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:39.513100 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvnsnt" event={"ID":"6d7843ed-9ada-42f3-954e-170c578d1115","Type":"ContainerStarted","Data":"548045554b901c67556835cf8f4a99933954aea99349bdca23c491cb97b588ac"} Oct 30 11:57:39.516117 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:39.516033 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-bt2fx" event={"ID":"b3c15d0a-6612-444e-9e5f-af10d8bc367d","Type":"ContainerStarted","Data":"856d2517e6fd54e51e61073ab0765bf6ecb1e8f713ca1182ca83609d68f3f9ab"} Oct 30 11:57:39.528057 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:39.527988 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-d9b94" event={"ID":"a5b1685b-e185-4b12-9c20-3c87e9837fd7","Type":"ContainerStarted","Data":"c33eae1ff5a10b776e2333b31251d70d4d881e0a6f6be490896eca127a32e637"} Oct 30 11:57:47.621678 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:47.620564 4908 generic.go:356] "Generic (PLEG): container finished" podID="baabd525-2d60-468e-a946-61cdb004deef" containerID="d55fc2142d3c9c7a730588c769c86cfe0915a04b07d258749316b0e499f0c8ec" exitCode=0 Oct 30 11:57:47.621678 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:47.620676 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71m7gfr" event={"ID":"baabd525-2d60-468e-a946-61cdb004deef","Type":"ContainerDied","Data":"d55fc2142d3c9c7a730588c769c86cfe0915a04b07d258749316b0e499f0c8ec"} Oct 30 11:57:47.630490 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:47.630006 4908 generic.go:356] "Generic (PLEG): container finished" podID="b4b41920-9ac9-412b-9afb-6c1d1ac23595" containerID="13da6c34579d66faa670e513bb17e5bdc7273aa8cadc5b7e1e3efcc4ad829cb5" exitCode=0 Oct 30 11:57:47.630490 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:47.630079 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvkfpf" event={"ID":"b4b41920-9ac9-412b-9afb-6c1d1ac23595","Type":"ContainerDied","Data":"13da6c34579d66faa670e513bb17e5bdc7273aa8cadc5b7e1e3efcc4ad829cb5"} Oct 30 11:57:47.795918 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:47.795207 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:57:47.795918 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:47.795300 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:57:47.795918 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:47.795346 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:57:47.835556 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:47.835515 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:57:47.835654 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:47.835615 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:57:47.835706 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:47.835665 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:57:48.642197 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:48.642159 4908 generic.go:356] "Generic (PLEG): container finished" podID="ced4bb34-b6ca-4e9a-990b-2aa6d0ee3843" containerID="9cdf21aec094c4b7f7ea21cb391dbbf0b401638fa686e30fb3487a616e30d3e0" exitCode=0 Oct 30 11:57:48.642690 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:48.642255 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69rqj8" event={"ID":"ced4bb34-b6ca-4e9a-990b-2aa6d0ee3843","Type":"ContainerDied","Data":"9cdf21aec094c4b7f7ea21cb391dbbf0b401638fa686e30fb3487a616e30d3e0"} Oct 30 11:57:48.646773 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:48.646747 4908 generic.go:356] "Generic (PLEG): container finished" podID="338c3005-c916-496a-b7af-6ea1b829b0fd" containerID="c13da3d522e970f1b77b8d4ecc0ab2beb390ede50d6abad9fe3d6fc970e60373" exitCode=0 Oct 30 11:57:48.646907 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:48.646816 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czcpfc" event={"ID":"338c3005-c916-496a-b7af-6ea1b829b0fd","Type":"ContainerDied","Data":"c13da3d522e970f1b77b8d4ecc0ab2beb390ede50d6abad9fe3d6fc970e60373"} Oct 30 11:57:48.650323 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:48.650297 4908 generic.go:356] "Generic (PLEG): container finished" podID="baabd525-2d60-468e-a946-61cdb004deef" containerID="357728831fcf360f38cf517fcc8ec8292e882579848a6d51258ddc301fe89b04" exitCode=0 Oct 30 11:57:48.650428 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:48.650375 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71m7gfr" event={"ID":"baabd525-2d60-468e-a946-61cdb004deef","Type":"ContainerDied","Data":"357728831fcf360f38cf517fcc8ec8292e882579848a6d51258ddc301fe89b04"} Oct 30 11:57:48.655855 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:48.655205 4908 generic.go:356] "Generic (PLEG): container finished" podID="bf5b2c54-0d4a-401f-b012-b3438ca0e40d" containerID="1e0251ee756b1987fb9a26b6fb99ce33d41a53be8b75e2094f4288b5656ab7de" exitCode=0 Oct 30 11:57:48.655855 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:48.655339 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ab4ddk" event={"ID":"bf5b2c54-0d4a-401f-b012-b3438ca0e40d","Type":"ContainerDied","Data":"1e0251ee756b1987fb9a26b6fb99ce33d41a53be8b75e2094f4288b5656ab7de"} Oct 30 11:57:48.660170 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:48.660001 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-d9b94" event={"ID":"a5b1685b-e185-4b12-9c20-3c87e9837fd7","Type":"ContainerStarted","Data":"e490ba37cb4c0f66a103fec7579c2a868db8760d8211a33dd6400e0d8c452b93"} Oct 30 11:57:48.663410 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:48.663385 4908 generic.go:356] "Generic (PLEG): container finished" podID="aa0ab887-afc9-4b8b-a76f-f43606327180" containerID="d38ef99c743b7d5e393b36edad9fdb1b751570acc8827e78fc5c9e309ef36056" exitCode=0 Oct 30 11:57:48.663993 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:48.663530 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w8g9t" event={"ID":"aa0ab887-afc9-4b8b-a76f-f43606327180","Type":"ContainerDied","Data":"d38ef99c743b7d5e393b36edad9fdb1b751570acc8827e78fc5c9e309ef36056"} Oct 30 11:57:48.667794 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:48.667427 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-bt2fx" event={"ID":"b3c15d0a-6612-444e-9e5f-af10d8bc367d","Type":"ContainerStarted","Data":"4557f8ed01898e1ece18f0722d50b668f55448e9e0c3acabe28b21944d2276cf"} Oct 30 11:57:48.774859 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:48.774799 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-bt2fx" podStartSLOduration=2.571983853 podStartE2EDuration="11.774784875s" totalImagesPullingTime="9.202801022s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:57:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:57:48.773667878 +0000 UTC m=+225.288096619" watchObservedRunningTime="2026-10-30 11:57:48.774784875 +0000 UTC m=+225.289213691" Oct 30 11:57:48.957778 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:48.957511 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-d9b94" podStartSLOduration=2.682681459 podStartE2EDuration="11.957497236s" totalImagesPullingTime="9.274815777s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:57:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:57:48.954369863 +0000 UTC m=+225.468798624" watchObservedRunningTime="2026-10-30 11:57:48.957497236 +0000 UTC m=+225.471926028" Oct 30 11:57:49.136821 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:49.136764 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-d9b94" Oct 30 11:57:49.142346 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:49.142320 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-d9b94" Oct 30 11:57:49.271754 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:49.271714 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-bt2fx" Oct 30 11:57:49.278013 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:49.277987 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-bt2fx" Oct 30 11:57:49.682090 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:49.680800 4908 generic.go:356] "Generic (PLEG): container finished" podID="2d7e3b95-ffe2-497f-baf4-86d71d7f3abe" containerID="11b80c76929846657c80dc60a1063555e7a7639b06d0fabe7a982d080d4c1fbc" exitCode=0 Oct 30 11:57:49.682090 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:49.680910 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644clgth" event={"ID":"2d7e3b95-ffe2-497f-baf4-86d71d7f3abe","Type":"ContainerDied","Data":"11b80c76929846657c80dc60a1063555e7a7639b06d0fabe7a982d080d4c1fbc"} Oct 30 11:57:49.690128 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:49.689427 4908 generic.go:356] "Generic (PLEG): container finished" podID="5eb05b86-ebf0-487d-b535-43f910fa5ff1" containerID="fd40cd6f6e43f9b242d6e2e608f9a966d49adf85ec379cd3a4dc15c54589cf3f" exitCode=0 Oct 30 11:57:49.690128 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:49.689729 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7155tk9" event={"ID":"5eb05b86-ebf0-487d-b535-43f910fa5ff1","Type":"ContainerDied","Data":"fd40cd6f6e43f9b242d6e2e608f9a966d49adf85ec379cd3a4dc15c54589cf3f"} Oct 30 11:57:49.691978 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:49.691846 4908 generic.go:356] "Generic (PLEG): container finished" podID="6d7843ed-9ada-42f3-954e-170c578d1115" containerID="c85588d3906e9a026d870cfccd248e3e372b0a2e0882464d2e0f1f223a141307" exitCode=0 Oct 30 11:57:49.691978 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:49.691956 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvnsnt" event={"ID":"6d7843ed-9ada-42f3-954e-170c578d1115","Type":"ContainerDied","Data":"c85588d3906e9a026d870cfccd248e3e372b0a2e0882464d2e0f1f223a141307"} Oct 30 11:57:49.693388 ip-10-0-141-251 kubenswrapper[4908]: E1030 11:57:49.692903 4908 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vn6vv" containerName="extractor" Oct 30 11:57:49.695273 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:49.695254 4908 generic.go:356] "Generic (PLEG): container finished" podID="ced4bb34-b6ca-4e9a-990b-2aa6d0ee3843" containerID="7410104d9f43d969fd887f8b50f3bcba4c6a8edb99101f4cbf1495ac968c0bc8" exitCode=0 Oct 30 11:57:49.695361 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:49.695333 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69rqj8" event={"ID":"ced4bb34-b6ca-4e9a-990b-2aa6d0ee3843","Type":"ContainerDied","Data":"7410104d9f43d969fd887f8b50f3bcba4c6a8edb99101f4cbf1495ac968c0bc8"} Oct 30 11:57:49.696345 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:49.696189 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-d9b94" Oct 30 11:57:49.696345 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:49.696232 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-bt2fx" Oct 30 11:57:49.696664 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:49.696643 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-d9b94" Oct 30 11:57:49.697133 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:49.697091 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-bt2fx" Oct 30 11:57:50.705271 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:50.705227 4908 generic.go:356] "Generic (PLEG): container finished" podID="338c3005-c916-496a-b7af-6ea1b829b0fd" containerID="cb473a62a956729ed89b6581105fb022a42b12f80132a4a7e7131a7a530f4a6d" exitCode=0 Oct 30 11:57:50.705752 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:50.705350 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czcpfc" event={"ID":"338c3005-c916-496a-b7af-6ea1b829b0fd","Type":"ContainerDied","Data":"cb473a62a956729ed89b6581105fb022a42b12f80132a4a7e7131a7a530f4a6d"} Oct 30 11:57:50.713870 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:50.713107 4908 generic.go:356] "Generic (PLEG): container finished" podID="bf5b2c54-0d4a-401f-b012-b3438ca0e40d" containerID="ada09317f7e00c0964e05d97f28b042a6a622074abbdcc13eccc706c63862124" exitCode=0 Oct 30 11:57:50.713870 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:50.713205 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ab4ddk" event={"ID":"bf5b2c54-0d4a-401f-b012-b3438ca0e40d","Type":"ContainerDied","Data":"ada09317f7e00c0964e05d97f28b042a6a622074abbdcc13eccc706c63862124"} Oct 30 11:57:50.718971 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:50.718322 4908 generic.go:356] "Generic (PLEG): container finished" podID="5eb05b86-ebf0-487d-b535-43f910fa5ff1" containerID="768898ea4248031cbc0294f7f2ef84b12c0d96b47441626cf2c35efbd75bbe26" exitCode=0 Oct 30 11:57:50.718971 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:50.718468 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7155tk9" event={"ID":"5eb05b86-ebf0-487d-b535-43f910fa5ff1","Type":"ContainerDied","Data":"768898ea4248031cbc0294f7f2ef84b12c0d96b47441626cf2c35efbd75bbe26"} Oct 30 11:57:50.725352 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:50.723733 4908 generic.go:356] "Generic (PLEG): container finished" podID="aa0ab887-afc9-4b8b-a76f-f43606327180" containerID="edf8ffdf569cce9e41a6adafbb1717b131e82640be3309fa6b23225193d17677" exitCode=0 Oct 30 11:57:50.725352 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:50.724643 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w8g9t" event={"ID":"aa0ab887-afc9-4b8b-a76f-f43606327180","Type":"ContainerDied","Data":"edf8ffdf569cce9e41a6adafbb1717b131e82640be3309fa6b23225193d17677"} Oct 30 11:57:51.742235 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:51.741154 4908 generic.go:356] "Generic (PLEG): container finished" podID="b4b41920-9ac9-412b-9afb-6c1d1ac23595" containerID="dc501574d36fb4805e832d01bb2023990549e92bcce2cee937b8a97b2add6dd8" exitCode=0 Oct 30 11:57:51.742235 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:51.741325 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvkfpf" event={"ID":"b4b41920-9ac9-412b-9afb-6c1d1ac23595","Type":"ContainerDied","Data":"dc501574d36fb4805e832d01bb2023990549e92bcce2cee937b8a97b2add6dd8"} Oct 30 11:57:51.747227 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:51.747195 4908 generic.go:356] "Generic (PLEG): container finished" podID="6d7843ed-9ada-42f3-954e-170c578d1115" containerID="c88d5b5385ec3c2d844a53917c5d0dc12b5150b597ac6bd339cf6559c078f2b0" exitCode=0 Oct 30 11:57:51.748328 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:51.748310 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvnsnt" event={"ID":"6d7843ed-9ada-42f3-954e-170c578d1115","Type":"ContainerDied","Data":"c88d5b5385ec3c2d844a53917c5d0dc12b5150b597ac6bd339cf6559c078f2b0"} Oct 30 11:57:56.828341 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:56.827246 4908 generic.go:356] "Generic (PLEG): container finished" podID="2d7e3b95-ffe2-497f-baf4-86d71d7f3abe" containerID="0b3b0c6a5bb3c201d8a892ba6d0a2e144f9d0c1fd6fe218e27049d6fa29d2b86" exitCode=0 Oct 30 11:57:56.828341 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:57:56.827362 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644clgth" event={"ID":"2d7e3b95-ffe2-497f-baf4-86d71d7f3abe","Type":"ContainerDied","Data":"0b3b0c6a5bb3c201d8a892ba6d0a2e144f9d0c1fd6fe218e27049d6fa29d2b86"} Oct 30 11:58:04.914200 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:04.914159 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71m7gfr" event={"ID":"baabd525-2d60-468e-a946-61cdb004deef","Type":"ContainerStarted","Data":"564c78b7d5f17dd0aed6f2be4fd4c0afa55848543193d1b149f8300dae488a51"} Oct 30 11:58:05.932709 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:05.932662 4908 generic.go:356] "Generic (PLEG): container finished" podID="baabd525-2d60-468e-a946-61cdb004deef" containerID="564c78b7d5f17dd0aed6f2be4fd4c0afa55848543193d1b149f8300dae488a51" exitCode=0 Oct 30 11:58:05.933228 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:05.932763 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71m7gfr" event={"ID":"baabd525-2d60-468e-a946-61cdb004deef","Type":"ContainerDied","Data":"564c78b7d5f17dd0aed6f2be4fd4c0afa55848543193d1b149f8300dae488a51"} Oct 30 11:58:07.157867 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:07.157816 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71m7gfr" Oct 30 11:58:07.288591 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:07.288554 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/baabd525-2d60-468e-a946-61cdb004deef-kube-api-access-rw8hf\" (UniqueName: \"kubernetes.io/projected/baabd525-2d60-468e-a946-61cdb004deef-kube-api-access-rw8hf\") pod \"baabd525-2d60-468e-a946-61cdb004deef\" (UID: \"baabd525-2d60-468e-a946-61cdb004deef\") " Oct 30 11:58:07.288757 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:07.288613 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/baabd525-2d60-468e-a946-61cdb004deef-util\" (UniqueName: \"kubernetes.io/empty-dir/baabd525-2d60-468e-a946-61cdb004deef-util\") pod \"baabd525-2d60-468e-a946-61cdb004deef\" (UID: \"baabd525-2d60-468e-a946-61cdb004deef\") " Oct 30 11:58:07.288757 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:07.288708 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/baabd525-2d60-468e-a946-61cdb004deef-bundle\" (UniqueName: \"kubernetes.io/empty-dir/baabd525-2d60-468e-a946-61cdb004deef-bundle\") pod \"baabd525-2d60-468e-a946-61cdb004deef\" (UID: \"baabd525-2d60-468e-a946-61cdb004deef\") " Oct 30 11:58:07.296580 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:07.293456 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baabd525-2d60-468e-a946-61cdb004deef-bundle" pod "baabd525-2d60-468e-a946-61cdb004deef" (UID: "baabd525-2d60-468e-a946-61cdb004deef"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:58:07.296580 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:07.295297 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/baabd525-2d60-468e-a946-61cdb004deef-kube-api-access-rw8hf" pod "baabd525-2d60-468e-a946-61cdb004deef" (UID: "baabd525-2d60-468e-a946-61cdb004deef"). InnerVolumeSpecName "kube-api-access-rw8hf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:58:07.299494 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:07.299460 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baabd525-2d60-468e-a946-61cdb004deef-util" pod "baabd525-2d60-468e-a946-61cdb004deef" (UID: "baabd525-2d60-468e-a946-61cdb004deef"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:58:07.390343 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:07.390305 4908 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/baabd525-2d60-468e-a946-61cdb004deef-bundle\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 11:58:07.390343 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:07.390345 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rw8hf\" (UniqueName: \"kubernetes.io/projected/baabd525-2d60-468e-a946-61cdb004deef-kube-api-access-rw8hf\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 11:58:07.390574 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:07.390362 4908 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/baabd525-2d60-468e-a946-61cdb004deef-util\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 11:58:07.948053 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:07.948010 4908 generic.go:356] "Generic (PLEG): container finished" podID="ced4bb34-b6ca-4e9a-990b-2aa6d0ee3843" containerID="c7c72dc6ac7cdd3f9892bb34681a186ef0ea6d8d82523e9e8c316024cc931cb3" exitCode=0 Oct 30 11:58:07.948214 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:07.948098 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69rqj8" event={"ID":"ced4bb34-b6ca-4e9a-990b-2aa6d0ee3843","Type":"ContainerDied","Data":"c7c72dc6ac7cdd3f9892bb34681a186ef0ea6d8d82523e9e8c316024cc931cb3"} Oct 30 11:58:07.950992 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:07.950957 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71m7gfr" event={"ID":"baabd525-2d60-468e-a946-61cdb004deef","Type":"ContainerDied","Data":"fa1ecd4a745dccd2c08a84592cc2e8e444fd9a6927563c8b75fabef44205247e"} Oct 30 11:58:07.951168 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:07.951018 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fa1ecd4a745dccd2c08a84592cc2e8e444fd9a6927563c8b75fabef44205247e" Oct 30 11:58:07.951168 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:07.950965 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71m7gfr" Oct 30 11:58:07.953756 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:07.953731 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ab4ddk" event={"ID":"bf5b2c54-0d4a-401f-b012-b3438ca0e40d","Type":"ContainerStarted","Data":"9b9351d70a20325ab32bd9d443efca5846e2e2c59bb16a66b9d1a0a9021db7b1"} Oct 30 11:58:08.044443 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:08.044397 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ab4ddk" podStartSLOduration=2.874866162 podStartE2EDuration="32.044381723s" totalImagesPullingTime="29.169515561s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:57:36 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:58:08.043960156 +0000 UTC m=+244.558388927" watchObservedRunningTime="2026-10-30 11:58:08.044381723 +0000 UTC m=+244.558810468" Oct 30 11:58:08.961702 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:08.961619 4908 generic.go:356] "Generic (PLEG): container finished" podID="aa0ab887-afc9-4b8b-a76f-f43606327180" containerID="390d820d216052e365fb1781ef5c1413de12dff834ed6c6549371177d1ef093c" exitCode=0 Oct 30 11:58:08.962128 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:08.961725 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w8g9t" event={"ID":"aa0ab887-afc9-4b8b-a76f-f43606327180","Type":"ContainerDied","Data":"390d820d216052e365fb1781ef5c1413de12dff834ed6c6549371177d1ef093c"} Oct 30 11:58:08.965044 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:08.965019 4908 generic.go:356] "Generic (PLEG): container finished" podID="338c3005-c916-496a-b7af-6ea1b829b0fd" containerID="e583bf842201988e3c47ffab63fcc91c08892c96593fd91e57b06c4143dad4c1" exitCode=0 Oct 30 11:58:08.965181 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:08.965112 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czcpfc" event={"ID":"338c3005-c916-496a-b7af-6ea1b829b0fd","Type":"ContainerDied","Data":"e583bf842201988e3c47ffab63fcc91c08892c96593fd91e57b06c4143dad4c1"} Oct 30 11:58:08.967739 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:08.967715 4908 generic.go:356] "Generic (PLEG): container finished" podID="bf5b2c54-0d4a-401f-b012-b3438ca0e40d" containerID="9b9351d70a20325ab32bd9d443efca5846e2e2c59bb16a66b9d1a0a9021db7b1" exitCode=0 Oct 30 11:58:08.967885 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:08.967752 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ab4ddk" event={"ID":"bf5b2c54-0d4a-401f-b012-b3438ca0e40d","Type":"ContainerDied","Data":"9b9351d70a20325ab32bd9d443efca5846e2e2c59bb16a66b9d1a0a9021db7b1"} Oct 30 11:58:08.970153 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:08.970128 4908 generic.go:356] "Generic (PLEG): container finished" podID="2d7e3b95-ffe2-497f-baf4-86d71d7f3abe" containerID="4baab75202a98be929fd9fb3c64d92095ca0684d6ad9ab9035d5ac932e4b1aae" exitCode=0 Oct 30 11:58:08.970255 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:08.970209 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644clgth" event={"ID":"2d7e3b95-ffe2-497f-baf4-86d71d7f3abe","Type":"ContainerDied","Data":"4baab75202a98be929fd9fb3c64d92095ca0684d6ad9ab9035d5ac932e4b1aae"} Oct 30 11:58:09.118767 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:09.118744 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69rqj8" Oct 30 11:58:09.208113 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:09.208082 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ced4bb34-b6ca-4e9a-990b-2aa6d0ee3843-bundle\" (UniqueName: \"kubernetes.io/empty-dir/ced4bb34-b6ca-4e9a-990b-2aa6d0ee3843-bundle\") pod \"ced4bb34-b6ca-4e9a-990b-2aa6d0ee3843\" (UID: \"ced4bb34-b6ca-4e9a-990b-2aa6d0ee3843\") " Oct 30 11:58:09.208288 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:09.208136 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ced4bb34-b6ca-4e9a-990b-2aa6d0ee3843-kube-api-access-stmzr\" (UniqueName: \"kubernetes.io/projected/ced4bb34-b6ca-4e9a-990b-2aa6d0ee3843-kube-api-access-stmzr\") pod \"ced4bb34-b6ca-4e9a-990b-2aa6d0ee3843\" (UID: \"ced4bb34-b6ca-4e9a-990b-2aa6d0ee3843\") " Oct 30 11:58:09.208288 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:09.208169 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ced4bb34-b6ca-4e9a-990b-2aa6d0ee3843-util\" (UniqueName: \"kubernetes.io/empty-dir/ced4bb34-b6ca-4e9a-990b-2aa6d0ee3843-util\") pod \"ced4bb34-b6ca-4e9a-990b-2aa6d0ee3843\" (UID: \"ced4bb34-b6ca-4e9a-990b-2aa6d0ee3843\") " Oct 30 11:58:09.208587 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:09.208556 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ced4bb34-b6ca-4e9a-990b-2aa6d0ee3843-bundle" pod "ced4bb34-b6ca-4e9a-990b-2aa6d0ee3843" (UID: "ced4bb34-b6ca-4e9a-990b-2aa6d0ee3843"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:58:09.210218 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:09.210196 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ced4bb34-b6ca-4e9a-990b-2aa6d0ee3843-kube-api-access-stmzr" pod "ced4bb34-b6ca-4e9a-990b-2aa6d0ee3843" (UID: "ced4bb34-b6ca-4e9a-990b-2aa6d0ee3843"). InnerVolumeSpecName "kube-api-access-stmzr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:58:09.213185 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:09.213136 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ced4bb34-b6ca-4e9a-990b-2aa6d0ee3843-util" pod "ced4bb34-b6ca-4e9a-990b-2aa6d0ee3843" (UID: "ced4bb34-b6ca-4e9a-990b-2aa6d0ee3843"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:58:09.308732 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:09.308713 4908 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ced4bb34-b6ca-4e9a-990b-2aa6d0ee3843-bundle\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 11:58:09.308801 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:09.308735 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-stmzr\" (UniqueName: \"kubernetes.io/projected/ced4bb34-b6ca-4e9a-990b-2aa6d0ee3843-kube-api-access-stmzr\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 11:58:09.308801 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:09.308745 4908 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ced4bb34-b6ca-4e9a-990b-2aa6d0ee3843-util\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 11:58:09.976899 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:09.976871 4908 generic.go:356] "Generic (PLEG): container finished" podID="5eb05b86-ebf0-487d-b535-43f910fa5ff1" containerID="8fa9b7e37c534c2e624bb253767d07d0685f209d571bde49686f9ded426be554" exitCode=0 Oct 30 11:58:09.977269 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:09.976961 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7155tk9" event={"ID":"5eb05b86-ebf0-487d-b535-43f910fa5ff1","Type":"ContainerDied","Data":"8fa9b7e37c534c2e624bb253767d07d0685f209d571bde49686f9ded426be554"} Oct 30 11:58:09.979213 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:09.979185 4908 generic.go:356] "Generic (PLEG): container finished" podID="b4b41920-9ac9-412b-9afb-6c1d1ac23595" containerID="7ad76e3336f8e16f9c2c20bbec857ff1ddf6031a939b8e0dc78c9a58e6e5cf9d" exitCode=0 Oct 30 11:58:09.979325 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:09.979268 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvkfpf" event={"ID":"b4b41920-9ac9-412b-9afb-6c1d1ac23595","Type":"ContainerDied","Data":"7ad76e3336f8e16f9c2c20bbec857ff1ddf6031a939b8e0dc78c9a58e6e5cf9d"} Oct 30 11:58:09.981315 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:09.981298 4908 generic.go:356] "Generic (PLEG): container finished" podID="6d7843ed-9ada-42f3-954e-170c578d1115" containerID="12259e775cd1c5374bf1b0b1a74789cd830129aaacfa2daff4942fc02949839e" exitCode=0 Oct 30 11:58:09.981392 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:09.981373 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvnsnt" event={"ID":"6d7843ed-9ada-42f3-954e-170c578d1115","Type":"ContainerDied","Data":"12259e775cd1c5374bf1b0b1a74789cd830129aaacfa2daff4942fc02949839e"} Oct 30 11:58:09.983295 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:09.983275 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69rqj8" Oct 30 11:58:09.983380 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:09.983311 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69rqj8" event={"ID":"ced4bb34-b6ca-4e9a-990b-2aa6d0ee3843","Type":"ContainerDied","Data":"63319d181b29ec1a7a8e471f64b8cfc90e845be3ccbf47d7c75526d5c60dc469"} Oct 30 11:58:09.983380 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:09.983344 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="63319d181b29ec1a7a8e471f64b8cfc90e845be3ccbf47d7c75526d5c60dc469" Oct 30 11:58:10.130478 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:10.130449 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w8g9t" Oct 30 11:58:10.205819 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:10.205796 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644clgth" Oct 30 11:58:10.208547 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:10.208529 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ab4ddk" Oct 30 11:58:10.211758 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:10.211739 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czcpfc" Oct 30 11:58:10.215079 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:10.215059 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa0ab887-afc9-4b8b-a76f-f43606327180-bundle\" (UniqueName: \"kubernetes.io/empty-dir/aa0ab887-afc9-4b8b-a76f-f43606327180-bundle\") pod \"aa0ab887-afc9-4b8b-a76f-f43606327180\" (UID: \"aa0ab887-afc9-4b8b-a76f-f43606327180\") " Oct 30 11:58:10.215207 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:10.215189 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa0ab887-afc9-4b8b-a76f-f43606327180-kube-api-access-tbfht\" (UniqueName: \"kubernetes.io/projected/aa0ab887-afc9-4b8b-a76f-f43606327180-kube-api-access-tbfht\") pod \"aa0ab887-afc9-4b8b-a76f-f43606327180\" (UID: \"aa0ab887-afc9-4b8b-a76f-f43606327180\") " Oct 30 11:58:10.215244 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:10.215231 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa0ab887-afc9-4b8b-a76f-f43606327180-util\" (UniqueName: \"kubernetes.io/empty-dir/aa0ab887-afc9-4b8b-a76f-f43606327180-util\") pod \"aa0ab887-afc9-4b8b-a76f-f43606327180\" (UID: \"aa0ab887-afc9-4b8b-a76f-f43606327180\") " Oct 30 11:58:10.215657 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:10.215633 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa0ab887-afc9-4b8b-a76f-f43606327180-bundle" pod "aa0ab887-afc9-4b8b-a76f-f43606327180" (UID: "aa0ab887-afc9-4b8b-a76f-f43606327180"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:58:10.217128 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:10.217108 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa0ab887-afc9-4b8b-a76f-f43606327180-kube-api-access-tbfht" pod "aa0ab887-afc9-4b8b-a76f-f43606327180" (UID: "aa0ab887-afc9-4b8b-a76f-f43606327180"). InnerVolumeSpecName "kube-api-access-tbfht". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:58:10.222916 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:10.222886 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa0ab887-afc9-4b8b-a76f-f43606327180-util" pod "aa0ab887-afc9-4b8b-a76f-f43606327180" (UID: "aa0ab887-afc9-4b8b-a76f-f43606327180"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:58:10.315947 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:10.315889 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/338c3005-c916-496a-b7af-6ea1b829b0fd-bundle\" (UniqueName: \"kubernetes.io/empty-dir/338c3005-c916-496a-b7af-6ea1b829b0fd-bundle\") pod \"338c3005-c916-496a-b7af-6ea1b829b0fd\" (UID: \"338c3005-c916-496a-b7af-6ea1b829b0fd\") " Oct 30 11:58:10.315947 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:10.315932 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/338c3005-c916-496a-b7af-6ea1b829b0fd-kube-api-access-p7cwb\" (UniqueName: \"kubernetes.io/projected/338c3005-c916-496a-b7af-6ea1b829b0fd-kube-api-access-p7cwb\") pod \"338c3005-c916-496a-b7af-6ea1b829b0fd\" (UID: \"338c3005-c916-496a-b7af-6ea1b829b0fd\") " Oct 30 11:58:10.316079 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:10.315951 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d7e3b95-ffe2-497f-baf4-86d71d7f3abe-bundle\" (UniqueName: \"kubernetes.io/empty-dir/2d7e3b95-ffe2-497f-baf4-86d71d7f3abe-bundle\") pod \"2d7e3b95-ffe2-497f-baf4-86d71d7f3abe\" (UID: \"2d7e3b95-ffe2-497f-baf4-86d71d7f3abe\") " Oct 30 11:58:10.316079 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:10.315979 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf5b2c54-0d4a-401f-b012-b3438ca0e40d-bundle\" (UniqueName: \"kubernetes.io/empty-dir/bf5b2c54-0d4a-401f-b012-b3438ca0e40d-bundle\") pod \"bf5b2c54-0d4a-401f-b012-b3438ca0e40d\" (UID: \"bf5b2c54-0d4a-401f-b012-b3438ca0e40d\") " Oct 30 11:58:10.316189 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:10.316128 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d7e3b95-ffe2-497f-baf4-86d71d7f3abe-util\" (UniqueName: \"kubernetes.io/empty-dir/2d7e3b95-ffe2-497f-baf4-86d71d7f3abe-util\") pod \"2d7e3b95-ffe2-497f-baf4-86d71d7f3abe\" (UID: \"2d7e3b95-ffe2-497f-baf4-86d71d7f3abe\") " Oct 30 11:58:10.316189 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:10.316180 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/338c3005-c916-496a-b7af-6ea1b829b0fd-util\" (UniqueName: \"kubernetes.io/empty-dir/338c3005-c916-496a-b7af-6ea1b829b0fd-util\") pod \"338c3005-c916-496a-b7af-6ea1b829b0fd\" (UID: \"338c3005-c916-496a-b7af-6ea1b829b0fd\") " Oct 30 11:58:10.316282 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:10.316249 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d7e3b95-ffe2-497f-baf4-86d71d7f3abe-kube-api-access-fmccn\" (UniqueName: \"kubernetes.io/projected/2d7e3b95-ffe2-497f-baf4-86d71d7f3abe-kube-api-access-fmccn\") pod \"2d7e3b95-ffe2-497f-baf4-86d71d7f3abe\" (UID: \"2d7e3b95-ffe2-497f-baf4-86d71d7f3abe\") " Oct 30 11:58:10.316335 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:10.316308 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf5b2c54-0d4a-401f-b012-b3438ca0e40d-kube-api-access-l8vzc\" (UniqueName: \"kubernetes.io/projected/bf5b2c54-0d4a-401f-b012-b3438ca0e40d-kube-api-access-l8vzc\") pod \"bf5b2c54-0d4a-401f-b012-b3438ca0e40d\" (UID: \"bf5b2c54-0d4a-401f-b012-b3438ca0e40d\") " Oct 30 11:58:10.316391 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:10.316326 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d7e3b95-ffe2-497f-baf4-86d71d7f3abe-bundle" pod "2d7e3b95-ffe2-497f-baf4-86d71d7f3abe" (UID: "2d7e3b95-ffe2-497f-baf4-86d71d7f3abe"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:58:10.316391 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:10.316340 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf5b2c54-0d4a-401f-b012-b3438ca0e40d-util\" (UniqueName: \"kubernetes.io/empty-dir/bf5b2c54-0d4a-401f-b012-b3438ca0e40d-util\") pod \"bf5b2c54-0d4a-401f-b012-b3438ca0e40d\" (UID: \"bf5b2c54-0d4a-401f-b012-b3438ca0e40d\") " Oct 30 11:58:10.316581 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:10.316564 4908 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aa0ab887-afc9-4b8b-a76f-f43606327180-util\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 11:58:10.316642 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:10.316587 4908 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aa0ab887-afc9-4b8b-a76f-f43606327180-bundle\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 11:58:10.316642 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:10.316599 4908 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2d7e3b95-ffe2-497f-baf4-86d71d7f3abe-bundle\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 11:58:10.316642 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:10.316614 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbfht\" (UniqueName: \"kubernetes.io/projected/aa0ab887-afc9-4b8b-a76f-f43606327180-kube-api-access-tbfht\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 11:58:10.316791 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:10.316725 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/338c3005-c916-496a-b7af-6ea1b829b0fd-bundle" pod "338c3005-c916-496a-b7af-6ea1b829b0fd" (UID: "338c3005-c916-496a-b7af-6ea1b829b0fd"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:58:10.317202 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:10.317163 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf5b2c54-0d4a-401f-b012-b3438ca0e40d-bundle" pod "bf5b2c54-0d4a-401f-b012-b3438ca0e40d" (UID: "bf5b2c54-0d4a-401f-b012-b3438ca0e40d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:58:10.318253 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:10.318232 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/338c3005-c916-496a-b7af-6ea1b829b0fd-kube-api-access-p7cwb" pod "338c3005-c916-496a-b7af-6ea1b829b0fd" (UID: "338c3005-c916-496a-b7af-6ea1b829b0fd"). InnerVolumeSpecName "kube-api-access-p7cwb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:58:10.318487 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:10.318471 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d7e3b95-ffe2-497f-baf4-86d71d7f3abe-kube-api-access-fmccn" pod "2d7e3b95-ffe2-497f-baf4-86d71d7f3abe" (UID: "2d7e3b95-ffe2-497f-baf4-86d71d7f3abe"). InnerVolumeSpecName "kube-api-access-fmccn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:58:10.318811 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:10.318785 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf5b2c54-0d4a-401f-b012-b3438ca0e40d-kube-api-access-l8vzc" pod "bf5b2c54-0d4a-401f-b012-b3438ca0e40d" (UID: "bf5b2c54-0d4a-401f-b012-b3438ca0e40d"). InnerVolumeSpecName "kube-api-access-l8vzc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:58:10.324367 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:10.324343 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf5b2c54-0d4a-401f-b012-b3438ca0e40d-util" pod "bf5b2c54-0d4a-401f-b012-b3438ca0e40d" (UID: "bf5b2c54-0d4a-401f-b012-b3438ca0e40d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:58:10.324975 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:10.324950 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d7e3b95-ffe2-497f-baf4-86d71d7f3abe-util" pod "2d7e3b95-ffe2-497f-baf4-86d71d7f3abe" (UID: "2d7e3b95-ffe2-497f-baf4-86d71d7f3abe"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:58:10.325277 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:10.325258 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/338c3005-c916-496a-b7af-6ea1b829b0fd-util" pod "338c3005-c916-496a-b7af-6ea1b829b0fd" (UID: "338c3005-c916-496a-b7af-6ea1b829b0fd"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:58:10.417975 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:10.417951 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmccn\" (UniqueName: \"kubernetes.io/projected/2d7e3b95-ffe2-497f-baf4-86d71d7f3abe-kube-api-access-fmccn\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 11:58:10.417975 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:10.417972 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8vzc\" (UniqueName: \"kubernetes.io/projected/bf5b2c54-0d4a-401f-b012-b3438ca0e40d-kube-api-access-l8vzc\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 11:58:10.418090 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:10.417981 4908 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bf5b2c54-0d4a-401f-b012-b3438ca0e40d-util\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 11:58:10.418090 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:10.417991 4908 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/338c3005-c916-496a-b7af-6ea1b829b0fd-bundle\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 11:58:10.418090 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:10.417999 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p7cwb\" (UniqueName: \"kubernetes.io/projected/338c3005-c916-496a-b7af-6ea1b829b0fd-kube-api-access-p7cwb\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 11:58:10.418090 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:10.418008 4908 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bf5b2c54-0d4a-401f-b012-b3438ca0e40d-bundle\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 11:58:10.418090 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:10.418016 4908 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2d7e3b95-ffe2-497f-baf4-86d71d7f3abe-util\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 11:58:10.418090 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:10.418024 4908 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/338c3005-c916-496a-b7af-6ea1b829b0fd-util\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 11:58:10.989613 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:10.989588 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w8g9t" Oct 30 11:58:10.990025 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:10.989590 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w8g9t" event={"ID":"aa0ab887-afc9-4b8b-a76f-f43606327180","Type":"ContainerDied","Data":"5b46ee25438148c64af80534168a3bcbf21fd086c60a56e373e3f717778e625e"} Oct 30 11:58:10.990025 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:10.989697 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5b46ee25438148c64af80534168a3bcbf21fd086c60a56e373e3f717778e625e" Oct 30 11:58:10.991616 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:10.991588 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czcpfc" Oct 30 11:58:10.991745 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:10.991612 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czcpfc" event={"ID":"338c3005-c916-496a-b7af-6ea1b829b0fd","Type":"ContainerDied","Data":"516126b009cf777057ba3a054881f8a6c292bd78cdba1625cce1083cf49ef308"} Oct 30 11:58:10.991745 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:10.991657 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="516126b009cf777057ba3a054881f8a6c292bd78cdba1625cce1083cf49ef308" Oct 30 11:58:10.993696 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:10.993670 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ab4ddk" event={"ID":"bf5b2c54-0d4a-401f-b012-b3438ca0e40d","Type":"ContainerDied","Data":"b37657b5c0659914e9fe5373cd3040d3fac60108764c80e1f60a258a05c5dea3"} Oct 30 11:58:10.993696 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:10.993688 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ab4ddk" Oct 30 11:58:10.993852 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:10.993702 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b37657b5c0659914e9fe5373cd3040d3fac60108764c80e1f60a258a05c5dea3" Oct 30 11:58:10.995742 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:10.995674 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644clgth" Oct 30 11:58:10.995742 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:10.995705 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644clgth" event={"ID":"2d7e3b95-ffe2-497f-baf4-86d71d7f3abe","Type":"ContainerDied","Data":"6597b86a4f7ae8b4f84349f9c41969ab61e35dc2baacdb403fefe9abb209a020"} Oct 30 11:58:10.995742 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:10.995738 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6597b86a4f7ae8b4f84349f9c41969ab61e35dc2baacdb403fefe9abb209a020" Oct 30 11:58:11.132906 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:11.132867 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvnsnt" Oct 30 11:58:11.136531 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:11.136512 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7155tk9" Oct 30 11:58:11.176918 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:11.176896 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvkfpf" Oct 30 11:58:11.224402 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:11.224381 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d7843ed-9ada-42f3-954e-170c578d1115-kube-api-access-224jb\" (UniqueName: \"kubernetes.io/projected/6d7843ed-9ada-42f3-954e-170c578d1115-kube-api-access-224jb\") pod \"6d7843ed-9ada-42f3-954e-170c578d1115\" (UID: \"6d7843ed-9ada-42f3-954e-170c578d1115\") " Oct 30 11:58:11.224546 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:11.224424 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d7843ed-9ada-42f3-954e-170c578d1115-bundle\" (UniqueName: \"kubernetes.io/empty-dir/6d7843ed-9ada-42f3-954e-170c578d1115-bundle\") pod \"6d7843ed-9ada-42f3-954e-170c578d1115\" (UID: \"6d7843ed-9ada-42f3-954e-170c578d1115\") " Oct 30 11:58:11.224546 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:11.224478 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5eb05b86-ebf0-487d-b535-43f910fa5ff1-kube-api-access-r4ltg\" (UniqueName: \"kubernetes.io/projected/5eb05b86-ebf0-487d-b535-43f910fa5ff1-kube-api-access-r4ltg\") pod \"5eb05b86-ebf0-487d-b535-43f910fa5ff1\" (UID: \"5eb05b86-ebf0-487d-b535-43f910fa5ff1\") " Oct 30 11:58:11.224546 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:11.224502 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5eb05b86-ebf0-487d-b535-43f910fa5ff1-util\" (UniqueName: \"kubernetes.io/empty-dir/5eb05b86-ebf0-487d-b535-43f910fa5ff1-util\") pod \"5eb05b86-ebf0-487d-b535-43f910fa5ff1\" (UID: \"5eb05b86-ebf0-487d-b535-43f910fa5ff1\") " Oct 30 11:58:11.224546 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:11.224520 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d7843ed-9ada-42f3-954e-170c578d1115-util\" (UniqueName: \"kubernetes.io/empty-dir/6d7843ed-9ada-42f3-954e-170c578d1115-util\") pod \"6d7843ed-9ada-42f3-954e-170c578d1115\" (UID: \"6d7843ed-9ada-42f3-954e-170c578d1115\") " Oct 30 11:58:11.224754 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:11.224557 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5eb05b86-ebf0-487d-b535-43f910fa5ff1-bundle\" (UniqueName: \"kubernetes.io/empty-dir/5eb05b86-ebf0-487d-b535-43f910fa5ff1-bundle\") pod \"5eb05b86-ebf0-487d-b535-43f910fa5ff1\" (UID: \"5eb05b86-ebf0-487d-b535-43f910fa5ff1\") " Oct 30 11:58:11.225283 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:11.225255 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d7843ed-9ada-42f3-954e-170c578d1115-bundle" pod "6d7843ed-9ada-42f3-954e-170c578d1115" (UID: "6d7843ed-9ada-42f3-954e-170c578d1115"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:58:11.225388 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:11.225322 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5eb05b86-ebf0-487d-b535-43f910fa5ff1-bundle" pod "5eb05b86-ebf0-487d-b535-43f910fa5ff1" (UID: "5eb05b86-ebf0-487d-b535-43f910fa5ff1"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:58:11.226610 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:11.226584 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d7843ed-9ada-42f3-954e-170c578d1115-kube-api-access-224jb" pod "6d7843ed-9ada-42f3-954e-170c578d1115" (UID: "6d7843ed-9ada-42f3-954e-170c578d1115"). InnerVolumeSpecName "kube-api-access-224jb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:58:11.228230 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:11.228203 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5eb05b86-ebf0-487d-b535-43f910fa5ff1-kube-api-access-r4ltg" pod "5eb05b86-ebf0-487d-b535-43f910fa5ff1" (UID: "5eb05b86-ebf0-487d-b535-43f910fa5ff1"). InnerVolumeSpecName "kube-api-access-r4ltg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:58:11.231185 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:11.231163 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d7843ed-9ada-42f3-954e-170c578d1115-util" pod "6d7843ed-9ada-42f3-954e-170c578d1115" (UID: "6d7843ed-9ada-42f3-954e-170c578d1115"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:58:11.232824 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:11.232799 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5eb05b86-ebf0-487d-b535-43f910fa5ff1-util" pod "5eb05b86-ebf0-487d-b535-43f910fa5ff1" (UID: "5eb05b86-ebf0-487d-b535-43f910fa5ff1"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:58:11.325490 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:11.325427 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4b41920-9ac9-412b-9afb-6c1d1ac23595-bundle\" (UniqueName: \"kubernetes.io/empty-dir/b4b41920-9ac9-412b-9afb-6c1d1ac23595-bundle\") pod \"b4b41920-9ac9-412b-9afb-6c1d1ac23595\" (UID: \"b4b41920-9ac9-412b-9afb-6c1d1ac23595\") " Oct 30 11:58:11.325490 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:11.325477 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4b41920-9ac9-412b-9afb-6c1d1ac23595-util\" (UniqueName: \"kubernetes.io/empty-dir/b4b41920-9ac9-412b-9afb-6c1d1ac23595-util\") pod \"b4b41920-9ac9-412b-9afb-6c1d1ac23595\" (UID: \"b4b41920-9ac9-412b-9afb-6c1d1ac23595\") " Oct 30 11:58:11.325649 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:11.325518 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4b41920-9ac9-412b-9afb-6c1d1ac23595-kube-api-access-6r8tt\" (UniqueName: \"kubernetes.io/projected/b4b41920-9ac9-412b-9afb-6c1d1ac23595-kube-api-access-6r8tt\") pod \"b4b41920-9ac9-412b-9afb-6c1d1ac23595\" (UID: \"b4b41920-9ac9-412b-9afb-6c1d1ac23595\") " Oct 30 11:58:11.325742 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:11.325727 4908 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6d7843ed-9ada-42f3-954e-170c578d1115-bundle\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 11:58:11.325805 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:11.325748 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4ltg\" (UniqueName: \"kubernetes.io/projected/5eb05b86-ebf0-487d-b535-43f910fa5ff1-kube-api-access-r4ltg\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 11:58:11.325805 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:11.325763 4908 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5eb05b86-ebf0-487d-b535-43f910fa5ff1-util\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 11:58:11.325805 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:11.325776 4908 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6d7843ed-9ada-42f3-954e-170c578d1115-util\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 11:58:11.325805 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:11.325789 4908 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5eb05b86-ebf0-487d-b535-43f910fa5ff1-bundle\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 11:58:11.325805 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:11.325802 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-224jb\" (UniqueName: \"kubernetes.io/projected/6d7843ed-9ada-42f3-954e-170c578d1115-kube-api-access-224jb\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 11:58:11.326057 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:11.326035 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4b41920-9ac9-412b-9afb-6c1d1ac23595-bundle" pod "b4b41920-9ac9-412b-9afb-6c1d1ac23595" (UID: "b4b41920-9ac9-412b-9afb-6c1d1ac23595"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:58:11.327390 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:11.327369 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4b41920-9ac9-412b-9afb-6c1d1ac23595-kube-api-access-6r8tt" pod "b4b41920-9ac9-412b-9afb-6c1d1ac23595" (UID: "b4b41920-9ac9-412b-9afb-6c1d1ac23595"). InnerVolumeSpecName "kube-api-access-6r8tt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:58:11.332590 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:11.332564 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4b41920-9ac9-412b-9afb-6c1d1ac23595-util" pod "b4b41920-9ac9-412b-9afb-6c1d1ac23595" (UID: "b4b41920-9ac9-412b-9afb-6c1d1ac23595"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:58:11.426234 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:11.426211 4908 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b4b41920-9ac9-412b-9afb-6c1d1ac23595-bundle\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 11:58:11.426234 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:11.426231 4908 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b4b41920-9ac9-412b-9afb-6c1d1ac23595-util\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 11:58:11.426360 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:11.426240 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6r8tt\" (UniqueName: \"kubernetes.io/projected/b4b41920-9ac9-412b-9afb-6c1d1ac23595-kube-api-access-6r8tt\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 11:58:12.002361 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:12.002331 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7155tk9" event={"ID":"5eb05b86-ebf0-487d-b535-43f910fa5ff1","Type":"ContainerDied","Data":"271f244be9d1e68f0ec194362fa0850feb47abe6f6c65725eb03a88254d81b62"} Oct 30 11:58:12.002784 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:12.002388 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="271f244be9d1e68f0ec194362fa0850feb47abe6f6c65725eb03a88254d81b62" Oct 30 11:58:12.002784 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:12.002339 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7155tk9" Oct 30 11:58:12.004440 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:12.004418 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvkfpf" Oct 30 11:58:12.004595 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:12.004460 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvkfpf" event={"ID":"b4b41920-9ac9-412b-9afb-6c1d1ac23595","Type":"ContainerDied","Data":"a3fcd5723d53461f9e01fe859e77f2642336d829619b3872a448da9057b00a69"} Oct 30 11:58:12.004595 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:12.004501 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a3fcd5723d53461f9e01fe859e77f2642336d829619b3872a448da9057b00a69" Oct 30 11:58:12.006608 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:12.006586 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvnsnt" event={"ID":"6d7843ed-9ada-42f3-954e-170c578d1115","Type":"ContainerDied","Data":"548045554b901c67556835cf8f4a99933954aea99349bdca23c491cb97b588ac"} Oct 30 11:58:12.006704 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:12.006625 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="548045554b901c67556835cf8f4a99933954aea99349bdca23c491cb97b588ac" Oct 30 11:58:12.006704 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:12.006635 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvnsnt" Oct 30 11:58:17.895891 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:17.895855 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-tqzld"] Oct 30 11:58:17.896274 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:17.896153 4908 cpu_manager.go:414] "containerMap: removing container" podUID="bf5b2c54-0d4a-401f-b012-b3438ca0e40d" containerName="pull" Oct 30 11:58:17.896274 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:17.896165 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf5b2c54-0d4a-401f-b012-b3438ca0e40d" containerName="pull" Oct 30 11:58:17.896274 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:17.896175 4908 cpu_manager.go:414] "containerMap: removing container" podUID="ced4bb34-b6ca-4e9a-990b-2aa6d0ee3843" containerName="extract" Oct 30 11:58:17.896274 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:17.896180 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ced4bb34-b6ca-4e9a-990b-2aa6d0ee3843" containerName="extract" Oct 30 11:58:17.896274 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:17.896188 4908 cpu_manager.go:414] "containerMap: removing container" podUID="b4b41920-9ac9-412b-9afb-6c1d1ac23595" containerName="extract" Oct 30 11:58:17.896274 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:17.896193 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4b41920-9ac9-412b-9afb-6c1d1ac23595" containerName="extract" Oct 30 11:58:17.896274 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:17.896204 4908 cpu_manager.go:414] "containerMap: removing container" podUID="ced4bb34-b6ca-4e9a-990b-2aa6d0ee3843" containerName="util" Oct 30 11:58:17.896274 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:17.896208 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ced4bb34-b6ca-4e9a-990b-2aa6d0ee3843" containerName="util" Oct 30 11:58:17.896274 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:17.896214 4908 cpu_manager.go:414] "containerMap: removing container" podUID="aa0ab887-afc9-4b8b-a76f-f43606327180" containerName="pull" Oct 30 11:58:17.896274 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:17.896219 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa0ab887-afc9-4b8b-a76f-f43606327180" containerName="pull" Oct 30 11:58:17.896274 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:17.896226 4908 cpu_manager.go:414] "containerMap: removing container" podUID="b4b41920-9ac9-412b-9afb-6c1d1ac23595" containerName="util" Oct 30 11:58:17.896274 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:17.896231 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4b41920-9ac9-412b-9afb-6c1d1ac23595" containerName="util" Oct 30 11:58:17.896274 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:17.896237 4908 cpu_manager.go:414] "containerMap: removing container" podUID="2d7e3b95-ffe2-497f-baf4-86d71d7f3abe" containerName="extract" Oct 30 11:58:17.896274 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:17.896241 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d7e3b95-ffe2-497f-baf4-86d71d7f3abe" containerName="extract" Oct 30 11:58:17.896274 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:17.896251 4908 cpu_manager.go:414] "containerMap: removing container" podUID="2d7e3b95-ffe2-497f-baf4-86d71d7f3abe" containerName="pull" Oct 30 11:58:17.896274 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:17.896255 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d7e3b95-ffe2-497f-baf4-86d71d7f3abe" containerName="pull" Oct 30 11:58:17.896274 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:17.896263 4908 cpu_manager.go:414] "containerMap: removing container" podUID="ced4bb34-b6ca-4e9a-990b-2aa6d0ee3843" containerName="pull" Oct 30 11:58:17.896274 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:17.896267 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ced4bb34-b6ca-4e9a-990b-2aa6d0ee3843" containerName="pull" Oct 30 11:58:17.896274 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:17.896274 4908 cpu_manager.go:414] "containerMap: removing container" podUID="5eb05b86-ebf0-487d-b535-43f910fa5ff1" containerName="pull" Oct 30 11:58:17.896274 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:17.896278 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5eb05b86-ebf0-487d-b535-43f910fa5ff1" containerName="pull" Oct 30 11:58:17.896274 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:17.896285 4908 cpu_manager.go:414] "containerMap: removing container" podUID="5eb05b86-ebf0-487d-b535-43f910fa5ff1" containerName="extract" Oct 30 11:58:17.896274 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:17.896289 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5eb05b86-ebf0-487d-b535-43f910fa5ff1" containerName="extract" Oct 30 11:58:17.896939 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:17.896297 4908 cpu_manager.go:414] "containerMap: removing container" podUID="338c3005-c916-496a-b7af-6ea1b829b0fd" containerName="pull" Oct 30 11:58:17.896939 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:17.896302 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="338c3005-c916-496a-b7af-6ea1b829b0fd" containerName="pull" Oct 30 11:58:17.896939 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:17.896308 4908 cpu_manager.go:414] "containerMap: removing container" podUID="6d7843ed-9ada-42f3-954e-170c578d1115" containerName="extract" Oct 30 11:58:17.896939 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:17.896313 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d7843ed-9ada-42f3-954e-170c578d1115" containerName="extract" Oct 30 11:58:17.896939 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:17.896320 4908 cpu_manager.go:414] "containerMap: removing container" podUID="6d7843ed-9ada-42f3-954e-170c578d1115" containerName="util" Oct 30 11:58:17.896939 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:17.896325 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d7843ed-9ada-42f3-954e-170c578d1115" containerName="util" Oct 30 11:58:17.896939 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:17.896331 4908 cpu_manager.go:414] "containerMap: removing container" podUID="bf5b2c54-0d4a-401f-b012-b3438ca0e40d" containerName="extract" Oct 30 11:58:17.896939 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:17.896336 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf5b2c54-0d4a-401f-b012-b3438ca0e40d" containerName="extract" Oct 30 11:58:17.896939 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:17.896344 4908 cpu_manager.go:414] "containerMap: removing container" podUID="aa0ab887-afc9-4b8b-a76f-f43606327180" containerName="util" Oct 30 11:58:17.896939 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:17.896349 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa0ab887-afc9-4b8b-a76f-f43606327180" containerName="util" Oct 30 11:58:17.896939 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:17.896357 4908 cpu_manager.go:414] "containerMap: removing container" podUID="baabd525-2d60-468e-a946-61cdb004deef" containerName="pull" Oct 30 11:58:17.896939 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:17.896361 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="baabd525-2d60-468e-a946-61cdb004deef" containerName="pull" Oct 30 11:58:17.896939 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:17.896366 4908 cpu_manager.go:414] "containerMap: removing container" podUID="aa0ab887-afc9-4b8b-a76f-f43606327180" containerName="extract" Oct 30 11:58:17.896939 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:17.896371 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa0ab887-afc9-4b8b-a76f-f43606327180" containerName="extract" Oct 30 11:58:17.896939 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:17.896377 4908 cpu_manager.go:414] "containerMap: removing container" podUID="338c3005-c916-496a-b7af-6ea1b829b0fd" containerName="util" Oct 30 11:58:17.896939 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:17.896381 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="338c3005-c916-496a-b7af-6ea1b829b0fd" containerName="util" Oct 30 11:58:17.896939 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:17.896386 4908 cpu_manager.go:414] "containerMap: removing container" podUID="2d7e3b95-ffe2-497f-baf4-86d71d7f3abe" containerName="util" Oct 30 11:58:17.896939 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:17.896390 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d7e3b95-ffe2-497f-baf4-86d71d7f3abe" containerName="util" Oct 30 11:58:17.896939 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:17.896395 4908 cpu_manager.go:414] "containerMap: removing container" podUID="baabd525-2d60-468e-a946-61cdb004deef" containerName="extract" Oct 30 11:58:17.896939 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:17.896400 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="baabd525-2d60-468e-a946-61cdb004deef" containerName="extract" Oct 30 11:58:17.896939 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:17.896405 4908 cpu_manager.go:414] "containerMap: removing container" podUID="338c3005-c916-496a-b7af-6ea1b829b0fd" containerName="extract" Oct 30 11:58:17.896939 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:17.896409 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="338c3005-c916-496a-b7af-6ea1b829b0fd" containerName="extract" Oct 30 11:58:17.896939 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:17.896415 4908 cpu_manager.go:414] "containerMap: removing container" podUID="5eb05b86-ebf0-487d-b535-43f910fa5ff1" containerName="util" Oct 30 11:58:17.896939 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:17.896420 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5eb05b86-ebf0-487d-b535-43f910fa5ff1" containerName="util" Oct 30 11:58:17.896939 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:17.896427 4908 cpu_manager.go:414] "containerMap: removing container" podUID="6d7843ed-9ada-42f3-954e-170c578d1115" containerName="pull" Oct 30 11:58:17.896939 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:17.896431 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d7843ed-9ada-42f3-954e-170c578d1115" containerName="pull" Oct 30 11:58:17.896939 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:17.896439 4908 cpu_manager.go:414] "containerMap: removing container" podUID="baabd525-2d60-468e-a946-61cdb004deef" containerName="util" Oct 30 11:58:17.896939 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:17.896443 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="baabd525-2d60-468e-a946-61cdb004deef" containerName="util" Oct 30 11:58:17.896939 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:17.896448 4908 cpu_manager.go:414] "containerMap: removing container" podUID="bf5b2c54-0d4a-401f-b012-b3438ca0e40d" containerName="util" Oct 30 11:58:17.896939 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:17.896452 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf5b2c54-0d4a-401f-b012-b3438ca0e40d" containerName="util" Oct 30 11:58:17.896939 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:17.896457 4908 cpu_manager.go:414] "containerMap: removing container" podUID="b4b41920-9ac9-412b-9afb-6c1d1ac23595" containerName="pull" Oct 30 11:58:17.896939 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:17.896461 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4b41920-9ac9-412b-9afb-6c1d1ac23595" containerName="pull" Oct 30 11:58:17.896939 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:17.896507 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d7e3b95-ffe2-497f-baf4-86d71d7f3abe" containerName="extract" Oct 30 11:58:17.896939 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:17.896517 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="5eb05b86-ebf0-487d-b535-43f910fa5ff1" containerName="extract" Oct 30 11:58:17.896939 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:17.896522 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d7843ed-9ada-42f3-954e-170c578d1115" containerName="extract" Oct 30 11:58:17.896939 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:17.896528 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf5b2c54-0d4a-401f-b012-b3438ca0e40d" containerName="extract" Oct 30 11:58:17.896939 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:17.896535 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa0ab887-afc9-4b8b-a76f-f43606327180" containerName="extract" Oct 30 11:58:17.896939 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:17.896540 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="baabd525-2d60-468e-a946-61cdb004deef" containerName="extract" Oct 30 11:58:17.896939 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:17.896544 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="338c3005-c916-496a-b7af-6ea1b829b0fd" containerName="extract" Oct 30 11:58:17.896939 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:17.896550 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4b41920-9ac9-412b-9afb-6c1d1ac23595" containerName="extract" Oct 30 11:58:17.896939 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:17.896554 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="ced4bb34-b6ca-4e9a-990b-2aa6d0ee3843" containerName="extract" Oct 30 11:58:17.899773 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:17.899754 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-tqzld" Oct 30 11:58:17.916202 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:17.916177 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-tqzld"] Oct 30 11:58:17.925231 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:17.925204 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Oct 30 11:58:17.925354 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:17.925243 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-dfdk8\"" Oct 30 11:58:17.925354 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:17.925249 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Oct 30 11:58:17.971713 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:17.971687 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/13a8af1c-f308-4eec-a543-f77f14ebfdf0-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-tqzld\" (UID: \"13a8af1c-f308-4eec-a543-f77f14ebfdf0\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-tqzld" Oct 30 11:58:17.971794 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:17.971720 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cdv4\" (UniqueName: \"kubernetes.io/projected/13a8af1c-f308-4eec-a543-f77f14ebfdf0-kube-api-access-9cdv4\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-tqzld\" (UID: \"13a8af1c-f308-4eec-a543-f77f14ebfdf0\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-tqzld" Oct 30 11:58:18.072870 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:18.072814 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/13a8af1c-f308-4eec-a543-f77f14ebfdf0-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-tqzld\" (UID: \"13a8af1c-f308-4eec-a543-f77f14ebfdf0\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-tqzld" Oct 30 11:58:18.072968 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:18.072879 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cdv4\" (UniqueName: \"kubernetes.io/projected/13a8af1c-f308-4eec-a543-f77f14ebfdf0-kube-api-access-9cdv4\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-tqzld\" (UID: \"13a8af1c-f308-4eec-a543-f77f14ebfdf0\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-tqzld" Oct 30 11:58:18.073161 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:18.073143 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/13a8af1c-f308-4eec-a543-f77f14ebfdf0-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-tqzld\" (UID: \"13a8af1c-f308-4eec-a543-f77f14ebfdf0\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-tqzld" Oct 30 11:58:18.098970 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:18.098947 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cdv4\" (UniqueName: \"kubernetes.io/projected/13a8af1c-f308-4eec-a543-f77f14ebfdf0-kube-api-access-9cdv4\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-tqzld\" (UID: \"13a8af1c-f308-4eec-a543-f77f14ebfdf0\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-tqzld" Oct 30 11:58:18.211766 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:18.211715 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-tqzld" Oct 30 11:58:18.537001 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:18.536974 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-tqzld"] Oct 30 11:58:18.558121 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:58:18.558067 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13a8af1c_f308_4eec_a543_f77f14ebfdf0.slice/crio-812b2bb428580e9d2a5f9a0939da5c55f94aaa6ad7250ad26d79506e775bce65 WatchSource:0}: Error finding container 812b2bb428580e9d2a5f9a0939da5c55f94aaa6ad7250ad26d79506e775bce65: Status 404 returned error can't find the container with id 812b2bb428580e9d2a5f9a0939da5c55f94aaa6ad7250ad26d79506e775bce65 Oct 30 11:58:19.039619 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:19.039584 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-tqzld" event={"ID":"13a8af1c-f308-4eec-a543-f77f14ebfdf0","Type":"ContainerStarted","Data":"812b2bb428580e9d2a5f9a0939da5c55f94aaa6ad7250ad26d79506e775bce65"} Oct 30 11:58:19.685605 ip-10-0-141-251 kubenswrapper[4908]: E1030 11:58:19.685572 4908 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vn6vv" containerName="extractor" Oct 30 11:58:19.825670 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:19.825638 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-64c9d5fd44-pmmwk"] Oct 30 11:58:19.829626 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:19.829600 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64c9d5fd44-pmmwk" Oct 30 11:58:19.849128 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:19.849105 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Oct 30 11:58:19.849128 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:19.849119 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Oct 30 11:58:19.849306 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:19.849286 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 30 11:58:19.849358 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:19.849342 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-b7n4r\"" Oct 30 11:58:19.860265 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:19.860236 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Oct 30 11:58:19.860383 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:19.860246 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Oct 30 11:58:19.863724 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:19.863697 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-64c9d5fd44-pmmwk"] Oct 30 11:58:19.988199 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:19.988166 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/17115a8e-8703-4a19-bc79-ddf4d1cc285b-manifests\") pod \"opendatahub-operator-controller-manager-64c9d5fd44-pmmwk\" (UID: \"17115a8e-8703-4a19-bc79-ddf4d1cc285b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64c9d5fd44-pmmwk" Oct 30 11:58:19.988345 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:19.988212 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgtvl\" (UniqueName: \"kubernetes.io/projected/17115a8e-8703-4a19-bc79-ddf4d1cc285b-kube-api-access-dgtvl\") pod \"opendatahub-operator-controller-manager-64c9d5fd44-pmmwk\" (UID: \"17115a8e-8703-4a19-bc79-ddf4d1cc285b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64c9d5fd44-pmmwk" Oct 30 11:58:19.988345 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:19.988317 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/17115a8e-8703-4a19-bc79-ddf4d1cc285b-webhook-cert\") pod \"opendatahub-operator-controller-manager-64c9d5fd44-pmmwk\" (UID: \"17115a8e-8703-4a19-bc79-ddf4d1cc285b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64c9d5fd44-pmmwk" Oct 30 11:58:19.988457 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:19.988361 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/17115a8e-8703-4a19-bc79-ddf4d1cc285b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-64c9d5fd44-pmmwk\" (UID: \"17115a8e-8703-4a19-bc79-ddf4d1cc285b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64c9d5fd44-pmmwk" Oct 30 11:58:19.988457 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:19.988427 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/17115a8e-8703-4a19-bc79-ddf4d1cc285b-metrics-cert\") pod \"opendatahub-operator-controller-manager-64c9d5fd44-pmmwk\" (UID: \"17115a8e-8703-4a19-bc79-ddf4d1cc285b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64c9d5fd44-pmmwk" Oct 30 11:58:19.988545 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:19.988526 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/17115a8e-8703-4a19-bc79-ddf4d1cc285b-tmp\") pod \"opendatahub-operator-controller-manager-64c9d5fd44-pmmwk\" (UID: \"17115a8e-8703-4a19-bc79-ddf4d1cc285b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64c9d5fd44-pmmwk" Oct 30 11:58:20.089552 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:20.089527 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/17115a8e-8703-4a19-bc79-ddf4d1cc285b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-64c9d5fd44-pmmwk\" (UID: \"17115a8e-8703-4a19-bc79-ddf4d1cc285b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64c9d5fd44-pmmwk" Oct 30 11:58:20.090012 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:20.089559 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/17115a8e-8703-4a19-bc79-ddf4d1cc285b-metrics-cert\") pod \"opendatahub-operator-controller-manager-64c9d5fd44-pmmwk\" (UID: \"17115a8e-8703-4a19-bc79-ddf4d1cc285b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64c9d5fd44-pmmwk" Oct 30 11:58:20.090012 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:20.089604 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/17115a8e-8703-4a19-bc79-ddf4d1cc285b-tmp\") pod \"opendatahub-operator-controller-manager-64c9d5fd44-pmmwk\" (UID: \"17115a8e-8703-4a19-bc79-ddf4d1cc285b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64c9d5fd44-pmmwk" Oct 30 11:58:20.090012 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:20.089632 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/17115a8e-8703-4a19-bc79-ddf4d1cc285b-manifests\") pod \"opendatahub-operator-controller-manager-64c9d5fd44-pmmwk\" (UID: \"17115a8e-8703-4a19-bc79-ddf4d1cc285b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64c9d5fd44-pmmwk" Oct 30 11:58:20.090012 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:20.089651 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgtvl\" (UniqueName: \"kubernetes.io/projected/17115a8e-8703-4a19-bc79-ddf4d1cc285b-kube-api-access-dgtvl\") pod \"opendatahub-operator-controller-manager-64c9d5fd44-pmmwk\" (UID: \"17115a8e-8703-4a19-bc79-ddf4d1cc285b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64c9d5fd44-pmmwk" Oct 30 11:58:20.090012 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:20.089674 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/17115a8e-8703-4a19-bc79-ddf4d1cc285b-webhook-cert\") pod \"opendatahub-operator-controller-manager-64c9d5fd44-pmmwk\" (UID: \"17115a8e-8703-4a19-bc79-ddf4d1cc285b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64c9d5fd44-pmmwk" Oct 30 11:58:20.090277 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:20.090063 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/17115a8e-8703-4a19-bc79-ddf4d1cc285b-manifests\") pod \"opendatahub-operator-controller-manager-64c9d5fd44-pmmwk\" (UID: \"17115a8e-8703-4a19-bc79-ddf4d1cc285b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64c9d5fd44-pmmwk" Oct 30 11:58:20.092421 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:20.092176 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/17115a8e-8703-4a19-bc79-ddf4d1cc285b-metrics-cert\") pod \"opendatahub-operator-controller-manager-64c9d5fd44-pmmwk\" (UID: \"17115a8e-8703-4a19-bc79-ddf4d1cc285b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64c9d5fd44-pmmwk" Oct 30 11:58:20.092421 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:20.092336 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/17115a8e-8703-4a19-bc79-ddf4d1cc285b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-64c9d5fd44-pmmwk\" (UID: \"17115a8e-8703-4a19-bc79-ddf4d1cc285b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64c9d5fd44-pmmwk" Oct 30 11:58:20.092556 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:20.092537 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/17115a8e-8703-4a19-bc79-ddf4d1cc285b-webhook-cert\") pod \"opendatahub-operator-controller-manager-64c9d5fd44-pmmwk\" (UID: \"17115a8e-8703-4a19-bc79-ddf4d1cc285b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64c9d5fd44-pmmwk" Oct 30 11:58:20.092597 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:20.092570 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/17115a8e-8703-4a19-bc79-ddf4d1cc285b-tmp\") pod \"opendatahub-operator-controller-manager-64c9d5fd44-pmmwk\" (UID: \"17115a8e-8703-4a19-bc79-ddf4d1cc285b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64c9d5fd44-pmmwk" Oct 30 11:58:20.111965 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:20.111945 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgtvl\" (UniqueName: \"kubernetes.io/projected/17115a8e-8703-4a19-bc79-ddf4d1cc285b-kube-api-access-dgtvl\") pod \"opendatahub-operator-controller-manager-64c9d5fd44-pmmwk\" (UID: \"17115a8e-8703-4a19-bc79-ddf4d1cc285b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64c9d5fd44-pmmwk" Oct 30 11:58:20.147003 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:20.146972 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64c9d5fd44-pmmwk" Oct 30 11:58:20.510713 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:20.510680 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-64c9d5fd44-pmmwk"] Oct 30 11:58:20.514299 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:58:20.514269 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17115a8e_8703_4a19_bc79_ddf4d1cc285b.slice/crio-c41d1e554cce7c903d18ea650a49d6cc98e4f1e3a2b5b78a45bd55aa77ab1b47 WatchSource:0}: Error finding container c41d1e554cce7c903d18ea650a49d6cc98e4f1e3a2b5b78a45bd55aa77ab1b47: Status 404 returned error can't find the container with id c41d1e554cce7c903d18ea650a49d6cc98e4f1e3a2b5b78a45bd55aa77ab1b47 Oct 30 11:58:21.051622 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:21.051585 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64c9d5fd44-pmmwk" event={"ID":"17115a8e-8703-4a19-bc79-ddf4d1cc285b","Type":"ContainerStarted","Data":"c41d1e554cce7c903d18ea650a49d6cc98e4f1e3a2b5b78a45bd55aa77ab1b47"} Oct 30 11:58:21.053449 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:21.053416 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-tqzld" event={"ID":"13a8af1c-f308-4eec-a543-f77f14ebfdf0","Type":"ContainerStarted","Data":"1e404da7c7bc4d69c6ca8a91404e18e5cd1aec38fc5d73f01091ba292a670d36"} Oct 30 11:58:21.097548 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:21.097494 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-tqzld" podStartSLOduration=2.565451751 podStartE2EDuration="4.097477567s" totalImagesPullingTime="1.532025816s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:58:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:58:21.095051641 +0000 UTC m=+257.609480387" watchObservedRunningTime="2026-10-30 11:58:21.097477567 +0000 UTC m=+257.611906312" Oct 30 11:58:24.070075 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:24.069998 4908 generic.go:356] "Generic (PLEG): container finished" podID="17115a8e-8703-4a19-bc79-ddf4d1cc285b" containerID="4df416dbfb750fa24bc5fff7b0f166cf1d068c07e8fed1dfd3ffceb14c4c3dad" exitCode=0 Oct 30 11:58:24.070483 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:24.070087 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64c9d5fd44-pmmwk" event={"ID":"17115a8e-8703-4a19-bc79-ddf4d1cc285b","Type":"ContainerDied","Data":"4df416dbfb750fa24bc5fff7b0f166cf1d068c07e8fed1dfd3ffceb14c4c3dad"} Oct 30 11:58:24.556291 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:24.556259 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-sztv9"] Oct 30 11:58:24.559679 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:24.559662 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-sztv9" Oct 30 11:58:24.577986 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:24.577955 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Oct 30 11:58:24.578093 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:24.578013 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Oct 30 11:58:24.578093 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:24.578053 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-97xz4\"" Oct 30 11:58:24.579893 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:24.579873 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-sztv9"] Oct 30 11:58:24.726186 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:24.726123 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vwsv\" (UniqueName: \"kubernetes.io/projected/5e0353a1-64bf-43c6-81f0-74a2e24fd1bd-kube-api-access-6vwsv\") pod \"cert-manager-cainjector-5c57bbd4fc-sztv9\" (UID: \"5e0353a1-64bf-43c6-81f0-74a2e24fd1bd\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-sztv9" Oct 30 11:58:24.726186 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:24.726165 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5e0353a1-64bf-43c6-81f0-74a2e24fd1bd-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-sztv9\" (UID: \"5e0353a1-64bf-43c6-81f0-74a2e24fd1bd\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-sztv9" Oct 30 11:58:24.827493 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:24.827457 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vwsv\" (UniqueName: \"kubernetes.io/projected/5e0353a1-64bf-43c6-81f0-74a2e24fd1bd-kube-api-access-6vwsv\") pod \"cert-manager-cainjector-5c57bbd4fc-sztv9\" (UID: \"5e0353a1-64bf-43c6-81f0-74a2e24fd1bd\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-sztv9" Oct 30 11:58:24.827621 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:24.827508 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5e0353a1-64bf-43c6-81f0-74a2e24fd1bd-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-sztv9\" (UID: \"5e0353a1-64bf-43c6-81f0-74a2e24fd1bd\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-sztv9" Oct 30 11:58:24.852655 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:24.852631 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vwsv\" (UniqueName: \"kubernetes.io/projected/5e0353a1-64bf-43c6-81f0-74a2e24fd1bd-kube-api-access-6vwsv\") pod \"cert-manager-cainjector-5c57bbd4fc-sztv9\" (UID: \"5e0353a1-64bf-43c6-81f0-74a2e24fd1bd\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-sztv9" Oct 30 11:58:24.854451 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:24.854429 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5e0353a1-64bf-43c6-81f0-74a2e24fd1bd-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-sztv9\" (UID: \"5e0353a1-64bf-43c6-81f0-74a2e24fd1bd\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-sztv9" Oct 30 11:58:24.889243 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:24.889220 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-sztv9" Oct 30 11:58:25.077044 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:25.077011 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64c9d5fd44-pmmwk" event={"ID":"17115a8e-8703-4a19-bc79-ddf4d1cc285b","Type":"ContainerStarted","Data":"89c82538ed3cfd4ea1f76f4dc8ea5811918fc6dfb20fafaf9947d8abd99a77cb"} Oct 30 11:58:25.077385 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:25.077147 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64c9d5fd44-pmmwk" Oct 30 11:58:25.198975 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:25.198950 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-sztv9"] Oct 30 11:58:25.210707 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:25.210667 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64c9d5fd44-pmmwk" podStartSLOduration=2.716527042 podStartE2EDuration="6.210656442s" totalImagesPullingTime="3.080058649s" totalInitContainerRuntime="414.070751ms" isStatefulPod=false podCreationTimestamp="2026-10-30 11:58:19 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:58:25.209144046 +0000 UTC m=+261.723572791" watchObservedRunningTime="2026-10-30 11:58:25.210656442 +0000 UTC m=+261.725085186" Oct 30 11:58:25.216680 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:58:25.216656 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5e0353a1_64bf_43c6_81f0_74a2e24fd1bd.slice/crio-17eab5ff71a03ba23bac6b10e5269bd91a113d8dc1e247eced1486ef8db7b954 WatchSource:0}: Error finding container 17eab5ff71a03ba23bac6b10e5269bd91a113d8dc1e247eced1486ef8db7b954: Status 404 returned error can't find the container with id 17eab5ff71a03ba23bac6b10e5269bd91a113d8dc1e247eced1486ef8db7b954 Oct 30 11:58:26.082769 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:26.082733 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-sztv9" event={"ID":"5e0353a1-64bf-43c6-81f0-74a2e24fd1bd","Type":"ContainerStarted","Data":"17eab5ff71a03ba23bac6b10e5269bd91a113d8dc1e247eced1486ef8db7b954"} Oct 30 11:58:28.095334 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:28.095241 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-sztv9" event={"ID":"5e0353a1-64bf-43c6-81f0-74a2e24fd1bd","Type":"ContainerStarted","Data":"1f9599d341f4403c13dc0e07650473324d6cca3781f8ebaa9b499d81dad1528a"} Oct 30 11:58:28.134050 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:28.134004 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-sztv9" podStartSLOduration=1.5755859349999999 podStartE2EDuration="4.133991919s" totalImagesPullingTime="2.558405984s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:58:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:58:28.132759133 +0000 UTC m=+264.647187878" watchObservedRunningTime="2026-10-30 11:58:28.133991919 +0000 UTC m=+264.648420662" Oct 30 11:58:35.674434 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:35.674400 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-7m749"] Oct 30 11:58:35.681366 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:35.681345 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-7m749" Oct 30 11:58:35.698731 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:35.698710 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-pnrpc\"" Oct 30 11:58:35.710162 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:35.710137 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 30 11:58:35.710258 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:35.710152 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 30 11:58:35.712034 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:35.712013 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-7m749"] Oct 30 11:58:35.812677 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:35.812654 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8zzq\" (UniqueName: \"kubernetes.io/projected/ca1be587-5f3e-4004-8d90-573130f6e9ed-kube-api-access-g8zzq\") pod \"authorino-operator-7b7bb4bc77-7m749\" (UID: \"ca1be587-5f3e-4004-8d90-573130f6e9ed\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-7m749" Oct 30 11:58:35.912950 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:35.912926 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8zzq\" (UniqueName: \"kubernetes.io/projected/ca1be587-5f3e-4004-8d90-573130f6e9ed-kube-api-access-g8zzq\") pod \"authorino-operator-7b7bb4bc77-7m749\" (UID: \"ca1be587-5f3e-4004-8d90-573130f6e9ed\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-7m749" Oct 30 11:58:35.935860 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:35.935779 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8zzq\" (UniqueName: \"kubernetes.io/projected/ca1be587-5f3e-4004-8d90-573130f6e9ed-kube-api-access-g8zzq\") pod \"authorino-operator-7b7bb4bc77-7m749\" (UID: \"ca1be587-5f3e-4004-8d90-573130f6e9ed\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-7m749" Oct 30 11:58:35.994735 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:35.994711 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-7m749" Oct 30 11:58:36.085095 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:36.085067 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64c9d5fd44-pmmwk" Oct 30 11:58:36.290546 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:36.290513 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-7m749"] Oct 30 11:58:36.295103 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:58:36.295076 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca1be587_5f3e_4004_8d90_573130f6e9ed.slice/crio-3f82cc12bc3aa942da4a0e2736630a1f8eb50fc86167692b86b6b2315d324f20 WatchSource:0}: Error finding container 3f82cc12bc3aa942da4a0e2736630a1f8eb50fc86167692b86b6b2315d324f20: Status 404 returned error can't find the container with id 3f82cc12bc3aa942da4a0e2736630a1f8eb50fc86167692b86b6b2315d324f20 Oct 30 11:58:37.141252 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:37.141218 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-7m749" event={"ID":"ca1be587-5f3e-4004-8d90-573130f6e9ed","Type":"ContainerStarted","Data":"3f82cc12bc3aa942da4a0e2736630a1f8eb50fc86167692b86b6b2315d324f20"} Oct 30 11:58:39.153480 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:39.153447 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-7m749" event={"ID":"ca1be587-5f3e-4004-8d90-573130f6e9ed","Type":"ContainerStarted","Data":"22c93b082e38fe0b7028466d612ce9744b402bdb9d9621f1f4c6a2ac94d2c870"} Oct 30 11:58:39.153857 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:39.153514 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-7m749" Oct 30 11:58:39.207698 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:39.207654 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-7b7bb4bc77-7m749" podStartSLOduration=1.985010039 podStartE2EDuration="4.207640899s" totalImagesPullingTime="2.22263086s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:58:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:58:39.206706115 +0000 UTC m=+275.721134860" watchObservedRunningTime="2026-10-30 11:58:39.207640899 +0000 UTC m=+275.722069654" Oct 30 11:58:39.259217 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:39.259195 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-lnqqz"] Oct 30 11:58:39.262626 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:39.262607 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-lnqqz" Oct 30 11:58:39.295365 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:39.295339 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-lnqqz"] Oct 30 11:58:39.297730 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:39.297708 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Oct 30 11:58:39.297817 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:39.297734 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-8hz5v\"" Oct 30 11:58:39.442860 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:39.442767 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-986rk\" (UniqueName: \"kubernetes.io/projected/1398a43d-5477-4f46-bfad-6e8dba0af13a-kube-api-access-986rk\") pod \"dns-operator-controller-manager-8557cbdb84-lnqqz\" (UID: \"1398a43d-5477-4f46-bfad-6e8dba0af13a\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-lnqqz" Oct 30 11:58:39.543425 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:39.543393 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-986rk\" (UniqueName: \"kubernetes.io/projected/1398a43d-5477-4f46-bfad-6e8dba0af13a-kube-api-access-986rk\") pod \"dns-operator-controller-manager-8557cbdb84-lnqqz\" (UID: \"1398a43d-5477-4f46-bfad-6e8dba0af13a\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-lnqqz" Oct 30 11:58:39.574474 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:39.574447 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-986rk\" (UniqueName: \"kubernetes.io/projected/1398a43d-5477-4f46-bfad-6e8dba0af13a-kube-api-access-986rk\") pod \"dns-operator-controller-manager-8557cbdb84-lnqqz\" (UID: \"1398a43d-5477-4f46-bfad-6e8dba0af13a\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-lnqqz" Oct 30 11:58:39.874711 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:39.874673 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-lnqqz" Oct 30 11:58:40.170986 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:40.170913 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-lnqqz"] Oct 30 11:58:40.174330 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:58:40.174301 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1398a43d_5477_4f46_bfad_6e8dba0af13a.slice/crio-698f827a298d12a188293343349159207d7814694cc4ca7c9d4a71352e0076b5 WatchSource:0}: Error finding container 698f827a298d12a188293343349159207d7814694cc4ca7c9d4a71352e0076b5: Status 404 returned error can't find the container with id 698f827a298d12a188293343349159207d7814694cc4ca7c9d4a71352e0076b5 Oct 30 11:58:41.165240 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:41.165204 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-lnqqz" event={"ID":"1398a43d-5477-4f46-bfad-6e8dba0af13a","Type":"ContainerStarted","Data":"698f827a298d12a188293343349159207d7814694cc4ca7c9d4a71352e0076b5"} Oct 30 11:58:42.172020 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:42.171943 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-lnqqz" event={"ID":"1398a43d-5477-4f46-bfad-6e8dba0af13a","Type":"ContainerStarted","Data":"9c365a87066ee631e6338d735b5c66f694d8922952730d349d4e3dad7448b0ac"} Oct 30 11:58:42.172403 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:42.172092 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-lnqqz" Oct 30 11:58:42.214295 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:42.214251 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-lnqqz" podStartSLOduration=1.75840267 podStartE2EDuration="3.214239017s" totalImagesPullingTime="1.455836347s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:58:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:58:42.212790891 +0000 UTC m=+278.727219633" watchObservedRunningTime="2026-10-30 11:58:42.214239017 +0000 UTC m=+278.728667762" Oct 30 11:58:44.447398 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:44.447366 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-kz5cc"] Oct 30 11:58:44.451081 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:44.451062 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-kz5cc" Oct 30 11:58:44.476925 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:44.476898 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfjkt\" (UniqueName: \"kubernetes.io/projected/3c4d8fbe-763c-47c6-9b31-f522275e7782-kube-api-access-nfjkt\") pod \"kuadrant-operator-controller-manager-678d87766f-kz5cc\" (UID: \"3c4d8fbe-763c-47c6-9b31-f522275e7782\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-kz5cc" Oct 30 11:58:44.477047 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:44.476926 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3c4d8fbe-763c-47c6-9b31-f522275e7782-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-kz5cc\" (UID: \"3c4d8fbe-763c-47c6-9b31-f522275e7782\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-kz5cc" Oct 30 11:58:44.488065 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:44.488042 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-bllgd\"" Oct 30 11:58:44.490715 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:44.490690 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-kz5cc"] Oct 30 11:58:44.577923 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:44.577900 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfjkt\" (UniqueName: \"kubernetes.io/projected/3c4d8fbe-763c-47c6-9b31-f522275e7782-kube-api-access-nfjkt\") pod \"kuadrant-operator-controller-manager-678d87766f-kz5cc\" (UID: \"3c4d8fbe-763c-47c6-9b31-f522275e7782\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-kz5cc" Oct 30 11:58:44.578035 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:44.577930 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3c4d8fbe-763c-47c6-9b31-f522275e7782-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-kz5cc\" (UID: \"3c4d8fbe-763c-47c6-9b31-f522275e7782\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-kz5cc" Oct 30 11:58:44.578227 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:44.578209 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3c4d8fbe-763c-47c6-9b31-f522275e7782-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-kz5cc\" (UID: \"3c4d8fbe-763c-47c6-9b31-f522275e7782\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-kz5cc" Oct 30 11:58:44.600104 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:44.600079 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfjkt\" (UniqueName: \"kubernetes.io/projected/3c4d8fbe-763c-47c6-9b31-f522275e7782-kube-api-access-nfjkt\") pod \"kuadrant-operator-controller-manager-678d87766f-kz5cc\" (UID: \"3c4d8fbe-763c-47c6-9b31-f522275e7782\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-kz5cc" Oct 30 11:58:44.764009 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:44.763988 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-kz5cc" Oct 30 11:58:45.107360 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:45.107339 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-kz5cc"] Oct 30 11:58:45.109590 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:58:45.109565 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c4d8fbe_763c_47c6_9b31_f522275e7782.slice/crio-983a324da8f7ec876daa40854ac4d0ba4059d21f7cae508fb83934b036ca0db4 WatchSource:0}: Error finding container 983a324da8f7ec876daa40854ac4d0ba4059d21f7cae508fb83934b036ca0db4: Status 404 returned error can't find the container with id 983a324da8f7ec876daa40854ac4d0ba4059d21f7cae508fb83934b036ca0db4 Oct 30 11:58:45.187338 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:45.187310 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-kz5cc" event={"ID":"3c4d8fbe-763c-47c6-9b31-f522275e7782","Type":"ContainerStarted","Data":"983a324da8f7ec876daa40854ac4d0ba4059d21f7cae508fb83934b036ca0db4"} Oct 30 11:58:46.461633 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:46.461598 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-wgm7f"] Oct 30 11:58:46.465641 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:46.465612 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-wgm7f" Oct 30 11:58:46.491265 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:46.491230 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gz2g\" (UniqueName: \"kubernetes.io/projected/15039390-67e6-48c7-ba22-8704aa2cc777-kube-api-access-7gz2g\") pod \"limitador-operator-controller-manager-5778bb4449-wgm7f\" (UID: \"15039390-67e6-48c7-ba22-8704aa2cc777\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-wgm7f" Oct 30 11:58:46.492096 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:46.491536 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-8x9q8\"" Oct 30 11:58:46.493935 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:46.493912 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-wgm7f"] Oct 30 11:58:46.592112 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:46.592077 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gz2g\" (UniqueName: \"kubernetes.io/projected/15039390-67e6-48c7-ba22-8704aa2cc777-kube-api-access-7gz2g\") pod \"limitador-operator-controller-manager-5778bb4449-wgm7f\" (UID: \"15039390-67e6-48c7-ba22-8704aa2cc777\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-wgm7f" Oct 30 11:58:46.628788 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:46.628758 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gz2g\" (UniqueName: \"kubernetes.io/projected/15039390-67e6-48c7-ba22-8704aa2cc777-kube-api-access-7gz2g\") pod \"limitador-operator-controller-manager-5778bb4449-wgm7f\" (UID: \"15039390-67e6-48c7-ba22-8704aa2cc777\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-wgm7f" Oct 30 11:58:46.781075 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:46.781042 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-wgm7f" Oct 30 11:58:47.090889 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:47.090780 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-wgm7f"] Oct 30 11:58:47.094678 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:58:47.094597 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15039390_67e6_48c7_ba22_8704aa2cc777.slice/crio-271376171cbd95a9cb6876f40a22aee83db9b6ca91263582eb15c13a719f6c44 WatchSource:0}: Error finding container 271376171cbd95a9cb6876f40a22aee83db9b6ca91263582eb15c13a719f6c44: Status 404 returned error can't find the container with id 271376171cbd95a9cb6876f40a22aee83db9b6ca91263582eb15c13a719f6c44 Oct 30 11:58:47.199929 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:47.199889 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-wgm7f" event={"ID":"15039390-67e6-48c7-ba22-8704aa2cc777","Type":"ContainerStarted","Data":"271376171cbd95a9cb6876f40a22aee83db9b6ca91263582eb15c13a719f6c44"} Oct 30 11:58:49.213367 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:49.213266 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-kz5cc" event={"ID":"3c4d8fbe-763c-47c6-9b31-f522275e7782","Type":"ContainerStarted","Data":"e8a509d7a74fd38025c82a1d170f82cab5a8282296e8d2dc996faf2803fd8408"} Oct 30 11:58:49.213823 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:49.213456 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-kz5cc" Oct 30 11:58:49.215331 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:49.215306 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-wgm7f" event={"ID":"15039390-67e6-48c7-ba22-8704aa2cc777","Type":"ContainerStarted","Data":"779432a8ae59b6bc90cdd513102d7a54b0e5d64e4a7821d0c02aa1e29c8a215a"} Oct 30 11:58:49.215422 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:49.215387 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-wgm7f" Oct 30 11:58:49.277673 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:49.277626 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-kz5cc" podStartSLOduration=1.626656211 podStartE2EDuration="5.277606517s" totalImagesPullingTime="3.650950306s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:58:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:58:49.266046451 +0000 UTC m=+285.780475196" watchObservedRunningTime="2026-10-30 11:58:49.277606517 +0000 UTC m=+285.792035261" Oct 30 11:58:49.305281 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:49.305242 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-wgm7f" podStartSLOduration=1.862893089 podStartE2EDuration="3.305230794s" totalImagesPullingTime="1.442337705s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:58:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:58:49.304714935 +0000 UTC m=+285.819143679" watchObservedRunningTime="2026-10-30 11:58:49.305230794 +0000 UTC m=+285.819659538" Oct 30 11:58:49.681751 ip-10-0-141-251 kubenswrapper[4908]: E1030 11:58:49.681721 4908 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vn6vv" containerName="extractor" Oct 30 11:58:50.160318 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:50.160286 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-7m749" Oct 30 11:58:51.533724 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:51.533685 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-85d774877-zkcdx"] Oct 30 11:58:51.537580 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:51.537555 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d774877-zkcdx" Oct 30 11:58:51.554634 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:51.554615 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-gjqmn\"" Oct 30 11:58:51.567726 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:51.567702 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-85d774877-zkcdx"] Oct 30 11:58:51.629982 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:51.629955 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6faa552b-c9a4-49a8-bc98-4d5a2cabf3e8-istio-token\") pod \"data-science-gateway-data-science-gateway-class-85d774877-zkcdx\" (UID: \"6faa552b-c9a4-49a8-bc98-4d5a2cabf3e8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d774877-zkcdx" Oct 30 11:58:51.630104 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:51.629989 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6faa552b-c9a4-49a8-bc98-4d5a2cabf3e8-istio-data\") pod \"data-science-gateway-data-science-gateway-class-85d774877-zkcdx\" (UID: \"6faa552b-c9a4-49a8-bc98-4d5a2cabf3e8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d774877-zkcdx" Oct 30 11:58:51.630104 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:51.630023 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6faa552b-c9a4-49a8-bc98-4d5a2cabf3e8-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-85d774877-zkcdx\" (UID: \"6faa552b-c9a4-49a8-bc98-4d5a2cabf3e8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d774877-zkcdx" Oct 30 11:58:51.630212 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:51.630127 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6faa552b-c9a4-49a8-bc98-4d5a2cabf3e8-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-85d774877-zkcdx\" (UID: \"6faa552b-c9a4-49a8-bc98-4d5a2cabf3e8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d774877-zkcdx" Oct 30 11:58:51.630270 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:51.630216 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6faa552b-c9a4-49a8-bc98-4d5a2cabf3e8-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-85d774877-zkcdx\" (UID: \"6faa552b-c9a4-49a8-bc98-4d5a2cabf3e8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d774877-zkcdx" Oct 30 11:58:51.630324 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:51.630263 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbl2p\" (UniqueName: \"kubernetes.io/projected/6faa552b-c9a4-49a8-bc98-4d5a2cabf3e8-kube-api-access-pbl2p\") pod \"data-science-gateway-data-science-gateway-class-85d774877-zkcdx\" (UID: \"6faa552b-c9a4-49a8-bc98-4d5a2cabf3e8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d774877-zkcdx" Oct 30 11:58:51.630324 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:51.630296 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6faa552b-c9a4-49a8-bc98-4d5a2cabf3e8-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-85d774877-zkcdx\" (UID: \"6faa552b-c9a4-49a8-bc98-4d5a2cabf3e8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d774877-zkcdx" Oct 30 11:58:51.630393 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:51.630324 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6faa552b-c9a4-49a8-bc98-4d5a2cabf3e8-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-85d774877-zkcdx\" (UID: \"6faa552b-c9a4-49a8-bc98-4d5a2cabf3e8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d774877-zkcdx" Oct 30 11:58:51.630432 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:51.630393 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6faa552b-c9a4-49a8-bc98-4d5a2cabf3e8-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-85d774877-zkcdx\" (UID: \"6faa552b-c9a4-49a8-bc98-4d5a2cabf3e8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d774877-zkcdx" Oct 30 11:58:51.731063 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:51.731034 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6faa552b-c9a4-49a8-bc98-4d5a2cabf3e8-istio-data\") pod \"data-science-gateway-data-science-gateway-class-85d774877-zkcdx\" (UID: \"6faa552b-c9a4-49a8-bc98-4d5a2cabf3e8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d774877-zkcdx" Oct 30 11:58:51.731187 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:51.731077 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6faa552b-c9a4-49a8-bc98-4d5a2cabf3e8-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-85d774877-zkcdx\" (UID: \"6faa552b-c9a4-49a8-bc98-4d5a2cabf3e8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d774877-zkcdx" Oct 30 11:58:51.731187 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:51.731102 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6faa552b-c9a4-49a8-bc98-4d5a2cabf3e8-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-85d774877-zkcdx\" (UID: \"6faa552b-c9a4-49a8-bc98-4d5a2cabf3e8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d774877-zkcdx" Oct 30 11:58:51.731304 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:51.731213 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6faa552b-c9a4-49a8-bc98-4d5a2cabf3e8-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-85d774877-zkcdx\" (UID: \"6faa552b-c9a4-49a8-bc98-4d5a2cabf3e8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d774877-zkcdx" Oct 30 11:58:51.731304 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:51.731260 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbl2p\" (UniqueName: \"kubernetes.io/projected/6faa552b-c9a4-49a8-bc98-4d5a2cabf3e8-kube-api-access-pbl2p\") pod \"data-science-gateway-data-science-gateway-class-85d774877-zkcdx\" (UID: \"6faa552b-c9a4-49a8-bc98-4d5a2cabf3e8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d774877-zkcdx" Oct 30 11:58:51.731304 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:51.731295 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6faa552b-c9a4-49a8-bc98-4d5a2cabf3e8-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-85d774877-zkcdx\" (UID: \"6faa552b-c9a4-49a8-bc98-4d5a2cabf3e8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d774877-zkcdx" Oct 30 11:58:51.731447 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:51.731323 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6faa552b-c9a4-49a8-bc98-4d5a2cabf3e8-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-85d774877-zkcdx\" (UID: \"6faa552b-c9a4-49a8-bc98-4d5a2cabf3e8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d774877-zkcdx" Oct 30 11:58:51.731447 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:51.731394 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6faa552b-c9a4-49a8-bc98-4d5a2cabf3e8-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-85d774877-zkcdx\" (UID: \"6faa552b-c9a4-49a8-bc98-4d5a2cabf3e8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d774877-zkcdx" Oct 30 11:58:51.731447 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:51.731428 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6faa552b-c9a4-49a8-bc98-4d5a2cabf3e8-istio-token\") pod \"data-science-gateway-data-science-gateway-class-85d774877-zkcdx\" (UID: \"6faa552b-c9a4-49a8-bc98-4d5a2cabf3e8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d774877-zkcdx" Oct 30 11:58:51.731597 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:51.731445 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6faa552b-c9a4-49a8-bc98-4d5a2cabf3e8-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-85d774877-zkcdx\" (UID: \"6faa552b-c9a4-49a8-bc98-4d5a2cabf3e8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d774877-zkcdx" Oct 30 11:58:51.731597 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:51.731511 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6faa552b-c9a4-49a8-bc98-4d5a2cabf3e8-istio-data\") pod \"data-science-gateway-data-science-gateway-class-85d774877-zkcdx\" (UID: \"6faa552b-c9a4-49a8-bc98-4d5a2cabf3e8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d774877-zkcdx" Oct 30 11:58:51.731707 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:51.731689 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6faa552b-c9a4-49a8-bc98-4d5a2cabf3e8-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-85d774877-zkcdx\" (UID: \"6faa552b-c9a4-49a8-bc98-4d5a2cabf3e8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d774877-zkcdx" Oct 30 11:58:51.731760 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:51.731727 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6faa552b-c9a4-49a8-bc98-4d5a2cabf3e8-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-85d774877-zkcdx\" (UID: \"6faa552b-c9a4-49a8-bc98-4d5a2cabf3e8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d774877-zkcdx" Oct 30 11:58:51.732100 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:51.732078 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6faa552b-c9a4-49a8-bc98-4d5a2cabf3e8-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-85d774877-zkcdx\" (UID: \"6faa552b-c9a4-49a8-bc98-4d5a2cabf3e8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d774877-zkcdx" Oct 30 11:58:51.733531 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:51.733503 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6faa552b-c9a4-49a8-bc98-4d5a2cabf3e8-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-85d774877-zkcdx\" (UID: \"6faa552b-c9a4-49a8-bc98-4d5a2cabf3e8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d774877-zkcdx" Oct 30 11:58:51.733779 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:51.733761 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6faa552b-c9a4-49a8-bc98-4d5a2cabf3e8-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-85d774877-zkcdx\" (UID: \"6faa552b-c9a4-49a8-bc98-4d5a2cabf3e8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d774877-zkcdx" Oct 30 11:58:51.758251 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:51.758229 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbl2p\" (UniqueName: \"kubernetes.io/projected/6faa552b-c9a4-49a8-bc98-4d5a2cabf3e8-kube-api-access-pbl2p\") pod \"data-science-gateway-data-science-gateway-class-85d774877-zkcdx\" (UID: \"6faa552b-c9a4-49a8-bc98-4d5a2cabf3e8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d774877-zkcdx" Oct 30 11:58:51.760978 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:51.760960 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6faa552b-c9a4-49a8-bc98-4d5a2cabf3e8-istio-token\") pod \"data-science-gateway-data-science-gateway-class-85d774877-zkcdx\" (UID: \"6faa552b-c9a4-49a8-bc98-4d5a2cabf3e8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d774877-zkcdx" Oct 30 11:58:51.851062 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:51.850985 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d774877-zkcdx" Oct 30 11:58:52.169198 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:52.169125 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-85d774877-zkcdx"] Oct 30 11:58:52.173008 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:58:52.172974 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6faa552b_c9a4_49a8_bc98_4d5a2cabf3e8.slice/crio-9c0626a4f84b0f2b7e6ee179338ea33687f3a699066aea52ebc65b0a5f8acde8 WatchSource:0}: Error finding container 9c0626a4f84b0f2b7e6ee179338ea33687f3a699066aea52ebc65b0a5f8acde8: Status 404 returned error can't find the container with id 9c0626a4f84b0f2b7e6ee179338ea33687f3a699066aea52ebc65b0a5f8acde8 Oct 30 11:58:52.175193 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:52.175164 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:58:52.175266 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:52.175234 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:58:52.175303 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:52.175263 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:58:52.232000 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:52.231974 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d774877-zkcdx" event={"ID":"6faa552b-c9a4-49a8-bc98-4d5a2cabf3e8","Type":"ContainerStarted","Data":"9c0626a4f84b0f2b7e6ee179338ea33687f3a699066aea52ebc65b0a5f8acde8"} Oct 30 11:58:53.180082 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:53.180051 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-lnqqz" Oct 30 11:58:53.238905 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:53.238866 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d774877-zkcdx" event={"ID":"6faa552b-c9a4-49a8-bc98-4d5a2cabf3e8","Type":"ContainerStarted","Data":"044f97651c89d5beb380c686b40cdcbeca967172c86569987ac60678e5a7e733"} Oct 30 11:58:53.282671 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:53.282623 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d774877-zkcdx" podStartSLOduration=2.282609972 podStartE2EDuration="2.282609972s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:58:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:58:53.280583279 +0000 UTC m=+289.795012018" watchObservedRunningTime="2026-10-30 11:58:53.282609972 +0000 UTC m=+289.797038788" Oct 30 11:58:53.851846 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:53.851802 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d774877-zkcdx" Oct 30 11:58:53.856382 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:53.856357 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d774877-zkcdx" Oct 30 11:58:54.244223 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:54.244200 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d774877-zkcdx" Oct 30 11:58:54.245010 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:54.244987 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d774877-zkcdx" Oct 30 11:58:59.451399 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:59.451366 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-vjg48"] Oct 30 11:58:59.457524 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:59.457502 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-vjg48" Oct 30 11:58:59.471542 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:59.471519 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 30 11:58:59.482119 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:59.482098 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 30 11:58:59.482208 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:59.482101 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-pfcx7\"" Oct 30 11:58:59.493515 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:59.493490 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpz2g\" (UniqueName: \"kubernetes.io/projected/706593d5-4331-41a7-a9d6-b66f2596e81b-kube-api-access-xpz2g\") pod \"kserve-module-controller-manager-7566cf5fc8-vjg48\" (UID: \"706593d5-4331-41a7-a9d6-b66f2596e81b\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-vjg48" Oct 30 11:58:59.493613 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:59.493573 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/706593d5-4331-41a7-a9d6-b66f2596e81b-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-vjg48\" (UID: \"706593d5-4331-41a7-a9d6-b66f2596e81b\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-vjg48" Oct 30 11:58:59.493684 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:59.493613 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/706593d5-4331-41a7-a9d6-b66f2596e81b-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-vjg48\" (UID: \"706593d5-4331-41a7-a9d6-b66f2596e81b\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-vjg48" Oct 30 11:58:59.493754 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:59.493724 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/706593d5-4331-41a7-a9d6-b66f2596e81b-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-vjg48\" (UID: \"706593d5-4331-41a7-a9d6-b66f2596e81b\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-vjg48" Oct 30 11:58:59.505388 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:59.505368 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\"" Oct 30 11:58:59.507421 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:59.507401 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-vjg48"] Oct 30 11:58:59.594786 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:59.594767 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpz2g\" (UniqueName: \"kubernetes.io/projected/706593d5-4331-41a7-a9d6-b66f2596e81b-kube-api-access-xpz2g\") pod \"kserve-module-controller-manager-7566cf5fc8-vjg48\" (UID: \"706593d5-4331-41a7-a9d6-b66f2596e81b\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-vjg48" Oct 30 11:58:59.594891 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:59.594807 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/706593d5-4331-41a7-a9d6-b66f2596e81b-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-vjg48\" (UID: \"706593d5-4331-41a7-a9d6-b66f2596e81b\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-vjg48" Oct 30 11:58:59.594891 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:59.594879 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/706593d5-4331-41a7-a9d6-b66f2596e81b-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-vjg48\" (UID: \"706593d5-4331-41a7-a9d6-b66f2596e81b\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-vjg48" Oct 30 11:58:59.594968 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:59.594926 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/706593d5-4331-41a7-a9d6-b66f2596e81b-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-vjg48\" (UID: \"706593d5-4331-41a7-a9d6-b66f2596e81b\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-vjg48" Oct 30 11:58:59.595137 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:59.595121 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/706593d5-4331-41a7-a9d6-b66f2596e81b-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-vjg48\" (UID: \"706593d5-4331-41a7-a9d6-b66f2596e81b\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-vjg48" Oct 30 11:58:59.595219 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:59.595204 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/706593d5-4331-41a7-a9d6-b66f2596e81b-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-vjg48\" (UID: \"706593d5-4331-41a7-a9d6-b66f2596e81b\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-vjg48" Oct 30 11:58:59.595773 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:59.595756 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/706593d5-4331-41a7-a9d6-b66f2596e81b-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-vjg48\" (UID: \"706593d5-4331-41a7-a9d6-b66f2596e81b\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-vjg48" Oct 30 11:58:59.616782 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:59.616763 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpz2g\" (UniqueName: \"kubernetes.io/projected/706593d5-4331-41a7-a9d6-b66f2596e81b-kube-api-access-xpz2g\") pod \"kserve-module-controller-manager-7566cf5fc8-vjg48\" (UID: \"706593d5-4331-41a7-a9d6-b66f2596e81b\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-vjg48" Oct 30 11:58:59.770610 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:58:59.770587 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-vjg48" Oct 30 11:59:00.079796 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:59:00.079731 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-vjg48"] Oct 30 11:59:00.084148 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:59:00.084123 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod706593d5_4331_41a7_a9d6_b66f2596e81b.slice/crio-a41f25f045eeeca0d1e67a43fe59cdf04f162535b9d810f1d97fd181193c6274 WatchSource:0}: Error finding container a41f25f045eeeca0d1e67a43fe59cdf04f162535b9d810f1d97fd181193c6274: Status 404 returned error can't find the container with id a41f25f045eeeca0d1e67a43fe59cdf04f162535b9d810f1d97fd181193c6274 Oct 30 11:59:00.224183 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:59:00.224160 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-kz5cc" Oct 30 11:59:00.224297 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:59:00.224246 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-wgm7f" Oct 30 11:59:00.280758 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:59:00.278816 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-vjg48" event={"ID":"706593d5-4331-41a7-a9d6-b66f2596e81b","Type":"ContainerStarted","Data":"a41f25f045eeeca0d1e67a43fe59cdf04f162535b9d810f1d97fd181193c6274"} Oct 30 11:59:02.291250 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:59:02.291214 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-vjg48" event={"ID":"706593d5-4331-41a7-a9d6-b66f2596e81b","Type":"ContainerStarted","Data":"2e37419d60e905173868790e2ad40742bf1d72bf5bddce24c0e865277146be0c"} Oct 30 11:59:02.291561 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:59:02.291289 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-vjg48" Oct 30 11:59:02.345410 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:59:02.345365 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-vjg48" podStartSLOduration=1.219915864 podStartE2EDuration="3.345349705s" totalImagesPullingTime="2.125433841s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:58:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:59:02.343087028 +0000 UTC m=+298.857515773" watchObservedRunningTime="2026-10-30 11:59:02.345349705 +0000 UTC m=+298.859778442" Oct 30 11:59:03.629797 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:59:03.629771 4908 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-plczt_a70ee11c-c20e-4a4f-8476-ac5eb02816d0/console-operator/3.log" Oct 30 11:59:03.630275 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:59:03.629934 4908 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-plczt_a70ee11c-c20e-4a4f-8476-ac5eb02816d0/console-operator/3.log" Oct 30 11:59:03.634584 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:59:03.634559 4908 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b8bxz_644e19b3-10ad-4145-9ed3-595d19ed9b07/ovn-acl-logging/0.log" Oct 30 11:59:03.634702 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:59:03.634601 4908 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b8bxz_644e19b3-10ad-4145-9ed3-595d19ed9b07/ovn-acl-logging/0.log" Oct 30 11:59:19.683477 ip-10-0-141-251 kubenswrapper[4908]: E1030 11:59:19.683394 4908 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vn6vv" containerName="extractor" Oct 30 11:59:23.297219 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:59:23.297189 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-vjg48" Oct 30 11:59:49.682277 ip-10-0-141-251 kubenswrapper[4908]: E1030 11:59:49.682239 4908 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vn6vv" containerName="extractor" Oct 30 11:59:55.362293 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:59:55.362262 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-7q7p5"] Oct 30 11:59:55.369480 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:59:55.369456 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-7q7p5" Oct 30 11:59:55.386324 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:59:55.386300 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-7q7p5"] Oct 30 11:59:55.395135 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:59:55.395114 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-wd2gh\"" Oct 30 11:59:55.397600 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:59:55.397583 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Oct 30 11:59:55.406796 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:59:55.406775 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-b5bdw"] Oct 30 11:59:55.410377 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:59:55.410352 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-b5bdw" Oct 30 11:59:55.424632 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:59:55.424612 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-5vq7b\"" Oct 30 11:59:55.424720 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:59:55.424662 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Oct 30 11:59:55.433288 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:59:55.433266 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-b5bdw"] Oct 30 11:59:55.529413 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:59:55.529381 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fftf9\" (UniqueName: \"kubernetes.io/projected/4c36eb48-c2b2-4500-a1eb-26a7611bc2b6-kube-api-access-fftf9\") pod \"kserve-controller-manager-776cc66d47-7q7p5\" (UID: \"4c36eb48-c2b2-4500-a1eb-26a7611bc2b6\") " pod="opendatahub/kserve-controller-manager-776cc66d47-7q7p5" Oct 30 11:59:55.529607 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:59:55.529429 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4c36eb48-c2b2-4500-a1eb-26a7611bc2b6-cert\") pod \"kserve-controller-manager-776cc66d47-7q7p5\" (UID: \"4c36eb48-c2b2-4500-a1eb-26a7611bc2b6\") " pod="opendatahub/kserve-controller-manager-776cc66d47-7q7p5" Oct 30 11:59:55.529607 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:59:55.529471 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pkvb\" (UniqueName: \"kubernetes.io/projected/f809408f-0f70-42e8-bb14-154253ccfb86-kube-api-access-5pkvb\") pod \"llmisvc-controller-manager-869d5dcc5d-b5bdw\" (UID: \"f809408f-0f70-42e8-bb14-154253ccfb86\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-b5bdw" Oct 30 11:59:55.529607 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:59:55.529573 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f809408f-0f70-42e8-bb14-154253ccfb86-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-b5bdw\" (UID: \"f809408f-0f70-42e8-bb14-154253ccfb86\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-b5bdw" Oct 30 11:59:55.630683 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:59:55.630610 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fftf9\" (UniqueName: \"kubernetes.io/projected/4c36eb48-c2b2-4500-a1eb-26a7611bc2b6-kube-api-access-fftf9\") pod \"kserve-controller-manager-776cc66d47-7q7p5\" (UID: \"4c36eb48-c2b2-4500-a1eb-26a7611bc2b6\") " pod="opendatahub/kserve-controller-manager-776cc66d47-7q7p5" Oct 30 11:59:55.630683 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:59:55.630660 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4c36eb48-c2b2-4500-a1eb-26a7611bc2b6-cert\") pod \"kserve-controller-manager-776cc66d47-7q7p5\" (UID: \"4c36eb48-c2b2-4500-a1eb-26a7611bc2b6\") " pod="opendatahub/kserve-controller-manager-776cc66d47-7q7p5" Oct 30 11:59:55.630867 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:59:55.630697 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pkvb\" (UniqueName: \"kubernetes.io/projected/f809408f-0f70-42e8-bb14-154253ccfb86-kube-api-access-5pkvb\") pod \"llmisvc-controller-manager-869d5dcc5d-b5bdw\" (UID: \"f809408f-0f70-42e8-bb14-154253ccfb86\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-b5bdw" Oct 30 11:59:55.630867 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:59:55.630716 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f809408f-0f70-42e8-bb14-154253ccfb86-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-b5bdw\" (UID: \"f809408f-0f70-42e8-bb14-154253ccfb86\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-b5bdw" Oct 30 11:59:55.633128 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:59:55.633106 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f809408f-0f70-42e8-bb14-154253ccfb86-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-b5bdw\" (UID: \"f809408f-0f70-42e8-bb14-154253ccfb86\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-b5bdw" Oct 30 11:59:55.633292 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:59:55.633270 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4c36eb48-c2b2-4500-a1eb-26a7611bc2b6-cert\") pod \"kserve-controller-manager-776cc66d47-7q7p5\" (UID: \"4c36eb48-c2b2-4500-a1eb-26a7611bc2b6\") " pod="opendatahub/kserve-controller-manager-776cc66d47-7q7p5" Oct 30 11:59:55.655563 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:59:55.655528 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pkvb\" (UniqueName: \"kubernetes.io/projected/f809408f-0f70-42e8-bb14-154253ccfb86-kube-api-access-5pkvb\") pod \"llmisvc-controller-manager-869d5dcc5d-b5bdw\" (UID: \"f809408f-0f70-42e8-bb14-154253ccfb86\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-b5bdw" Oct 30 11:59:55.656013 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:59:55.655993 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fftf9\" (UniqueName: \"kubernetes.io/projected/4c36eb48-c2b2-4500-a1eb-26a7611bc2b6-kube-api-access-fftf9\") pod \"kserve-controller-manager-776cc66d47-7q7p5\" (UID: \"4c36eb48-c2b2-4500-a1eb-26a7611bc2b6\") " pod="opendatahub/kserve-controller-manager-776cc66d47-7q7p5" Oct 30 11:59:55.682104 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:59:55.682083 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-7q7p5" Oct 30 11:59:55.722300 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:59:55.722274 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-b5bdw" Oct 30 11:59:55.995589 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:59:55.995558 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-7q7p5"] Oct 30 11:59:55.999119 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:59:55.999093 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c36eb48_c2b2_4500_a1eb_26a7611bc2b6.slice/crio-8f290415253f42399b6900a9b0159a94af4199d41c21e4355ac06b8740ea1d41 WatchSource:0}: Error finding container 8f290415253f42399b6900a9b0159a94af4199d41c21e4355ac06b8740ea1d41: Status 404 returned error can't find the container with id 8f290415253f42399b6900a9b0159a94af4199d41c21e4355ac06b8740ea1d41 Oct 30 11:59:56.000476 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:59:56.000455 4908 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 11:59:56.033038 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:59:56.033014 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-b5bdw"] Oct 30 11:59:56.036347 ip-10-0-141-251 kubenswrapper[4908]: W1030 11:59:56.036324 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podf809408f_0f70_42e8_bb14_154253ccfb86.slice/crio-dd48565438bbc7048aff189274d765a1fde93f077eb7b73957128205ae4cfda5 WatchSource:0}: Error finding container dd48565438bbc7048aff189274d765a1fde93f077eb7b73957128205ae4cfda5: Status 404 returned error can't find the container with id dd48565438bbc7048aff189274d765a1fde93f077eb7b73957128205ae4cfda5 Oct 30 11:59:56.585572 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:59:56.585532 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-b5bdw" event={"ID":"f809408f-0f70-42e8-bb14-154253ccfb86","Type":"ContainerStarted","Data":"dd48565438bbc7048aff189274d765a1fde93f077eb7b73957128205ae4cfda5"} Oct 30 11:59:56.586783 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:59:56.586754 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-7q7p5" event={"ID":"4c36eb48-c2b2-4500-a1eb-26a7611bc2b6","Type":"ContainerStarted","Data":"8f290415253f42399b6900a9b0159a94af4199d41c21e4355ac06b8740ea1d41"} Oct 30 11:59:58.600489 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:59:58.600457 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-b5bdw" event={"ID":"f809408f-0f70-42e8-bb14-154253ccfb86","Type":"ContainerStarted","Data":"c3065846c8d10d248e82a0b302f526c1bb84e862a1cdc8393815fcf0e17250c2"} Oct 30 11:59:58.600929 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:59:58.600531 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-b5bdw" Oct 30 11:59:58.602088 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:59:58.602067 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-7q7p5" event={"ID":"4c36eb48-c2b2-4500-a1eb-26a7611bc2b6","Type":"ContainerStarted","Data":"53ebbb2e709c683330906bca9293ffe8fdcf40f503cc96c699c5f7db3ecb7562"} Oct 30 11:59:58.602211 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:59:58.602157 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-7q7p5" Oct 30 11:59:58.643071 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:59:58.643028 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-b5bdw" podStartSLOduration=1.545029785 podStartE2EDuration="3.643017623s" totalImagesPullingTime="2.097987838s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:59:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:59:58.640949339 +0000 UTC m=+355.155378084" watchObservedRunningTime="2026-10-30 11:59:58.643017623 +0000 UTC m=+355.157446367" Oct 30 11:59:58.685339 ip-10-0-141-251 kubenswrapper[4908]: I1030 11:59:58.685294 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-7q7p5" podStartSLOduration=1.547093958 podStartE2EDuration="3.685283247s" totalImagesPullingTime="2.138189289s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:59:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:59:58.685134716 +0000 UTC m=+355.199563462" watchObservedRunningTime="2026-10-30 11:59:58.685283247 +0000 UTC m=+355.199711991" Oct 30 12:00:19.682302 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:00:19.682270 4908 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vn6vv" containerName="extractor" Oct 30 12:00:29.610710 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:00:29.610682 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-b5bdw" Oct 30 12:00:29.613499 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:00:29.613476 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-7q7p5" Oct 30 12:00:49.682063 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:00:49.681978 4908 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vn6vv" containerName="extractor" Oct 30 12:01:19.681928 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:01:19.681891 4908 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vn6vv" containerName="extractor" Oct 30 12:01:49.682900 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:01:49.682858 4908 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vn6vv" containerName="extractor" Oct 30 12:02:19.682131 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:02:19.682059 4908 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vn6vv" containerName="extractor" Oct 30 12:02:49.685717 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:02:49.685672 4908 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vn6vv" containerName="extractor" Oct 30 12:03:19.683278 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:03:19.683243 4908 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vn6vv" containerName="extractor" Oct 30 12:03:49.681582 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:03:49.681500 4908 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vn6vv" containerName="extractor" Oct 30 12:04:03.667004 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:04:03.666976 4908 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-plczt_a70ee11c-c20e-4a4f-8476-ac5eb02816d0/console-operator/3.log" Oct 30 12:04:03.667562 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:04:03.667362 4908 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-plczt_a70ee11c-c20e-4a4f-8476-ac5eb02816d0/console-operator/3.log" Oct 30 12:04:03.672232 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:04:03.672208 4908 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b8bxz_644e19b3-10ad-4145-9ed3-595d19ed9b07/ovn-acl-logging/0.log" Oct 30 12:04:03.672459 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:04:03.672441 4908 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b8bxz_644e19b3-10ad-4145-9ed3-595d19ed9b07/ovn-acl-logging/0.log" Oct 30 12:04:19.685375 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:04:19.685331 4908 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vn6vv" containerName="extractor" Oct 30 12:04:49.681939 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:04:49.681906 4908 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vn6vv" containerName="extractor" Oct 30 12:05:19.681246 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:05:19.681171 4908 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vn6vv" containerName="extractor" Oct 30 12:05:49.682581 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:05:49.682545 4908 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vn6vv" containerName="extractor" Oct 30 12:06:19.682374 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:06:19.682336 4908 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vn6vv" containerName="extractor" Oct 30 12:06:49.681999 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:06:49.681911 4908 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vn6vv" containerName="extractor" Oct 30 12:07:19.682132 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:07:19.682095 4908 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vn6vv" containerName="extractor" Oct 30 12:07:49.683208 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:07:49.683173 4908 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vn6vv" containerName="extractor" Oct 30 12:08:19.684960 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:08:19.684865 4908 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vn6vv" containerName="extractor" Oct 30 12:08:49.682313 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:08:49.682274 4908 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vn6vv" containerName="extractor" Oct 30 12:09:03.702387 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:09:03.702355 4908 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-plczt_a70ee11c-c20e-4a4f-8476-ac5eb02816d0/console-operator/3.log" Oct 30 12:09:03.704430 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:09:03.704403 4908 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-plczt_a70ee11c-c20e-4a4f-8476-ac5eb02816d0/console-operator/3.log" Oct 30 12:09:03.707207 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:09:03.707184 4908 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b8bxz_644e19b3-10ad-4145-9ed3-595d19ed9b07/ovn-acl-logging/0.log" Oct 30 12:09:03.709707 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:09:03.709689 4908 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b8bxz_644e19b3-10ad-4145-9ed3-595d19ed9b07/ovn-acl-logging/0.log" Oct 30 12:09:12.319308 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:09:12.319274 4908 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-kz5cc_3c4d8fbe-763c-47c6-9b31-f522275e7782/manager/0.log" Oct 30 12:09:12.564233 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:09:12.564201 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-kz5cc"] Oct 30 12:09:12.564421 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:09:12.564387 4908 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-kz5cc" podUID="3c4d8fbe-763c-47c6-9b31-f522275e7782" containerName="manager" containerID="cri-o://e8a509d7a74fd38025c82a1d170f82cab5a8282296e8d2dc996faf2803fd8408" gracePeriod=10 Oct 30 12:09:12.590887 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:09:12.590862 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-5z7d8"] Oct 30 12:09:12.594194 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:09:12.594179 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-5z7d8" Oct 30 12:09:12.634881 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:09:12.634854 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-5z7d8"] Oct 30 12:09:12.702453 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:09:12.702425 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e2bcf742-b9a1-46e1-865d-e7e062786c05-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-5z7d8\" (UID: \"e2bcf742-b9a1-46e1-865d-e7e062786c05\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-5z7d8" Oct 30 12:09:12.702582 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:09:12.702466 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gcs2\" (UniqueName: \"kubernetes.io/projected/e2bcf742-b9a1-46e1-865d-e7e062786c05-kube-api-access-5gcs2\") pod \"kuadrant-operator-controller-manager-678d87766f-5z7d8\" (UID: \"e2bcf742-b9a1-46e1-865d-e7e062786c05\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-5z7d8" Oct 30 12:09:12.803571 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:09:12.803542 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gcs2\" (UniqueName: \"kubernetes.io/projected/e2bcf742-b9a1-46e1-865d-e7e062786c05-kube-api-access-5gcs2\") pod \"kuadrant-operator-controller-manager-678d87766f-5z7d8\" (UID: \"e2bcf742-b9a1-46e1-865d-e7e062786c05\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-5z7d8" Oct 30 12:09:12.803678 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:09:12.803664 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e2bcf742-b9a1-46e1-865d-e7e062786c05-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-5z7d8\" (UID: \"e2bcf742-b9a1-46e1-865d-e7e062786c05\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-5z7d8" Oct 30 12:09:12.804039 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:09:12.804020 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e2bcf742-b9a1-46e1-865d-e7e062786c05-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-5z7d8\" (UID: \"e2bcf742-b9a1-46e1-865d-e7e062786c05\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-5z7d8" Oct 30 12:09:12.815469 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:09:12.815451 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-kz5cc" Oct 30 12:09:12.825861 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:09:12.825784 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gcs2\" (UniqueName: \"kubernetes.io/projected/e2bcf742-b9a1-46e1-865d-e7e062786c05-kube-api-access-5gcs2\") pod \"kuadrant-operator-controller-manager-678d87766f-5z7d8\" (UID: \"e2bcf742-b9a1-46e1-865d-e7e062786c05\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-5z7d8" Oct 30 12:09:12.904306 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:09:12.904279 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c4d8fbe-763c-47c6-9b31-f522275e7782-kube-api-access-nfjkt\" (UniqueName: \"kubernetes.io/projected/3c4d8fbe-763c-47c6-9b31-f522275e7782-kube-api-access-nfjkt\") pod \"3c4d8fbe-763c-47c6-9b31-f522275e7782\" (UID: \"3c4d8fbe-763c-47c6-9b31-f522275e7782\") " Oct 30 12:09:12.904440 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:09:12.904314 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c4d8fbe-763c-47c6-9b31-f522275e7782-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3c4d8fbe-763c-47c6-9b31-f522275e7782-extensions-socket-volume\") pod \"3c4d8fbe-763c-47c6-9b31-f522275e7782\" (UID: \"3c4d8fbe-763c-47c6-9b31-f522275e7782\") " Oct 30 12:09:12.904730 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:09:12.904707 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c4d8fbe-763c-47c6-9b31-f522275e7782-extensions-socket-volume" pod "3c4d8fbe-763c-47c6-9b31-f522275e7782" (UID: "3c4d8fbe-763c-47c6-9b31-f522275e7782"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:09:12.906530 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:09:12.906510 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c4d8fbe-763c-47c6-9b31-f522275e7782-kube-api-access-nfjkt" pod "3c4d8fbe-763c-47c6-9b31-f522275e7782" (UID: "3c4d8fbe-763c-47c6-9b31-f522275e7782"). InnerVolumeSpecName "kube-api-access-nfjkt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:09:12.963869 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:09:12.963846 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-5z7d8" Oct 30 12:09:13.006079 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:09:13.006050 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nfjkt\" (UniqueName: \"kubernetes.io/projected/3c4d8fbe-763c-47c6-9b31-f522275e7782-kube-api-access-nfjkt\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:09:13.006079 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:09:13.006077 4908 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3c4d8fbe-763c-47c6-9b31-f522275e7782-extensions-socket-volume\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:09:13.278694 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:09:13.278668 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-5z7d8"] Oct 30 12:09:13.282251 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:09:13.282223 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2bcf742_b9a1_46e1_865d_e7e062786c05.slice/crio-9d748dbeeeca9493977464b33bd23cefdf010578cd25b6de5449609d685aba31 WatchSource:0}: Error finding container 9d748dbeeeca9493977464b33bd23cefdf010578cd25b6de5449609d685aba31: Status 404 returned error can't find the container with id 9d748dbeeeca9493977464b33bd23cefdf010578cd25b6de5449609d685aba31 Oct 30 12:09:13.418994 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:09:13.418966 4908 generic.go:356] "Generic (PLEG): container finished" podID="3c4d8fbe-763c-47c6-9b31-f522275e7782" containerID="e8a509d7a74fd38025c82a1d170f82cab5a8282296e8d2dc996faf2803fd8408" exitCode=0 Oct 30 12:09:13.419400 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:09:13.419010 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-kz5cc" Oct 30 12:09:13.419400 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:09:13.419047 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-kz5cc" event={"ID":"3c4d8fbe-763c-47c6-9b31-f522275e7782","Type":"ContainerDied","Data":"e8a509d7a74fd38025c82a1d170f82cab5a8282296e8d2dc996faf2803fd8408"} Oct 30 12:09:13.419400 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:09:13.419101 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-kz5cc" event={"ID":"3c4d8fbe-763c-47c6-9b31-f522275e7782","Type":"ContainerDied","Data":"983a324da8f7ec876daa40854ac4d0ba4059d21f7cae508fb83934b036ca0db4"} Oct 30 12:09:13.419400 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:09:13.419125 4908 scope.go:122] "RemoveContainer" containerID="e8a509d7a74fd38025c82a1d170f82cab5a8282296e8d2dc996faf2803fd8408" Oct 30 12:09:13.428037 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:09:13.427993 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-5z7d8" event={"ID":"e2bcf742-b9a1-46e1-865d-e7e062786c05","Type":"ContainerStarted","Data":"cfd07976d412b6745a61f4a8f5ff001e85abb5f914d034859b64fed7b809fda2"} Oct 30 12:09:13.428174 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:09:13.428065 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-5z7d8" event={"ID":"e2bcf742-b9a1-46e1-865d-e7e062786c05","Type":"ContainerStarted","Data":"9d748dbeeeca9493977464b33bd23cefdf010578cd25b6de5449609d685aba31"} Oct 30 12:09:13.428240 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:09:13.428176 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-5z7d8" Oct 30 12:09:13.435898 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:09:13.435878 4908 scope.go:122] "RemoveContainer" containerID="e8a509d7a74fd38025c82a1d170f82cab5a8282296e8d2dc996faf2803fd8408" Oct 30 12:09:13.436149 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:09:13.436130 4908 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8a509d7a74fd38025c82a1d170f82cab5a8282296e8d2dc996faf2803fd8408\": container with ID starting with e8a509d7a74fd38025c82a1d170f82cab5a8282296e8d2dc996faf2803fd8408 not found: ID does not exist" containerID="e8a509d7a74fd38025c82a1d170f82cab5a8282296e8d2dc996faf2803fd8408" Oct 30 12:09:13.436198 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:09:13.436157 4908 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8a509d7a74fd38025c82a1d170f82cab5a8282296e8d2dc996faf2803fd8408"} err="failed to get container status \"e8a509d7a74fd38025c82a1d170f82cab5a8282296e8d2dc996faf2803fd8408\": rpc error: code = NotFound desc = could not find container \"e8a509d7a74fd38025c82a1d170f82cab5a8282296e8d2dc996faf2803fd8408\": container with ID starting with e8a509d7a74fd38025c82a1d170f82cab5a8282296e8d2dc996faf2803fd8408 not found: ID does not exist" Oct 30 12:09:13.542169 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:09:13.542090 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-kz5cc"] Oct 30 12:09:13.552619 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:09:13.552588 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-kz5cc"] Oct 30 12:09:13.588702 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:09:13.588654 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-5z7d8" podStartSLOduration=1.5886403009999999 podStartE2EDuration="1.588640301s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:09:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:09:13.587368506 +0000 UTC m=+910.101797251" watchObservedRunningTime="2026-10-30 12:09:13.588640301 +0000 UTC m=+910.103069037" Oct 30 12:09:13.604336 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:09:13.604311 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c4d8fbe-763c-47c6-9b31-f522275e7782" path="/var/lib/kubelet/pods/3c4d8fbe-763c-47c6-9b31-f522275e7782/volumes" Oct 30 12:09:19.681789 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:09:19.681754 4908 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vn6vv" containerName="extractor" Oct 30 12:09:24.436623 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:09:24.436590 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-5z7d8" Oct 30 12:09:32.488250 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:09:32.488214 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9wwfc"] Oct 30 12:09:32.488783 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:09:32.488761 4908 cpu_manager.go:414] "containerMap: removing container" podUID="3c4d8fbe-763c-47c6-9b31-f522275e7782" containerName="manager" Oct 30 12:09:32.488904 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:09:32.488785 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c4d8fbe-763c-47c6-9b31-f522275e7782" containerName="manager" Oct 30 12:09:32.488904 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:09:32.488900 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c4d8fbe-763c-47c6-9b31-f522275e7782" containerName="manager" Oct 30 12:09:32.492042 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:09:32.492018 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-9wwfc" Oct 30 12:09:32.510379 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:09:32.510357 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Oct 30 12:09:32.512928 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:09:32.512907 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9wwfc"] Oct 30 12:09:32.521541 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:09:32.521511 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-jg5cf\"" Oct 30 12:09:32.580835 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:09:32.580813 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9wwfc"] Oct 30 12:09:32.648509 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:09:32.648489 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jk4mw\" (UniqueName: \"kubernetes.io/projected/f01cbe07-9124-4bc8-80fd-902b23e57f67-kube-api-access-jk4mw\") pod \"limitador-limitador-7c9b87d574-9wwfc\" (UID: \"f01cbe07-9124-4bc8-80fd-902b23e57f67\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-9wwfc" Oct 30 12:09:32.648606 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:09:32.648518 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/f01cbe07-9124-4bc8-80fd-902b23e57f67-config-file\") pod \"limitador-limitador-7c9b87d574-9wwfc\" (UID: \"f01cbe07-9124-4bc8-80fd-902b23e57f67\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-9wwfc" Oct 30 12:09:32.749791 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:09:32.749727 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jk4mw\" (UniqueName: \"kubernetes.io/projected/f01cbe07-9124-4bc8-80fd-902b23e57f67-kube-api-access-jk4mw\") pod \"limitador-limitador-7c9b87d574-9wwfc\" (UID: \"f01cbe07-9124-4bc8-80fd-902b23e57f67\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-9wwfc" Oct 30 12:09:32.749791 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:09:32.749765 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/f01cbe07-9124-4bc8-80fd-902b23e57f67-config-file\") pod \"limitador-limitador-7c9b87d574-9wwfc\" (UID: \"f01cbe07-9124-4bc8-80fd-902b23e57f67\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-9wwfc" Oct 30 12:09:32.750391 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:09:32.750370 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/f01cbe07-9124-4bc8-80fd-902b23e57f67-config-file\") pod \"limitador-limitador-7c9b87d574-9wwfc\" (UID: \"f01cbe07-9124-4bc8-80fd-902b23e57f67\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-9wwfc" Oct 30 12:09:32.771611 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:09:32.771584 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jk4mw\" (UniqueName: \"kubernetes.io/projected/f01cbe07-9124-4bc8-80fd-902b23e57f67-kube-api-access-jk4mw\") pod \"limitador-limitador-7c9b87d574-9wwfc\" (UID: \"f01cbe07-9124-4bc8-80fd-902b23e57f67\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-9wwfc" Oct 30 12:09:32.805384 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:09:32.805359 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-9wwfc" Oct 30 12:09:33.098207 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:09:33.098125 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9wwfc"] Oct 30 12:09:33.101965 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:09:33.101936 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf01cbe07_9124_4bc8_80fd_902b23e57f67.slice/crio-9df7f70054a5434d691f42005bfa99b884ec936cc9c04d2aefe3d51b20764625 WatchSource:0}: Error finding container 9df7f70054a5434d691f42005bfa99b884ec936cc9c04d2aefe3d51b20764625: Status 404 returned error can't find the container with id 9df7f70054a5434d691f42005bfa99b884ec936cc9c04d2aefe3d51b20764625 Oct 30 12:09:33.104192 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:09:33.104169 4908 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 12:09:33.538609 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:09:33.538580 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-9wwfc" event={"ID":"f01cbe07-9124-4bc8-80fd-902b23e57f67","Type":"ContainerStarted","Data":"9df7f70054a5434d691f42005bfa99b884ec936cc9c04d2aefe3d51b20764625"} Oct 30 12:09:34.545291 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:09:34.545261 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-9wwfc" event={"ID":"f01cbe07-9124-4bc8-80fd-902b23e57f67","Type":"ContainerStarted","Data":"9768bcf1d61c15446cce9be923d6ba1df74db885e2b6591daf1752e803a4d85e"} Oct 30 12:09:34.545748 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:09:34.545419 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-9wwfc" Oct 30 12:09:34.585806 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:09:34.585762 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-9wwfc" podStartSLOduration=1.72824986 podStartE2EDuration="2.585750977s" totalImagesPullingTime="857.501117ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:09:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:09:34.583400796 +0000 UTC m=+931.097829539" watchObservedRunningTime="2026-10-30 12:09:34.585750977 +0000 UTC m=+931.100179750" Oct 30 12:09:45.551564 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:09:45.551532 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-9wwfc" Oct 30 12:09:49.681514 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:09:49.681476 4908 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vn6vv" containerName="extractor" Oct 30 12:10:19.682157 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:10:19.682119 4908 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vn6vv" containerName="extractor" Oct 30 12:10:49.681906 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:10:49.681866 4908 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vn6vv" containerName="extractor" Oct 30 12:10:51.770994 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:10:51.770962 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9wwfc"] Oct 30 12:11:00.977749 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:11:00.977720 4908 generic.go:356] "Generic (PLEG): container finished" podID="a5b1685b-e185-4b12-9c20-3c87e9837fd7" containerID="e490ba37cb4c0f66a103fec7579c2a868db8760d8211a33dd6400e0d8c452b93" exitCode=0 Oct 30 12:11:00.978178 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:11:00.977780 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-d9b94" event={"ID":"a5b1685b-e185-4b12-9c20-3c87e9837fd7","Type":"ContainerDied","Data":"e490ba37cb4c0f66a103fec7579c2a868db8760d8211a33dd6400e0d8c452b93"} Oct 30 12:11:00.978178 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:11:00.978066 4908 scope.go:122] "RemoveContainer" containerID="e490ba37cb4c0f66a103fec7579c2a868db8760d8211a33dd6400e0d8c452b93" Oct 30 12:11:00.979096 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:11:00.979061 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:11:00.979222 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:11:00.979145 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:11:00.979222 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:11:00.979194 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:11:01.991732 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:11:01.991685 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-d9b94" event={"ID":"a5b1685b-e185-4b12-9c20-3c87e9837fd7","Type":"ContainerStarted","Data":"4631e9016d55731633f9fff9d825d54981feba64ae6b2918e84ed19e90831a3b"} Oct 30 12:11:02.136610 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:11:02.136578 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-d9b94" Oct 30 12:11:02.137951 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:11:02.137927 4908 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-d9b94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" start-of-body= Oct 30 12:11:02.138066 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:11:02.137977 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-d9b94" podUID="a5b1685b-e185-4b12-9c20-3c87e9837fd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" Oct 30 12:11:02.980175 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:11:02.980125 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-nsldm"] Oct 30 12:11:02.990270 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:11:02.984748 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-nsldm" Oct 30 12:11:02.997631 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:11:02.997585 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-nsldm"] Oct 30 12:11:03.136345 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:11:03.136308 4908 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-d9b94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" start-of-body= Oct 30 12:11:03.136498 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:11:03.136378 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-d9b94" podUID="a5b1685b-e185-4b12-9c20-3c87e9837fd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" Oct 30 12:11:03.184195 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:11:03.184161 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zc5g\" (UniqueName: \"kubernetes.io/projected/37fef44c-01b0-430e-a657-e9f22e60b580-kube-api-access-7zc5g\") pod \"llmisvc-controller-manager-6466c554d8-nsldm\" (UID: \"37fef44c-01b0-430e-a657-e9f22e60b580\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-nsldm" Oct 30 12:11:03.184346 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:11:03.184206 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/37fef44c-01b0-430e-a657-e9f22e60b580-cert\") pod \"llmisvc-controller-manager-6466c554d8-nsldm\" (UID: \"37fef44c-01b0-430e-a657-e9f22e60b580\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-nsldm" Oct 30 12:11:03.285307 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:11:03.285228 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/37fef44c-01b0-430e-a657-e9f22e60b580-cert\") pod \"llmisvc-controller-manager-6466c554d8-nsldm\" (UID: \"37fef44c-01b0-430e-a657-e9f22e60b580\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-nsldm" Oct 30 12:11:03.285437 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:11:03.285333 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zc5g\" (UniqueName: \"kubernetes.io/projected/37fef44c-01b0-430e-a657-e9f22e60b580-kube-api-access-7zc5g\") pod \"llmisvc-controller-manager-6466c554d8-nsldm\" (UID: \"37fef44c-01b0-430e-a657-e9f22e60b580\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-nsldm" Oct 30 12:11:03.287720 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:11:03.287690 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/37fef44c-01b0-430e-a657-e9f22e60b580-cert\") pod \"llmisvc-controller-manager-6466c554d8-nsldm\" (UID: \"37fef44c-01b0-430e-a657-e9f22e60b580\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-nsldm" Oct 30 12:11:03.307960 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:11:03.307931 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zc5g\" (UniqueName: \"kubernetes.io/projected/37fef44c-01b0-430e-a657-e9f22e60b580-kube-api-access-7zc5g\") pod \"llmisvc-controller-manager-6466c554d8-nsldm\" (UID: \"37fef44c-01b0-430e-a657-e9f22e60b580\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-nsldm" Oct 30 12:11:03.604077 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:11:03.603990 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-nsldm" Oct 30 12:11:03.915303 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:11:03.915221 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-nsldm"] Oct 30 12:11:03.919246 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:11:03.919209 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37fef44c_01b0_430e_a657_e9f22e60b580.slice/crio-9ed80b4d21eec4da6217d0a5ebf082229632269598c88fcc890e680a235a6f85 WatchSource:0}: Error finding container 9ed80b4d21eec4da6217d0a5ebf082229632269598c88fcc890e680a235a6f85: Status 404 returned error can't find the container with id 9ed80b4d21eec4da6217d0a5ebf082229632269598c88fcc890e680a235a6f85 Oct 30 12:11:04.007578 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:11:04.007538 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-nsldm" event={"ID":"37fef44c-01b0-430e-a657-e9f22e60b580","Type":"ContainerStarted","Data":"9ed80b4d21eec4da6217d0a5ebf082229632269598c88fcc890e680a235a6f85"} Oct 30 12:11:04.136383 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:11:04.136348 4908 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-d9b94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" start-of-body= Oct 30 12:11:04.136528 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:11:04.136395 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-d9b94" podUID="a5b1685b-e185-4b12-9c20-3c87e9837fd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" Oct 30 12:11:04.695658 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:11:04.695574 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-d9b94" Oct 30 12:11:05.017248 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:11:05.017212 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-nsldm" event={"ID":"37fef44c-01b0-430e-a657-e9f22e60b580","Type":"ContainerStarted","Data":"bc4c9c30c8ef64a08f6091a2de657daad44f1a75bc58e7c888912ef25043b59f"} Oct 30 12:11:05.017615 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:11:05.017309 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-nsldm" Oct 30 12:11:05.057725 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:11:05.057671 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-nsldm" podStartSLOduration=2.645477494 podStartE2EDuration="3.05765866s" totalImagesPullingTime="412.181166ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:11:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:11:05.055322371 +0000 UTC m=+1021.569751145" watchObservedRunningTime="2026-10-30 12:11:05.05765866 +0000 UTC m=+1021.572087404" Oct 30 12:11:05.136532 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:11:05.136499 4908 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-d9b94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" start-of-body= Oct 30 12:11:05.136684 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:11:05.136555 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-d9b94" podUID="a5b1685b-e185-4b12-9c20-3c87e9837fd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" Oct 30 12:11:06.136319 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:11:06.136287 4908 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-d9b94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" start-of-body= Oct 30 12:11:06.136688 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:11:06.136344 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-d9b94" podUID="a5b1685b-e185-4b12-9c20-3c87e9837fd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" Oct 30 12:11:07.140438 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:11:07.140410 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-d9b94" Oct 30 12:11:07.141454 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:11:07.141437 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-d9b94" Oct 30 12:11:19.683011 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:11:19.682927 4908 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vn6vv" containerName="extractor" Oct 30 12:11:36.024630 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:11:36.024601 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-nsldm" Oct 30 12:11:36.095092 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:11:36.095058 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-b5bdw"] Oct 30 12:11:36.095352 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:11:36.095298 4908 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-b5bdw" podUID="f809408f-0f70-42e8-bb14-154253ccfb86" containerName="manager" containerID="cri-o://c3065846c8d10d248e82a0b302f526c1bb84e862a1cdc8393815fcf0e17250c2" gracePeriod=30 Oct 30 12:11:36.341373 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:11:36.341351 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-b5bdw" Oct 30 12:11:36.453086 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:11:36.453057 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f809408f-0f70-42e8-bb14-154253ccfb86-cert\" (UniqueName: \"kubernetes.io/secret/f809408f-0f70-42e8-bb14-154253ccfb86-cert\") pod \"f809408f-0f70-42e8-bb14-154253ccfb86\" (UID: \"f809408f-0f70-42e8-bb14-154253ccfb86\") " Oct 30 12:11:36.453242 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:11:36.453100 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f809408f-0f70-42e8-bb14-154253ccfb86-kube-api-access-5pkvb\" (UniqueName: \"kubernetes.io/projected/f809408f-0f70-42e8-bb14-154253ccfb86-kube-api-access-5pkvb\") pod \"f809408f-0f70-42e8-bb14-154253ccfb86\" (UID: \"f809408f-0f70-42e8-bb14-154253ccfb86\") " Oct 30 12:11:36.455215 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:11:36.455183 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f809408f-0f70-42e8-bb14-154253ccfb86-kube-api-access-5pkvb" pod "f809408f-0f70-42e8-bb14-154253ccfb86" (UID: "f809408f-0f70-42e8-bb14-154253ccfb86"). InnerVolumeSpecName "kube-api-access-5pkvb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:11:36.455323 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:11:36.455254 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f809408f-0f70-42e8-bb14-154253ccfb86-cert" pod "f809408f-0f70-42e8-bb14-154253ccfb86" (UID: "f809408f-0f70-42e8-bb14-154253ccfb86"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:11:36.553958 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:11:36.553928 4908 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f809408f-0f70-42e8-bb14-154253ccfb86-cert\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:11:36.553958 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:11:36.553953 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5pkvb\" (UniqueName: \"kubernetes.io/projected/f809408f-0f70-42e8-bb14-154253ccfb86-kube-api-access-5pkvb\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:11:37.190052 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:11:37.190017 4908 generic.go:356] "Generic (PLEG): container finished" podID="f809408f-0f70-42e8-bb14-154253ccfb86" containerID="c3065846c8d10d248e82a0b302f526c1bb84e862a1cdc8393815fcf0e17250c2" exitCode=0 Oct 30 12:11:37.190492 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:11:37.190070 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-b5bdw" Oct 30 12:11:37.190492 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:11:37.190107 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-b5bdw" event={"ID":"f809408f-0f70-42e8-bb14-154253ccfb86","Type":"ContainerDied","Data":"c3065846c8d10d248e82a0b302f526c1bb84e862a1cdc8393815fcf0e17250c2"} Oct 30 12:11:37.190492 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:11:37.190202 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-b5bdw" event={"ID":"f809408f-0f70-42e8-bb14-154253ccfb86","Type":"ContainerDied","Data":"dd48565438bbc7048aff189274d765a1fde93f077eb7b73957128205ae4cfda5"} Oct 30 12:11:37.190492 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:11:37.190233 4908 scope.go:122] "RemoveContainer" containerID="c3065846c8d10d248e82a0b302f526c1bb84e862a1cdc8393815fcf0e17250c2" Oct 30 12:11:37.199806 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:11:37.199784 4908 scope.go:122] "RemoveContainer" containerID="c3065846c8d10d248e82a0b302f526c1bb84e862a1cdc8393815fcf0e17250c2" Oct 30 12:11:37.200233 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:11:37.200213 4908 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3065846c8d10d248e82a0b302f526c1bb84e862a1cdc8393815fcf0e17250c2\": container with ID starting with c3065846c8d10d248e82a0b302f526c1bb84e862a1cdc8393815fcf0e17250c2 not found: ID does not exist" containerID="c3065846c8d10d248e82a0b302f526c1bb84e862a1cdc8393815fcf0e17250c2" Oct 30 12:11:37.200279 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:11:37.200241 4908 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3065846c8d10d248e82a0b302f526c1bb84e862a1cdc8393815fcf0e17250c2"} err="failed to get container status \"c3065846c8d10d248e82a0b302f526c1bb84e862a1cdc8393815fcf0e17250c2\": rpc error: code = NotFound desc = could not find container \"c3065846c8d10d248e82a0b302f526c1bb84e862a1cdc8393815fcf0e17250c2\": container with ID starting with c3065846c8d10d248e82a0b302f526c1bb84e862a1cdc8393815fcf0e17250c2 not found: ID does not exist" Oct 30 12:11:37.238202 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:11:37.238179 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-b5bdw"] Oct 30 12:11:37.252658 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:11:37.252636 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-b5bdw"] Oct 30 12:11:37.604434 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:11:37.604401 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f809408f-0f70-42e8-bb14-154253ccfb86" path="/var/lib/kubelet/pods/f809408f-0f70-42e8-bb14-154253ccfb86/volumes" Oct 30 12:11:40.191100 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:11:40.191067 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-b2fvc"] Oct 30 12:11:40.191600 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:11:40.191580 4908 cpu_manager.go:414] "containerMap: removing container" podUID="f809408f-0f70-42e8-bb14-154253ccfb86" containerName="manager" Oct 30 12:11:40.191650 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:11:40.191603 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f809408f-0f70-42e8-bb14-154253ccfb86" containerName="manager" Oct 30 12:11:40.191708 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:11:40.191698 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="f809408f-0f70-42e8-bb14-154253ccfb86" containerName="manager" Oct 30 12:11:40.196298 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:11:40.196279 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-b2fvc" Oct 30 12:11:40.223680 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:11:40.223652 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-b2fvc"] Oct 30 12:11:40.289309 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:11:40.289277 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/486f71e9-0730-4dbc-b456-31a57942cd2b-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-b2fvc\" (UID: \"486f71e9-0730-4dbc-b456-31a57942cd2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-b2fvc" Oct 30 12:11:40.289309 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:11:40.289310 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/486f71e9-0730-4dbc-b456-31a57942cd2b-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-b2fvc\" (UID: \"486f71e9-0730-4dbc-b456-31a57942cd2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-b2fvc" Oct 30 12:11:40.289510 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:11:40.289346 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/486f71e9-0730-4dbc-b456-31a57942cd2b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-b2fvc\" (UID: \"486f71e9-0730-4dbc-b456-31a57942cd2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-b2fvc" Oct 30 12:11:40.289510 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:11:40.289461 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2h4wr\" (UniqueName: \"kubernetes.io/projected/486f71e9-0730-4dbc-b456-31a57942cd2b-kube-api-access-2h4wr\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-b2fvc\" (UID: \"486f71e9-0730-4dbc-b456-31a57942cd2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-b2fvc" Oct 30 12:11:40.289510 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:11:40.289499 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/486f71e9-0730-4dbc-b456-31a57942cd2b-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-b2fvc\" (UID: \"486f71e9-0730-4dbc-b456-31a57942cd2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-b2fvc" Oct 30 12:11:40.289665 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:11:40.289529 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/486f71e9-0730-4dbc-b456-31a57942cd2b-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-b2fvc\" (UID: \"486f71e9-0730-4dbc-b456-31a57942cd2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-b2fvc" Oct 30 12:11:40.289665 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:11:40.289613 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/486f71e9-0730-4dbc-b456-31a57942cd2b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-b2fvc\" (UID: \"486f71e9-0730-4dbc-b456-31a57942cd2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-b2fvc" Oct 30 12:11:40.289778 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:11:40.289664 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/486f71e9-0730-4dbc-b456-31a57942cd2b-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-b2fvc\" (UID: \"486f71e9-0730-4dbc-b456-31a57942cd2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-b2fvc" Oct 30 12:11:40.289778 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:11:40.289722 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/486f71e9-0730-4dbc-b456-31a57942cd2b-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-b2fvc\" (UID: \"486f71e9-0730-4dbc-b456-31a57942cd2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-b2fvc" Oct 30 12:11:40.390673 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:11:40.390646 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/486f71e9-0730-4dbc-b456-31a57942cd2b-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-b2fvc\" (UID: \"486f71e9-0730-4dbc-b456-31a57942cd2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-b2fvc" Oct 30 12:11:40.390851 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:11:40.390677 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/486f71e9-0730-4dbc-b456-31a57942cd2b-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-b2fvc\" (UID: \"486f71e9-0730-4dbc-b456-31a57942cd2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-b2fvc" Oct 30 12:11:40.390851 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:11:40.390710 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/486f71e9-0730-4dbc-b456-31a57942cd2b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-b2fvc\" (UID: \"486f71e9-0730-4dbc-b456-31a57942cd2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-b2fvc" Oct 30 12:11:40.390851 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:11:40.390816 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2h4wr\" (UniqueName: \"kubernetes.io/projected/486f71e9-0730-4dbc-b456-31a57942cd2b-kube-api-access-2h4wr\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-b2fvc\" (UID: \"486f71e9-0730-4dbc-b456-31a57942cd2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-b2fvc" Oct 30 12:11:40.391027 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:11:40.390876 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/486f71e9-0730-4dbc-b456-31a57942cd2b-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-b2fvc\" (UID: \"486f71e9-0730-4dbc-b456-31a57942cd2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-b2fvc" Oct 30 12:11:40.391027 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:11:40.390906 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/486f71e9-0730-4dbc-b456-31a57942cd2b-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-b2fvc\" (UID: \"486f71e9-0730-4dbc-b456-31a57942cd2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-b2fvc" Oct 30 12:11:40.391027 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:11:40.390925 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/486f71e9-0730-4dbc-b456-31a57942cd2b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-b2fvc\" (UID: \"486f71e9-0730-4dbc-b456-31a57942cd2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-b2fvc" Oct 30 12:11:40.391027 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:11:40.390960 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/486f71e9-0730-4dbc-b456-31a57942cd2b-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-b2fvc\" (UID: \"486f71e9-0730-4dbc-b456-31a57942cd2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-b2fvc" Oct 30 12:11:40.391027 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:11:40.391008 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/486f71e9-0730-4dbc-b456-31a57942cd2b-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-b2fvc\" (UID: \"486f71e9-0730-4dbc-b456-31a57942cd2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-b2fvc" Oct 30 12:11:40.391257 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:11:40.391078 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/486f71e9-0730-4dbc-b456-31a57942cd2b-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-b2fvc\" (UID: \"486f71e9-0730-4dbc-b456-31a57942cd2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-b2fvc" Oct 30 12:11:40.391257 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:11:40.391176 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/486f71e9-0730-4dbc-b456-31a57942cd2b-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-b2fvc\" (UID: \"486f71e9-0730-4dbc-b456-31a57942cd2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-b2fvc" Oct 30 12:11:40.391363 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:11:40.391341 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/486f71e9-0730-4dbc-b456-31a57942cd2b-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-b2fvc\" (UID: \"486f71e9-0730-4dbc-b456-31a57942cd2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-b2fvc" Oct 30 12:11:40.391428 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:11:40.391405 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/486f71e9-0730-4dbc-b456-31a57942cd2b-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-b2fvc\" (UID: \"486f71e9-0730-4dbc-b456-31a57942cd2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-b2fvc" Oct 30 12:11:40.391550 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:11:40.391532 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/486f71e9-0730-4dbc-b456-31a57942cd2b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-b2fvc\" (UID: \"486f71e9-0730-4dbc-b456-31a57942cd2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-b2fvc" Oct 30 12:11:40.393255 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:11:40.393230 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/486f71e9-0730-4dbc-b456-31a57942cd2b-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-b2fvc\" (UID: \"486f71e9-0730-4dbc-b456-31a57942cd2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-b2fvc" Oct 30 12:11:40.393364 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:11:40.393319 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/486f71e9-0730-4dbc-b456-31a57942cd2b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-b2fvc\" (UID: \"486f71e9-0730-4dbc-b456-31a57942cd2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-b2fvc" Oct 30 12:11:40.412915 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:11:40.412890 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/486f71e9-0730-4dbc-b456-31a57942cd2b-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-b2fvc\" (UID: \"486f71e9-0730-4dbc-b456-31a57942cd2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-b2fvc" Oct 30 12:11:40.413029 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:11:40.413018 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2h4wr\" (UniqueName: \"kubernetes.io/projected/486f71e9-0730-4dbc-b456-31a57942cd2b-kube-api-access-2h4wr\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-b2fvc\" (UID: \"486f71e9-0730-4dbc-b456-31a57942cd2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-b2fvc" Oct 30 12:11:40.510542 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:11:40.510514 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-b2fvc" Oct 30 12:11:40.821148 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:11:40.821059 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-b2fvc"] Oct 30 12:11:40.825079 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:11:40.825051 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod486f71e9_0730_4dbc_b456_31a57942cd2b.slice/crio-b440307840bfc54c22b28d6fe927555a02b4b60cbeb6d89287fa96f0195787fe WatchSource:0}: Error finding container b440307840bfc54c22b28d6fe927555a02b4b60cbeb6d89287fa96f0195787fe: Status 404 returned error can't find the container with id b440307840bfc54c22b28d6fe927555a02b4b60cbeb6d89287fa96f0195787fe Oct 30 12:11:40.827022 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:11:40.826996 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:11:40.827089 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:11:40.827057 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:11:40.827124 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:11:40.827087 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:11:41.215219 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:11:41.215139 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-b2fvc" event={"ID":"486f71e9-0730-4dbc-b456-31a57942cd2b","Type":"ContainerStarted","Data":"4fa9870ea7935d8bf00699f570c5eecdb4bb51681c5f9fc446be527510b3b10a"} Oct 30 12:11:41.215219 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:11:41.215188 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-b2fvc" event={"ID":"486f71e9-0730-4dbc-b456-31a57942cd2b","Type":"ContainerStarted","Data":"b440307840bfc54c22b28d6fe927555a02b4b60cbeb6d89287fa96f0195787fe"} Oct 30 12:11:41.289339 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:11:41.289280 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-b2fvc" podStartSLOduration=1.289264308 podStartE2EDuration="1.289264308s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:11:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:11:41.277274935 +0000 UTC m=+1057.791703680" watchObservedRunningTime="2026-10-30 12:11:41.289264308 +0000 UTC m=+1057.803693053" Oct 30 12:11:41.510684 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:11:41.510643 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-b2fvc" Oct 30 12:11:41.511918 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:11:41.511890 4908 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-b2fvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Oct 30 12:11:41.512023 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:11:41.511949 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-b2fvc" podUID="486f71e9-0730-4dbc-b456-31a57942cd2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Oct 30 12:11:42.511126 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:11:42.511088 4908 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-b2fvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Oct 30 12:11:42.511522 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:11:42.511156 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-b2fvc" podUID="486f71e9-0730-4dbc-b456-31a57942cd2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Oct 30 12:11:43.510809 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:11:43.510778 4908 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-b2fvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Oct 30 12:11:43.510992 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:11:43.510858 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-b2fvc" podUID="486f71e9-0730-4dbc-b456-31a57942cd2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Oct 30 12:11:44.511441 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:11:44.511409 4908 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-b2fvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Oct 30 12:11:44.511851 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:11:44.511471 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-b2fvc" podUID="486f71e9-0730-4dbc-b456-31a57942cd2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Oct 30 12:11:45.511226 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:11:45.511194 4908 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-b2fvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Oct 30 12:11:45.511398 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:11:45.511243 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-b2fvc" podUID="486f71e9-0730-4dbc-b456-31a57942cd2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Oct 30 12:11:46.511448 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:11:46.511411 4908 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-b2fvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Oct 30 12:11:46.511874 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:11:46.511467 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-b2fvc" podUID="486f71e9-0730-4dbc-b456-31a57942cd2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Oct 30 12:11:47.514548 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:11:47.514516 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-b2fvc" Oct 30 12:11:47.514978 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:11:47.514760 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-b2fvc" Oct 30 12:11:47.515484 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:11:47.515467 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-b2fvc" Oct 30 12:11:49.683138 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:11:49.683103 4908 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vn6vv" containerName="extractor" Oct 30 12:11:55.225319 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:11:55.225285 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-sn5mv"] Oct 30 12:11:55.229437 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:11:55.229413 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-sn5mv" Oct 30 12:11:55.257069 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:11:55.257040 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-sn5mv"] Oct 30 12:11:55.322808 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:11:55.322768 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2806f1c-03ec-42ee-a217-040d43646d9d-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-sn5mv\" (UID: \"a2806f1c-03ec-42ee-a217-040d43646d9d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-sn5mv" Oct 30 12:11:55.322808 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:11:55.322809 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cvvg\" (UniqueName: \"kubernetes.io/projected/a2806f1c-03ec-42ee-a217-040d43646d9d-kube-api-access-5cvvg\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-sn5mv\" (UID: \"a2806f1c-03ec-42ee-a217-040d43646d9d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-sn5mv" Oct 30 12:11:55.323018 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:11:55.322921 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2806f1c-03ec-42ee-a217-040d43646d9d-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-sn5mv\" (UID: \"a2806f1c-03ec-42ee-a217-040d43646d9d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-sn5mv" Oct 30 12:11:55.323018 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:11:55.322945 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2806f1c-03ec-42ee-a217-040d43646d9d-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-sn5mv\" (UID: \"a2806f1c-03ec-42ee-a217-040d43646d9d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-sn5mv" Oct 30 12:11:55.323018 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:11:55.322973 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2806f1c-03ec-42ee-a217-040d43646d9d-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-sn5mv\" (UID: \"a2806f1c-03ec-42ee-a217-040d43646d9d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-sn5mv" Oct 30 12:11:55.323144 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:11:55.323037 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2806f1c-03ec-42ee-a217-040d43646d9d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-sn5mv\" (UID: \"a2806f1c-03ec-42ee-a217-040d43646d9d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-sn5mv" Oct 30 12:11:55.323180 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:11:55.323153 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2806f1c-03ec-42ee-a217-040d43646d9d-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-sn5mv\" (UID: \"a2806f1c-03ec-42ee-a217-040d43646d9d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-sn5mv" Oct 30 12:11:55.323180 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:11:55.323173 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2806f1c-03ec-42ee-a217-040d43646d9d-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-sn5mv\" (UID: \"a2806f1c-03ec-42ee-a217-040d43646d9d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-sn5mv" Oct 30 12:11:55.323251 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:11:55.323239 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2806f1c-03ec-42ee-a217-040d43646d9d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-sn5mv\" (UID: \"a2806f1c-03ec-42ee-a217-040d43646d9d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-sn5mv" Oct 30 12:11:55.424124 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:11:55.424095 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2806f1c-03ec-42ee-a217-040d43646d9d-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-sn5mv\" (UID: \"a2806f1c-03ec-42ee-a217-040d43646d9d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-sn5mv" Oct 30 12:11:55.424286 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:11:55.424143 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2806f1c-03ec-42ee-a217-040d43646d9d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-sn5mv\" (UID: \"a2806f1c-03ec-42ee-a217-040d43646d9d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-sn5mv" Oct 30 12:11:55.424286 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:11:55.424182 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2806f1c-03ec-42ee-a217-040d43646d9d-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-sn5mv\" (UID: \"a2806f1c-03ec-42ee-a217-040d43646d9d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-sn5mv" Oct 30 12:11:55.424286 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:11:55.424198 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cvvg\" (UniqueName: \"kubernetes.io/projected/a2806f1c-03ec-42ee-a217-040d43646d9d-kube-api-access-5cvvg\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-sn5mv\" (UID: \"a2806f1c-03ec-42ee-a217-040d43646d9d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-sn5mv" Oct 30 12:11:55.424286 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:11:55.424224 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2806f1c-03ec-42ee-a217-040d43646d9d-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-sn5mv\" (UID: \"a2806f1c-03ec-42ee-a217-040d43646d9d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-sn5mv" Oct 30 12:11:55.424286 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:11:55.424244 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2806f1c-03ec-42ee-a217-040d43646d9d-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-sn5mv\" (UID: \"a2806f1c-03ec-42ee-a217-040d43646d9d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-sn5mv" Oct 30 12:11:55.424286 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:11:55.424277 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2806f1c-03ec-42ee-a217-040d43646d9d-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-sn5mv\" (UID: \"a2806f1c-03ec-42ee-a217-040d43646d9d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-sn5mv" Oct 30 12:11:55.424533 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:11:55.424309 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2806f1c-03ec-42ee-a217-040d43646d9d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-sn5mv\" (UID: \"a2806f1c-03ec-42ee-a217-040d43646d9d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-sn5mv" Oct 30 12:11:55.424533 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:11:55.424350 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2806f1c-03ec-42ee-a217-040d43646d9d-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-sn5mv\" (UID: \"a2806f1c-03ec-42ee-a217-040d43646d9d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-sn5mv" Oct 30 12:11:55.424533 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:11:55.424521 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2806f1c-03ec-42ee-a217-040d43646d9d-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-sn5mv\" (UID: \"a2806f1c-03ec-42ee-a217-040d43646d9d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-sn5mv" Oct 30 12:11:55.424821 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:11:55.424794 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2806f1c-03ec-42ee-a217-040d43646d9d-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-sn5mv\" (UID: \"a2806f1c-03ec-42ee-a217-040d43646d9d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-sn5mv" Oct 30 12:11:55.424821 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:11:55.424846 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2806f1c-03ec-42ee-a217-040d43646d9d-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-sn5mv\" (UID: \"a2806f1c-03ec-42ee-a217-040d43646d9d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-sn5mv" Oct 30 12:11:55.425013 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:11:55.424943 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2806f1c-03ec-42ee-a217-040d43646d9d-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-sn5mv\" (UID: \"a2806f1c-03ec-42ee-a217-040d43646d9d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-sn5mv" Oct 30 12:11:55.425057 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:11:55.425039 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2806f1c-03ec-42ee-a217-040d43646d9d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-sn5mv\" (UID: \"a2806f1c-03ec-42ee-a217-040d43646d9d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-sn5mv" Oct 30 12:11:55.426353 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:11:55.426334 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2806f1c-03ec-42ee-a217-040d43646d9d-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-sn5mv\" (UID: \"a2806f1c-03ec-42ee-a217-040d43646d9d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-sn5mv" Oct 30 12:11:55.426643 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:11:55.426624 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2806f1c-03ec-42ee-a217-040d43646d9d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-sn5mv\" (UID: \"a2806f1c-03ec-42ee-a217-040d43646d9d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-sn5mv" Oct 30 12:11:55.446257 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:11:55.446229 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2806f1c-03ec-42ee-a217-040d43646d9d-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-sn5mv\" (UID: \"a2806f1c-03ec-42ee-a217-040d43646d9d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-sn5mv" Oct 30 12:11:55.446423 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:11:55.446401 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cvvg\" (UniqueName: \"kubernetes.io/projected/a2806f1c-03ec-42ee-a217-040d43646d9d-kube-api-access-5cvvg\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-sn5mv\" (UID: \"a2806f1c-03ec-42ee-a217-040d43646d9d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-sn5mv" Oct 30 12:11:55.542772 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:11:55.542737 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-sn5mv" Oct 30 12:11:55.856285 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:11:55.856215 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-sn5mv"] Oct 30 12:11:55.859344 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:11:55.859320 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2806f1c_03ec_42ee_a217_040d43646d9d.slice/crio-1ba77a0ec9b6f5b4f99eecb54722df6c5b687f7c0b988c9f265efd879351abb5 WatchSource:0}: Error finding container 1ba77a0ec9b6f5b4f99eecb54722df6c5b687f7c0b988c9f265efd879351abb5: Status 404 returned error can't find the container with id 1ba77a0ec9b6f5b4f99eecb54722df6c5b687f7c0b988c9f265efd879351abb5 Oct 30 12:11:55.861444 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:11:55.861412 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:11:55.861535 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:11:55.861490 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:11:55.861596 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:11:55.861537 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:11:56.302450 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:11:56.302411 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-sn5mv" event={"ID":"a2806f1c-03ec-42ee-a217-040d43646d9d","Type":"ContainerStarted","Data":"89a39b6d77b062b4f7662480aab1f806c19ab69a85a13fdff103025375f3876e"} Oct 30 12:11:56.302966 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:11:56.302471 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-sn5mv" event={"ID":"a2806f1c-03ec-42ee-a217-040d43646d9d","Type":"ContainerStarted","Data":"1ba77a0ec9b6f5b4f99eecb54722df6c5b687f7c0b988c9f265efd879351abb5"} Oct 30 12:11:56.378853 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:11:56.369425 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-sn5mv" podStartSLOduration=1.369403559 podStartE2EDuration="1.369403559s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:11:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:11:56.356615466 +0000 UTC m=+1072.871044235" watchObservedRunningTime="2026-10-30 12:11:56.369403559 +0000 UTC m=+1072.883832305" Oct 30 12:11:56.543634 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:11:56.543601 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-sn5mv" Oct 30 12:11:56.544754 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:11:56.544733 4908 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-sn5mv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Oct 30 12:11:56.544822 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:11:56.544778 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-sn5mv" podUID="a2806f1c-03ec-42ee-a217-040d43646d9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Oct 30 12:11:57.543751 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:11:57.543715 4908 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-sn5mv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Oct 30 12:11:57.544236 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:11:57.543773 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-sn5mv" podUID="a2806f1c-03ec-42ee-a217-040d43646d9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Oct 30 12:11:58.543784 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:11:58.543751 4908 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-sn5mv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Oct 30 12:11:58.544196 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:11:58.543800 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-sn5mv" podUID="a2806f1c-03ec-42ee-a217-040d43646d9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Oct 30 12:11:59.543352 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:11:59.543318 4908 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-sn5mv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Oct 30 12:11:59.543530 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:11:59.543369 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-sn5mv" podUID="a2806f1c-03ec-42ee-a217-040d43646d9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Oct 30 12:12:00.543864 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:12:00.543817 4908 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-sn5mv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Oct 30 12:12:00.544315 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:12:00.543892 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-sn5mv" podUID="a2806f1c-03ec-42ee-a217-040d43646d9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Oct 30 12:12:01.543303 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:12:01.543264 4908 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-sn5mv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Oct 30 12:12:01.543483 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:12:01.543320 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-sn5mv" podUID="a2806f1c-03ec-42ee-a217-040d43646d9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Oct 30 12:12:02.546809 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:12:02.546778 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-sn5mv" Oct 30 12:12:02.547203 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:12:02.547055 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-sn5mv" Oct 30 12:12:02.547743 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:12:02.547726 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-sn5mv" Oct 30 12:12:15.011234 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:12:15.011197 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9wwfc"] Oct 30 12:12:19.682734 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:12:19.682694 4908 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vn6vv" containerName="extractor" Oct 30 12:12:20.633371 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:12:20.633333 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9wwfc"] Oct 30 12:12:25.531941 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:12:25.531908 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9wwfc"] Oct 30 12:12:33.592564 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:12:33.592528 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9wwfc"] Oct 30 12:12:49.682337 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:12:49.682252 4908 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vn6vv" containerName="extractor" Oct 30 12:13:19.682195 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:13:19.682157 4908 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vn6vv" containerName="extractor" Oct 30 12:13:31.834215 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:13:31.834180 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9wwfc"] Oct 30 12:13:40.074679 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:13:40.074642 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-307283d2-4m29n"] Oct 30 12:13:40.079032 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:13:40.079014 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-307283d2-4m29n" Oct 30 12:13:40.093697 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:13:40.093672 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 12:13:40.104737 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:13:40.104711 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 12:13:40.104868 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:13:40.104758 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 12:13:40.104936 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:13:40.104881 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s75hq\"" Oct 30 12:13:40.107423 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:13:40.107400 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-307283d2-4m29n"] Oct 30 12:13:40.205872 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:13:40.205814 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e23ac155-6e41-458e-97d3-83e9cc24c03e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-307283d2-4m29n\" (UID: \"e23ac155-6e41-458e-97d3-83e9cc24c03e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-307283d2-4m29n" Oct 30 12:13:40.206017 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:13:40.205881 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgztj\" (UniqueName: \"kubernetes.io/projected/e23ac155-6e41-458e-97d3-83e9cc24c03e-kube-api-access-dgztj\") pod \"maas-api-revoke-keys-e2e-migrate-307283d2-4m29n\" (UID: \"e23ac155-6e41-458e-97d3-83e9cc24c03e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-307283d2-4m29n" Oct 30 12:13:40.206017 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:13:40.205992 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e23ac155-6e41-458e-97d3-83e9cc24c03e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-307283d2-4m29n\" (UID: \"e23ac155-6e41-458e-97d3-83e9cc24c03e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-307283d2-4m29n" Oct 30 12:13:40.307009 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:13:40.306983 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e23ac155-6e41-458e-97d3-83e9cc24c03e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-307283d2-4m29n\" (UID: \"e23ac155-6e41-458e-97d3-83e9cc24c03e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-307283d2-4m29n" Oct 30 12:13:40.307133 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:13:40.307023 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgztj\" (UniqueName: \"kubernetes.io/projected/e23ac155-6e41-458e-97d3-83e9cc24c03e-kube-api-access-dgztj\") pod \"maas-api-revoke-keys-e2e-migrate-307283d2-4m29n\" (UID: \"e23ac155-6e41-458e-97d3-83e9cc24c03e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-307283d2-4m29n" Oct 30 12:13:40.307133 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:13:40.307047 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e23ac155-6e41-458e-97d3-83e9cc24c03e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-307283d2-4m29n\" (UID: \"e23ac155-6e41-458e-97d3-83e9cc24c03e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-307283d2-4m29n" Oct 30 12:13:40.307606 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:13:40.307586 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e23ac155-6e41-458e-97d3-83e9cc24c03e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-307283d2-4m29n\" (UID: \"e23ac155-6e41-458e-97d3-83e9cc24c03e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-307283d2-4m29n" Oct 30 12:13:40.309381 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:13:40.309359 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e23ac155-6e41-458e-97d3-83e9cc24c03e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-307283d2-4m29n\" (UID: \"e23ac155-6e41-458e-97d3-83e9cc24c03e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-307283d2-4m29n" Oct 30 12:13:40.329004 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:13:40.328955 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgztj\" (UniqueName: \"kubernetes.io/projected/e23ac155-6e41-458e-97d3-83e9cc24c03e-kube-api-access-dgztj\") pod \"maas-api-revoke-keys-e2e-migrate-307283d2-4m29n\" (UID: \"e23ac155-6e41-458e-97d3-83e9cc24c03e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-307283d2-4m29n" Oct 30 12:13:40.392017 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:13:40.391994 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-307283d2-4m29n" Oct 30 12:13:40.689170 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:13:40.689093 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-307283d2-4m29n"] Oct 30 12:13:40.693240 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:13:40.693213 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode23ac155_6e41_458e_97d3_83e9cc24c03e.slice/crio-803f87d71162d377fa6782e1aafa2109519dc288f09f107681b03ab3185063ef WatchSource:0}: Error finding container 803f87d71162d377fa6782e1aafa2109519dc288f09f107681b03ab3185063ef: Status 404 returned error can't find the container with id 803f87d71162d377fa6782e1aafa2109519dc288f09f107681b03ab3185063ef Oct 30 12:13:40.913973 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:13:40.913933 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-307283d2-4m29n" event={"ID":"e23ac155-6e41-458e-97d3-83e9cc24c03e","Type":"ContainerStarted","Data":"803f87d71162d377fa6782e1aafa2109519dc288f09f107681b03ab3185063ef"} Oct 30 12:13:41.046341 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:13:41.046303 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9wwfc"] Oct 30 12:13:43.935296 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:13:43.935266 4908 generic.go:356] "Generic (PLEG): container finished" podID="e23ac155-6e41-458e-97d3-83e9cc24c03e" containerID="d49837e4250ee2677404f86d1053b22de1e8df5ea0d5a7b693608d931120b6bc" exitCode=0 Oct 30 12:13:43.935696 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:13:43.935322 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-307283d2-4m29n" event={"ID":"e23ac155-6e41-458e-97d3-83e9cc24c03e","Type":"ContainerDied","Data":"d49837e4250ee2677404f86d1053b22de1e8df5ea0d5a7b693608d931120b6bc"} Oct 30 12:13:45.074586 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:13:45.074562 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-307283d2-4m29n" Oct 30 12:13:45.156408 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:13:45.156379 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e23ac155-6e41-458e-97d3-83e9cc24c03e-kube-api-access-dgztj\" (UniqueName: \"kubernetes.io/projected/e23ac155-6e41-458e-97d3-83e9cc24c03e-kube-api-access-dgztj\") pod \"e23ac155-6e41-458e-97d3-83e9cc24c03e\" (UID: \"e23ac155-6e41-458e-97d3-83e9cc24c03e\") " Oct 30 12:13:45.156552 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:13:45.156428 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e23ac155-6e41-458e-97d3-83e9cc24c03e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e23ac155-6e41-458e-97d3-83e9cc24c03e-maas-api-openshift-ca\") pod \"e23ac155-6e41-458e-97d3-83e9cc24c03e\" (UID: \"e23ac155-6e41-458e-97d3-83e9cc24c03e\") " Oct 30 12:13:45.156552 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:13:45.156461 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e23ac155-6e41-458e-97d3-83e9cc24c03e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e23ac155-6e41-458e-97d3-83e9cc24c03e-maas-api-xks-ca\") pod \"e23ac155-6e41-458e-97d3-83e9cc24c03e\" (UID: \"e23ac155-6e41-458e-97d3-83e9cc24c03e\") " Oct 30 12:13:45.156815 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:13:45.156784 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e23ac155-6e41-458e-97d3-83e9cc24c03e-maas-api-openshift-ca" pod "e23ac155-6e41-458e-97d3-83e9cc24c03e" (UID: "e23ac155-6e41-458e-97d3-83e9cc24c03e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:13:45.158564 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:13:45.158542 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e23ac155-6e41-458e-97d3-83e9cc24c03e-maas-api-xks-ca" pod "e23ac155-6e41-458e-97d3-83e9cc24c03e" (UID: "e23ac155-6e41-458e-97d3-83e9cc24c03e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:13:45.158648 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:13:45.158601 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e23ac155-6e41-458e-97d3-83e9cc24c03e-kube-api-access-dgztj" pod "e23ac155-6e41-458e-97d3-83e9cc24c03e" (UID: "e23ac155-6e41-458e-97d3-83e9cc24c03e"). InnerVolumeSpecName "kube-api-access-dgztj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:13:45.257655 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:13:45.257630 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgztj\" (UniqueName: \"kubernetes.io/projected/e23ac155-6e41-458e-97d3-83e9cc24c03e-kube-api-access-dgztj\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:13:45.257655 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:13:45.257656 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e23ac155-6e41-458e-97d3-83e9cc24c03e-maas-api-openshift-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:13:45.257885 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:13:45.257666 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e23ac155-6e41-458e-97d3-83e9cc24c03e-maas-api-xks-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:13:45.947114 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:13:45.947044 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-307283d2-4m29n" Oct 30 12:13:45.947114 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:13:45.947069 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-307283d2-4m29n" event={"ID":"e23ac155-6e41-458e-97d3-83e9cc24c03e","Type":"ContainerDied","Data":"803f87d71162d377fa6782e1aafa2109519dc288f09f107681b03ab3185063ef"} Oct 30 12:13:45.947114 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:13:45.947110 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="803f87d71162d377fa6782e1aafa2109519dc288f09f107681b03ab3185063ef" Oct 30 12:13:46.171659 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:13:46.171620 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9wwfc"] Oct 30 12:13:46.880776 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:13:46.880742 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c85bd66-trnpj"] Oct 30 12:13:46.881203 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:13:46.881189 4908 cpu_manager.go:414] "containerMap: removing container" podUID="e23ac155-6e41-458e-97d3-83e9cc24c03e" containerName="revoke-keys" Oct 30 12:13:46.881251 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:13:46.881205 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e23ac155-6e41-458e-97d3-83e9cc24c03e" containerName="revoke-keys" Oct 30 12:13:46.881286 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:13:46.881270 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="e23ac155-6e41-458e-97d3-83e9cc24c03e" containerName="revoke-keys" Oct 30 12:13:46.884435 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:13:46.884414 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c85bd66-trnpj" Oct 30 12:13:46.901760 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:13:46.901742 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s75hq\"" Oct 30 12:13:46.904355 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:13:46.904332 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c85bd66-trnpj"] Oct 30 12:13:46.912958 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:13:46.912939 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 12:13:46.914361 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:13:46.914343 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 12:13:46.914664 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:13:46.914642 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 12:13:46.975616 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:13:46.975581 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04d78a3d-786c-4aa7-87f4-c5789b2046d9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-1c85bd66-trnpj\" (UID: \"04d78a3d-786c-4aa7-87f4-c5789b2046d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c85bd66-trnpj" Oct 30 12:13:46.975860 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:13:46.975824 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04d78a3d-786c-4aa7-87f4-c5789b2046d9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-1c85bd66-trnpj\" (UID: \"04d78a3d-786c-4aa7-87f4-c5789b2046d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c85bd66-trnpj" Oct 30 12:13:46.976214 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:13:46.976194 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4hx9\" (UniqueName: \"kubernetes.io/projected/04d78a3d-786c-4aa7-87f4-c5789b2046d9-kube-api-access-n4hx9\") pod \"maas-api-revoke-keys-e2e-ait-1c85bd66-trnpj\" (UID: \"04d78a3d-786c-4aa7-87f4-c5789b2046d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c85bd66-trnpj" Oct 30 12:13:47.076911 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:13:47.076883 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04d78a3d-786c-4aa7-87f4-c5789b2046d9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-1c85bd66-trnpj\" (UID: \"04d78a3d-786c-4aa7-87f4-c5789b2046d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c85bd66-trnpj" Oct 30 12:13:47.076911 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:13:47.076917 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04d78a3d-786c-4aa7-87f4-c5789b2046d9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-1c85bd66-trnpj\" (UID: \"04d78a3d-786c-4aa7-87f4-c5789b2046d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c85bd66-trnpj" Oct 30 12:13:47.077123 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:13:47.076939 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4hx9\" (UniqueName: \"kubernetes.io/projected/04d78a3d-786c-4aa7-87f4-c5789b2046d9-kube-api-access-n4hx9\") pod \"maas-api-revoke-keys-e2e-ait-1c85bd66-trnpj\" (UID: \"04d78a3d-786c-4aa7-87f4-c5789b2046d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c85bd66-trnpj" Oct 30 12:13:47.077584 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:13:47.077557 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04d78a3d-786c-4aa7-87f4-c5789b2046d9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-1c85bd66-trnpj\" (UID: \"04d78a3d-786c-4aa7-87f4-c5789b2046d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c85bd66-trnpj" Oct 30 12:13:47.079561 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:13:47.079525 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04d78a3d-786c-4aa7-87f4-c5789b2046d9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-1c85bd66-trnpj\" (UID: \"04d78a3d-786c-4aa7-87f4-c5789b2046d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c85bd66-trnpj" Oct 30 12:13:47.106224 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:13:47.106200 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4hx9\" (UniqueName: \"kubernetes.io/projected/04d78a3d-786c-4aa7-87f4-c5789b2046d9-kube-api-access-n4hx9\") pod \"maas-api-revoke-keys-e2e-ait-1c85bd66-trnpj\" (UID: \"04d78a3d-786c-4aa7-87f4-c5789b2046d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c85bd66-trnpj" Oct 30 12:13:47.197281 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:13:47.197215 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c85bd66-trnpj" Oct 30 12:13:47.505503 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:13:47.505481 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c85bd66-trnpj"] Oct 30 12:13:47.507671 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:13:47.507628 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04d78a3d_786c_4aa7_87f4_c5789b2046d9.slice/crio-13cb03adcb3f8ba481c413ffee744600493e2681eb63bf2b03068074b4eaedd9 WatchSource:0}: Error finding container 13cb03adcb3f8ba481c413ffee744600493e2681eb63bf2b03068074b4eaedd9: Status 404 returned error can't find the container with id 13cb03adcb3f8ba481c413ffee744600493e2681eb63bf2b03068074b4eaedd9 Oct 30 12:13:47.958155 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:13:47.958077 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c85bd66-trnpj" event={"ID":"04d78a3d-786c-4aa7-87f4-c5789b2046d9","Type":"ContainerStarted","Data":"7d3ba06bae620988613653f8cd2c0a3682cccec5434f6a132a1e7f7cdb944915"} Oct 30 12:13:47.958155 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:13:47.958137 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c85bd66-trnpj" event={"ID":"04d78a3d-786c-4aa7-87f4-c5789b2046d9","Type":"ContainerStarted","Data":"13cb03adcb3f8ba481c413ffee744600493e2681eb63bf2b03068074b4eaedd9"} Oct 30 12:13:47.998153 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:13:47.998094 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c85bd66-trnpj" podStartSLOduration=1.998075896 podStartE2EDuration="1.998075896s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:13:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:13:47.99537352 +0000 UTC m=+1184.509802265" watchObservedRunningTime="2026-10-30 12:13:47.998075896 +0000 UTC m=+1184.512504635" Oct 30 12:13:48.967460 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:13:48.967425 4908 generic.go:356] "Generic (PLEG): container finished" podID="04d78a3d-786c-4aa7-87f4-c5789b2046d9" containerID="7d3ba06bae620988613653f8cd2c0a3682cccec5434f6a132a1e7f7cdb944915" exitCode=0 Oct 30 12:13:48.967850 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:13:48.967504 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c85bd66-trnpj" event={"ID":"04d78a3d-786c-4aa7-87f4-c5789b2046d9","Type":"ContainerDied","Data":"7d3ba06bae620988613653f8cd2c0a3682cccec5434f6a132a1e7f7cdb944915"} Oct 30 12:13:49.682742 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:13:49.682707 4908 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vn6vv" containerName="extractor" Oct 30 12:13:50.104765 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:13:50.104737 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c85bd66-trnpj" Oct 30 12:13:50.204273 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:13:50.204243 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/04d78a3d-786c-4aa7-87f4-c5789b2046d9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04d78a3d-786c-4aa7-87f4-c5789b2046d9-maas-api-xks-ca\") pod \"04d78a3d-786c-4aa7-87f4-c5789b2046d9\" (UID: \"04d78a3d-786c-4aa7-87f4-c5789b2046d9\") " Oct 30 12:13:50.204425 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:13:50.204297 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04d78a3d-786c-4aa7-87f4-c5789b2046d9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04d78a3d-786c-4aa7-87f4-c5789b2046d9-maas-api-openshift-ca\") pod \"04d78a3d-786c-4aa7-87f4-c5789b2046d9\" (UID: \"04d78a3d-786c-4aa7-87f4-c5789b2046d9\") " Oct 30 12:13:50.204476 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:13:50.204428 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04d78a3d-786c-4aa7-87f4-c5789b2046d9-kube-api-access-n4hx9\" (UniqueName: \"kubernetes.io/projected/04d78a3d-786c-4aa7-87f4-c5789b2046d9-kube-api-access-n4hx9\") pod \"04d78a3d-786c-4aa7-87f4-c5789b2046d9\" (UID: \"04d78a3d-786c-4aa7-87f4-c5789b2046d9\") " Oct 30 12:13:50.204652 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:13:50.204628 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04d78a3d-786c-4aa7-87f4-c5789b2046d9-maas-api-openshift-ca" pod "04d78a3d-786c-4aa7-87f4-c5789b2046d9" (UID: "04d78a3d-786c-4aa7-87f4-c5789b2046d9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:13:50.206244 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:13:50.206212 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04d78a3d-786c-4aa7-87f4-c5789b2046d9-maas-api-xks-ca" pod "04d78a3d-786c-4aa7-87f4-c5789b2046d9" (UID: "04d78a3d-786c-4aa7-87f4-c5789b2046d9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:13:50.206350 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:13:50.206326 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04d78a3d-786c-4aa7-87f4-c5789b2046d9-kube-api-access-n4hx9" pod "04d78a3d-786c-4aa7-87f4-c5789b2046d9" (UID: "04d78a3d-786c-4aa7-87f4-c5789b2046d9"). InnerVolumeSpecName "kube-api-access-n4hx9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:13:50.305653 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:13:50.305631 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4hx9\" (UniqueName: \"kubernetes.io/projected/04d78a3d-786c-4aa7-87f4-c5789b2046d9-kube-api-access-n4hx9\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:13:50.305653 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:13:50.305652 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04d78a3d-786c-4aa7-87f4-c5789b2046d9-maas-api-xks-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:13:50.305781 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:13:50.305661 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04d78a3d-786c-4aa7-87f4-c5789b2046d9-maas-api-openshift-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:13:50.678824 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:13:50.678740 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-80143318-4slg8"] Oct 30 12:13:50.679156 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:13:50.679143 4908 cpu_manager.go:414] "containerMap: removing container" podUID="04d78a3d-786c-4aa7-87f4-c5789b2046d9" containerName="revoke-keys" Oct 30 12:13:50.679203 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:13:50.679157 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04d78a3d-786c-4aa7-87f4-c5789b2046d9" containerName="revoke-keys" Oct 30 12:13:50.679249 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:13:50.679241 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="04d78a3d-786c-4aa7-87f4-c5789b2046d9" containerName="revoke-keys" Oct 30 12:13:50.682373 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:13:50.682357 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-80143318-4slg8" Oct 30 12:13:50.700393 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:13:50.700372 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-80143318-4slg8"] Oct 30 12:13:50.810683 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:13:50.810649 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbsqd\" (UniqueName: \"kubernetes.io/projected/3f070e57-e304-4c20-b996-c3f428f8ef08-kube-api-access-hbsqd\") pod \"maas-api-revoke-keys-e2e-ait-80143318-4slg8\" (UID: \"3f070e57-e304-4c20-b996-c3f428f8ef08\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-80143318-4slg8" Oct 30 12:13:50.810905 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:13:50.810724 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f070e57-e304-4c20-b996-c3f428f8ef08-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-80143318-4slg8\" (UID: \"3f070e57-e304-4c20-b996-c3f428f8ef08\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-80143318-4slg8" Oct 30 12:13:50.810905 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:13:50.810817 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f070e57-e304-4c20-b996-c3f428f8ef08-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-80143318-4slg8\" (UID: \"3f070e57-e304-4c20-b996-c3f428f8ef08\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-80143318-4slg8" Oct 30 12:13:50.911711 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:13:50.911684 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f070e57-e304-4c20-b996-c3f428f8ef08-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-80143318-4slg8\" (UID: \"3f070e57-e304-4c20-b996-c3f428f8ef08\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-80143318-4slg8" Oct 30 12:13:50.911862 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:13:50.911720 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f070e57-e304-4c20-b996-c3f428f8ef08-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-80143318-4slg8\" (UID: \"3f070e57-e304-4c20-b996-c3f428f8ef08\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-80143318-4slg8" Oct 30 12:13:50.911862 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:13:50.911762 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbsqd\" (UniqueName: \"kubernetes.io/projected/3f070e57-e304-4c20-b996-c3f428f8ef08-kube-api-access-hbsqd\") pod \"maas-api-revoke-keys-e2e-ait-80143318-4slg8\" (UID: \"3f070e57-e304-4c20-b996-c3f428f8ef08\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-80143318-4slg8" Oct 30 12:13:50.912371 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:13:50.912353 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f070e57-e304-4c20-b996-c3f428f8ef08-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-80143318-4slg8\" (UID: \"3f070e57-e304-4c20-b996-c3f428f8ef08\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-80143318-4slg8" Oct 30 12:13:50.913939 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:13:50.913914 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f070e57-e304-4c20-b996-c3f428f8ef08-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-80143318-4slg8\" (UID: \"3f070e57-e304-4c20-b996-c3f428f8ef08\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-80143318-4slg8" Oct 30 12:13:50.933471 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:13:50.933418 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbsqd\" (UniqueName: \"kubernetes.io/projected/3f070e57-e304-4c20-b996-c3f428f8ef08-kube-api-access-hbsqd\") pod \"maas-api-revoke-keys-e2e-ait-80143318-4slg8\" (UID: \"3f070e57-e304-4c20-b996-c3f428f8ef08\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-80143318-4slg8" Oct 30 12:13:50.978889 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:13:50.978868 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c85bd66-trnpj" Oct 30 12:13:50.978889 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:13:50.978876 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c85bd66-trnpj" event={"ID":"04d78a3d-786c-4aa7-87f4-c5789b2046d9","Type":"ContainerDied","Data":"13cb03adcb3f8ba481c413ffee744600493e2681eb63bf2b03068074b4eaedd9"} Oct 30 12:13:50.979016 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:13:50.978919 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="13cb03adcb3f8ba481c413ffee744600493e2681eb63bf2b03068074b4eaedd9" Oct 30 12:13:50.998272 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:13:50.998251 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-80143318-4slg8" Oct 30 12:13:51.295403 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:13:51.295374 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-80143318-4slg8"] Oct 30 12:13:51.297886 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:13:51.297858 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f070e57_e304_4c20_b996_c3f428f8ef08.slice/crio-81338a0b33926aec2180d5a92f414a82a90e37eac7bbc756ef5517cefa891104 WatchSource:0}: Error finding container 81338a0b33926aec2180d5a92f414a82a90e37eac7bbc756ef5517cefa891104: Status 404 returned error can't find the container with id 81338a0b33926aec2180d5a92f414a82a90e37eac7bbc756ef5517cefa891104 Oct 30 12:13:51.990126 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:13:51.990077 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-80143318-4slg8" event={"ID":"3f070e57-e304-4c20-b996-c3f428f8ef08","Type":"ContainerStarted","Data":"76b21b36f8f7a30e6f4f9d0c0616e068c21fb97c98ed0fd4c372a071dc8172e9"} Oct 30 12:13:51.990311 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:13:51.990154 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-80143318-4slg8" event={"ID":"3f070e57-e304-4c20-b996-c3f428f8ef08","Type":"ContainerStarted","Data":"81338a0b33926aec2180d5a92f414a82a90e37eac7bbc756ef5517cefa891104"} Oct 30 12:13:52.028935 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:13:52.028889 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-80143318-4slg8" podStartSLOduration=2.028875903 podStartE2EDuration="2.028875903s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:13:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:13:52.027802242 +0000 UTC m=+1188.542230988" watchObservedRunningTime="2026-10-30 12:13:52.028875903 +0000 UTC m=+1188.543304646" Oct 30 12:13:52.997688 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:13:52.997655 4908 generic.go:356] "Generic (PLEG): container finished" podID="3f070e57-e304-4c20-b996-c3f428f8ef08" containerID="76b21b36f8f7a30e6f4f9d0c0616e068c21fb97c98ed0fd4c372a071dc8172e9" exitCode=0 Oct 30 12:13:52.998079 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:13:52.997699 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-80143318-4slg8" event={"ID":"3f070e57-e304-4c20-b996-c3f428f8ef08","Type":"ContainerDied","Data":"76b21b36f8f7a30e6f4f9d0c0616e068c21fb97c98ed0fd4c372a071dc8172e9"} Oct 30 12:13:53.216614 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:13:53.216583 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e636da41-7m7hr"] Oct 30 12:13:53.220358 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:13:53.220339 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e636da41-7m7hr" Oct 30 12:13:53.238838 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:13:53.238784 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e636da41-7m7hr"] Oct 30 12:13:53.336151 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:13:53.336068 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5a9c296-07a4-4fe2-8a5c-593b799bd8fa-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-e636da41-7m7hr\" (UID: \"d5a9c296-07a4-4fe2-8a5c-593b799bd8fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e636da41-7m7hr" Oct 30 12:13:53.336151 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:13:53.336104 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n88d8\" (UniqueName: \"kubernetes.io/projected/d5a9c296-07a4-4fe2-8a5c-593b799bd8fa-kube-api-access-n88d8\") pod \"maas-api-revoke-keys-e2e-ait-e636da41-7m7hr\" (UID: \"d5a9c296-07a4-4fe2-8a5c-593b799bd8fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e636da41-7m7hr" Oct 30 12:13:53.336151 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:13:53.336143 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5a9c296-07a4-4fe2-8a5c-593b799bd8fa-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-e636da41-7m7hr\" (UID: \"d5a9c296-07a4-4fe2-8a5c-593b799bd8fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e636da41-7m7hr" Oct 30 12:13:53.436933 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:13:53.436899 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5a9c296-07a4-4fe2-8a5c-593b799bd8fa-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-e636da41-7m7hr\" (UID: \"d5a9c296-07a4-4fe2-8a5c-593b799bd8fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e636da41-7m7hr" Oct 30 12:13:53.437074 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:13:53.436941 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n88d8\" (UniqueName: \"kubernetes.io/projected/d5a9c296-07a4-4fe2-8a5c-593b799bd8fa-kube-api-access-n88d8\") pod \"maas-api-revoke-keys-e2e-ait-e636da41-7m7hr\" (UID: \"d5a9c296-07a4-4fe2-8a5c-593b799bd8fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e636da41-7m7hr" Oct 30 12:13:53.437074 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:13:53.436997 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5a9c296-07a4-4fe2-8a5c-593b799bd8fa-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-e636da41-7m7hr\" (UID: \"d5a9c296-07a4-4fe2-8a5c-593b799bd8fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e636da41-7m7hr" Oct 30 12:13:53.437657 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:13:53.437632 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5a9c296-07a4-4fe2-8a5c-593b799bd8fa-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-e636da41-7m7hr\" (UID: \"d5a9c296-07a4-4fe2-8a5c-593b799bd8fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e636da41-7m7hr" Oct 30 12:13:53.439288 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:13:53.439265 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5a9c296-07a4-4fe2-8a5c-593b799bd8fa-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-e636da41-7m7hr\" (UID: \"d5a9c296-07a4-4fe2-8a5c-593b799bd8fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e636da41-7m7hr" Oct 30 12:13:53.458657 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:13:53.458630 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n88d8\" (UniqueName: \"kubernetes.io/projected/d5a9c296-07a4-4fe2-8a5c-593b799bd8fa-kube-api-access-n88d8\") pod \"maas-api-revoke-keys-e2e-ait-e636da41-7m7hr\" (UID: \"d5a9c296-07a4-4fe2-8a5c-593b799bd8fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e636da41-7m7hr" Oct 30 12:13:53.533397 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:13:53.533374 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e636da41-7m7hr" Oct 30 12:13:53.831124 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:13:53.831098 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e636da41-7m7hr"] Oct 30 12:13:53.834523 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:13:53.834499 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5a9c296_07a4_4fe2_8a5c_593b799bd8fa.slice/crio-852d05301442ae7c516ef0405a5dd2783fe1b3d65c7b3ad2e2b578807c873efe WatchSource:0}: Error finding container 852d05301442ae7c516ef0405a5dd2783fe1b3d65c7b3ad2e2b578807c873efe: Status 404 returned error can't find the container with id 852d05301442ae7c516ef0405a5dd2783fe1b3d65c7b3ad2e2b578807c873efe Oct 30 12:13:54.004022 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:13:54.003993 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e636da41-7m7hr" event={"ID":"d5a9c296-07a4-4fe2-8a5c-593b799bd8fa","Type":"ContainerStarted","Data":"852d05301442ae7c516ef0405a5dd2783fe1b3d65c7b3ad2e2b578807c873efe"} Oct 30 12:13:54.129772 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:13:54.129747 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-80143318-4slg8" Oct 30 12:13:54.242343 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:13:54.242316 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f070e57-e304-4c20-b996-c3f428f8ef08-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f070e57-e304-4c20-b996-c3f428f8ef08-maas-api-xks-ca\") pod \"3f070e57-e304-4c20-b996-c3f428f8ef08\" (UID: \"3f070e57-e304-4c20-b996-c3f428f8ef08\") " Oct 30 12:13:54.242487 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:13:54.242384 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f070e57-e304-4c20-b996-c3f428f8ef08-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f070e57-e304-4c20-b996-c3f428f8ef08-maas-api-openshift-ca\") pod \"3f070e57-e304-4c20-b996-c3f428f8ef08\" (UID: \"3f070e57-e304-4c20-b996-c3f428f8ef08\") " Oct 30 12:13:54.242487 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:13:54.242478 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f070e57-e304-4c20-b996-c3f428f8ef08-kube-api-access-hbsqd\" (UniqueName: \"kubernetes.io/projected/3f070e57-e304-4c20-b996-c3f428f8ef08-kube-api-access-hbsqd\") pod \"3f070e57-e304-4c20-b996-c3f428f8ef08\" (UID: \"3f070e57-e304-4c20-b996-c3f428f8ef08\") " Oct 30 12:13:54.242765 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:13:54.242738 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f070e57-e304-4c20-b996-c3f428f8ef08-maas-api-openshift-ca" pod "3f070e57-e304-4c20-b996-c3f428f8ef08" (UID: "3f070e57-e304-4c20-b996-c3f428f8ef08"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:13:54.244371 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:13:54.244342 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f070e57-e304-4c20-b996-c3f428f8ef08-maas-api-xks-ca" pod "3f070e57-e304-4c20-b996-c3f428f8ef08" (UID: "3f070e57-e304-4c20-b996-c3f428f8ef08"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:13:54.244495 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:13:54.244475 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f070e57-e304-4c20-b996-c3f428f8ef08-kube-api-access-hbsqd" pod "3f070e57-e304-4c20-b996-c3f428f8ef08" (UID: "3f070e57-e304-4c20-b996-c3f428f8ef08"). InnerVolumeSpecName "kube-api-access-hbsqd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:13:54.343325 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:13:54.343300 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbsqd\" (UniqueName: \"kubernetes.io/projected/3f070e57-e304-4c20-b996-c3f428f8ef08-kube-api-access-hbsqd\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:13:54.343325 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:13:54.343324 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f070e57-e304-4c20-b996-c3f428f8ef08-maas-api-xks-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:13:54.343471 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:13:54.343334 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f070e57-e304-4c20-b996-c3f428f8ef08-maas-api-openshift-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:13:55.011644 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:13:55.011556 4908 generic.go:356] "Generic (PLEG): container finished" podID="d5a9c296-07a4-4fe2-8a5c-593b799bd8fa" containerID="ff1502cb89ada31dd2b2dfcbcf8e41d629eaddc0c51786985575fa91f36d9f4f" exitCode=0 Oct 30 12:13:55.012048 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:13:55.011644 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e636da41-7m7hr" event={"ID":"d5a9c296-07a4-4fe2-8a5c-593b799bd8fa","Type":"ContainerDied","Data":"ff1502cb89ada31dd2b2dfcbcf8e41d629eaddc0c51786985575fa91f36d9f4f"} Oct 30 12:13:55.012974 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:13:55.012953 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-80143318-4slg8" Oct 30 12:13:55.012974 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:13:55.012965 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-80143318-4slg8" event={"ID":"3f070e57-e304-4c20-b996-c3f428f8ef08","Type":"ContainerDied","Data":"81338a0b33926aec2180d5a92f414a82a90e37eac7bbc756ef5517cefa891104"} Oct 30 12:13:55.013128 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:13:55.013006 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="81338a0b33926aec2180d5a92f414a82a90e37eac7bbc756ef5517cefa891104" Oct 30 12:13:56.147011 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:13:56.146988 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e636da41-7m7hr" Oct 30 12:13:56.158126 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:13:56.158101 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d5a9c296-07a4-4fe2-8a5c-593b799bd8fa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5a9c296-07a4-4fe2-8a5c-593b799bd8fa-maas-api-xks-ca\") pod \"d5a9c296-07a4-4fe2-8a5c-593b799bd8fa\" (UID: \"d5a9c296-07a4-4fe2-8a5c-593b799bd8fa\") " Oct 30 12:13:56.158231 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:13:56.158150 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5a9c296-07a4-4fe2-8a5c-593b799bd8fa-kube-api-access-n88d8\" (UniqueName: \"kubernetes.io/projected/d5a9c296-07a4-4fe2-8a5c-593b799bd8fa-kube-api-access-n88d8\") pod \"d5a9c296-07a4-4fe2-8a5c-593b799bd8fa\" (UID: \"d5a9c296-07a4-4fe2-8a5c-593b799bd8fa\") " Oct 30 12:13:56.158231 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:13:56.158188 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5a9c296-07a4-4fe2-8a5c-593b799bd8fa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5a9c296-07a4-4fe2-8a5c-593b799bd8fa-maas-api-openshift-ca\") pod \"d5a9c296-07a4-4fe2-8a5c-593b799bd8fa\" (UID: \"d5a9c296-07a4-4fe2-8a5c-593b799bd8fa\") " Oct 30 12:13:56.158572 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:13:56.158548 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5a9c296-07a4-4fe2-8a5c-593b799bd8fa-maas-api-openshift-ca" pod "d5a9c296-07a4-4fe2-8a5c-593b799bd8fa" (UID: "d5a9c296-07a4-4fe2-8a5c-593b799bd8fa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:13:56.160089 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:13:56.160064 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5a9c296-07a4-4fe2-8a5c-593b799bd8fa-maas-api-xks-ca" pod "d5a9c296-07a4-4fe2-8a5c-593b799bd8fa" (UID: "d5a9c296-07a4-4fe2-8a5c-593b799bd8fa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:13:56.160195 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:13:56.160127 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5a9c296-07a4-4fe2-8a5c-593b799bd8fa-kube-api-access-n88d8" pod "d5a9c296-07a4-4fe2-8a5c-593b799bd8fa" (UID: "d5a9c296-07a4-4fe2-8a5c-593b799bd8fa"). InnerVolumeSpecName "kube-api-access-n88d8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:13:56.258714 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:13:56.258689 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n88d8\" (UniqueName: \"kubernetes.io/projected/d5a9c296-07a4-4fe2-8a5c-593b799bd8fa-kube-api-access-n88d8\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:13:56.258714 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:13:56.258712 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5a9c296-07a4-4fe2-8a5c-593b799bd8fa-maas-api-openshift-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:13:56.258889 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:13:56.258722 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5a9c296-07a4-4fe2-8a5c-593b799bd8fa-maas-api-xks-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:13:57.025514 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:13:57.025484 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e636da41-7m7hr" Oct 30 12:13:57.025694 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:13:57.025516 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e636da41-7m7hr" event={"ID":"d5a9c296-07a4-4fe2-8a5c-593b799bd8fa","Type":"ContainerDied","Data":"852d05301442ae7c516ef0405a5dd2783fe1b3d65c7b3ad2e2b578807c873efe"} Oct 30 12:13:57.025694 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:13:57.025559 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="852d05301442ae7c516ef0405a5dd2783fe1b3d65c7b3ad2e2b578807c873efe" Oct 30 12:13:57.916647 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:13:57.916615 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-dfd96f0e-9nt75"] Oct 30 12:13:57.917024 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:13:57.917010 4908 cpu_manager.go:414] "containerMap: removing container" podUID="3f070e57-e304-4c20-b996-c3f428f8ef08" containerName="revoke-keys" Oct 30 12:13:57.917076 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:13:57.917025 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f070e57-e304-4c20-b996-c3f428f8ef08" containerName="revoke-keys" Oct 30 12:13:57.917076 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:13:57.917048 4908 cpu_manager.go:414] "containerMap: removing container" podUID="d5a9c296-07a4-4fe2-8a5c-593b799bd8fa" containerName="revoke-keys" Oct 30 12:13:57.917076 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:13:57.917053 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5a9c296-07a4-4fe2-8a5c-593b799bd8fa" containerName="revoke-keys" Oct 30 12:13:57.917169 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:13:57.917110 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5a9c296-07a4-4fe2-8a5c-593b799bd8fa" containerName="revoke-keys" Oct 30 12:13:57.917169 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:13:57.917123 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f070e57-e304-4c20-b996-c3f428f8ef08" containerName="revoke-keys" Oct 30 12:13:57.920210 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:13:57.920196 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-dfd96f0e-9nt75" Oct 30 12:13:57.938404 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:13:57.938379 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-dfd96f0e-9nt75"] Oct 30 12:13:57.946612 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:13:57.946584 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 12:13:57.946734 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:13:57.946626 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 12:13:57.946734 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:13:57.946651 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 12:13:57.946734 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:13:57.946685 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s75hq\"" Oct 30 12:13:57.970306 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:13:57.970284 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84893168-3fa1-45c6-8b56-d7a3c28636f9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-dfd96f0e-9nt75\" (UID: \"84893168-3fa1-45c6-8b56-d7a3c28636f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-dfd96f0e-9nt75" Oct 30 12:13:57.970417 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:13:57.970349 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j924c\" (UniqueName: \"kubernetes.io/projected/84893168-3fa1-45c6-8b56-d7a3c28636f9-kube-api-access-j924c\") pod \"maas-api-revoke-keys-e2e-derive-dfd96f0e-9nt75\" (UID: \"84893168-3fa1-45c6-8b56-d7a3c28636f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-dfd96f0e-9nt75" Oct 30 12:13:57.970417 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:13:57.970389 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84893168-3fa1-45c6-8b56-d7a3c28636f9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-dfd96f0e-9nt75\" (UID: \"84893168-3fa1-45c6-8b56-d7a3c28636f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-dfd96f0e-9nt75" Oct 30 12:13:58.070893 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:13:58.070866 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j924c\" (UniqueName: \"kubernetes.io/projected/84893168-3fa1-45c6-8b56-d7a3c28636f9-kube-api-access-j924c\") pod \"maas-api-revoke-keys-e2e-derive-dfd96f0e-9nt75\" (UID: \"84893168-3fa1-45c6-8b56-d7a3c28636f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-dfd96f0e-9nt75" Oct 30 12:13:58.071001 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:13:58.070904 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84893168-3fa1-45c6-8b56-d7a3c28636f9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-dfd96f0e-9nt75\" (UID: \"84893168-3fa1-45c6-8b56-d7a3c28636f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-dfd96f0e-9nt75" Oct 30 12:13:58.071001 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:13:58.070961 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84893168-3fa1-45c6-8b56-d7a3c28636f9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-dfd96f0e-9nt75\" (UID: \"84893168-3fa1-45c6-8b56-d7a3c28636f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-dfd96f0e-9nt75" Oct 30 12:13:58.071546 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:13:58.071522 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84893168-3fa1-45c6-8b56-d7a3c28636f9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-dfd96f0e-9nt75\" (UID: \"84893168-3fa1-45c6-8b56-d7a3c28636f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-dfd96f0e-9nt75" Oct 30 12:13:58.073141 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:13:58.073122 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84893168-3fa1-45c6-8b56-d7a3c28636f9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-dfd96f0e-9nt75\" (UID: \"84893168-3fa1-45c6-8b56-d7a3c28636f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-dfd96f0e-9nt75" Oct 30 12:13:58.092252 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:13:58.092227 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j924c\" (UniqueName: \"kubernetes.io/projected/84893168-3fa1-45c6-8b56-d7a3c28636f9-kube-api-access-j924c\") pod \"maas-api-revoke-keys-e2e-derive-dfd96f0e-9nt75\" (UID: \"84893168-3fa1-45c6-8b56-d7a3c28636f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-dfd96f0e-9nt75" Oct 30 12:13:58.232181 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:13:58.232156 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-dfd96f0e-9nt75" Oct 30 12:13:58.530483 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:13:58.530396 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-dfd96f0e-9nt75"] Oct 30 12:13:58.534077 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:13:58.534050 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84893168_3fa1_45c6_8b56_d7a3c28636f9.slice/crio-df7cc1016e88d36fa2a52b88d1d1f7ae2db8f45ee63191124775f30ec7e94bf9 WatchSource:0}: Error finding container df7cc1016e88d36fa2a52b88d1d1f7ae2db8f45ee63191124775f30ec7e94bf9: Status 404 returned error can't find the container with id df7cc1016e88d36fa2a52b88d1d1f7ae2db8f45ee63191124775f30ec7e94bf9 Oct 30 12:13:59.037107 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:13:59.037074 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-dfd96f0e-9nt75" event={"ID":"84893168-3fa1-45c6-8b56-d7a3c28636f9","Type":"ContainerStarted","Data":"e03de40fb42cb31c9361facc54b227acdd9afbb115b21e5bca29dbb7af57fe0d"} Oct 30 12:13:59.037469 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:13:59.037125 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-dfd96f0e-9nt75" event={"ID":"84893168-3fa1-45c6-8b56-d7a3c28636f9","Type":"ContainerStarted","Data":"df7cc1016e88d36fa2a52b88d1d1f7ae2db8f45ee63191124775f30ec7e94bf9"} Oct 30 12:13:59.075857 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:13:59.075796 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-dfd96f0e-9nt75" podStartSLOduration=2.075781146 podStartE2EDuration="2.075781146s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:13:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:13:59.074022391 +0000 UTC m=+1195.588451155" watchObservedRunningTime="2026-10-30 12:13:59.075781146 +0000 UTC m=+1195.590209937" Oct 30 12:14:00.044192 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:14:00.044155 4908 generic.go:356] "Generic (PLEG): container finished" podID="84893168-3fa1-45c6-8b56-d7a3c28636f9" containerID="e03de40fb42cb31c9361facc54b227acdd9afbb115b21e5bca29dbb7af57fe0d" exitCode=0 Oct 30 12:14:00.044623 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:14:00.044246 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-dfd96f0e-9nt75" event={"ID":"84893168-3fa1-45c6-8b56-d7a3c28636f9","Type":"ContainerDied","Data":"e03de40fb42cb31c9361facc54b227acdd9afbb115b21e5bca29dbb7af57fe0d"} Oct 30 12:14:00.181463 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:14:00.181427 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-307283d2-4m29n"] Oct 30 12:14:00.184915 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:14:00.184890 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-307283d2-4m29n"] Oct 30 12:14:01.193237 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:14:01.193207 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-dfd96f0e-9nt75" Oct 30 12:14:01.198552 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:14:01.198526 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84893168-3fa1-45c6-8b56-d7a3c28636f9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84893168-3fa1-45c6-8b56-d7a3c28636f9-maas-api-openshift-ca\") pod \"84893168-3fa1-45c6-8b56-d7a3c28636f9\" (UID: \"84893168-3fa1-45c6-8b56-d7a3c28636f9\") " Oct 30 12:14:01.198705 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:14:01.198666 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84893168-3fa1-45c6-8b56-d7a3c28636f9-kube-api-access-j924c\" (UniqueName: \"kubernetes.io/projected/84893168-3fa1-45c6-8b56-d7a3c28636f9-kube-api-access-j924c\") pod \"84893168-3fa1-45c6-8b56-d7a3c28636f9\" (UID: \"84893168-3fa1-45c6-8b56-d7a3c28636f9\") " Oct 30 12:14:01.198705 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:14:01.198696 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84893168-3fa1-45c6-8b56-d7a3c28636f9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84893168-3fa1-45c6-8b56-d7a3c28636f9-maas-api-xks-ca\") pod \"84893168-3fa1-45c6-8b56-d7a3c28636f9\" (UID: \"84893168-3fa1-45c6-8b56-d7a3c28636f9\") " Oct 30 12:14:01.199400 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:14:01.199375 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84893168-3fa1-45c6-8b56-d7a3c28636f9-maas-api-openshift-ca" pod "84893168-3fa1-45c6-8b56-d7a3c28636f9" (UID: "84893168-3fa1-45c6-8b56-d7a3c28636f9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:14:01.203968 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:14:01.203947 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84893168-3fa1-45c6-8b56-d7a3c28636f9-maas-api-xks-ca" pod "84893168-3fa1-45c6-8b56-d7a3c28636f9" (UID: "84893168-3fa1-45c6-8b56-d7a3c28636f9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:14:01.204056 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:14:01.203963 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84893168-3fa1-45c6-8b56-d7a3c28636f9-kube-api-access-j924c" pod "84893168-3fa1-45c6-8b56-d7a3c28636f9" (UID: "84893168-3fa1-45c6-8b56-d7a3c28636f9"). InnerVolumeSpecName "kube-api-access-j924c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:14:01.300041 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:14:01.299973 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84893168-3fa1-45c6-8b56-d7a3c28636f9-maas-api-openshift-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:14:01.300041 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:14:01.299999 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j924c\" (UniqueName: \"kubernetes.io/projected/84893168-3fa1-45c6-8b56-d7a3c28636f9-kube-api-access-j924c\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:14:01.300041 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:14:01.300010 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84893168-3fa1-45c6-8b56-d7a3c28636f9-maas-api-xks-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:14:01.605092 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:14:01.605004 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e23ac155-6e41-458e-97d3-83e9cc24c03e" path="/var/lib/kubelet/pods/e23ac155-6e41-458e-97d3-83e9cc24c03e/volumes" Oct 30 12:14:02.057946 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:14:02.057904 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-dfd96f0e-9nt75" event={"ID":"84893168-3fa1-45c6-8b56-d7a3c28636f9","Type":"ContainerDied","Data":"df7cc1016e88d36fa2a52b88d1d1f7ae2db8f45ee63191124775f30ec7e94bf9"} Oct 30 12:14:02.058146 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:14:02.057991 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="df7cc1016e88d36fa2a52b88d1d1f7ae2db8f45ee63191124775f30ec7e94bf9" Oct 30 12:14:02.058326 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:14:02.058306 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-dfd96f0e-9nt75" Oct 30 12:14:02.147974 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:14:02.147929 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9wwfc"] Oct 30 12:14:02.717161 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:14:02.717133 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c85bd66-trnpj"] Oct 30 12:14:02.727799 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:14:02.727768 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c85bd66-trnpj"] Oct 30 12:14:03.606357 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:14:03.606321 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04d78a3d-786c-4aa7-87f4-c5789b2046d9" path="/var/lib/kubelet/pods/04d78a3d-786c-4aa7-87f4-c5789b2046d9/volumes" Oct 30 12:14:03.746213 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:14:03.746185 4908 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-plczt_a70ee11c-c20e-4a4f-8476-ac5eb02816d0/console-operator/3.log" Oct 30 12:14:03.749258 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:14:03.749231 4908 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-plczt_a70ee11c-c20e-4a4f-8476-ac5eb02816d0/console-operator/3.log" Oct 30 12:14:03.756777 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:14:03.756753 4908 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b8bxz_644e19b3-10ad-4145-9ed3-595d19ed9b07/ovn-acl-logging/0.log" Oct 30 12:14:03.760159 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:14:03.760137 4908 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b8bxz_644e19b3-10ad-4145-9ed3-595d19ed9b07/ovn-acl-logging/0.log" Oct 30 12:14:07.223237 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:14:07.223202 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-80143318-4slg8"] Oct 30 12:14:07.234197 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:14:07.234169 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-80143318-4slg8"] Oct 30 12:14:07.604488 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:14:07.604453 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f070e57-e304-4c20-b996-c3f428f8ef08" path="/var/lib/kubelet/pods/3f070e57-e304-4c20-b996-c3f428f8ef08/volumes" Oct 30 12:14:07.858799 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:14:07.858713 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e636da41-7m7hr"] Oct 30 12:14:07.871806 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:14:07.871762 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e636da41-7m7hr"] Oct 30 12:14:09.605187 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:14:09.605154 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5a9c296-07a4-4fe2-8a5c-593b799bd8fa" path="/var/lib/kubelet/pods/d5a9c296-07a4-4fe2-8a5c-593b799bd8fa/volumes" Oct 30 12:14:12.365464 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:14:12.365432 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9wwfc"] Oct 30 12:14:16.711996 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:14:16.711912 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-dfd96f0e-9nt75"] Oct 30 12:14:16.724134 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:14:16.724097 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-dfd96f0e-9nt75"] Oct 30 12:14:17.604677 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:14:17.604639 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84893168-3fa1-45c6-8b56-d7a3c28636f9" path="/var/lib/kubelet/pods/84893168-3fa1-45c6-8b56-d7a3c28636f9/volumes" Oct 30 12:14:19.684960 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:14:19.684925 4908 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vn6vv" containerName="extractor" Oct 30 12:14:21.966898 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:14:21.966864 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9wwfc"] Oct 30 12:14:26.531670 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:14:26.531621 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9wwfc"] Oct 30 12:14:32.390768 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:14:32.390723 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-512ec5/simulator-w4-512ec5-kserve-7df7674f8-8rqwb"] Oct 30 12:14:32.392862 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:14:32.391788 4908 cpu_manager.go:414] "containerMap: removing container" podUID="84893168-3fa1-45c6-8b56-d7a3c28636f9" containerName="revoke-keys" Oct 30 12:14:32.392862 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:14:32.391815 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84893168-3fa1-45c6-8b56-d7a3c28636f9" containerName="revoke-keys" Oct 30 12:14:32.392862 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:14:32.392404 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="84893168-3fa1-45c6-8b56-d7a3c28636f9" containerName="revoke-keys" Oct 30 12:14:32.398220 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:14:32.398190 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-512ec5/simulator-w4-512ec5-kserve-7df7674f8-8rqwb" Oct 30 12:14:32.417279 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:14:32.417249 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-512ec5/simulator-w4-512ec5-kserve-7df7674f8-8rqwb"] Oct 30 12:14:32.423633 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:14:32.423599 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-512ec5\"/\"kube-root-ca.crt\"" Oct 30 12:14:32.423633 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:14:32.423604 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-512ec5\"/\"simulator-w4-512ec5-kserve-self-signed-certs\"" Oct 30 12:14:32.423870 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:14:32.423610 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-512ec5\"/\"openshift-service-ca.crt\"" Oct 30 12:14:32.423870 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:14:32.423702 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-512ec5\"/\"default-dockercfg-2qxls\"" Oct 30 12:14:32.495501 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:14:32.495459 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a5dfcc9-dec3-4bc6-b0a8-c928350b3596-home\") pod \"simulator-w4-512ec5-kserve-7df7674f8-8rqwb\" (UID: \"8a5dfcc9-dec3-4bc6-b0a8-c928350b3596\") " pod="e2e-models-e2e-worker-w4-512ec5/simulator-w4-512ec5-kserve-7df7674f8-8rqwb" Oct 30 12:14:32.495501 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:14:32.495499 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a5dfcc9-dec3-4bc6-b0a8-c928350b3596-model-cache\") pod \"simulator-w4-512ec5-kserve-7df7674f8-8rqwb\" (UID: \"8a5dfcc9-dec3-4bc6-b0a8-c928350b3596\") " pod="e2e-models-e2e-worker-w4-512ec5/simulator-w4-512ec5-kserve-7df7674f8-8rqwb" Oct 30 12:14:32.495745 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:14:32.495600 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a5dfcc9-dec3-4bc6-b0a8-c928350b3596-tmp-dir\") pod \"simulator-w4-512ec5-kserve-7df7674f8-8rqwb\" (UID: \"8a5dfcc9-dec3-4bc6-b0a8-c928350b3596\") " pod="e2e-models-e2e-worker-w4-512ec5/simulator-w4-512ec5-kserve-7df7674f8-8rqwb" Oct 30 12:14:32.495745 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:14:32.495651 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xp6wz\" (UniqueName: \"kubernetes.io/projected/8a5dfcc9-dec3-4bc6-b0a8-c928350b3596-kube-api-access-xp6wz\") pod \"simulator-w4-512ec5-kserve-7df7674f8-8rqwb\" (UID: \"8a5dfcc9-dec3-4bc6-b0a8-c928350b3596\") " pod="e2e-models-e2e-worker-w4-512ec5/simulator-w4-512ec5-kserve-7df7674f8-8rqwb" Oct 30 12:14:32.495895 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:14:32.495748 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a5dfcc9-dec3-4bc6-b0a8-c928350b3596-tls-certs\") pod \"simulator-w4-512ec5-kserve-7df7674f8-8rqwb\" (UID: \"8a5dfcc9-dec3-4bc6-b0a8-c928350b3596\") " pod="e2e-models-e2e-worker-w4-512ec5/simulator-w4-512ec5-kserve-7df7674f8-8rqwb" Oct 30 12:14:32.495895 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:14:32.495799 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a5dfcc9-dec3-4bc6-b0a8-c928350b3596-dshm\") pod \"simulator-w4-512ec5-kserve-7df7674f8-8rqwb\" (UID: \"8a5dfcc9-dec3-4bc6-b0a8-c928350b3596\") " pod="e2e-models-e2e-worker-w4-512ec5/simulator-w4-512ec5-kserve-7df7674f8-8rqwb" Oct 30 12:14:32.597024 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:14:32.596987 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a5dfcc9-dec3-4bc6-b0a8-c928350b3596-tls-certs\") pod \"simulator-w4-512ec5-kserve-7df7674f8-8rqwb\" (UID: \"8a5dfcc9-dec3-4bc6-b0a8-c928350b3596\") " pod="e2e-models-e2e-worker-w4-512ec5/simulator-w4-512ec5-kserve-7df7674f8-8rqwb" Oct 30 12:14:32.597024 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:14:32.597028 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a5dfcc9-dec3-4bc6-b0a8-c928350b3596-dshm\") pod \"simulator-w4-512ec5-kserve-7df7674f8-8rqwb\" (UID: \"8a5dfcc9-dec3-4bc6-b0a8-c928350b3596\") " pod="e2e-models-e2e-worker-w4-512ec5/simulator-w4-512ec5-kserve-7df7674f8-8rqwb" Oct 30 12:14:32.597274 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:14:32.597219 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a5dfcc9-dec3-4bc6-b0a8-c928350b3596-home\") pod \"simulator-w4-512ec5-kserve-7df7674f8-8rqwb\" (UID: \"8a5dfcc9-dec3-4bc6-b0a8-c928350b3596\") " pod="e2e-models-e2e-worker-w4-512ec5/simulator-w4-512ec5-kserve-7df7674f8-8rqwb" Oct 30 12:14:32.597274 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:14:32.597266 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a5dfcc9-dec3-4bc6-b0a8-c928350b3596-model-cache\") pod \"simulator-w4-512ec5-kserve-7df7674f8-8rqwb\" (UID: \"8a5dfcc9-dec3-4bc6-b0a8-c928350b3596\") " pod="e2e-models-e2e-worker-w4-512ec5/simulator-w4-512ec5-kserve-7df7674f8-8rqwb" Oct 30 12:14:32.597381 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:14:32.597296 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a5dfcc9-dec3-4bc6-b0a8-c928350b3596-tmp-dir\") pod \"simulator-w4-512ec5-kserve-7df7674f8-8rqwb\" (UID: \"8a5dfcc9-dec3-4bc6-b0a8-c928350b3596\") " pod="e2e-models-e2e-worker-w4-512ec5/simulator-w4-512ec5-kserve-7df7674f8-8rqwb" Oct 30 12:14:32.597381 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:14:32.597331 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xp6wz\" (UniqueName: \"kubernetes.io/projected/8a5dfcc9-dec3-4bc6-b0a8-c928350b3596-kube-api-access-xp6wz\") pod \"simulator-w4-512ec5-kserve-7df7674f8-8rqwb\" (UID: \"8a5dfcc9-dec3-4bc6-b0a8-c928350b3596\") " pod="e2e-models-e2e-worker-w4-512ec5/simulator-w4-512ec5-kserve-7df7674f8-8rqwb" Oct 30 12:14:32.597607 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:14:32.597582 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a5dfcc9-dec3-4bc6-b0a8-c928350b3596-home\") pod \"simulator-w4-512ec5-kserve-7df7674f8-8rqwb\" (UID: \"8a5dfcc9-dec3-4bc6-b0a8-c928350b3596\") " pod="e2e-models-e2e-worker-w4-512ec5/simulator-w4-512ec5-kserve-7df7674f8-8rqwb" Oct 30 12:14:32.597777 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:14:32.597643 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a5dfcc9-dec3-4bc6-b0a8-c928350b3596-model-cache\") pod \"simulator-w4-512ec5-kserve-7df7674f8-8rqwb\" (UID: \"8a5dfcc9-dec3-4bc6-b0a8-c928350b3596\") " pod="e2e-models-e2e-worker-w4-512ec5/simulator-w4-512ec5-kserve-7df7674f8-8rqwb" Oct 30 12:14:32.597882 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:14:32.597712 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a5dfcc9-dec3-4bc6-b0a8-c928350b3596-tmp-dir\") pod \"simulator-w4-512ec5-kserve-7df7674f8-8rqwb\" (UID: \"8a5dfcc9-dec3-4bc6-b0a8-c928350b3596\") " pod="e2e-models-e2e-worker-w4-512ec5/simulator-w4-512ec5-kserve-7df7674f8-8rqwb" Oct 30 12:14:32.599287 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:14:32.599266 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a5dfcc9-dec3-4bc6-b0a8-c928350b3596-dshm\") pod \"simulator-w4-512ec5-kserve-7df7674f8-8rqwb\" (UID: \"8a5dfcc9-dec3-4bc6-b0a8-c928350b3596\") " pod="e2e-models-e2e-worker-w4-512ec5/simulator-w4-512ec5-kserve-7df7674f8-8rqwb" Oct 30 12:14:32.599652 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:14:32.599635 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a5dfcc9-dec3-4bc6-b0a8-c928350b3596-tls-certs\") pod \"simulator-w4-512ec5-kserve-7df7674f8-8rqwb\" (UID: \"8a5dfcc9-dec3-4bc6-b0a8-c928350b3596\") " pod="e2e-models-e2e-worker-w4-512ec5/simulator-w4-512ec5-kserve-7df7674f8-8rqwb" Oct 30 12:14:32.619942 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:14:32.619906 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xp6wz\" (UniqueName: \"kubernetes.io/projected/8a5dfcc9-dec3-4bc6-b0a8-c928350b3596-kube-api-access-xp6wz\") pod \"simulator-w4-512ec5-kserve-7df7674f8-8rqwb\" (UID: \"8a5dfcc9-dec3-4bc6-b0a8-c928350b3596\") " pod="e2e-models-e2e-worker-w4-512ec5/simulator-w4-512ec5-kserve-7df7674f8-8rqwb" Oct 30 12:14:32.712950 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:14:32.712860 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-512ec5/simulator-w4-512ec5-kserve-7df7674f8-8rqwb" Oct 30 12:14:33.031872 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:14:33.031821 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-512ec5/simulator-w4-512ec5-kserve-7df7674f8-8rqwb"] Oct 30 12:14:33.036761 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:14:33.036728 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8a5dfcc9_dec3_4bc6_b0a8_c928350b3596.slice/crio-21b33880b3e1d77d6ef4ea777af938cfcb78718c8c5a43e345fd1e70a62e0e95 WatchSource:0}: Error finding container 21b33880b3e1d77d6ef4ea777af938cfcb78718c8c5a43e345fd1e70a62e0e95: Status 404 returned error can't find the container with id 21b33880b3e1d77d6ef4ea777af938cfcb78718c8c5a43e345fd1e70a62e0e95 Oct 30 12:14:33.251425 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:14:33.251386 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-512ec5/simulator-w4-512ec5-kserve-7df7674f8-8rqwb" event={"ID":"8a5dfcc9-dec3-4bc6-b0a8-c928350b3596","Type":"ContainerStarted","Data":"21b33880b3e1d77d6ef4ea777af938cfcb78718c8c5a43e345fd1e70a62e0e95"} Oct 30 12:14:35.266329 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:14:35.266285 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-512ec5/simulator-w4-512ec5-kserve-7df7674f8-8rqwb" event={"ID":"8a5dfcc9-dec3-4bc6-b0a8-c928350b3596","Type":"ContainerStarted","Data":"6d87953e0148ef1a9ab936760bbceec78ad43bbf2d54319c94c7cbded807b647"} Oct 30 12:14:35.310332 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:14:35.310084 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-512ec5/simulator-w4-512ec5-kserve-7df7674f8-8rqwb" podStartSLOduration=2.111807302 podStartE2EDuration="3.310064663s" totalImagesPullingTime="1.198257361s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:14:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:14:35.309077198 +0000 UTC m=+1231.823505964" watchObservedRunningTime="2026-10-30 12:14:35.310064663 +0000 UTC m=+1231.824493411" Oct 30 12:14:35.713693 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:14:35.713600 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-512ec5/simulator-w4-512ec5-kserve-7df7674f8-8rqwb" Oct 30 12:14:36.281862 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:14:36.270723 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9wwfc"] Oct 30 12:14:42.713811 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:14:42.713766 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-512ec5/simulator-w4-512ec5-kserve-7df7674f8-8rqwb" Oct 30 12:14:42.730908 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:14:42.730864 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-512ec5/simulator-w4-512ec5-kserve-7df7674f8-8rqwb" Oct 30 12:14:43.349585 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:14:43.349547 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-512ec5/simulator-w4-512ec5-kserve-7df7674f8-8rqwb" Oct 30 12:14:45.573056 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:14:45.572998 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9wwfc"] Oct 30 12:14:49.685400 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:14:49.685362 4908 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vn6vv" containerName="extractor" Oct 30 12:14:50.882926 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:14:50.882885 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-f920f7/simulator-w0-f920f7-kserve-599b7ff45f-g79d7"] Oct 30 12:14:50.888255 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:14:50.888222 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f920f7/simulator-w0-f920f7-kserve-599b7ff45f-g79d7" Oct 30 12:14:50.902629 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:14:50.902597 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-f920f7\"/\"default-dockercfg-px7j9\"" Oct 30 12:14:50.914048 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:14:50.914014 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-f920f7\"/\"openshift-service-ca.crt\"" Oct 30 12:14:50.914371 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:14:50.914020 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-f920f7\"/\"kube-root-ca.crt\"" Oct 30 12:14:50.917260 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:14:50.917233 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-f920f7/simulator-w0-f920f7-kserve-599b7ff45f-g79d7"] Oct 30 12:14:50.977902 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:14:50.977808 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a85e562f-d655-4c22-b4f4-a299a4e494a0-tls-certs\") pod \"simulator-w0-f920f7-kserve-599b7ff45f-g79d7\" (UID: \"a85e562f-d655-4c22-b4f4-a299a4e494a0\") " pod="e2e-models-e2e-worker-w0-f920f7/simulator-w0-f920f7-kserve-599b7ff45f-g79d7" Oct 30 12:14:50.978106 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:14:50.977990 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqbsz\" (UniqueName: \"kubernetes.io/projected/a85e562f-d655-4c22-b4f4-a299a4e494a0-kube-api-access-cqbsz\") pod \"simulator-w0-f920f7-kserve-599b7ff45f-g79d7\" (UID: \"a85e562f-d655-4c22-b4f4-a299a4e494a0\") " pod="e2e-models-e2e-worker-w0-f920f7/simulator-w0-f920f7-kserve-599b7ff45f-g79d7" Oct 30 12:14:50.978106 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:14:50.978036 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a85e562f-d655-4c22-b4f4-a299a4e494a0-dshm\") pod \"simulator-w0-f920f7-kserve-599b7ff45f-g79d7\" (UID: \"a85e562f-d655-4c22-b4f4-a299a4e494a0\") " pod="e2e-models-e2e-worker-w0-f920f7/simulator-w0-f920f7-kserve-599b7ff45f-g79d7" Oct 30 12:14:50.978208 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:14:50.978117 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a85e562f-d655-4c22-b4f4-a299a4e494a0-model-cache\") pod \"simulator-w0-f920f7-kserve-599b7ff45f-g79d7\" (UID: \"a85e562f-d655-4c22-b4f4-a299a4e494a0\") " pod="e2e-models-e2e-worker-w0-f920f7/simulator-w0-f920f7-kserve-599b7ff45f-g79d7" Oct 30 12:14:50.978257 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:14:50.978232 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a85e562f-d655-4c22-b4f4-a299a4e494a0-tmp-dir\") pod \"simulator-w0-f920f7-kserve-599b7ff45f-g79d7\" (UID: \"a85e562f-d655-4c22-b4f4-a299a4e494a0\") " pod="e2e-models-e2e-worker-w0-f920f7/simulator-w0-f920f7-kserve-599b7ff45f-g79d7" Oct 30 12:14:50.978316 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:14:50.978298 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a85e562f-d655-4c22-b4f4-a299a4e494a0-home\") pod \"simulator-w0-f920f7-kserve-599b7ff45f-g79d7\" (UID: \"a85e562f-d655-4c22-b4f4-a299a4e494a0\") " pod="e2e-models-e2e-worker-w0-f920f7/simulator-w0-f920f7-kserve-599b7ff45f-g79d7" Oct 30 12:14:50.991735 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:14:50.991697 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-f920f7\"/\"simulator-w0-f920f7-kserve-self-signed-certs\"" Oct 30 12:14:51.078890 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:14:51.078822 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a85e562f-d655-4c22-b4f4-a299a4e494a0-home\") pod \"simulator-w0-f920f7-kserve-599b7ff45f-g79d7\" (UID: \"a85e562f-d655-4c22-b4f4-a299a4e494a0\") " pod="e2e-models-e2e-worker-w0-f920f7/simulator-w0-f920f7-kserve-599b7ff45f-g79d7" Oct 30 12:14:51.079089 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:14:51.078931 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a85e562f-d655-4c22-b4f4-a299a4e494a0-tls-certs\") pod \"simulator-w0-f920f7-kserve-599b7ff45f-g79d7\" (UID: \"a85e562f-d655-4c22-b4f4-a299a4e494a0\") " pod="e2e-models-e2e-worker-w0-f920f7/simulator-w0-f920f7-kserve-599b7ff45f-g79d7" Oct 30 12:14:51.079089 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:14:51.078968 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqbsz\" (UniqueName: \"kubernetes.io/projected/a85e562f-d655-4c22-b4f4-a299a4e494a0-kube-api-access-cqbsz\") pod \"simulator-w0-f920f7-kserve-599b7ff45f-g79d7\" (UID: \"a85e562f-d655-4c22-b4f4-a299a4e494a0\") " pod="e2e-models-e2e-worker-w0-f920f7/simulator-w0-f920f7-kserve-599b7ff45f-g79d7" Oct 30 12:14:51.079089 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:14:51.078996 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a85e562f-d655-4c22-b4f4-a299a4e494a0-dshm\") pod \"simulator-w0-f920f7-kserve-599b7ff45f-g79d7\" (UID: \"a85e562f-d655-4c22-b4f4-a299a4e494a0\") " pod="e2e-models-e2e-worker-w0-f920f7/simulator-w0-f920f7-kserve-599b7ff45f-g79d7" Oct 30 12:14:51.079089 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:14:51.079028 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a85e562f-d655-4c22-b4f4-a299a4e494a0-model-cache\") pod \"simulator-w0-f920f7-kserve-599b7ff45f-g79d7\" (UID: \"a85e562f-d655-4c22-b4f4-a299a4e494a0\") " pod="e2e-models-e2e-worker-w0-f920f7/simulator-w0-f920f7-kserve-599b7ff45f-g79d7" Oct 30 12:14:51.079089 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:14:51.079054 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a85e562f-d655-4c22-b4f4-a299a4e494a0-tmp-dir\") pod \"simulator-w0-f920f7-kserve-599b7ff45f-g79d7\" (UID: \"a85e562f-d655-4c22-b4f4-a299a4e494a0\") " pod="e2e-models-e2e-worker-w0-f920f7/simulator-w0-f920f7-kserve-599b7ff45f-g79d7" Oct 30 12:14:51.079358 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:14:51.079334 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a85e562f-d655-4c22-b4f4-a299a4e494a0-home\") pod \"simulator-w0-f920f7-kserve-599b7ff45f-g79d7\" (UID: \"a85e562f-d655-4c22-b4f4-a299a4e494a0\") " pod="e2e-models-e2e-worker-w0-f920f7/simulator-w0-f920f7-kserve-599b7ff45f-g79d7" Oct 30 12:14:51.079461 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:14:51.079405 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a85e562f-d655-4c22-b4f4-a299a4e494a0-model-cache\") pod \"simulator-w0-f920f7-kserve-599b7ff45f-g79d7\" (UID: \"a85e562f-d655-4c22-b4f4-a299a4e494a0\") " pod="e2e-models-e2e-worker-w0-f920f7/simulator-w0-f920f7-kserve-599b7ff45f-g79d7" Oct 30 12:14:51.079577 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:14:51.079418 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a85e562f-d655-4c22-b4f4-a299a4e494a0-tmp-dir\") pod \"simulator-w0-f920f7-kserve-599b7ff45f-g79d7\" (UID: \"a85e562f-d655-4c22-b4f4-a299a4e494a0\") " pod="e2e-models-e2e-worker-w0-f920f7/simulator-w0-f920f7-kserve-599b7ff45f-g79d7" Oct 30 12:14:51.081626 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:14:51.081590 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a85e562f-d655-4c22-b4f4-a299a4e494a0-dshm\") pod \"simulator-w0-f920f7-kserve-599b7ff45f-g79d7\" (UID: \"a85e562f-d655-4c22-b4f4-a299a4e494a0\") " pod="e2e-models-e2e-worker-w0-f920f7/simulator-w0-f920f7-kserve-599b7ff45f-g79d7" Oct 30 12:14:51.081745 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:14:51.081698 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a85e562f-d655-4c22-b4f4-a299a4e494a0-tls-certs\") pod \"simulator-w0-f920f7-kserve-599b7ff45f-g79d7\" (UID: \"a85e562f-d655-4c22-b4f4-a299a4e494a0\") " pod="e2e-models-e2e-worker-w0-f920f7/simulator-w0-f920f7-kserve-599b7ff45f-g79d7" Oct 30 12:14:51.100908 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:14:51.100878 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqbsz\" (UniqueName: \"kubernetes.io/projected/a85e562f-d655-4c22-b4f4-a299a4e494a0-kube-api-access-cqbsz\") pod \"simulator-w0-f920f7-kserve-599b7ff45f-g79d7\" (UID: \"a85e562f-d655-4c22-b4f4-a299a4e494a0\") " pod="e2e-models-e2e-worker-w0-f920f7/simulator-w0-f920f7-kserve-599b7ff45f-g79d7" Oct 30 12:14:51.204838 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:14:51.204732 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f920f7/simulator-w0-f920f7-kserve-599b7ff45f-g79d7" Oct 30 12:14:51.516313 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:14:51.516271 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-f920f7/simulator-w0-f920f7-kserve-599b7ff45f-g79d7"] Oct 30 12:14:51.520235 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:14:51.520202 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda85e562f_d655_4c22_b4f4_a299a4e494a0.slice/crio-f89ae38dbad086be146586a7ae696320fd88fed35b6cc9b084f56de79e996028 WatchSource:0}: Error finding container f89ae38dbad086be146586a7ae696320fd88fed35b6cc9b084f56de79e996028: Status 404 returned error can't find the container with id f89ae38dbad086be146586a7ae696320fd88fed35b6cc9b084f56de79e996028 Oct 30 12:14:52.388939 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:14:52.388887 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f920f7/simulator-w0-f920f7-kserve-599b7ff45f-g79d7" event={"ID":"a85e562f-d655-4c22-b4f4-a299a4e494a0","Type":"ContainerStarted","Data":"d904798341eedab6a9fa053fc321f05682c35fccd03374ef779c718d44358680"} Oct 30 12:14:52.389447 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:14:52.388949 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f920f7/simulator-w0-f920f7-kserve-599b7ff45f-g79d7" event={"ID":"a85e562f-d655-4c22-b4f4-a299a4e494a0","Type":"ContainerStarted","Data":"f89ae38dbad086be146586a7ae696320fd88fed35b6cc9b084f56de79e996028"} Oct 30 12:14:52.428676 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:14:52.428602 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-f920f7/simulator-w0-f920f7-kserve-599b7ff45f-g79d7" podStartSLOduration=2.428585694 podStartE2EDuration="2.428585694s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:14:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:14:52.426921059 +0000 UTC m=+1248.941349807" watchObservedRunningTime="2026-10-30 12:14:52.428585694 +0000 UTC m=+1248.943014440" Oct 30 12:14:53.204899 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:14:53.204859 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f920f7/simulator-w0-f920f7-kserve-599b7ff45f-g79d7" Oct 30 12:14:56.767582 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:14:56.767536 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9wwfc"] Oct 30 12:15:00.164661 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:00.164626 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a7148-29889375-mp2g4"] Oct 30 12:15:00.168802 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:00.168771 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-47fd99-29889375-6nl92"] Oct 30 12:15:00.169799 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:00.169774 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a7148-29889375-mp2g4" Oct 30 12:15:00.172403 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:00.172377 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f920f7-29889375-w9862"] Oct 30 12:15:00.172531 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:00.172511 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-47fd99-29889375-6nl92" Oct 30 12:15:00.176165 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:00.176138 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-512ec5-29889375-hqs69"] Oct 30 12:15:00.176287 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:00.176249 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f920f7-29889375-w9862" Oct 30 12:15:00.180329 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:00.180307 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-512ec5-29889375-hqs69" Oct 30 12:15:00.188023 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:00.187900 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-47fd99-29889375-6nl92"] Oct 30 12:15:00.191309 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:00.191280 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 12:15:00.191553 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:00.191528 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 12:15:00.200021 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:00.199983 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a7148-29889375-mp2g4"] Oct 30 12:15:00.200151 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:00.200066 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f920f7-29889375-w9862"] Oct 30 12:15:00.202321 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:00.202296 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s75hq\"" Oct 30 12:15:00.206822 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:00.206788 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-512ec5-29889375-hqs69"] Oct 30 12:15:00.258624 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:00.258591 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33c3769e-0c1e-48e0-8668-80e56fdb8ebe-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-6a7148-29889375-mp2g4\" (UID: \"33c3769e-0c1e-48e0-8668-80e56fdb8ebe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a7148-29889375-mp2g4" Oct 30 12:15:00.258817 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:00.258641 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62729517-29af-4c57-92ee-5fcc191ffd18-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-f920f7-29889375-w9862\" (UID: \"62729517-29af-4c57-92ee-5fcc191ffd18\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f920f7-29889375-w9862" Oct 30 12:15:00.258817 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:00.258660 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e0e6051-0a79-4f02-9f8c-813aff322d11-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-47fd99-29889375-6nl92\" (UID: \"6e0e6051-0a79-4f02-9f8c-813aff322d11\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-47fd99-29889375-6nl92" Oct 30 12:15:00.258817 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:00.258779 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tq22n\" (UniqueName: \"kubernetes.io/projected/6e0e6051-0a79-4f02-9f8c-813aff322d11-kube-api-access-tq22n\") pod \"maas-api-key-cleanup-e2e-worker-w2-47fd99-29889375-6nl92\" (UID: \"6e0e6051-0a79-4f02-9f8c-813aff322d11\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-47fd99-29889375-6nl92" Oct 30 12:15:00.258972 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:00.258862 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwnj4\" (UniqueName: \"kubernetes.io/projected/33c3769e-0c1e-48e0-8668-80e56fdb8ebe-kube-api-access-hwnj4\") pod \"maas-api-key-cleanup-e2e-worker-w3-6a7148-29889375-mp2g4\" (UID: \"33c3769e-0c1e-48e0-8668-80e56fdb8ebe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a7148-29889375-mp2g4" Oct 30 12:15:00.258972 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:00.258892 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fabf3a57-8eac-42d7-b47e-a183437610a8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-512ec5-29889375-hqs69\" (UID: \"fabf3a57-8eac-42d7-b47e-a183437610a8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-512ec5-29889375-hqs69" Oct 30 12:15:00.258972 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:00.258908 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qsw7\" (UniqueName: \"kubernetes.io/projected/fabf3a57-8eac-42d7-b47e-a183437610a8-kube-api-access-4qsw7\") pod \"maas-api-key-cleanup-e2e-worker-w4-512ec5-29889375-hqs69\" (UID: \"fabf3a57-8eac-42d7-b47e-a183437610a8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-512ec5-29889375-hqs69" Oct 30 12:15:00.258972 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:00.258926 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8sblc\" (UniqueName: \"kubernetes.io/projected/62729517-29af-4c57-92ee-5fcc191ffd18-kube-api-access-8sblc\") pod \"maas-api-key-cleanup-e2e-worker-w0-f920f7-29889375-w9862\" (UID: \"62729517-29af-4c57-92ee-5fcc191ffd18\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f920f7-29889375-w9862" Oct 30 12:15:00.359796 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:00.359757 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33c3769e-0c1e-48e0-8668-80e56fdb8ebe-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-6a7148-29889375-mp2g4\" (UID: \"33c3769e-0c1e-48e0-8668-80e56fdb8ebe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a7148-29889375-mp2g4" Oct 30 12:15:00.360006 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:00.359823 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62729517-29af-4c57-92ee-5fcc191ffd18-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-f920f7-29889375-w9862\" (UID: \"62729517-29af-4c57-92ee-5fcc191ffd18\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f920f7-29889375-w9862" Oct 30 12:15:00.360006 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:00.359884 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e0e6051-0a79-4f02-9f8c-813aff322d11-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-47fd99-29889375-6nl92\" (UID: \"6e0e6051-0a79-4f02-9f8c-813aff322d11\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-47fd99-29889375-6nl92" Oct 30 12:15:00.360006 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:00.359925 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tq22n\" (UniqueName: \"kubernetes.io/projected/6e0e6051-0a79-4f02-9f8c-813aff322d11-kube-api-access-tq22n\") pod \"maas-api-key-cleanup-e2e-worker-w2-47fd99-29889375-6nl92\" (UID: \"6e0e6051-0a79-4f02-9f8c-813aff322d11\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-47fd99-29889375-6nl92" Oct 30 12:15:00.360146 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:00.360069 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwnj4\" (UniqueName: \"kubernetes.io/projected/33c3769e-0c1e-48e0-8668-80e56fdb8ebe-kube-api-access-hwnj4\") pod \"maas-api-key-cleanup-e2e-worker-w3-6a7148-29889375-mp2g4\" (UID: \"33c3769e-0c1e-48e0-8668-80e56fdb8ebe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a7148-29889375-mp2g4" Oct 30 12:15:00.360194 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:00.360138 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fabf3a57-8eac-42d7-b47e-a183437610a8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-512ec5-29889375-hqs69\" (UID: \"fabf3a57-8eac-42d7-b47e-a183437610a8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-512ec5-29889375-hqs69" Oct 30 12:15:00.360194 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:00.360168 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qsw7\" (UniqueName: \"kubernetes.io/projected/fabf3a57-8eac-42d7-b47e-a183437610a8-kube-api-access-4qsw7\") pod \"maas-api-key-cleanup-e2e-worker-w4-512ec5-29889375-hqs69\" (UID: \"fabf3a57-8eac-42d7-b47e-a183437610a8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-512ec5-29889375-hqs69" Oct 30 12:15:00.360291 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:00.360197 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8sblc\" (UniqueName: \"kubernetes.io/projected/62729517-29af-4c57-92ee-5fcc191ffd18-kube-api-access-8sblc\") pod \"maas-api-key-cleanup-e2e-worker-w0-f920f7-29889375-w9862\" (UID: \"62729517-29af-4c57-92ee-5fcc191ffd18\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f920f7-29889375-w9862" Oct 30 12:15:00.360685 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:00.360660 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33c3769e-0c1e-48e0-8668-80e56fdb8ebe-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-6a7148-29889375-mp2g4\" (UID: \"33c3769e-0c1e-48e0-8668-80e56fdb8ebe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a7148-29889375-mp2g4" Oct 30 12:15:00.360797 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:00.360723 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e0e6051-0a79-4f02-9f8c-813aff322d11-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-47fd99-29889375-6nl92\" (UID: \"6e0e6051-0a79-4f02-9f8c-813aff322d11\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-47fd99-29889375-6nl92" Oct 30 12:15:00.360797 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:00.360751 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62729517-29af-4c57-92ee-5fcc191ffd18-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-f920f7-29889375-w9862\" (UID: \"62729517-29af-4c57-92ee-5fcc191ffd18\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f920f7-29889375-w9862" Oct 30 12:15:00.360959 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:00.360910 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fabf3a57-8eac-42d7-b47e-a183437610a8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-512ec5-29889375-hqs69\" (UID: \"fabf3a57-8eac-42d7-b47e-a183437610a8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-512ec5-29889375-hqs69" Oct 30 12:15:00.383172 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:00.383135 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8sblc\" (UniqueName: \"kubernetes.io/projected/62729517-29af-4c57-92ee-5fcc191ffd18-kube-api-access-8sblc\") pod \"maas-api-key-cleanup-e2e-worker-w0-f920f7-29889375-w9862\" (UID: \"62729517-29af-4c57-92ee-5fcc191ffd18\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f920f7-29889375-w9862" Oct 30 12:15:00.383385 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:00.383192 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tq22n\" (UniqueName: \"kubernetes.io/projected/6e0e6051-0a79-4f02-9f8c-813aff322d11-kube-api-access-tq22n\") pod \"maas-api-key-cleanup-e2e-worker-w2-47fd99-29889375-6nl92\" (UID: \"6e0e6051-0a79-4f02-9f8c-813aff322d11\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-47fd99-29889375-6nl92" Oct 30 12:15:00.383385 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:00.383263 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwnj4\" (UniqueName: \"kubernetes.io/projected/33c3769e-0c1e-48e0-8668-80e56fdb8ebe-kube-api-access-hwnj4\") pod \"maas-api-key-cleanup-e2e-worker-w3-6a7148-29889375-mp2g4\" (UID: \"33c3769e-0c1e-48e0-8668-80e56fdb8ebe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a7148-29889375-mp2g4" Oct 30 12:15:00.383512 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:00.383457 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qsw7\" (UniqueName: \"kubernetes.io/projected/fabf3a57-8eac-42d7-b47e-a183437610a8-kube-api-access-4qsw7\") pod \"maas-api-key-cleanup-e2e-worker-w4-512ec5-29889375-hqs69\" (UID: \"fabf3a57-8eac-42d7-b47e-a183437610a8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-512ec5-29889375-hqs69" Oct 30 12:15:00.487400 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:00.487358 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a7148-29889375-mp2g4" Oct 30 12:15:00.506963 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:00.506927 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-47fd99-29889375-6nl92" Oct 30 12:15:00.518222 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:00.518191 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f920f7-29889375-w9862" Oct 30 12:15:00.536503 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:00.536462 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-512ec5-29889375-hqs69" Oct 30 12:15:00.808165 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:00.807383 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a7148-29889375-mp2g4"] Oct 30 12:15:00.812622 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:15:00.812585 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33c3769e_0c1e_48e0_8668_80e56fdb8ebe.slice/crio-54d818624630cdea3e56ea297de83a6b4ed03fc95a1eeeee804b984435ae7a47 WatchSource:0}: Error finding container 54d818624630cdea3e56ea297de83a6b4ed03fc95a1eeeee804b984435ae7a47: Status 404 returned error can't find the container with id 54d818624630cdea3e56ea297de83a6b4ed03fc95a1eeeee804b984435ae7a47 Oct 30 12:15:00.852087 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:00.851887 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-47fd99-29889375-6nl92"] Oct 30 12:15:00.855923 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:15:00.855890 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e0e6051_0a79_4f02_9f8c_813aff322d11.slice/crio-25c1620b1dcc66a32fe779d5fe24c6ebac2856969ef293b5d882806c687fb7c8 WatchSource:0}: Error finding container 25c1620b1dcc66a32fe779d5fe24c6ebac2856969ef293b5d882806c687fb7c8: Status 404 returned error can't find the container with id 25c1620b1dcc66a32fe779d5fe24c6ebac2856969ef293b5d882806c687fb7c8 Oct 30 12:15:00.882000 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:00.881819 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f920f7-29889375-w9862"] Oct 30 12:15:00.884541 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:15:00.884507 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62729517_29af_4c57_92ee_5fcc191ffd18.slice/crio-c8c8e689ee701fbe78d4db6b6dd916d1147c82237694bd8f0b65752ada5bf411 WatchSource:0}: Error finding container c8c8e689ee701fbe78d4db6b6dd916d1147c82237694bd8f0b65752ada5bf411: Status 404 returned error can't find the container with id c8c8e689ee701fbe78d4db6b6dd916d1147c82237694bd8f0b65752ada5bf411 Oct 30 12:15:00.920108 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:00.920074 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-512ec5-29889375-hqs69"] Oct 30 12:15:00.924066 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:15:00.924031 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfabf3a57_8eac_42d7_b47e_a183437610a8.slice/crio-e5faf53b31daa11e340a74c319bc03c2795832b1f76cbce1de39914d4c1c8222 WatchSource:0}: Error finding container e5faf53b31daa11e340a74c319bc03c2795832b1f76cbce1de39914d4c1c8222: Status 404 returned error can't find the container with id e5faf53b31daa11e340a74c319bc03c2795832b1f76cbce1de39914d4c1c8222 Oct 30 12:15:01.205735 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:01.205701 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-f920f7/simulator-w0-f920f7-kserve-599b7ff45f-g79d7" Oct 30 12:15:01.218769 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:01.218735 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-f920f7/simulator-w0-f920f7-kserve-599b7ff45f-g79d7" Oct 30 12:15:01.449503 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:01.449386 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-512ec5-29889375-hqs69" event={"ID":"fabf3a57-8eac-42d7-b47e-a183437610a8","Type":"ContainerStarted","Data":"40a204b8e7cff86fd8eb96a1d311a82752d1041e7f0fee729f2eecef174e8f53"} Oct 30 12:15:01.449503 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:01.449461 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-512ec5-29889375-hqs69" event={"ID":"fabf3a57-8eac-42d7-b47e-a183437610a8","Type":"ContainerStarted","Data":"e5faf53b31daa11e340a74c319bc03c2795832b1f76cbce1de39914d4c1c8222"} Oct 30 12:15:01.451270 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:01.451240 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a7148-29889375-mp2g4" event={"ID":"33c3769e-0c1e-48e0-8668-80e56fdb8ebe","Type":"ContainerStarted","Data":"8051cea2321db70c5fd5e505aa5d59f01e64862a010d6cb9f5ad65c9943af8b5"} Oct 30 12:15:01.451378 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:01.451293 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a7148-29889375-mp2g4" event={"ID":"33c3769e-0c1e-48e0-8668-80e56fdb8ebe","Type":"ContainerStarted","Data":"54d818624630cdea3e56ea297de83a6b4ed03fc95a1eeeee804b984435ae7a47"} Oct 30 12:15:01.453559 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:01.453268 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-47fd99-29889375-6nl92" event={"ID":"6e0e6051-0a79-4f02-9f8c-813aff322d11","Type":"ContainerStarted","Data":"f383e956e28f75d63c0c8697b4c44192dd6c88a5922d9e68145d0d26ddeb5188"} Oct 30 12:15:01.453559 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:01.453316 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-47fd99-29889375-6nl92" event={"ID":"6e0e6051-0a79-4f02-9f8c-813aff322d11","Type":"ContainerStarted","Data":"25c1620b1dcc66a32fe779d5fe24c6ebac2856969ef293b5d882806c687fb7c8"} Oct 30 12:15:01.455048 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:01.455019 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f920f7-29889375-w9862" event={"ID":"62729517-29af-4c57-92ee-5fcc191ffd18","Type":"ContainerStarted","Data":"83ce62fbb180dd70c13489f8a1621a6d5f26c699cccdefeb8820c3c59683884a"} Oct 30 12:15:01.455175 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:01.455076 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f920f7-29889375-w9862" event={"ID":"62729517-29af-4c57-92ee-5fcc191ffd18","Type":"ContainerStarted","Data":"c8c8e689ee701fbe78d4db6b6dd916d1147c82237694bd8f0b65752ada5bf411"} Oct 30 12:15:01.471085 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:01.471053 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f920f7/simulator-w0-f920f7-kserve-599b7ff45f-g79d7" Oct 30 12:15:01.490393 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:01.490322 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-512ec5-29889375-hqs69" podStartSLOduration=1.490301061 podStartE2EDuration="1.490301061s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:15:01.486586836 +0000 UTC m=+1258.001015607" watchObservedRunningTime="2026-10-30 12:15:01.490301061 +0000 UTC m=+1258.004729810" Oct 30 12:15:01.526952 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:01.526898 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-47fd99-29889375-6nl92" podStartSLOduration=1.5268825050000001 podStartE2EDuration="1.526882505s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:15:01.525226575 +0000 UTC m=+1258.039655322" watchObservedRunningTime="2026-10-30 12:15:01.526882505 +0000 UTC m=+1258.041311251" Oct 30 12:15:01.603758 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:01.602876 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a7148-29889375-mp2g4" podStartSLOduration=1.602524484 podStartE2EDuration="1.602524484s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:15:01.601586937 +0000 UTC m=+1258.116015684" watchObservedRunningTime="2026-10-30 12:15:01.602524484 +0000 UTC m=+1258.116953231" Oct 30 12:15:01.607051 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:01.605807 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f920f7-29889375-w9862" podStartSLOduration=1.605795208 podStartE2EDuration="1.605795208s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:15:01.564748691 +0000 UTC m=+1258.079177437" watchObservedRunningTime="2026-10-30 12:15:01.605795208 +0000 UTC m=+1258.120223955" Oct 30 12:15:02.464573 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:02.464473 4908 generic.go:356] "Generic (PLEG): container finished" podID="fabf3a57-8eac-42d7-b47e-a183437610a8" containerID="40a204b8e7cff86fd8eb96a1d311a82752d1041e7f0fee729f2eecef174e8f53" exitCode=60 Oct 30 12:15:02.465109 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:02.464562 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-512ec5-29889375-hqs69" event={"ID":"fabf3a57-8eac-42d7-b47e-a183437610a8","Type":"ContainerDied","Data":"40a204b8e7cff86fd8eb96a1d311a82752d1041e7f0fee729f2eecef174e8f53"} Oct 30 12:15:02.465109 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:02.464877 4908 scope.go:122] "RemoveContainer" containerID="40a204b8e7cff86fd8eb96a1d311a82752d1041e7f0fee729f2eecef174e8f53" Oct 30 12:15:02.466599 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:02.466578 4908 generic.go:356] "Generic (PLEG): container finished" podID="33c3769e-0c1e-48e0-8668-80e56fdb8ebe" containerID="8051cea2321db70c5fd5e505aa5d59f01e64862a010d6cb9f5ad65c9943af8b5" exitCode=60 Oct 30 12:15:02.467217 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:02.466651 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a7148-29889375-mp2g4" event={"ID":"33c3769e-0c1e-48e0-8668-80e56fdb8ebe","Type":"ContainerDied","Data":"8051cea2321db70c5fd5e505aa5d59f01e64862a010d6cb9f5ad65c9943af8b5"} Oct 30 12:15:02.467217 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:02.466933 4908 scope.go:122] "RemoveContainer" containerID="8051cea2321db70c5fd5e505aa5d59f01e64862a010d6cb9f5ad65c9943af8b5" Oct 30 12:15:02.469514 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:02.469489 4908 generic.go:356] "Generic (PLEG): container finished" podID="6e0e6051-0a79-4f02-9f8c-813aff322d11" containerID="f383e956e28f75d63c0c8697b4c44192dd6c88a5922d9e68145d0d26ddeb5188" exitCode=60 Oct 30 12:15:02.469613 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:02.469541 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-47fd99-29889375-6nl92" event={"ID":"6e0e6051-0a79-4f02-9f8c-813aff322d11","Type":"ContainerDied","Data":"f383e956e28f75d63c0c8697b4c44192dd6c88a5922d9e68145d0d26ddeb5188"} Oct 30 12:15:02.469795 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:02.469781 4908 scope.go:122] "RemoveContainer" containerID="f383e956e28f75d63c0c8697b4c44192dd6c88a5922d9e68145d0d26ddeb5188" Oct 30 12:15:02.471517 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:02.471486 4908 generic.go:356] "Generic (PLEG): container finished" podID="62729517-29af-4c57-92ee-5fcc191ffd18" containerID="83ce62fbb180dd70c13489f8a1621a6d5f26c699cccdefeb8820c3c59683884a" exitCode=60 Oct 30 12:15:02.471624 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:02.471584 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f920f7-29889375-w9862" event={"ID":"62729517-29af-4c57-92ee-5fcc191ffd18","Type":"ContainerDied","Data":"83ce62fbb180dd70c13489f8a1621a6d5f26c699cccdefeb8820c3c59683884a"} Oct 30 12:15:02.471765 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:02.471750 4908 scope.go:122] "RemoveContainer" containerID="83ce62fbb180dd70c13489f8a1621a6d5f26c699cccdefeb8820c3c59683884a" Oct 30 12:15:03.481272 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:03.481228 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-512ec5-29889375-hqs69" event={"ID":"fabf3a57-8eac-42d7-b47e-a183437610a8","Type":"ContainerStarted","Data":"d9718a728c62459fa660d2c11d2a2ce238519d1de9d27d60493b62f2fb1212f9"} Oct 30 12:15:03.483784 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:03.483740 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a7148-29889375-mp2g4" event={"ID":"33c3769e-0c1e-48e0-8668-80e56fdb8ebe","Type":"ContainerStarted","Data":"9f6488b2927a35bdfb94a8a08fca33b338abc0bf373fe4ae8885a0a6c0a1baf7"} Oct 30 12:15:03.485614 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:03.485587 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-47fd99-29889375-6nl92" event={"ID":"6e0e6051-0a79-4f02-9f8c-813aff322d11","Type":"ContainerStarted","Data":"0c8752e77d63f2a243400bcaf81e82d1df9b8190678d2dc2ca049452be7d4b4c"} Oct 30 12:15:03.487545 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:03.487524 4908 generic.go:356] "Generic (PLEG): container finished" podID="62729517-29af-4c57-92ee-5fcc191ffd18" containerID="d97c3ac47daf68f64f9a9067f4934d1051e79d13359f94ce5799a4eeda9a91cf" exitCode=60 Oct 30 12:15:03.487660 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:03.487587 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f920f7-29889375-w9862" event={"ID":"62729517-29af-4c57-92ee-5fcc191ffd18","Type":"ContainerDied","Data":"d97c3ac47daf68f64f9a9067f4934d1051e79d13359f94ce5799a4eeda9a91cf"} Oct 30 12:15:03.487660 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:03.487624 4908 scope.go:122] "RemoveContainer" containerID="83ce62fbb180dd70c13489f8a1621a6d5f26c699cccdefeb8820c3c59683884a" Oct 30 12:15:03.487861 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:03.487812 4908 scope.go:122] "RemoveContainer" containerID="d97c3ac47daf68f64f9a9067f4934d1051e79d13359f94ce5799a4eeda9a91cf" Oct 30 12:15:03.488102 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:15:03.488077 4908 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-f920f7-29889375-w9862_odh-ai-gateway-infra(62729517-29af-4c57-92ee-5fcc191ffd18)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f920f7-29889375-w9862" podUID="62729517-29af-4c57-92ee-5fcc191ffd18" Oct 30 12:15:04.498411 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:04.498366 4908 generic.go:356] "Generic (PLEG): container finished" podID="6e0e6051-0a79-4f02-9f8c-813aff322d11" containerID="0c8752e77d63f2a243400bcaf81e82d1df9b8190678d2dc2ca049452be7d4b4c" exitCode=60 Oct 30 12:15:04.498930 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:04.498444 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-47fd99-29889375-6nl92" event={"ID":"6e0e6051-0a79-4f02-9f8c-813aff322d11","Type":"ContainerDied","Data":"0c8752e77d63f2a243400bcaf81e82d1df9b8190678d2dc2ca049452be7d4b4c"} Oct 30 12:15:04.498930 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:04.498512 4908 scope.go:122] "RemoveContainer" containerID="f383e956e28f75d63c0c8697b4c44192dd6c88a5922d9e68145d0d26ddeb5188" Oct 30 12:15:04.498930 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:04.498784 4908 scope.go:122] "RemoveContainer" containerID="0c8752e77d63f2a243400bcaf81e82d1df9b8190678d2dc2ca049452be7d4b4c" Oct 30 12:15:04.499119 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:15:04.499095 4908 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-47fd99-29889375-6nl92_odh-ai-gateway-infra(6e0e6051-0a79-4f02-9f8c-813aff322d11)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-47fd99-29889375-6nl92" podUID="6e0e6051-0a79-4f02-9f8c-813aff322d11" Oct 30 12:15:04.501695 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:04.501665 4908 scope.go:122] "RemoveContainer" containerID="d97c3ac47daf68f64f9a9067f4934d1051e79d13359f94ce5799a4eeda9a91cf" Oct 30 12:15:04.501982 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:15:04.501954 4908 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-f920f7-29889375-w9862_odh-ai-gateway-infra(62729517-29af-4c57-92ee-5fcc191ffd18)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f920f7-29889375-w9862" podUID="62729517-29af-4c57-92ee-5fcc191ffd18" Oct 30 12:15:04.503676 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:04.503653 4908 generic.go:356] "Generic (PLEG): container finished" podID="fabf3a57-8eac-42d7-b47e-a183437610a8" containerID="d9718a728c62459fa660d2c11d2a2ce238519d1de9d27d60493b62f2fb1212f9" exitCode=60 Oct 30 12:15:04.503801 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:04.503747 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-512ec5-29889375-hqs69" event={"ID":"fabf3a57-8eac-42d7-b47e-a183437610a8","Type":"ContainerDied","Data":"d9718a728c62459fa660d2c11d2a2ce238519d1de9d27d60493b62f2fb1212f9"} Oct 30 12:15:04.503975 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:04.503960 4908 scope.go:122] "RemoveContainer" containerID="d9718a728c62459fa660d2c11d2a2ce238519d1de9d27d60493b62f2fb1212f9" Oct 30 12:15:04.504224 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:15:04.504198 4908 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-512ec5-29889375-hqs69_odh-ai-gateway-infra(fabf3a57-8eac-42d7-b47e-a183437610a8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-512ec5-29889375-hqs69" podUID="fabf3a57-8eac-42d7-b47e-a183437610a8" Oct 30 12:15:04.505722 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:04.505700 4908 generic.go:356] "Generic (PLEG): container finished" podID="33c3769e-0c1e-48e0-8668-80e56fdb8ebe" containerID="9f6488b2927a35bdfb94a8a08fca33b338abc0bf373fe4ae8885a0a6c0a1baf7" exitCode=60 Oct 30 12:15:04.505818 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:04.505790 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a7148-29889375-mp2g4" event={"ID":"33c3769e-0c1e-48e0-8668-80e56fdb8ebe","Type":"ContainerDied","Data":"9f6488b2927a35bdfb94a8a08fca33b338abc0bf373fe4ae8885a0a6c0a1baf7"} Oct 30 12:15:04.506037 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:04.506019 4908 scope.go:122] "RemoveContainer" containerID="9f6488b2927a35bdfb94a8a08fca33b338abc0bf373fe4ae8885a0a6c0a1baf7" Oct 30 12:15:04.506293 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:15:04.506270 4908 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-6a7148-29889375-mp2g4_odh-ai-gateway-infra(33c3769e-0c1e-48e0-8668-80e56fdb8ebe)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a7148-29889375-mp2g4" podUID="33c3769e-0c1e-48e0-8668-80e56fdb8ebe" Oct 30 12:15:04.515174 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:04.515149 4908 scope.go:122] "RemoveContainer" containerID="40a204b8e7cff86fd8eb96a1d311a82752d1041e7f0fee729f2eecef174e8f53" Oct 30 12:15:04.528541 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:04.528521 4908 scope.go:122] "RemoveContainer" containerID="8051cea2321db70c5fd5e505aa5d59f01e64862a010d6cb9f5ad65c9943af8b5" Oct 30 12:15:05.516354 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:05.516313 4908 scope.go:122] "RemoveContainer" containerID="0c8752e77d63f2a243400bcaf81e82d1df9b8190678d2dc2ca049452be7d4b4c" Oct 30 12:15:05.516935 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:15:05.516666 4908 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-47fd99-29889375-6nl92_odh-ai-gateway-infra(6e0e6051-0a79-4f02-9f8c-813aff322d11)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-47fd99-29889375-6nl92" podUID="6e0e6051-0a79-4f02-9f8c-813aff322d11" Oct 30 12:15:05.518353 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:05.518323 4908 scope.go:122] "RemoveContainer" containerID="d9718a728c62459fa660d2c11d2a2ce238519d1de9d27d60493b62f2fb1212f9" Oct 30 12:15:05.518661 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:15:05.518637 4908 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-512ec5-29889375-hqs69_odh-ai-gateway-infra(fabf3a57-8eac-42d7-b47e-a183437610a8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-512ec5-29889375-hqs69" podUID="fabf3a57-8eac-42d7-b47e-a183437610a8" Oct 30 12:15:05.520210 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:05.520188 4908 scope.go:122] "RemoveContainer" containerID="9f6488b2927a35bdfb94a8a08fca33b338abc0bf373fe4ae8885a0a6c0a1baf7" Oct 30 12:15:05.520435 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:15:05.520413 4908 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-6a7148-29889375-mp2g4_odh-ai-gateway-infra(33c3769e-0c1e-48e0-8668-80e56fdb8ebe)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a7148-29889375-mp2g4" podUID="33c3769e-0c1e-48e0-8668-80e56fdb8ebe" Oct 30 12:15:07.706235 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:07.706194 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-47fd99/premium-w2-47fd99-kserve-5d85f9bb95-hqkmf"] Oct 30 12:15:07.712325 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:07.712300 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-47fd99/premium-w2-47fd99-kserve-5d85f9bb95-hqkmf" Oct 30 12:15:07.727056 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:07.727025 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-47fd99\"/\"default-dockercfg-tkn4k\"" Oct 30 12:15:07.738384 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:07.738354 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-47fd99\"/\"premium-w2-47fd99-kserve-self-signed-certs\"" Oct 30 12:15:07.741218 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:07.741191 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-47fd99\"/\"openshift-service-ca.crt\"" Oct 30 12:15:07.741218 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:07.741210 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-47fd99\"/\"kube-root-ca.crt\"" Oct 30 12:15:07.743778 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:07.743758 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-47fd99/premium-w2-47fd99-kserve-5d85f9bb95-hqkmf"] Oct 30 12:15:07.745776 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:07.745752 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd2cbeda-dbc4-41f6-a730-e1c6f9343c5e-dshm\") pod \"premium-w2-47fd99-kserve-5d85f9bb95-hqkmf\" (UID: \"fd2cbeda-dbc4-41f6-a730-e1c6f9343c5e\") " pod="e2e-models-e2e-worker-w2-47fd99/premium-w2-47fd99-kserve-5d85f9bb95-hqkmf" Oct 30 12:15:07.745868 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:07.745812 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd2cbeda-dbc4-41f6-a730-e1c6f9343c5e-model-cache\") pod \"premium-w2-47fd99-kserve-5d85f9bb95-hqkmf\" (UID: \"fd2cbeda-dbc4-41f6-a730-e1c6f9343c5e\") " pod="e2e-models-e2e-worker-w2-47fd99/premium-w2-47fd99-kserve-5d85f9bb95-hqkmf" Oct 30 12:15:07.746016 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:07.745994 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd2cbeda-dbc4-41f6-a730-e1c6f9343c5e-home\") pod \"premium-w2-47fd99-kserve-5d85f9bb95-hqkmf\" (UID: \"fd2cbeda-dbc4-41f6-a730-e1c6f9343c5e\") " pod="e2e-models-e2e-worker-w2-47fd99/premium-w2-47fd99-kserve-5d85f9bb95-hqkmf" Oct 30 12:15:07.746087 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:07.746030 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd2cbeda-dbc4-41f6-a730-e1c6f9343c5e-tmp-dir\") pod \"premium-w2-47fd99-kserve-5d85f9bb95-hqkmf\" (UID: \"fd2cbeda-dbc4-41f6-a730-e1c6f9343c5e\") " pod="e2e-models-e2e-worker-w2-47fd99/premium-w2-47fd99-kserve-5d85f9bb95-hqkmf" Oct 30 12:15:07.746087 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:07.746065 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd2cbeda-dbc4-41f6-a730-e1c6f9343c5e-tls-certs\") pod \"premium-w2-47fd99-kserve-5d85f9bb95-hqkmf\" (UID: \"fd2cbeda-dbc4-41f6-a730-e1c6f9343c5e\") " pod="e2e-models-e2e-worker-w2-47fd99/premium-w2-47fd99-kserve-5d85f9bb95-hqkmf" Oct 30 12:15:07.746167 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:07.746101 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6z4w\" (UniqueName: \"kubernetes.io/projected/fd2cbeda-dbc4-41f6-a730-e1c6f9343c5e-kube-api-access-l6z4w\") pod \"premium-w2-47fd99-kserve-5d85f9bb95-hqkmf\" (UID: \"fd2cbeda-dbc4-41f6-a730-e1c6f9343c5e\") " pod="e2e-models-e2e-worker-w2-47fd99/premium-w2-47fd99-kserve-5d85f9bb95-hqkmf" Oct 30 12:15:07.847307 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:07.847260 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd2cbeda-dbc4-41f6-a730-e1c6f9343c5e-home\") pod \"premium-w2-47fd99-kserve-5d85f9bb95-hqkmf\" (UID: \"fd2cbeda-dbc4-41f6-a730-e1c6f9343c5e\") " pod="e2e-models-e2e-worker-w2-47fd99/premium-w2-47fd99-kserve-5d85f9bb95-hqkmf" Oct 30 12:15:07.847481 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:07.847318 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd2cbeda-dbc4-41f6-a730-e1c6f9343c5e-tmp-dir\") pod \"premium-w2-47fd99-kserve-5d85f9bb95-hqkmf\" (UID: \"fd2cbeda-dbc4-41f6-a730-e1c6f9343c5e\") " pod="e2e-models-e2e-worker-w2-47fd99/premium-w2-47fd99-kserve-5d85f9bb95-hqkmf" Oct 30 12:15:07.847481 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:07.847350 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd2cbeda-dbc4-41f6-a730-e1c6f9343c5e-tls-certs\") pod \"premium-w2-47fd99-kserve-5d85f9bb95-hqkmf\" (UID: \"fd2cbeda-dbc4-41f6-a730-e1c6f9343c5e\") " pod="e2e-models-e2e-worker-w2-47fd99/premium-w2-47fd99-kserve-5d85f9bb95-hqkmf" Oct 30 12:15:07.847481 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:07.847376 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6z4w\" (UniqueName: \"kubernetes.io/projected/fd2cbeda-dbc4-41f6-a730-e1c6f9343c5e-kube-api-access-l6z4w\") pod \"premium-w2-47fd99-kserve-5d85f9bb95-hqkmf\" (UID: \"fd2cbeda-dbc4-41f6-a730-e1c6f9343c5e\") " pod="e2e-models-e2e-worker-w2-47fd99/premium-w2-47fd99-kserve-5d85f9bb95-hqkmf" Oct 30 12:15:07.847671 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:07.847516 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd2cbeda-dbc4-41f6-a730-e1c6f9343c5e-dshm\") pod \"premium-w2-47fd99-kserve-5d85f9bb95-hqkmf\" (UID: \"fd2cbeda-dbc4-41f6-a730-e1c6f9343c5e\") " pod="e2e-models-e2e-worker-w2-47fd99/premium-w2-47fd99-kserve-5d85f9bb95-hqkmf" Oct 30 12:15:07.847671 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:07.847561 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd2cbeda-dbc4-41f6-a730-e1c6f9343c5e-model-cache\") pod \"premium-w2-47fd99-kserve-5d85f9bb95-hqkmf\" (UID: \"fd2cbeda-dbc4-41f6-a730-e1c6f9343c5e\") " pod="e2e-models-e2e-worker-w2-47fd99/premium-w2-47fd99-kserve-5d85f9bb95-hqkmf" Oct 30 12:15:07.847799 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:07.847773 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd2cbeda-dbc4-41f6-a730-e1c6f9343c5e-home\") pod \"premium-w2-47fd99-kserve-5d85f9bb95-hqkmf\" (UID: \"fd2cbeda-dbc4-41f6-a730-e1c6f9343c5e\") " pod="e2e-models-e2e-worker-w2-47fd99/premium-w2-47fd99-kserve-5d85f9bb95-hqkmf" Oct 30 12:15:07.848189 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:07.848162 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd2cbeda-dbc4-41f6-a730-e1c6f9343c5e-model-cache\") pod \"premium-w2-47fd99-kserve-5d85f9bb95-hqkmf\" (UID: \"fd2cbeda-dbc4-41f6-a730-e1c6f9343c5e\") " pod="e2e-models-e2e-worker-w2-47fd99/premium-w2-47fd99-kserve-5d85f9bb95-hqkmf" Oct 30 12:15:07.848189 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:07.848173 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd2cbeda-dbc4-41f6-a730-e1c6f9343c5e-tmp-dir\") pod \"premium-w2-47fd99-kserve-5d85f9bb95-hqkmf\" (UID: \"fd2cbeda-dbc4-41f6-a730-e1c6f9343c5e\") " pod="e2e-models-e2e-worker-w2-47fd99/premium-w2-47fd99-kserve-5d85f9bb95-hqkmf" Oct 30 12:15:07.850335 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:07.850305 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd2cbeda-dbc4-41f6-a730-e1c6f9343c5e-tls-certs\") pod \"premium-w2-47fd99-kserve-5d85f9bb95-hqkmf\" (UID: \"fd2cbeda-dbc4-41f6-a730-e1c6f9343c5e\") " pod="e2e-models-e2e-worker-w2-47fd99/premium-w2-47fd99-kserve-5d85f9bb95-hqkmf" Oct 30 12:15:07.850466 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:07.850445 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd2cbeda-dbc4-41f6-a730-e1c6f9343c5e-dshm\") pod \"premium-w2-47fd99-kserve-5d85f9bb95-hqkmf\" (UID: \"fd2cbeda-dbc4-41f6-a730-e1c6f9343c5e\") " pod="e2e-models-e2e-worker-w2-47fd99/premium-w2-47fd99-kserve-5d85f9bb95-hqkmf" Oct 30 12:15:07.871005 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:07.870965 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6z4w\" (UniqueName: \"kubernetes.io/projected/fd2cbeda-dbc4-41f6-a730-e1c6f9343c5e-kube-api-access-l6z4w\") pod \"premium-w2-47fd99-kserve-5d85f9bb95-hqkmf\" (UID: \"fd2cbeda-dbc4-41f6-a730-e1c6f9343c5e\") " pod="e2e-models-e2e-worker-w2-47fd99/premium-w2-47fd99-kserve-5d85f9bb95-hqkmf" Oct 30 12:15:08.028567 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:08.028527 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-47fd99/premium-w2-47fd99-kserve-5d85f9bb95-hqkmf" Oct 30 12:15:08.337481 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:08.337394 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-47fd99/premium-w2-47fd99-kserve-5d85f9bb95-hqkmf"] Oct 30 12:15:08.340674 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:15:08.340647 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfd2cbeda_dbc4_41f6_a730_e1c6f9343c5e.slice/crio-e7dbc7689dbc250d757eec337519d4af08683b8918cd9c5d9655764bef4b0813 WatchSource:0}: Error finding container e7dbc7689dbc250d757eec337519d4af08683b8918cd9c5d9655764bef4b0813: Status 404 returned error can't find the container with id e7dbc7689dbc250d757eec337519d4af08683b8918cd9c5d9655764bef4b0813 Oct 30 12:15:08.542007 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:08.541971 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-47fd99/premium-w2-47fd99-kserve-5d85f9bb95-hqkmf" event={"ID":"fd2cbeda-dbc4-41f6-a730-e1c6f9343c5e","Type":"ContainerStarted","Data":"2a87eadca80163145063eb0f8e3c3dd7d99f1798fee7cc2c36b29a26e00a86d2"} Oct 30 12:15:08.542179 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:08.542021 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-47fd99/premium-w2-47fd99-kserve-5d85f9bb95-hqkmf" event={"ID":"fd2cbeda-dbc4-41f6-a730-e1c6f9343c5e","Type":"ContainerStarted","Data":"e7dbc7689dbc250d757eec337519d4af08683b8918cd9c5d9655764bef4b0813"} Oct 30 12:15:08.582901 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:08.582819 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-47fd99/premium-w2-47fd99-kserve-5d85f9bb95-hqkmf" podStartSLOduration=1.582803997 podStartE2EDuration="1.582803997s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:15:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:15:08.581252811 +0000 UTC m=+1265.095681549" watchObservedRunningTime="2026-10-30 12:15:08.582803997 +0000 UTC m=+1265.097232742" Oct 30 12:15:08.636067 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:08.635986 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9wwfc"] Oct 30 12:15:09.029186 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:09.029151 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-47fd99/premium-w2-47fd99-kserve-5d85f9bb95-hqkmf" Oct 30 12:15:14.599207 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:14.599170 4908 scope.go:122] "RemoveContainer" containerID="d97c3ac47daf68f64f9a9067f4934d1051e79d13359f94ce5799a4eeda9a91cf" Oct 30 12:15:14.599619 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:14.599323 4908 scope.go:122] "RemoveContainer" containerID="0c8752e77d63f2a243400bcaf81e82d1df9b8190678d2dc2ca049452be7d4b4c" Oct 30 12:15:15.591078 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:15.591041 4908 generic.go:356] "Generic (PLEG): container finished" podID="6e0e6051-0a79-4f02-9f8c-813aff322d11" containerID="c5638a790b7c2b39b532264e3e7e44adf5f5e8a5647434022b02266a77e07be3" exitCode=60 Oct 30 12:15:15.591255 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:15.591115 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-47fd99-29889375-6nl92" event={"ID":"6e0e6051-0a79-4f02-9f8c-813aff322d11","Type":"ContainerDied","Data":"c5638a790b7c2b39b532264e3e7e44adf5f5e8a5647434022b02266a77e07be3"} Oct 30 12:15:15.591255 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:15.591192 4908 scope.go:122] "RemoveContainer" containerID="0c8752e77d63f2a243400bcaf81e82d1df9b8190678d2dc2ca049452be7d4b4c" Oct 30 12:15:15.591573 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:15.591551 4908 scope.go:122] "RemoveContainer" containerID="c5638a790b7c2b39b532264e3e7e44adf5f5e8a5647434022b02266a77e07be3" Oct 30 12:15:15.591807 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:15:15.591790 4908 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-47fd99-29889375-6nl92_odh-ai-gateway-infra(6e0e6051-0a79-4f02-9f8c-813aff322d11)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-47fd99-29889375-6nl92" podUID="6e0e6051-0a79-4f02-9f8c-813aff322d11" Oct 30 12:15:15.594172 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:15.594153 4908 generic.go:356] "Generic (PLEG): container finished" podID="62729517-29af-4c57-92ee-5fcc191ffd18" containerID="9a0b366d67890fff477c2ae50003efb7f59d03b5051f7f7de40bbd1f87ce8927" exitCode=60 Oct 30 12:15:15.594277 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:15.594227 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f920f7-29889375-w9862" event={"ID":"62729517-29af-4c57-92ee-5fcc191ffd18","Type":"ContainerDied","Data":"9a0b366d67890fff477c2ae50003efb7f59d03b5051f7f7de40bbd1f87ce8927"} Oct 30 12:15:15.594614 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:15.594598 4908 scope.go:122] "RemoveContainer" containerID="9a0b366d67890fff477c2ae50003efb7f59d03b5051f7f7de40bbd1f87ce8927" Oct 30 12:15:15.594896 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:15:15.594875 4908 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-f920f7-29889375-w9862_odh-ai-gateway-infra(62729517-29af-4c57-92ee-5fcc191ffd18)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f920f7-29889375-w9862" podUID="62729517-29af-4c57-92ee-5fcc191ffd18" Oct 30 12:15:15.605994 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:15.605776 4908 scope.go:122] "RemoveContainer" containerID="d97c3ac47daf68f64f9a9067f4934d1051e79d13359f94ce5799a4eeda9a91cf" Oct 30 12:15:16.598909 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:16.598876 4908 scope.go:122] "RemoveContainer" containerID="9f6488b2927a35bdfb94a8a08fca33b338abc0bf373fe4ae8885a0a6c0a1baf7" Oct 30 12:15:16.599123 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:16.598936 4908 scope.go:122] "RemoveContainer" containerID="d9718a728c62459fa660d2c11d2a2ce238519d1de9d27d60493b62f2fb1212f9" Oct 30 12:15:16.640599 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:16.640567 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-47fd99-29889375-6nl92"] Oct 30 12:15:16.678680 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:16.677889 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f920f7-29889375-w9862"] Oct 30 12:15:16.820414 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:16.820387 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-47fd99-29889375-6nl92" Oct 30 12:15:16.938864 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:16.938812 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f920f7-29889375-w9862" Oct 30 12:15:16.946230 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:16.946198 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e0e6051-0a79-4f02-9f8c-813aff322d11-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e0e6051-0a79-4f02-9f8c-813aff322d11-maas-api-service-ca\") pod \"6e0e6051-0a79-4f02-9f8c-813aff322d11\" (UID: \"6e0e6051-0a79-4f02-9f8c-813aff322d11\") " Oct 30 12:15:16.947179 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:16.946261 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e0e6051-0a79-4f02-9f8c-813aff322d11-kube-api-access-tq22n\" (UniqueName: \"kubernetes.io/projected/6e0e6051-0a79-4f02-9f8c-813aff322d11-kube-api-access-tq22n\") pod \"6e0e6051-0a79-4f02-9f8c-813aff322d11\" (UID: \"6e0e6051-0a79-4f02-9f8c-813aff322d11\") " Oct 30 12:15:16.947179 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:16.946597 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e0e6051-0a79-4f02-9f8c-813aff322d11-maas-api-service-ca" pod "6e0e6051-0a79-4f02-9f8c-813aff322d11" (UID: "6e0e6051-0a79-4f02-9f8c-813aff322d11"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:15:16.948731 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:16.948708 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e0e6051-0a79-4f02-9f8c-813aff322d11-kube-api-access-tq22n" pod "6e0e6051-0a79-4f02-9f8c-813aff322d11" (UID: "6e0e6051-0a79-4f02-9f8c-813aff322d11"). InnerVolumeSpecName "kube-api-access-tq22n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:15:17.046952 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:17.046906 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62729517-29af-4c57-92ee-5fcc191ffd18-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62729517-29af-4c57-92ee-5fcc191ffd18-maas-api-service-ca\") pod \"62729517-29af-4c57-92ee-5fcc191ffd18\" (UID: \"62729517-29af-4c57-92ee-5fcc191ffd18\") " Oct 30 12:15:17.047395 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:17.047365 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62729517-29af-4c57-92ee-5fcc191ffd18-maas-api-service-ca" pod "62729517-29af-4c57-92ee-5fcc191ffd18" (UID: "62729517-29af-4c57-92ee-5fcc191ffd18"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:15:17.047605 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:17.047589 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62729517-29af-4c57-92ee-5fcc191ffd18-kube-api-access-8sblc\" (UniqueName: \"kubernetes.io/projected/62729517-29af-4c57-92ee-5fcc191ffd18-kube-api-access-8sblc\") pod \"62729517-29af-4c57-92ee-5fcc191ffd18\" (UID: \"62729517-29af-4c57-92ee-5fcc191ffd18\") " Oct 30 12:15:17.047913 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:17.047896 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e0e6051-0a79-4f02-9f8c-813aff322d11-maas-api-service-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:15:17.047952 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:17.047916 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tq22n\" (UniqueName: \"kubernetes.io/projected/6e0e6051-0a79-4f02-9f8c-813aff322d11-kube-api-access-tq22n\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:15:17.047952 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:17.047925 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62729517-29af-4c57-92ee-5fcc191ffd18-maas-api-service-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:15:17.049972 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:17.049936 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62729517-29af-4c57-92ee-5fcc191ffd18-kube-api-access-8sblc" pod "62729517-29af-4c57-92ee-5fcc191ffd18" (UID: "62729517-29af-4c57-92ee-5fcc191ffd18"). InnerVolumeSpecName "kube-api-access-8sblc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:15:17.149100 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:17.149019 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8sblc\" (UniqueName: \"kubernetes.io/projected/62729517-29af-4c57-92ee-5fcc191ffd18-kube-api-access-8sblc\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:15:17.616046 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:17.615998 4908 generic.go:356] "Generic (PLEG): container finished" podID="33c3769e-0c1e-48e0-8668-80e56fdb8ebe" containerID="9789b15668f4f1fb57e229a3d53e86b8f6f2fb06fd5587318fd20d9ffe2e006a" exitCode=60 Oct 30 12:15:17.616243 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:17.616064 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a7148-29889375-mp2g4" event={"ID":"33c3769e-0c1e-48e0-8668-80e56fdb8ebe","Type":"ContainerDied","Data":"9789b15668f4f1fb57e229a3d53e86b8f6f2fb06fd5587318fd20d9ffe2e006a"} Oct 30 12:15:17.616243 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:17.616145 4908 scope.go:122] "RemoveContainer" containerID="9f6488b2927a35bdfb94a8a08fca33b338abc0bf373fe4ae8885a0a6c0a1baf7" Oct 30 12:15:17.617373 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:17.616928 4908 scope.go:122] "RemoveContainer" containerID="9789b15668f4f1fb57e229a3d53e86b8f6f2fb06fd5587318fd20d9ffe2e006a" Oct 30 12:15:17.617373 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:15:17.617203 4908 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-6a7148-29889375-mp2g4_odh-ai-gateway-infra(33c3769e-0c1e-48e0-8668-80e56fdb8ebe)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a7148-29889375-mp2g4" podUID="33c3769e-0c1e-48e0-8668-80e56fdb8ebe" Oct 30 12:15:17.618579 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:17.618548 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-47fd99-29889375-6nl92" event={"ID":"6e0e6051-0a79-4f02-9f8c-813aff322d11","Type":"ContainerDied","Data":"25c1620b1dcc66a32fe779d5fe24c6ebac2856969ef293b5d882806c687fb7c8"} Oct 30 12:15:17.618688 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:17.618577 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-47fd99-29889375-6nl92" Oct 30 12:15:17.619996 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:17.619972 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f920f7-29889375-w9862" event={"ID":"62729517-29af-4c57-92ee-5fcc191ffd18","Type":"ContainerDied","Data":"c8c8e689ee701fbe78d4db6b6dd916d1147c82237694bd8f0b65752ada5bf411"} Oct 30 12:15:17.620107 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:17.620079 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f920f7-29889375-w9862" Oct 30 12:15:17.624447 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:17.624423 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-512ec5-29889375-hqs69" event={"ID":"fabf3a57-8eac-42d7-b47e-a183437610a8","Type":"ContainerStarted","Data":"43610339019b722cb7a7a8114ba40d0cdf1c7b3c4b2eb338aef0ef8be3afbc83"} Oct 30 12:15:17.634098 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:17.634078 4908 scope.go:122] "RemoveContainer" containerID="c5638a790b7c2b39b532264e3e7e44adf5f5e8a5647434022b02266a77e07be3" Oct 30 12:15:17.645453 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:17.645430 4908 scope.go:122] "RemoveContainer" containerID="9a0b366d67890fff477c2ae50003efb7f59d03b5051f7f7de40bbd1f87ce8927" Oct 30 12:15:17.692645 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:17.692615 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-47fd99-29889375-6nl92"] Oct 30 12:15:17.704196 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:17.704165 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-47fd99-29889375-6nl92"] Oct 30 12:15:17.729713 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:17.729681 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f920f7-29889375-w9862"] Oct 30 12:15:17.733880 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:17.733853 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f920f7-29889375-w9862"] Oct 30 12:15:18.029340 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:18.029255 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-47fd99/premium-w2-47fd99-kserve-5d85f9bb95-hqkmf" Oct 30 12:15:18.043922 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:18.043893 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-47fd99/premium-w2-47fd99-kserve-5d85f9bb95-hqkmf" Oct 30 12:15:18.633084 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:18.633053 4908 generic.go:356] "Generic (PLEG): container finished" podID="fabf3a57-8eac-42d7-b47e-a183437610a8" containerID="43610339019b722cb7a7a8114ba40d0cdf1c7b3c4b2eb338aef0ef8be3afbc83" exitCode=60 Oct 30 12:15:18.633272 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:18.633131 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-512ec5-29889375-hqs69" event={"ID":"fabf3a57-8eac-42d7-b47e-a183437610a8","Type":"ContainerDied","Data":"43610339019b722cb7a7a8114ba40d0cdf1c7b3c4b2eb338aef0ef8be3afbc83"} Oct 30 12:15:18.633272 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:18.633193 4908 scope.go:122] "RemoveContainer" containerID="d9718a728c62459fa660d2c11d2a2ce238519d1de9d27d60493b62f2fb1212f9" Oct 30 12:15:18.633561 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:18.633539 4908 scope.go:122] "RemoveContainer" containerID="43610339019b722cb7a7a8114ba40d0cdf1c7b3c4b2eb338aef0ef8be3afbc83" Oct 30 12:15:18.633850 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:15:18.633809 4908 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-512ec5-29889375-hqs69_odh-ai-gateway-infra(fabf3a57-8eac-42d7-b47e-a183437610a8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-512ec5-29889375-hqs69" podUID="fabf3a57-8eac-42d7-b47e-a183437610a8" Oct 30 12:15:18.648170 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:18.648146 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-47fd99/premium-w2-47fd99-kserve-5d85f9bb95-hqkmf" Oct 30 12:15:18.671668 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:18.671634 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a7148-29889375-mp2g4"] Oct 30 12:15:18.782848 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:18.782411 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-512ec5-29889375-hqs69"] Oct 30 12:15:18.887823 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:18.887748 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a7148-29889375-mp2g4" Oct 30 12:15:18.968933 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:18.968902 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33c3769e-0c1e-48e0-8668-80e56fdb8ebe-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33c3769e-0c1e-48e0-8668-80e56fdb8ebe-maas-api-service-ca\") pod \"33c3769e-0c1e-48e0-8668-80e56fdb8ebe\" (UID: \"33c3769e-0c1e-48e0-8668-80e56fdb8ebe\") " Oct 30 12:15:18.969122 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:18.969031 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33c3769e-0c1e-48e0-8668-80e56fdb8ebe-kube-api-access-hwnj4\" (UniqueName: \"kubernetes.io/projected/33c3769e-0c1e-48e0-8668-80e56fdb8ebe-kube-api-access-hwnj4\") pod \"33c3769e-0c1e-48e0-8668-80e56fdb8ebe\" (UID: \"33c3769e-0c1e-48e0-8668-80e56fdb8ebe\") " Oct 30 12:15:18.969351 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:18.969327 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33c3769e-0c1e-48e0-8668-80e56fdb8ebe-maas-api-service-ca" pod "33c3769e-0c1e-48e0-8668-80e56fdb8ebe" (UID: "33c3769e-0c1e-48e0-8668-80e56fdb8ebe"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:15:18.969426 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:18.969410 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33c3769e-0c1e-48e0-8668-80e56fdb8ebe-maas-api-service-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:15:18.972989 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:18.972949 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33c3769e-0c1e-48e0-8668-80e56fdb8ebe-kube-api-access-hwnj4" pod "33c3769e-0c1e-48e0-8668-80e56fdb8ebe" (UID: "33c3769e-0c1e-48e0-8668-80e56fdb8ebe"). InnerVolumeSpecName "kube-api-access-hwnj4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:15:19.070314 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:19.070270 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hwnj4\" (UniqueName: \"kubernetes.io/projected/33c3769e-0c1e-48e0-8668-80e56fdb8ebe-kube-api-access-hwnj4\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:15:19.120747 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:19.120704 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9wwfc"] Oct 30 12:15:19.606036 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:19.605999 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62729517-29af-4c57-92ee-5fcc191ffd18" path="/var/lib/kubelet/pods/62729517-29af-4c57-92ee-5fcc191ffd18/volumes" Oct 30 12:15:19.606394 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:19.606380 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e0e6051-0a79-4f02-9f8c-813aff322d11" path="/var/lib/kubelet/pods/6e0e6051-0a79-4f02-9f8c-813aff322d11/volumes" Oct 30 12:15:19.644228 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:19.644197 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a7148-29889375-mp2g4" Oct 30 12:15:19.644228 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:19.644207 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a7148-29889375-mp2g4" event={"ID":"33c3769e-0c1e-48e0-8668-80e56fdb8ebe","Type":"ContainerDied","Data":"54d818624630cdea3e56ea297de83a6b4ed03fc95a1eeeee804b984435ae7a47"} Oct 30 12:15:19.644442 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:19.644275 4908 scope.go:122] "RemoveContainer" containerID="9789b15668f4f1fb57e229a3d53e86b8f6f2fb06fd5587318fd20d9ffe2e006a" Oct 30 12:15:19.688870 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:15:19.688804 4908 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vn6vv" containerName="extractor" Oct 30 12:15:19.739050 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:19.739011 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a7148-29889375-mp2g4"] Oct 30 12:15:19.743042 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:19.743009 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a7148-29889375-mp2g4"] Oct 30 12:15:19.808650 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:19.808624 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-512ec5-29889375-hqs69" Oct 30 12:15:19.877713 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:19.877618 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fabf3a57-8eac-42d7-b47e-a183437610a8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fabf3a57-8eac-42d7-b47e-a183437610a8-maas-api-service-ca\") pod \"fabf3a57-8eac-42d7-b47e-a183437610a8\" (UID: \"fabf3a57-8eac-42d7-b47e-a183437610a8\") " Oct 30 12:15:19.877898 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:19.877755 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fabf3a57-8eac-42d7-b47e-a183437610a8-kube-api-access-4qsw7\" (UniqueName: \"kubernetes.io/projected/fabf3a57-8eac-42d7-b47e-a183437610a8-kube-api-access-4qsw7\") pod \"fabf3a57-8eac-42d7-b47e-a183437610a8\" (UID: \"fabf3a57-8eac-42d7-b47e-a183437610a8\") " Oct 30 12:15:19.878095 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:19.878065 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fabf3a57-8eac-42d7-b47e-a183437610a8-maas-api-service-ca" pod "fabf3a57-8eac-42d7-b47e-a183437610a8" (UID: "fabf3a57-8eac-42d7-b47e-a183437610a8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:15:19.880675 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:19.880645 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fabf3a57-8eac-42d7-b47e-a183437610a8-kube-api-access-4qsw7" pod "fabf3a57-8eac-42d7-b47e-a183437610a8" (UID: "fabf3a57-8eac-42d7-b47e-a183437610a8"). InnerVolumeSpecName "kube-api-access-4qsw7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:15:19.979345 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:19.979310 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fabf3a57-8eac-42d7-b47e-a183437610a8-maas-api-service-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:15:19.979345 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:19.979341 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4qsw7\" (UniqueName: \"kubernetes.io/projected/fabf3a57-8eac-42d7-b47e-a183437610a8-kube-api-access-4qsw7\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:15:20.654349 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:20.654311 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-512ec5-29889375-hqs69" Oct 30 12:15:20.654545 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:20.654319 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-512ec5-29889375-hqs69" event={"ID":"fabf3a57-8eac-42d7-b47e-a183437610a8","Type":"ContainerDied","Data":"e5faf53b31daa11e340a74c319bc03c2795832b1f76cbce1de39914d4c1c8222"} Oct 30 12:15:20.654545 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:20.654471 4908 scope.go:122] "RemoveContainer" containerID="43610339019b722cb7a7a8114ba40d0cdf1c7b3c4b2eb338aef0ef8be3afbc83" Oct 30 12:15:20.703442 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:20.703405 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-512ec5-29889375-hqs69"] Oct 30 12:15:20.713015 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:20.712988 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-512ec5-29889375-hqs69"] Oct 30 12:15:21.605504 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:21.605470 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33c3769e-0c1e-48e0-8668-80e56fdb8ebe" path="/var/lib/kubelet/pods/33c3769e-0c1e-48e0-8668-80e56fdb8ebe/volumes" Oct 30 12:15:21.605900 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:21.605886 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fabf3a57-8eac-42d7-b47e-a183437610a8" path="/var/lib/kubelet/pods/fabf3a57-8eac-42d7-b47e-a183437610a8/volumes" Oct 30 12:15:24.009558 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:24.009513 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-qrkpg"] Oct 30 12:15:24.010143 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:24.010125 4908 cpu_manager.go:414] "containerMap: removing container" podUID="fabf3a57-8eac-42d7-b47e-a183437610a8" containerName="cleanup" Oct 30 12:15:24.010186 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:24.010147 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fabf3a57-8eac-42d7-b47e-a183437610a8" containerName="cleanup" Oct 30 12:15:24.010186 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:24.010169 4908 cpu_manager.go:414] "containerMap: removing container" podUID="6e0e6051-0a79-4f02-9f8c-813aff322d11" containerName="cleanup" Oct 30 12:15:24.010186 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:24.010177 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e0e6051-0a79-4f02-9f8c-813aff322d11" containerName="cleanup" Oct 30 12:15:24.010307 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:24.010191 4908 cpu_manager.go:414] "containerMap: removing container" podUID="62729517-29af-4c57-92ee-5fcc191ffd18" containerName="cleanup" Oct 30 12:15:24.010307 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:24.010199 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62729517-29af-4c57-92ee-5fcc191ffd18" containerName="cleanup" Oct 30 12:15:24.010307 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:24.010212 4908 cpu_manager.go:414] "containerMap: removing container" podUID="33c3769e-0c1e-48e0-8668-80e56fdb8ebe" containerName="cleanup" Oct 30 12:15:24.010307 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:24.010219 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33c3769e-0c1e-48e0-8668-80e56fdb8ebe" containerName="cleanup" Oct 30 12:15:24.010307 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:24.010228 4908 cpu_manager.go:414] "containerMap: removing container" podUID="6e0e6051-0a79-4f02-9f8c-813aff322d11" containerName="cleanup" Oct 30 12:15:24.010307 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:24.010236 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e0e6051-0a79-4f02-9f8c-813aff322d11" containerName="cleanup" Oct 30 12:15:24.010307 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:24.010246 4908 cpu_manager.go:414] "containerMap: removing container" podUID="fabf3a57-8eac-42d7-b47e-a183437610a8" containerName="cleanup" Oct 30 12:15:24.010307 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:24.010252 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fabf3a57-8eac-42d7-b47e-a183437610a8" containerName="cleanup" Oct 30 12:15:24.010307 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:24.010278 4908 cpu_manager.go:414] "containerMap: removing container" podUID="62729517-29af-4c57-92ee-5fcc191ffd18" containerName="cleanup" Oct 30 12:15:24.010307 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:24.010285 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62729517-29af-4c57-92ee-5fcc191ffd18" containerName="cleanup" Oct 30 12:15:24.010307 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:24.010300 4908 cpu_manager.go:414] "containerMap: removing container" podUID="33c3769e-0c1e-48e0-8668-80e56fdb8ebe" containerName="cleanup" Oct 30 12:15:24.010307 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:24.010306 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33c3769e-0c1e-48e0-8668-80e56fdb8ebe" containerName="cleanup" Oct 30 12:15:24.010634 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:24.010326 4908 cpu_manager.go:414] "containerMap: removing container" podUID="62729517-29af-4c57-92ee-5fcc191ffd18" containerName="cleanup" Oct 30 12:15:24.010634 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:24.010333 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62729517-29af-4c57-92ee-5fcc191ffd18" containerName="cleanup" Oct 30 12:15:24.010634 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:24.010341 4908 cpu_manager.go:414] "containerMap: removing container" podUID="33c3769e-0c1e-48e0-8668-80e56fdb8ebe" containerName="cleanup" Oct 30 12:15:24.010634 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:24.010348 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33c3769e-0c1e-48e0-8668-80e56fdb8ebe" containerName="cleanup" Oct 30 12:15:24.010634 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:24.010437 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="62729517-29af-4c57-92ee-5fcc191ffd18" containerName="cleanup" Oct 30 12:15:24.010634 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:24.010448 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e0e6051-0a79-4f02-9f8c-813aff322d11" containerName="cleanup" Oct 30 12:15:24.010634 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:24.010459 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="33c3769e-0c1e-48e0-8668-80e56fdb8ebe" containerName="cleanup" Oct 30 12:15:24.010634 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:24.010468 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="62729517-29af-4c57-92ee-5fcc191ffd18" containerName="cleanup" Oct 30 12:15:24.010634 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:24.010476 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="33c3769e-0c1e-48e0-8668-80e56fdb8ebe" containerName="cleanup" Oct 30 12:15:24.010634 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:24.010485 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="fabf3a57-8eac-42d7-b47e-a183437610a8" containerName="cleanup" Oct 30 12:15:24.010634 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:24.010495 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="fabf3a57-8eac-42d7-b47e-a183437610a8" containerName="cleanup" Oct 30 12:15:24.010634 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:24.010506 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e0e6051-0a79-4f02-9f8c-813aff322d11" containerName="cleanup" Oct 30 12:15:24.013967 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:24.013939 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-qrkpg" Oct 30 12:15:24.027738 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:24.027712 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-47fd99-openshift-default-dockercfg-lmqzd\"" Oct 30 12:15:24.041813 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:24.041778 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-qrkpg"] Oct 30 12:15:24.120002 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:24.119958 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8cf0a817-eff3-454d-8ddc-0a811a951647-istio-envoy\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-qrkpg\" (UID: \"8cf0a817-eff3-454d-8ddc-0a811a951647\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-qrkpg" Oct 30 12:15:24.120192 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:24.120019 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cf0a817-eff3-454d-8ddc-0a811a951647-credential-socket\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-qrkpg\" (UID: \"8cf0a817-eff3-454d-8ddc-0a811a951647\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-qrkpg" Oct 30 12:15:24.120192 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:24.120139 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cf0a817-eff3-454d-8ddc-0a811a951647-workload-socket\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-qrkpg\" (UID: \"8cf0a817-eff3-454d-8ddc-0a811a951647\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-qrkpg" Oct 30 12:15:24.120192 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:24.120185 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8cf0a817-eff3-454d-8ddc-0a811a951647-istiod-ca-cert\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-qrkpg\" (UID: \"8cf0a817-eff3-454d-8ddc-0a811a951647\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-qrkpg" Oct 30 12:15:24.120287 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:24.120252 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8cf0a817-eff3-454d-8ddc-0a811a951647-istio-data\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-qrkpg\" (UID: \"8cf0a817-eff3-454d-8ddc-0a811a951647\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-qrkpg" Oct 30 12:15:24.120325 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:24.120301 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8cf0a817-eff3-454d-8ddc-0a811a951647-istio-token\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-qrkpg\" (UID: \"8cf0a817-eff3-454d-8ddc-0a811a951647\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-qrkpg" Oct 30 12:15:24.120359 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:24.120329 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8cf0a817-eff3-454d-8ddc-0a811a951647-workload-certs\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-qrkpg\" (UID: \"8cf0a817-eff3-454d-8ddc-0a811a951647\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-qrkpg" Oct 30 12:15:24.120402 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:24.120370 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zw5n7\" (UniqueName: \"kubernetes.io/projected/8cf0a817-eff3-454d-8ddc-0a811a951647-kube-api-access-zw5n7\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-qrkpg\" (UID: \"8cf0a817-eff3-454d-8ddc-0a811a951647\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-qrkpg" Oct 30 12:15:24.120489 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:24.120474 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8cf0a817-eff3-454d-8ddc-0a811a951647-istio-podinfo\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-qrkpg\" (UID: \"8cf0a817-eff3-454d-8ddc-0a811a951647\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-qrkpg" Oct 30 12:15:24.221166 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:24.221128 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8cf0a817-eff3-454d-8ddc-0a811a951647-workload-certs\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-qrkpg\" (UID: \"8cf0a817-eff3-454d-8ddc-0a811a951647\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-qrkpg" Oct 30 12:15:24.221376 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:24.221189 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zw5n7\" (UniqueName: \"kubernetes.io/projected/8cf0a817-eff3-454d-8ddc-0a811a951647-kube-api-access-zw5n7\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-qrkpg\" (UID: \"8cf0a817-eff3-454d-8ddc-0a811a951647\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-qrkpg" Oct 30 12:15:24.221376 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:24.221228 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8cf0a817-eff3-454d-8ddc-0a811a951647-istio-podinfo\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-qrkpg\" (UID: \"8cf0a817-eff3-454d-8ddc-0a811a951647\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-qrkpg" Oct 30 12:15:24.221376 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:24.221278 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8cf0a817-eff3-454d-8ddc-0a811a951647-istio-envoy\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-qrkpg\" (UID: \"8cf0a817-eff3-454d-8ddc-0a811a951647\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-qrkpg" Oct 30 12:15:24.221376 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:24.221320 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cf0a817-eff3-454d-8ddc-0a811a951647-credential-socket\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-qrkpg\" (UID: \"8cf0a817-eff3-454d-8ddc-0a811a951647\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-qrkpg" Oct 30 12:15:24.221376 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:24.221365 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cf0a817-eff3-454d-8ddc-0a811a951647-workload-socket\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-qrkpg\" (UID: \"8cf0a817-eff3-454d-8ddc-0a811a951647\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-qrkpg" Oct 30 12:15:24.221644 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:24.221389 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8cf0a817-eff3-454d-8ddc-0a811a951647-istiod-ca-cert\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-qrkpg\" (UID: \"8cf0a817-eff3-454d-8ddc-0a811a951647\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-qrkpg" Oct 30 12:15:24.221644 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:24.221415 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8cf0a817-eff3-454d-8ddc-0a811a951647-istio-data\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-qrkpg\" (UID: \"8cf0a817-eff3-454d-8ddc-0a811a951647\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-qrkpg" Oct 30 12:15:24.221644 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:24.221441 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8cf0a817-eff3-454d-8ddc-0a811a951647-istio-token\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-qrkpg\" (UID: \"8cf0a817-eff3-454d-8ddc-0a811a951647\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-qrkpg" Oct 30 12:15:24.221644 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:24.221607 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8cf0a817-eff3-454d-8ddc-0a811a951647-workload-certs\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-qrkpg\" (UID: \"8cf0a817-eff3-454d-8ddc-0a811a951647\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-qrkpg" Oct 30 12:15:24.221903 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:24.221866 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cf0a817-eff3-454d-8ddc-0a811a951647-workload-socket\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-qrkpg\" (UID: \"8cf0a817-eff3-454d-8ddc-0a811a951647\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-qrkpg" Oct 30 12:15:24.222061 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:24.222034 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8cf0a817-eff3-454d-8ddc-0a811a951647-istio-data\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-qrkpg\" (UID: \"8cf0a817-eff3-454d-8ddc-0a811a951647\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-qrkpg" Oct 30 12:15:24.222127 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:24.222064 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cf0a817-eff3-454d-8ddc-0a811a951647-credential-socket\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-qrkpg\" (UID: \"8cf0a817-eff3-454d-8ddc-0a811a951647\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-qrkpg" Oct 30 12:15:24.222297 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:24.222276 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8cf0a817-eff3-454d-8ddc-0a811a951647-istiod-ca-cert\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-qrkpg\" (UID: \"8cf0a817-eff3-454d-8ddc-0a811a951647\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-qrkpg" Oct 30 12:15:24.224246 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:24.224220 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8cf0a817-eff3-454d-8ddc-0a811a951647-istio-envoy\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-qrkpg\" (UID: \"8cf0a817-eff3-454d-8ddc-0a811a951647\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-qrkpg" Oct 30 12:15:24.224346 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:24.224293 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8cf0a817-eff3-454d-8ddc-0a811a951647-istio-podinfo\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-qrkpg\" (UID: \"8cf0a817-eff3-454d-8ddc-0a811a951647\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-qrkpg" Oct 30 12:15:24.243822 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:24.243773 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8cf0a817-eff3-454d-8ddc-0a811a951647-istio-token\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-qrkpg\" (UID: \"8cf0a817-eff3-454d-8ddc-0a811a951647\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-qrkpg" Oct 30 12:15:24.244128 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:24.244109 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zw5n7\" (UniqueName: \"kubernetes.io/projected/8cf0a817-eff3-454d-8ddc-0a811a951647-kube-api-access-zw5n7\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-qrkpg\" (UID: \"8cf0a817-eff3-454d-8ddc-0a811a951647\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-qrkpg" Oct 30 12:15:24.328360 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:24.328256 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-qrkpg" Oct 30 12:15:24.645228 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:24.645146 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-qrkpg"] Oct 30 12:15:24.649721 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:15:24.649693 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cf0a817_eff3_454d_8ddc_0a811a951647.slice/crio-8fde4183f1a0a7afca13fc2809839e9b2028f38e2f813efb6404b78ed535b78a WatchSource:0}: Error finding container 8fde4183f1a0a7afca13fc2809839e9b2028f38e2f813efb6404b78ed535b78a: Status 404 returned error can't find the container with id 8fde4183f1a0a7afca13fc2809839e9b2028f38e2f813efb6404b78ed535b78a Oct 30 12:15:24.652104 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:24.652071 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:15:24.652173 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:24.652140 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:15:24.652217 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:24.652171 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:15:24.683900 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:24.683863 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-qrkpg" event={"ID":"8cf0a817-eff3-454d-8ddc-0a811a951647","Type":"ContainerStarted","Data":"8fde4183f1a0a7afca13fc2809839e9b2028f38e2f813efb6404b78ed535b78a"} Oct 30 12:15:24.921892 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:24.921773 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-6a7148/premium-w3-6a7148-kserve-67b976c888-tgd6c"] Oct 30 12:15:24.922467 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:24.922442 4908 cpu_manager.go:414] "containerMap: removing container" podUID="fabf3a57-8eac-42d7-b47e-a183437610a8" containerName="cleanup" Oct 30 12:15:24.922533 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:24.922469 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fabf3a57-8eac-42d7-b47e-a183437610a8" containerName="cleanup" Oct 30 12:15:24.922533 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:24.922498 4908 cpu_manager.go:414] "containerMap: removing container" podUID="6e0e6051-0a79-4f02-9f8c-813aff322d11" containerName="cleanup" Oct 30 12:15:24.922533 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:24.922505 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e0e6051-0a79-4f02-9f8c-813aff322d11" containerName="cleanup" Oct 30 12:15:24.922693 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:24.922633 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="33c3769e-0c1e-48e0-8668-80e56fdb8ebe" containerName="cleanup" Oct 30 12:15:24.922693 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:24.922650 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="fabf3a57-8eac-42d7-b47e-a183437610a8" containerName="cleanup" Oct 30 12:15:24.922693 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:24.922665 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="62729517-29af-4c57-92ee-5fcc191ffd18" containerName="cleanup" Oct 30 12:15:24.922693 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:24.922676 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e0e6051-0a79-4f02-9f8c-813aff322d11" containerName="cleanup" Oct 30 12:15:24.926836 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:24.926794 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-6a7148/premium-w3-6a7148-kserve-67b976c888-tgd6c" Oct 30 12:15:24.942465 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:24.942426 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-6a7148\"/\"openshift-service-ca.crt\"" Oct 30 12:15:24.942465 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:24.942453 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-6a7148\"/\"kube-root-ca.crt\"" Oct 30 12:15:24.942465 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:24.942426 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-6a7148\"/\"default-dockercfg-d5dmb\"" Oct 30 12:15:24.953097 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:24.953065 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-6a7148\"/\"premium-w3-6a7148-kserve-self-signed-certs\"" Oct 30 12:15:24.955856 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:24.955804 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-6a7148/premium-w3-6a7148-kserve-67b976c888-tgd6c"] Oct 30 12:15:25.029902 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:25.029848 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13de4441-e7f6-4cbd-97d9-0f079d11cb8e-home\") pod \"premium-w3-6a7148-kserve-67b976c888-tgd6c\" (UID: \"13de4441-e7f6-4cbd-97d9-0f079d11cb8e\") " pod="e2e-models-e2e-worker-w3-6a7148/premium-w3-6a7148-kserve-67b976c888-tgd6c" Oct 30 12:15:25.030400 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:25.029984 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13de4441-e7f6-4cbd-97d9-0f079d11cb8e-tls-certs\") pod \"premium-w3-6a7148-kserve-67b976c888-tgd6c\" (UID: \"13de4441-e7f6-4cbd-97d9-0f079d11cb8e\") " pod="e2e-models-e2e-worker-w3-6a7148/premium-w3-6a7148-kserve-67b976c888-tgd6c" Oct 30 12:15:25.030400 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:25.030044 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13de4441-e7f6-4cbd-97d9-0f079d11cb8e-tmp-dir\") pod \"premium-w3-6a7148-kserve-67b976c888-tgd6c\" (UID: \"13de4441-e7f6-4cbd-97d9-0f079d11cb8e\") " pod="e2e-models-e2e-worker-w3-6a7148/premium-w3-6a7148-kserve-67b976c888-tgd6c" Oct 30 12:15:25.030400 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:25.030079 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbwmq\" (UniqueName: \"kubernetes.io/projected/13de4441-e7f6-4cbd-97d9-0f079d11cb8e-kube-api-access-tbwmq\") pod \"premium-w3-6a7148-kserve-67b976c888-tgd6c\" (UID: \"13de4441-e7f6-4cbd-97d9-0f079d11cb8e\") " pod="e2e-models-e2e-worker-w3-6a7148/premium-w3-6a7148-kserve-67b976c888-tgd6c" Oct 30 12:15:25.030400 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:25.030220 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13de4441-e7f6-4cbd-97d9-0f079d11cb8e-model-cache\") pod \"premium-w3-6a7148-kserve-67b976c888-tgd6c\" (UID: \"13de4441-e7f6-4cbd-97d9-0f079d11cb8e\") " pod="e2e-models-e2e-worker-w3-6a7148/premium-w3-6a7148-kserve-67b976c888-tgd6c" Oct 30 12:15:25.030400 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:25.030266 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13de4441-e7f6-4cbd-97d9-0f079d11cb8e-dshm\") pod \"premium-w3-6a7148-kserve-67b976c888-tgd6c\" (UID: \"13de4441-e7f6-4cbd-97d9-0f079d11cb8e\") " pod="e2e-models-e2e-worker-w3-6a7148/premium-w3-6a7148-kserve-67b976c888-tgd6c" Oct 30 12:15:25.131625 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:25.131580 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13de4441-e7f6-4cbd-97d9-0f079d11cb8e-tls-certs\") pod \"premium-w3-6a7148-kserve-67b976c888-tgd6c\" (UID: \"13de4441-e7f6-4cbd-97d9-0f079d11cb8e\") " pod="e2e-models-e2e-worker-w3-6a7148/premium-w3-6a7148-kserve-67b976c888-tgd6c" Oct 30 12:15:25.131625 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:25.131647 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13de4441-e7f6-4cbd-97d9-0f079d11cb8e-tmp-dir\") pod \"premium-w3-6a7148-kserve-67b976c888-tgd6c\" (UID: \"13de4441-e7f6-4cbd-97d9-0f079d11cb8e\") " pod="e2e-models-e2e-worker-w3-6a7148/premium-w3-6a7148-kserve-67b976c888-tgd6c" Oct 30 12:15:25.131949 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:25.131683 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbwmq\" (UniqueName: \"kubernetes.io/projected/13de4441-e7f6-4cbd-97d9-0f079d11cb8e-kube-api-access-tbwmq\") pod \"premium-w3-6a7148-kserve-67b976c888-tgd6c\" (UID: \"13de4441-e7f6-4cbd-97d9-0f079d11cb8e\") " pod="e2e-models-e2e-worker-w3-6a7148/premium-w3-6a7148-kserve-67b976c888-tgd6c" Oct 30 12:15:25.131949 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:25.131741 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13de4441-e7f6-4cbd-97d9-0f079d11cb8e-model-cache\") pod \"premium-w3-6a7148-kserve-67b976c888-tgd6c\" (UID: \"13de4441-e7f6-4cbd-97d9-0f079d11cb8e\") " pod="e2e-models-e2e-worker-w3-6a7148/premium-w3-6a7148-kserve-67b976c888-tgd6c" Oct 30 12:15:25.131949 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:25.131785 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13de4441-e7f6-4cbd-97d9-0f079d11cb8e-dshm\") pod \"premium-w3-6a7148-kserve-67b976c888-tgd6c\" (UID: \"13de4441-e7f6-4cbd-97d9-0f079d11cb8e\") " pod="e2e-models-e2e-worker-w3-6a7148/premium-w3-6a7148-kserve-67b976c888-tgd6c" Oct 30 12:15:25.131949 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:25.131877 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13de4441-e7f6-4cbd-97d9-0f079d11cb8e-home\") pod \"premium-w3-6a7148-kserve-67b976c888-tgd6c\" (UID: \"13de4441-e7f6-4cbd-97d9-0f079d11cb8e\") " pod="e2e-models-e2e-worker-w3-6a7148/premium-w3-6a7148-kserve-67b976c888-tgd6c" Oct 30 12:15:25.132158 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:25.132135 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13de4441-e7f6-4cbd-97d9-0f079d11cb8e-tmp-dir\") pod \"premium-w3-6a7148-kserve-67b976c888-tgd6c\" (UID: \"13de4441-e7f6-4cbd-97d9-0f079d11cb8e\") " pod="e2e-models-e2e-worker-w3-6a7148/premium-w3-6a7148-kserve-67b976c888-tgd6c" Oct 30 12:15:25.132217 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:25.132170 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13de4441-e7f6-4cbd-97d9-0f079d11cb8e-home\") pod \"premium-w3-6a7148-kserve-67b976c888-tgd6c\" (UID: \"13de4441-e7f6-4cbd-97d9-0f079d11cb8e\") " pod="e2e-models-e2e-worker-w3-6a7148/premium-w3-6a7148-kserve-67b976c888-tgd6c" Oct 30 12:15:25.132705 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:25.132399 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13de4441-e7f6-4cbd-97d9-0f079d11cb8e-model-cache\") pod \"premium-w3-6a7148-kserve-67b976c888-tgd6c\" (UID: \"13de4441-e7f6-4cbd-97d9-0f079d11cb8e\") " pod="e2e-models-e2e-worker-w3-6a7148/premium-w3-6a7148-kserve-67b976c888-tgd6c" Oct 30 12:15:25.135915 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:25.135104 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13de4441-e7f6-4cbd-97d9-0f079d11cb8e-tls-certs\") pod \"premium-w3-6a7148-kserve-67b976c888-tgd6c\" (UID: \"13de4441-e7f6-4cbd-97d9-0f079d11cb8e\") " pod="e2e-models-e2e-worker-w3-6a7148/premium-w3-6a7148-kserve-67b976c888-tgd6c" Oct 30 12:15:25.136077 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:25.136054 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13de4441-e7f6-4cbd-97d9-0f079d11cb8e-dshm\") pod \"premium-w3-6a7148-kserve-67b976c888-tgd6c\" (UID: \"13de4441-e7f6-4cbd-97d9-0f079d11cb8e\") " pod="e2e-models-e2e-worker-w3-6a7148/premium-w3-6a7148-kserve-67b976c888-tgd6c" Oct 30 12:15:25.155457 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:25.155422 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbwmq\" (UniqueName: \"kubernetes.io/projected/13de4441-e7f6-4cbd-97d9-0f079d11cb8e-kube-api-access-tbwmq\") pod \"premium-w3-6a7148-kserve-67b976c888-tgd6c\" (UID: \"13de4441-e7f6-4cbd-97d9-0f079d11cb8e\") " pod="e2e-models-e2e-worker-w3-6a7148/premium-w3-6a7148-kserve-67b976c888-tgd6c" Oct 30 12:15:25.242252 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:25.242219 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-6a7148/premium-w3-6a7148-kserve-67b976c888-tgd6c" Oct 30 12:15:25.554510 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:25.554419 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-6a7148/premium-w3-6a7148-kserve-67b976c888-tgd6c"] Oct 30 12:15:25.559055 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:15:25.559026 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod13de4441_e7f6_4cbd_97d9_0f079d11cb8e.slice/crio-30a5a68a1d364f0a126a56f395a30865b8f4c48d5be11457c3d2e60d32a77dc1 WatchSource:0}: Error finding container 30a5a68a1d364f0a126a56f395a30865b8f4c48d5be11457c3d2e60d32a77dc1: Status 404 returned error can't find the container with id 30a5a68a1d364f0a126a56f395a30865b8f4c48d5be11457c3d2e60d32a77dc1 Oct 30 12:15:25.697912 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:25.697867 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-6a7148/premium-w3-6a7148-kserve-67b976c888-tgd6c" event={"ID":"13de4441-e7f6-4cbd-97d9-0f079d11cb8e","Type":"ContainerStarted","Data":"0c2b0c30219fc8431eaeece2d6b59fba5f7f81e9b1b3409e606e6f926c591388"} Oct 30 12:15:25.698121 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:25.697924 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-6a7148/premium-w3-6a7148-kserve-67b976c888-tgd6c" event={"ID":"13de4441-e7f6-4cbd-97d9-0f079d11cb8e","Type":"ContainerStarted","Data":"30a5a68a1d364f0a126a56f395a30865b8f4c48d5be11457c3d2e60d32a77dc1"} Oct 30 12:15:25.700105 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:25.700070 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-qrkpg" event={"ID":"8cf0a817-eff3-454d-8ddc-0a811a951647","Type":"ContainerStarted","Data":"f0026a8a3630be0abba041c0b62b4bf389e9db608c517bb959d11f4d1f27b561"} Oct 30 12:15:25.740279 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:25.740225 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-6a7148/premium-w3-6a7148-kserve-67b976c888-tgd6c" podStartSLOduration=1.740210814 podStartE2EDuration="1.740210814s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:15:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:15:25.736744287 +0000 UTC m=+1282.251173034" watchObservedRunningTime="2026-10-30 12:15:25.740210814 +0000 UTC m=+1282.254639559" Oct 30 12:15:26.242983 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:26.242946 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-6a7148/premium-w3-6a7148-kserve-67b976c888-tgd6c" Oct 30 12:15:26.328639 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:26.328602 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-qrkpg" Oct 30 12:15:26.330505 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:26.330472 4908 patch_prober.go:29] interesting pod/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-qrkpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Oct 30 12:15:26.330624 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:26.330534 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-qrkpg" podUID="8cf0a817-eff3-454d-8ddc-0a811a951647" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Oct 30 12:15:27.329231 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:27.329192 4908 patch_prober.go:29] interesting pod/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-qrkpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Oct 30 12:15:27.329632 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:27.329248 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-qrkpg" podUID="8cf0a817-eff3-454d-8ddc-0a811a951647" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Oct 30 12:15:28.328678 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:28.328637 4908 patch_prober.go:29] interesting pod/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-qrkpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Oct 30 12:15:28.328918 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:28.328693 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-qrkpg" podUID="8cf0a817-eff3-454d-8ddc-0a811a951647" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Oct 30 12:15:29.328900 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:29.328865 4908 patch_prober.go:29] interesting pod/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-qrkpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Oct 30 12:15:29.329403 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:29.328927 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-qrkpg" podUID="8cf0a817-eff3-454d-8ddc-0a811a951647" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Oct 30 12:15:30.328817 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:30.328772 4908 patch_prober.go:29] interesting pod/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-qrkpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Oct 30 12:15:30.329000 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:30.328851 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-qrkpg" podUID="8cf0a817-eff3-454d-8ddc-0a811a951647" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Oct 30 12:15:31.328916 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:31.328876 4908 patch_prober.go:29] interesting pod/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-qrkpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Oct 30 12:15:31.329085 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:31.328932 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-qrkpg" podUID="8cf0a817-eff3-454d-8ddc-0a811a951647" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Oct 30 12:15:32.334009 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:32.333975 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-qrkpg" Oct 30 12:15:32.334455 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:32.334376 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-qrkpg" Oct 30 12:15:32.335267 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:32.335250 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-qrkpg" Oct 30 12:15:32.393791 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:32.393722 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-qrkpg" podStartSLOduration=9.393694782 podStartE2EDuration="9.393694782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:15:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:15:25.79324507 +0000 UTC m=+1282.307673834" watchObservedRunningTime="2026-10-30 12:15:32.393694782 +0000 UTC m=+1288.908123529" Oct 30 12:15:33.598320 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:33.598272 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9wwfc"] Oct 30 12:15:35.242880 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:35.242817 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-6a7148/premium-w3-6a7148-kserve-67b976c888-tgd6c" Oct 30 12:15:35.256703 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:35.256666 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-6a7148/premium-w3-6a7148-kserve-67b976c888-tgd6c" Oct 30 12:15:35.779356 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:35.779316 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-6a7148/premium-w3-6a7148-kserve-67b976c888-tgd6c" Oct 30 12:15:46.649524 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:46.649431 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2a841b0-wnzdf"] Oct 30 12:15:46.655040 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:46.655013 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2a841b0-wnzdf" Oct 30 12:15:46.668534 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:46.668503 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 12:15:46.680294 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:46.680266 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s75hq\"" Oct 30 12:15:46.682903 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:46.682874 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 12:15:46.683082 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:46.682972 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 12:15:46.687493 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:46.687464 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2a841b0-wnzdf"] Oct 30 12:15:46.691195 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:46.691162 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/025f63d3-09a6-401f-8403-98241d3a4e24-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2a841b0-wnzdf\" (UID: \"025f63d3-09a6-401f-8403-98241d3a4e24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2a841b0-wnzdf" Oct 30 12:15:46.691320 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:46.691227 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/025f63d3-09a6-401f-8403-98241d3a4e24-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2a841b0-wnzdf\" (UID: \"025f63d3-09a6-401f-8403-98241d3a4e24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2a841b0-wnzdf" Oct 30 12:15:46.691428 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:46.691407 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjs46\" (UniqueName: \"kubernetes.io/projected/025f63d3-09a6-401f-8403-98241d3a4e24-kube-api-access-wjs46\") pod \"maas-api-revoke-sub-e2a841b0-wnzdf\" (UID: \"025f63d3-09a6-401f-8403-98241d3a4e24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2a841b0-wnzdf" Oct 30 12:15:46.792001 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:46.791962 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/025f63d3-09a6-401f-8403-98241d3a4e24-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2a841b0-wnzdf\" (UID: \"025f63d3-09a6-401f-8403-98241d3a4e24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2a841b0-wnzdf" Oct 30 12:15:46.792214 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:46.792012 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/025f63d3-09a6-401f-8403-98241d3a4e24-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2a841b0-wnzdf\" (UID: \"025f63d3-09a6-401f-8403-98241d3a4e24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2a841b0-wnzdf" Oct 30 12:15:46.792214 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:46.792074 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjs46\" (UniqueName: \"kubernetes.io/projected/025f63d3-09a6-401f-8403-98241d3a4e24-kube-api-access-wjs46\") pod \"maas-api-revoke-sub-e2a841b0-wnzdf\" (UID: \"025f63d3-09a6-401f-8403-98241d3a4e24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2a841b0-wnzdf" Oct 30 12:15:46.792921 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:46.792890 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/025f63d3-09a6-401f-8403-98241d3a4e24-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2a841b0-wnzdf\" (UID: \"025f63d3-09a6-401f-8403-98241d3a4e24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2a841b0-wnzdf" Oct 30 12:15:46.794548 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:46.794517 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/025f63d3-09a6-401f-8403-98241d3a4e24-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2a841b0-wnzdf\" (UID: \"025f63d3-09a6-401f-8403-98241d3a4e24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2a841b0-wnzdf" Oct 30 12:15:46.814177 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:46.814145 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjs46\" (UniqueName: \"kubernetes.io/projected/025f63d3-09a6-401f-8403-98241d3a4e24-kube-api-access-wjs46\") pod \"maas-api-revoke-sub-e2a841b0-wnzdf\" (UID: \"025f63d3-09a6-401f-8403-98241d3a4e24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2a841b0-wnzdf" Oct 30 12:15:46.970046 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:46.969943 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2a841b0-wnzdf" Oct 30 12:15:47.273561 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:47.273522 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2a841b0-wnzdf"] Oct 30 12:15:47.277710 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:15:47.277672 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod025f63d3_09a6_401f_8403_98241d3a4e24.slice/crio-de1e5be05f85c0dcf4a71697623de2f9d09b22f020832e988406a1e085e5559d WatchSource:0}: Error finding container de1e5be05f85c0dcf4a71697623de2f9d09b22f020832e988406a1e085e5559d: Status 404 returned error can't find the container with id de1e5be05f85c0dcf4a71697623de2f9d09b22f020832e988406a1e085e5559d Oct 30 12:15:47.847464 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:47.847372 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2a841b0-wnzdf" event={"ID":"025f63d3-09a6-401f-8403-98241d3a4e24","Type":"ContainerStarted","Data":"ad7d08afbd2bc0372f06b82b997ae0659db7c10388b55545f92d849665bfb202"} Oct 30 12:15:47.847464 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:47.847446 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2a841b0-wnzdf" event={"ID":"025f63d3-09a6-401f-8403-98241d3a4e24","Type":"ContainerStarted","Data":"de1e5be05f85c0dcf4a71697623de2f9d09b22f020832e988406a1e085e5559d"} Oct 30 12:15:47.886617 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:47.886558 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2a841b0-wnzdf" podStartSLOduration=1.886544571 podStartE2EDuration="1.886544571s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:15:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:15:47.884611764 +0000 UTC m=+1304.399040512" watchObservedRunningTime="2026-10-30 12:15:47.886544571 +0000 UTC m=+1304.400973331" Oct 30 12:15:48.301518 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:48.301472 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9wwfc"] Oct 30 12:15:48.863973 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:48.863933 4908 generic.go:356] "Generic (PLEG): container finished" podID="025f63d3-09a6-401f-8403-98241d3a4e24" containerID="ad7d08afbd2bc0372f06b82b997ae0659db7c10388b55545f92d849665bfb202" exitCode=0 Oct 30 12:15:48.864568 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:48.864004 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2a841b0-wnzdf" event={"ID":"025f63d3-09a6-401f-8403-98241d3a4e24","Type":"ContainerDied","Data":"ad7d08afbd2bc0372f06b82b997ae0659db7c10388b55545f92d849665bfb202"} Oct 30 12:15:49.684328 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:15:49.684290 4908 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vn6vv" containerName="extractor" Oct 30 12:15:50.017069 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:50.017038 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2a841b0-wnzdf" Oct 30 12:15:50.133102 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:50.133064 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/025f63d3-09a6-401f-8403-98241d3a4e24-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/025f63d3-09a6-401f-8403-98241d3a4e24-maas-api-xks-ca\") pod \"025f63d3-09a6-401f-8403-98241d3a4e24\" (UID: \"025f63d3-09a6-401f-8403-98241d3a4e24\") " Oct 30 12:15:50.133315 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:50.133241 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/025f63d3-09a6-401f-8403-98241d3a4e24-kube-api-access-wjs46\" (UniqueName: \"kubernetes.io/projected/025f63d3-09a6-401f-8403-98241d3a4e24-kube-api-access-wjs46\") pod \"025f63d3-09a6-401f-8403-98241d3a4e24\" (UID: \"025f63d3-09a6-401f-8403-98241d3a4e24\") " Oct 30 12:15:50.133383 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:50.133313 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/025f63d3-09a6-401f-8403-98241d3a4e24-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/025f63d3-09a6-401f-8403-98241d3a4e24-maas-api-openshift-ca\") pod \"025f63d3-09a6-401f-8403-98241d3a4e24\" (UID: \"025f63d3-09a6-401f-8403-98241d3a4e24\") " Oct 30 12:15:50.133747 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:50.133713 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/025f63d3-09a6-401f-8403-98241d3a4e24-maas-api-openshift-ca" pod "025f63d3-09a6-401f-8403-98241d3a4e24" (UID: "025f63d3-09a6-401f-8403-98241d3a4e24"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:15:50.135445 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:50.135418 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/025f63d3-09a6-401f-8403-98241d3a4e24-kube-api-access-wjs46" pod "025f63d3-09a6-401f-8403-98241d3a4e24" (UID: "025f63d3-09a6-401f-8403-98241d3a4e24"). InnerVolumeSpecName "kube-api-access-wjs46". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:15:50.135597 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:50.135577 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/025f63d3-09a6-401f-8403-98241d3a4e24-maas-api-xks-ca" pod "025f63d3-09a6-401f-8403-98241d3a4e24" (UID: "025f63d3-09a6-401f-8403-98241d3a4e24"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:15:50.234952 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:50.234917 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/025f63d3-09a6-401f-8403-98241d3a4e24-maas-api-xks-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:15:50.234952 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:50.234946 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjs46\" (UniqueName: \"kubernetes.io/projected/025f63d3-09a6-401f-8403-98241d3a4e24-kube-api-access-wjs46\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:15:50.234952 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:50.234956 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/025f63d3-09a6-401f-8403-98241d3a4e24-maas-api-openshift-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:15:50.878438 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:50.878404 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2a841b0-wnzdf" Oct 30 12:15:50.878438 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:50.878427 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2a841b0-wnzdf" event={"ID":"025f63d3-09a6-401f-8403-98241d3a4e24","Type":"ContainerDied","Data":"de1e5be05f85c0dcf4a71697623de2f9d09b22f020832e988406a1e085e5559d"} Oct 30 12:15:50.878724 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:15:50.878482 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="de1e5be05f85c0dcf4a71697623de2f9d09b22f020832e988406a1e085e5559d" Oct 30 12:16:00.795567 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:00.795536 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7024660-sqz4g"] Oct 30 12:16:00.797969 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:00.796047 4908 cpu_manager.go:414] "containerMap: removing container" podUID="025f63d3-09a6-401f-8403-98241d3a4e24" containerName="revoke-keys" Oct 30 12:16:00.797969 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:00.796069 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="025f63d3-09a6-401f-8403-98241d3a4e24" containerName="revoke-keys" Oct 30 12:16:00.797969 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:00.796169 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="025f63d3-09a6-401f-8403-98241d3a4e24" containerName="revoke-keys" Oct 30 12:16:00.799233 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:00.799218 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7024660-sqz4g" Oct 30 12:16:00.814976 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:00.814959 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 12:16:00.826033 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:00.826005 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 12:16:00.826171 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:00.826040 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 12:16:00.826171 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:00.826053 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s75hq\"" Oct 30 12:16:00.828914 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:00.828889 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7024660-sqz4g"] Oct 30 12:16:00.944378 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:00.944340 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t87m7\" (UniqueName: \"kubernetes.io/projected/c36c58da-716d-47d8-ac5d-8230d92f60f3-kube-api-access-t87m7\") pod \"maas-api-revoke-sub-d7024660-sqz4g\" (UID: \"c36c58da-716d-47d8-ac5d-8230d92f60f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7024660-sqz4g" Oct 30 12:16:00.944551 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:00.944457 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c36c58da-716d-47d8-ac5d-8230d92f60f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d7024660-sqz4g\" (UID: \"c36c58da-716d-47d8-ac5d-8230d92f60f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7024660-sqz4g" Oct 30 12:16:00.944551 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:00.944528 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c36c58da-716d-47d8-ac5d-8230d92f60f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d7024660-sqz4g\" (UID: \"c36c58da-716d-47d8-ac5d-8230d92f60f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7024660-sqz4g" Oct 30 12:16:01.046148 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:01.046048 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t87m7\" (UniqueName: \"kubernetes.io/projected/c36c58da-716d-47d8-ac5d-8230d92f60f3-kube-api-access-t87m7\") pod \"maas-api-revoke-sub-d7024660-sqz4g\" (UID: \"c36c58da-716d-47d8-ac5d-8230d92f60f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7024660-sqz4g" Oct 30 12:16:01.046350 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:01.046161 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c36c58da-716d-47d8-ac5d-8230d92f60f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d7024660-sqz4g\" (UID: \"c36c58da-716d-47d8-ac5d-8230d92f60f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7024660-sqz4g" Oct 30 12:16:01.046350 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:01.046319 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c36c58da-716d-47d8-ac5d-8230d92f60f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d7024660-sqz4g\" (UID: \"c36c58da-716d-47d8-ac5d-8230d92f60f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7024660-sqz4g" Oct 30 12:16:01.047058 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:01.047030 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c36c58da-716d-47d8-ac5d-8230d92f60f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d7024660-sqz4g\" (UID: \"c36c58da-716d-47d8-ac5d-8230d92f60f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7024660-sqz4g" Oct 30 12:16:01.048811 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:01.048786 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c36c58da-716d-47d8-ac5d-8230d92f60f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d7024660-sqz4g\" (UID: \"c36c58da-716d-47d8-ac5d-8230d92f60f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7024660-sqz4g" Oct 30 12:16:01.068430 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:01.068396 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t87m7\" (UniqueName: \"kubernetes.io/projected/c36c58da-716d-47d8-ac5d-8230d92f60f3-kube-api-access-t87m7\") pod \"maas-api-revoke-sub-d7024660-sqz4g\" (UID: \"c36c58da-716d-47d8-ac5d-8230d92f60f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7024660-sqz4g" Oct 30 12:16:01.113068 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:01.113039 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7024660-sqz4g" Oct 30 12:16:01.422861 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:01.422771 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7024660-sqz4g"] Oct 30 12:16:01.426562 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:16:01.426527 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc36c58da_716d_47d8_ac5d_8230d92f60f3.slice/crio-51c7f97460fcd473061029ce6997463ee8c84feab543c48d274452adf31cea88 WatchSource:0}: Error finding container 51c7f97460fcd473061029ce6997463ee8c84feab543c48d274452adf31cea88: Status 404 returned error can't find the container with id 51c7f97460fcd473061029ce6997463ee8c84feab543c48d274452adf31cea88 Oct 30 12:16:01.948052 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:01.948021 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7024660-sqz4g" event={"ID":"c36c58da-716d-47d8-ac5d-8230d92f60f3","Type":"ContainerStarted","Data":"9a2dc1d3afb0fc36dd8fa5a5f60dfa3397a176f33b44c018966c9f61cb28d8ff"} Oct 30 12:16:01.948421 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:01.948074 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7024660-sqz4g" event={"ID":"c36c58da-716d-47d8-ac5d-8230d92f60f3","Type":"ContainerStarted","Data":"51c7f97460fcd473061029ce6997463ee8c84feab543c48d274452adf31cea88"} Oct 30 12:16:01.987970 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:01.987901 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7024660-sqz4g" podStartSLOduration=1.987882967 podStartE2EDuration="1.987882967s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:16:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:16:01.985673867 +0000 UTC m=+1318.500102642" watchObservedRunningTime="2026-10-30 12:16:01.987882967 +0000 UTC m=+1318.502311707" Oct 30 12:16:02.808747 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:02.808710 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e534bd0-rhl8b"] Oct 30 12:16:02.812986 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:02.812965 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e534bd0-rhl8b" Oct 30 12:16:02.833001 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:02.832976 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e534bd0-rhl8b"] Oct 30 12:16:02.955217 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:02.955183 4908 generic.go:356] "Generic (PLEG): container finished" podID="c36c58da-716d-47d8-ac5d-8230d92f60f3" containerID="9a2dc1d3afb0fc36dd8fa5a5f60dfa3397a176f33b44c018966c9f61cb28d8ff" exitCode=0 Oct 30 12:16:02.955575 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:02.955266 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7024660-sqz4g" event={"ID":"c36c58da-716d-47d8-ac5d-8230d92f60f3","Type":"ContainerDied","Data":"9a2dc1d3afb0fc36dd8fa5a5f60dfa3397a176f33b44c018966c9f61cb28d8ff"} Oct 30 12:16:02.964512 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:02.964488 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5db1278-dd68-45f3-a5ed-384b3d9a721d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e534bd0-rhl8b\" (UID: \"f5db1278-dd68-45f3-a5ed-384b3d9a721d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e534bd0-rhl8b" Oct 30 12:16:02.964645 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:02.964535 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5db1278-dd68-45f3-a5ed-384b3d9a721d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e534bd0-rhl8b\" (UID: \"f5db1278-dd68-45f3-a5ed-384b3d9a721d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e534bd0-rhl8b" Oct 30 12:16:02.964686 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:02.964655 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vgpm\" (UniqueName: \"kubernetes.io/projected/f5db1278-dd68-45f3-a5ed-384b3d9a721d-kube-api-access-4vgpm\") pod \"maas-api-revoke-sub-7e534bd0-rhl8b\" (UID: \"f5db1278-dd68-45f3-a5ed-384b3d9a721d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e534bd0-rhl8b" Oct 30 12:16:03.065775 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:03.065681 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5db1278-dd68-45f3-a5ed-384b3d9a721d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e534bd0-rhl8b\" (UID: \"f5db1278-dd68-45f3-a5ed-384b3d9a721d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e534bd0-rhl8b" Oct 30 12:16:03.065941 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:03.065800 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vgpm\" (UniqueName: \"kubernetes.io/projected/f5db1278-dd68-45f3-a5ed-384b3d9a721d-kube-api-access-4vgpm\") pod \"maas-api-revoke-sub-7e534bd0-rhl8b\" (UID: \"f5db1278-dd68-45f3-a5ed-384b3d9a721d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e534bd0-rhl8b" Oct 30 12:16:03.065941 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:03.065908 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5db1278-dd68-45f3-a5ed-384b3d9a721d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e534bd0-rhl8b\" (UID: \"f5db1278-dd68-45f3-a5ed-384b3d9a721d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e534bd0-rhl8b" Oct 30 12:16:03.066522 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:03.066497 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5db1278-dd68-45f3-a5ed-384b3d9a721d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e534bd0-rhl8b\" (UID: \"f5db1278-dd68-45f3-a5ed-384b3d9a721d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e534bd0-rhl8b" Oct 30 12:16:03.068119 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:03.068092 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5db1278-dd68-45f3-a5ed-384b3d9a721d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e534bd0-rhl8b\" (UID: \"f5db1278-dd68-45f3-a5ed-384b3d9a721d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e534bd0-rhl8b" Oct 30 12:16:03.087598 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:03.087568 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vgpm\" (UniqueName: \"kubernetes.io/projected/f5db1278-dd68-45f3-a5ed-384b3d9a721d-kube-api-access-4vgpm\") pod \"maas-api-revoke-sub-7e534bd0-rhl8b\" (UID: \"f5db1278-dd68-45f3-a5ed-384b3d9a721d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e534bd0-rhl8b" Oct 30 12:16:03.126408 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:03.126381 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e534bd0-rhl8b" Oct 30 12:16:03.430008 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:03.429929 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e534bd0-rhl8b"] Oct 30 12:16:03.433963 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:16:03.433937 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5db1278_dd68_45f3_a5ed_384b3d9a721d.slice/crio-6c3af86e8d466adf41de20936c21d6e9fb28b61ec03d50463a4e9dc2536aef76 WatchSource:0}: Error finding container 6c3af86e8d466adf41de20936c21d6e9fb28b61ec03d50463a4e9dc2536aef76: Status 404 returned error can't find the container with id 6c3af86e8d466adf41de20936c21d6e9fb28b61ec03d50463a4e9dc2536aef76 Oct 30 12:16:03.965318 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:03.965277 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e534bd0-rhl8b" event={"ID":"f5db1278-dd68-45f3-a5ed-384b3d9a721d","Type":"ContainerStarted","Data":"9a1867aeeb0939f788a96922b46eda07aecfdff6db2e9fff725dab1a86849d68"} Oct 30 12:16:03.965732 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:03.965336 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e534bd0-rhl8b" event={"ID":"f5db1278-dd68-45f3-a5ed-384b3d9a721d","Type":"ContainerStarted","Data":"6c3af86e8d466adf41de20936c21d6e9fb28b61ec03d50463a4e9dc2536aef76"} Oct 30 12:16:04.005945 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:04.005870 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e534bd0-rhl8b" podStartSLOduration=2.005848741 podStartE2EDuration="2.005848741s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:16:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:16:04.004151816 +0000 UTC m=+1320.518580563" watchObservedRunningTime="2026-10-30 12:16:04.005848741 +0000 UTC m=+1320.520277501" Oct 30 12:16:04.136591 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:04.136561 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7024660-sqz4g" Oct 30 12:16:04.281110 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:04.281015 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c36c58da-716d-47d8-ac5d-8230d92f60f3-kube-api-access-t87m7\" (UniqueName: \"kubernetes.io/projected/c36c58da-716d-47d8-ac5d-8230d92f60f3-kube-api-access-t87m7\") pod \"c36c58da-716d-47d8-ac5d-8230d92f60f3\" (UID: \"c36c58da-716d-47d8-ac5d-8230d92f60f3\") " Oct 30 12:16:04.281283 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:04.281229 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c36c58da-716d-47d8-ac5d-8230d92f60f3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c36c58da-716d-47d8-ac5d-8230d92f60f3-maas-api-xks-ca\") pod \"c36c58da-716d-47d8-ac5d-8230d92f60f3\" (UID: \"c36c58da-716d-47d8-ac5d-8230d92f60f3\") " Oct 30 12:16:04.281359 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:04.281309 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c36c58da-716d-47d8-ac5d-8230d92f60f3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c36c58da-716d-47d8-ac5d-8230d92f60f3-maas-api-openshift-ca\") pod \"c36c58da-716d-47d8-ac5d-8230d92f60f3\" (UID: \"c36c58da-716d-47d8-ac5d-8230d92f60f3\") " Oct 30 12:16:04.281720 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:04.281687 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c36c58da-716d-47d8-ac5d-8230d92f60f3-maas-api-openshift-ca" pod "c36c58da-716d-47d8-ac5d-8230d92f60f3" (UID: "c36c58da-716d-47d8-ac5d-8230d92f60f3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:16:04.283392 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:04.283361 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c36c58da-716d-47d8-ac5d-8230d92f60f3-maas-api-xks-ca" pod "c36c58da-716d-47d8-ac5d-8230d92f60f3" (UID: "c36c58da-716d-47d8-ac5d-8230d92f60f3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:16:04.283500 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:04.283483 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c36c58da-716d-47d8-ac5d-8230d92f60f3-kube-api-access-t87m7" pod "c36c58da-716d-47d8-ac5d-8230d92f60f3" (UID: "c36c58da-716d-47d8-ac5d-8230d92f60f3"). InnerVolumeSpecName "kube-api-access-t87m7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:16:04.382557 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:04.382523 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c36c58da-716d-47d8-ac5d-8230d92f60f3-maas-api-xks-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:16:04.382557 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:04.382552 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c36c58da-716d-47d8-ac5d-8230d92f60f3-maas-api-openshift-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:16:04.382557 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:04.382561 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t87m7\" (UniqueName: \"kubernetes.io/projected/c36c58da-716d-47d8-ac5d-8230d92f60f3-kube-api-access-t87m7\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:16:04.973927 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:04.973887 4908 generic.go:356] "Generic (PLEG): container finished" podID="f5db1278-dd68-45f3-a5ed-384b3d9a721d" containerID="9a1867aeeb0939f788a96922b46eda07aecfdff6db2e9fff725dab1a86849d68" exitCode=0 Oct 30 12:16:04.974395 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:04.973981 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e534bd0-rhl8b" event={"ID":"f5db1278-dd68-45f3-a5ed-384b3d9a721d","Type":"ContainerDied","Data":"9a1867aeeb0939f788a96922b46eda07aecfdff6db2e9fff725dab1a86849d68"} Oct 30 12:16:04.975414 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:04.975384 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7024660-sqz4g" event={"ID":"c36c58da-716d-47d8-ac5d-8230d92f60f3","Type":"ContainerDied","Data":"51c7f97460fcd473061029ce6997463ee8c84feab543c48d274452adf31cea88"} Oct 30 12:16:04.975531 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:04.975415 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7024660-sqz4g" Oct 30 12:16:04.975531 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:04.975432 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="51c7f97460fcd473061029ce6997463ee8c84feab543c48d274452adf31cea88" Oct 30 12:16:06.126869 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:06.126823 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e534bd0-rhl8b" Oct 30 12:16:06.301981 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:06.301903 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5db1278-dd68-45f3-a5ed-384b3d9a721d-kube-api-access-4vgpm\" (UniqueName: \"kubernetes.io/projected/f5db1278-dd68-45f3-a5ed-384b3d9a721d-kube-api-access-4vgpm\") pod \"f5db1278-dd68-45f3-a5ed-384b3d9a721d\" (UID: \"f5db1278-dd68-45f3-a5ed-384b3d9a721d\") " Oct 30 12:16:06.302221 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:06.302028 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f5db1278-dd68-45f3-a5ed-384b3d9a721d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5db1278-dd68-45f3-a5ed-384b3d9a721d-maas-api-xks-ca\") pod \"f5db1278-dd68-45f3-a5ed-384b3d9a721d\" (UID: \"f5db1278-dd68-45f3-a5ed-384b3d9a721d\") " Oct 30 12:16:06.302221 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:06.302074 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5db1278-dd68-45f3-a5ed-384b3d9a721d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5db1278-dd68-45f3-a5ed-384b3d9a721d-maas-api-openshift-ca\") pod \"f5db1278-dd68-45f3-a5ed-384b3d9a721d\" (UID: \"f5db1278-dd68-45f3-a5ed-384b3d9a721d\") " Oct 30 12:16:06.302508 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:06.302480 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5db1278-dd68-45f3-a5ed-384b3d9a721d-maas-api-openshift-ca" pod "f5db1278-dd68-45f3-a5ed-384b3d9a721d" (UID: "f5db1278-dd68-45f3-a5ed-384b3d9a721d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:16:06.304109 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:06.304081 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5db1278-dd68-45f3-a5ed-384b3d9a721d-maas-api-xks-ca" pod "f5db1278-dd68-45f3-a5ed-384b3d9a721d" (UID: "f5db1278-dd68-45f3-a5ed-384b3d9a721d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:16:06.304286 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:06.304262 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5db1278-dd68-45f3-a5ed-384b3d9a721d-kube-api-access-4vgpm" pod "f5db1278-dd68-45f3-a5ed-384b3d9a721d" (UID: "f5db1278-dd68-45f3-a5ed-384b3d9a721d"). InnerVolumeSpecName "kube-api-access-4vgpm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:16:06.403247 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:06.403205 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5db1278-dd68-45f3-a5ed-384b3d9a721d-maas-api-xks-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:16:06.403247 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:06.403236 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5db1278-dd68-45f3-a5ed-384b3d9a721d-maas-api-openshift-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:16:06.403247 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:06.403246 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vgpm\" (UniqueName: \"kubernetes.io/projected/f5db1278-dd68-45f3-a5ed-384b3d9a721d-kube-api-access-4vgpm\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:16:06.990502 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:06.990472 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e534bd0-rhl8b" Oct 30 12:16:06.990681 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:06.990463 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e534bd0-rhl8b" event={"ID":"f5db1278-dd68-45f3-a5ed-384b3d9a721d","Type":"ContainerDied","Data":"6c3af86e8d466adf41de20936c21d6e9fb28b61ec03d50463a4e9dc2536aef76"} Oct 30 12:16:06.990681 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:06.990625 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6c3af86e8d466adf41de20936c21d6e9fb28b61ec03d50463a4e9dc2536aef76" Oct 30 12:16:10.631509 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:10.631473 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9wwfc"] Oct 30 12:16:15.367078 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:15.367025 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d554a235-x5ccx"] Oct 30 12:16:15.367712 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:15.367683 4908 cpu_manager.go:414] "containerMap: removing container" podUID="f5db1278-dd68-45f3-a5ed-384b3d9a721d" containerName="revoke-keys" Oct 30 12:16:15.367712 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:15.367710 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5db1278-dd68-45f3-a5ed-384b3d9a721d" containerName="revoke-keys" Oct 30 12:16:15.367967 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:15.367729 4908 cpu_manager.go:414] "containerMap: removing container" podUID="c36c58da-716d-47d8-ac5d-8230d92f60f3" containerName="revoke-keys" Oct 30 12:16:15.367967 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:15.367743 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c36c58da-716d-47d8-ac5d-8230d92f60f3" containerName="revoke-keys" Oct 30 12:16:15.367967 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:15.367858 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5db1278-dd68-45f3-a5ed-384b3d9a721d" containerName="revoke-keys" Oct 30 12:16:15.367967 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:15.367872 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="c36c58da-716d-47d8-ac5d-8230d92f60f3" containerName="revoke-keys" Oct 30 12:16:15.371414 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:15.371390 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b703a455-llp76"] Oct 30 12:16:15.371564 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:15.371531 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d554a235-x5ccx" Oct 30 12:16:15.375051 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:15.375031 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b703a455-llp76" Oct 30 12:16:15.385031 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:15.385009 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 12:16:15.385324 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:15.385290 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s75hq\"" Oct 30 12:16:15.385324 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:15.385317 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 12:16:15.388713 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:15.388686 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfeb8f6c-afd6-4dd2-99af-a4829db10a20-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d554a235-x5ccx\" (UID: \"dfeb8f6c-afd6-4dd2-99af-a4829db10a20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d554a235-x5ccx" Oct 30 12:16:15.388914 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:15.388893 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjr25\" (UniqueName: \"kubernetes.io/projected/dfeb8f6c-afd6-4dd2-99af-a4829db10a20-kube-api-access-bjr25\") pod \"maas-api-revoke-sub-d554a235-x5ccx\" (UID: \"dfeb8f6c-afd6-4dd2-99af-a4829db10a20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d554a235-x5ccx" Oct 30 12:16:15.388974 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:15.388942 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfeb8f6c-afd6-4dd2-99af-a4829db10a20-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d554a235-x5ccx\" (UID: \"dfeb8f6c-afd6-4dd2-99af-a4829db10a20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d554a235-x5ccx" Oct 30 12:16:15.396610 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:15.396584 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 12:16:15.402938 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:15.402894 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b703a455-llp76"] Oct 30 12:16:15.405252 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:15.405218 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d554a235-x5ccx"] Oct 30 12:16:15.490264 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:15.490222 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76c0e83d-ebe5-4e89-be6b-043568973731-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b703a455-llp76\" (UID: \"76c0e83d-ebe5-4e89-be6b-043568973731\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b703a455-llp76" Oct 30 12:16:15.490264 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:15.490273 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76c0e83d-ebe5-4e89-be6b-043568973731-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b703a455-llp76\" (UID: \"76c0e83d-ebe5-4e89-be6b-043568973731\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b703a455-llp76" Oct 30 12:16:15.490635 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:15.490385 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjr25\" (UniqueName: \"kubernetes.io/projected/dfeb8f6c-afd6-4dd2-99af-a4829db10a20-kube-api-access-bjr25\") pod \"maas-api-revoke-sub-d554a235-x5ccx\" (UID: \"dfeb8f6c-afd6-4dd2-99af-a4829db10a20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d554a235-x5ccx" Oct 30 12:16:15.490635 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:15.490434 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfeb8f6c-afd6-4dd2-99af-a4829db10a20-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d554a235-x5ccx\" (UID: \"dfeb8f6c-afd6-4dd2-99af-a4829db10a20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d554a235-x5ccx" Oct 30 12:16:15.490635 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:15.490503 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfeb8f6c-afd6-4dd2-99af-a4829db10a20-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d554a235-x5ccx\" (UID: \"dfeb8f6c-afd6-4dd2-99af-a4829db10a20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d554a235-x5ccx" Oct 30 12:16:15.490635 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:15.490584 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzwlt\" (UniqueName: \"kubernetes.io/projected/76c0e83d-ebe5-4e89-be6b-043568973731-kube-api-access-lzwlt\") pod \"maas-api-revoke-sub-b703a455-llp76\" (UID: \"76c0e83d-ebe5-4e89-be6b-043568973731\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b703a455-llp76" Oct 30 12:16:15.491316 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:15.491295 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfeb8f6c-afd6-4dd2-99af-a4829db10a20-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d554a235-x5ccx\" (UID: \"dfeb8f6c-afd6-4dd2-99af-a4829db10a20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d554a235-x5ccx" Oct 30 12:16:15.493148 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:15.493117 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfeb8f6c-afd6-4dd2-99af-a4829db10a20-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d554a235-x5ccx\" (UID: \"dfeb8f6c-afd6-4dd2-99af-a4829db10a20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d554a235-x5ccx" Oct 30 12:16:15.512627 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:15.512596 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjr25\" (UniqueName: \"kubernetes.io/projected/dfeb8f6c-afd6-4dd2-99af-a4829db10a20-kube-api-access-bjr25\") pod \"maas-api-revoke-sub-d554a235-x5ccx\" (UID: \"dfeb8f6c-afd6-4dd2-99af-a4829db10a20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d554a235-x5ccx" Oct 30 12:16:15.592021 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:15.591978 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzwlt\" (UniqueName: \"kubernetes.io/projected/76c0e83d-ebe5-4e89-be6b-043568973731-kube-api-access-lzwlt\") pod \"maas-api-revoke-sub-b703a455-llp76\" (UID: \"76c0e83d-ebe5-4e89-be6b-043568973731\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b703a455-llp76" Oct 30 12:16:15.592237 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:15.592069 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76c0e83d-ebe5-4e89-be6b-043568973731-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b703a455-llp76\" (UID: \"76c0e83d-ebe5-4e89-be6b-043568973731\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b703a455-llp76" Oct 30 12:16:15.592237 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:15.592102 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76c0e83d-ebe5-4e89-be6b-043568973731-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b703a455-llp76\" (UID: \"76c0e83d-ebe5-4e89-be6b-043568973731\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b703a455-llp76" Oct 30 12:16:15.593056 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:15.593016 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76c0e83d-ebe5-4e89-be6b-043568973731-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b703a455-llp76\" (UID: \"76c0e83d-ebe5-4e89-be6b-043568973731\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b703a455-llp76" Oct 30 12:16:15.601242 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:15.601208 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76c0e83d-ebe5-4e89-be6b-043568973731-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b703a455-llp76\" (UID: \"76c0e83d-ebe5-4e89-be6b-043568973731\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b703a455-llp76" Oct 30 12:16:15.614965 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:15.614933 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzwlt\" (UniqueName: \"kubernetes.io/projected/76c0e83d-ebe5-4e89-be6b-043568973731-kube-api-access-lzwlt\") pod \"maas-api-revoke-sub-b703a455-llp76\" (UID: \"76c0e83d-ebe5-4e89-be6b-043568973731\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b703a455-llp76" Oct 30 12:16:15.691995 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:15.691899 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d554a235-x5ccx" Oct 30 12:16:15.706363 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:15.706334 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b703a455-llp76" Oct 30 12:16:16.011425 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:16.011374 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d554a235-x5ccx"] Oct 30 12:16:16.013876 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:16:16.013845 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfeb8f6c_afd6_4dd2_99af_a4829db10a20.slice/crio-58dcfb37ffc41173596a2b72874d49acdd0090e4bcf85484b2a5d9625049e313 WatchSource:0}: Error finding container 58dcfb37ffc41173596a2b72874d49acdd0090e4bcf85484b2a5d9625049e313: Status 404 returned error can't find the container with id 58dcfb37ffc41173596a2b72874d49acdd0090e4bcf85484b2a5d9625049e313 Oct 30 12:16:16.022632 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:16.022601 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b703a455-llp76"] Oct 30 12:16:16.027822 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:16:16.027790 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76c0e83d_ebe5_4e89_be6b_043568973731.slice/crio-0b47c1272148f80e91cde1abc58ae62092e06e6663c9cc709cd3c347bca2fc38 WatchSource:0}: Error finding container 0b47c1272148f80e91cde1abc58ae62092e06e6663c9cc709cd3c347bca2fc38: Status 404 returned error can't find the container with id 0b47c1272148f80e91cde1abc58ae62092e06e6663c9cc709cd3c347bca2fc38 Oct 30 12:16:16.049433 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:16.049398 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b703a455-llp76" event={"ID":"76c0e83d-ebe5-4e89-be6b-043568973731","Type":"ContainerStarted","Data":"0b47c1272148f80e91cde1abc58ae62092e06e6663c9cc709cd3c347bca2fc38"} Oct 30 12:16:16.050821 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:16.050795 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d554a235-x5ccx" event={"ID":"dfeb8f6c-afd6-4dd2-99af-a4829db10a20","Type":"ContainerStarted","Data":"58dcfb37ffc41173596a2b72874d49acdd0090e4bcf85484b2a5d9625049e313"} Oct 30 12:16:17.059549 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:17.059498 4908 generic.go:356] "Generic (PLEG): container finished" podID="76c0e83d-ebe5-4e89-be6b-043568973731" containerID="6d0f0517fc6dc1f8f584c840277c316d6e140a762958499df768c036369af6fb" exitCode=0 Oct 30 12:16:17.060068 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:17.059625 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b703a455-llp76" event={"ID":"76c0e83d-ebe5-4e89-be6b-043568973731","Type":"ContainerDied","Data":"6d0f0517fc6dc1f8f584c840277c316d6e140a762958499df768c036369af6fb"} Oct 30 12:16:17.061775 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:17.061747 4908 generic.go:356] "Generic (PLEG): container finished" podID="dfeb8f6c-afd6-4dd2-99af-a4829db10a20" containerID="7756d5d15b4ff9f2a4fa1bcbe9ab639fd92451caaf336330fce2951e701ad21b" exitCode=0 Oct 30 12:16:17.061936 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:17.061783 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d554a235-x5ccx" event={"ID":"dfeb8f6c-afd6-4dd2-99af-a4829db10a20","Type":"ContainerDied","Data":"7756d5d15b4ff9f2a4fa1bcbe9ab639fd92451caaf336330fce2951e701ad21b"} Oct 30 12:16:17.088710 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:17.088671 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-wq96b"] Oct 30 12:16:17.092744 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:17.092693 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-wq96b" Oct 30 12:16:17.107047 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:17.107015 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-6a7148-openshift-default-dockercfg-49s7b\"" Oct 30 12:16:17.115942 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:17.115789 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-wq96b"] Oct 30 12:16:17.211697 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:17.211662 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8aaac2b2-a4e5-4736-96ec-30f545797512-workload-socket\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-wq96b\" (UID: \"8aaac2b2-a4e5-4736-96ec-30f545797512\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-wq96b" Oct 30 12:16:17.211697 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:17.211703 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5mk6\" (UniqueName: \"kubernetes.io/projected/8aaac2b2-a4e5-4736-96ec-30f545797512-kube-api-access-l5mk6\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-wq96b\" (UID: \"8aaac2b2-a4e5-4736-96ec-30f545797512\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-wq96b" Oct 30 12:16:17.211956 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:17.211734 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8aaac2b2-a4e5-4736-96ec-30f545797512-istio-token\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-wq96b\" (UID: \"8aaac2b2-a4e5-4736-96ec-30f545797512\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-wq96b" Oct 30 12:16:17.211956 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:17.211885 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8aaac2b2-a4e5-4736-96ec-30f545797512-istio-podinfo\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-wq96b\" (UID: \"8aaac2b2-a4e5-4736-96ec-30f545797512\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-wq96b" Oct 30 12:16:17.212025 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:17.211974 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8aaac2b2-a4e5-4736-96ec-30f545797512-istiod-ca-cert\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-wq96b\" (UID: \"8aaac2b2-a4e5-4736-96ec-30f545797512\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-wq96b" Oct 30 12:16:17.212025 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:17.212003 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8aaac2b2-a4e5-4736-96ec-30f545797512-workload-certs\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-wq96b\" (UID: \"8aaac2b2-a4e5-4736-96ec-30f545797512\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-wq96b" Oct 30 12:16:17.212025 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:17.212020 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8aaac2b2-a4e5-4736-96ec-30f545797512-istio-data\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-wq96b\" (UID: \"8aaac2b2-a4e5-4736-96ec-30f545797512\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-wq96b" Oct 30 12:16:17.212130 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:17.212118 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8aaac2b2-a4e5-4736-96ec-30f545797512-istio-envoy\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-wq96b\" (UID: \"8aaac2b2-a4e5-4736-96ec-30f545797512\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-wq96b" Oct 30 12:16:17.212177 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:17.212166 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8aaac2b2-a4e5-4736-96ec-30f545797512-credential-socket\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-wq96b\" (UID: \"8aaac2b2-a4e5-4736-96ec-30f545797512\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-wq96b" Oct 30 12:16:17.313498 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:17.313457 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8aaac2b2-a4e5-4736-96ec-30f545797512-workload-socket\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-wq96b\" (UID: \"8aaac2b2-a4e5-4736-96ec-30f545797512\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-wq96b" Oct 30 12:16:17.313498 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:17.313500 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5mk6\" (UniqueName: \"kubernetes.io/projected/8aaac2b2-a4e5-4736-96ec-30f545797512-kube-api-access-l5mk6\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-wq96b\" (UID: \"8aaac2b2-a4e5-4736-96ec-30f545797512\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-wq96b" Oct 30 12:16:17.313765 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:17.313520 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8aaac2b2-a4e5-4736-96ec-30f545797512-istio-token\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-wq96b\" (UID: \"8aaac2b2-a4e5-4736-96ec-30f545797512\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-wq96b" Oct 30 12:16:17.313765 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:17.313544 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8aaac2b2-a4e5-4736-96ec-30f545797512-istio-podinfo\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-wq96b\" (UID: \"8aaac2b2-a4e5-4736-96ec-30f545797512\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-wq96b" Oct 30 12:16:17.313765 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:17.313580 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8aaac2b2-a4e5-4736-96ec-30f545797512-istiod-ca-cert\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-wq96b\" (UID: \"8aaac2b2-a4e5-4736-96ec-30f545797512\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-wq96b" Oct 30 12:16:17.313765 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:17.313602 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8aaac2b2-a4e5-4736-96ec-30f545797512-workload-certs\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-wq96b\" (UID: \"8aaac2b2-a4e5-4736-96ec-30f545797512\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-wq96b" Oct 30 12:16:17.313765 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:17.313622 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8aaac2b2-a4e5-4736-96ec-30f545797512-istio-data\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-wq96b\" (UID: \"8aaac2b2-a4e5-4736-96ec-30f545797512\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-wq96b" Oct 30 12:16:17.313765 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:17.313669 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8aaac2b2-a4e5-4736-96ec-30f545797512-istio-envoy\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-wq96b\" (UID: \"8aaac2b2-a4e5-4736-96ec-30f545797512\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-wq96b" Oct 30 12:16:17.313765 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:17.313733 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8aaac2b2-a4e5-4736-96ec-30f545797512-credential-socket\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-wq96b\" (UID: \"8aaac2b2-a4e5-4736-96ec-30f545797512\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-wq96b" Oct 30 12:16:17.314168 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:17.314015 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8aaac2b2-a4e5-4736-96ec-30f545797512-workload-socket\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-wq96b\" (UID: \"8aaac2b2-a4e5-4736-96ec-30f545797512\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-wq96b" Oct 30 12:16:17.314168 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:17.314134 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8aaac2b2-a4e5-4736-96ec-30f545797512-workload-certs\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-wq96b\" (UID: \"8aaac2b2-a4e5-4736-96ec-30f545797512\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-wq96b" Oct 30 12:16:17.314283 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:17.314180 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8aaac2b2-a4e5-4736-96ec-30f545797512-istio-data\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-wq96b\" (UID: \"8aaac2b2-a4e5-4736-96ec-30f545797512\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-wq96b" Oct 30 12:16:17.314283 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:17.314260 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8aaac2b2-a4e5-4736-96ec-30f545797512-credential-socket\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-wq96b\" (UID: \"8aaac2b2-a4e5-4736-96ec-30f545797512\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-wq96b" Oct 30 12:16:17.314389 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:17.314318 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8aaac2b2-a4e5-4736-96ec-30f545797512-istiod-ca-cert\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-wq96b\" (UID: \"8aaac2b2-a4e5-4736-96ec-30f545797512\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-wq96b" Oct 30 12:16:17.316128 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:17.316105 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8aaac2b2-a4e5-4736-96ec-30f545797512-istio-envoy\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-wq96b\" (UID: \"8aaac2b2-a4e5-4736-96ec-30f545797512\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-wq96b" Oct 30 12:16:17.316447 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:17.316424 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8aaac2b2-a4e5-4736-96ec-30f545797512-istio-podinfo\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-wq96b\" (UID: \"8aaac2b2-a4e5-4736-96ec-30f545797512\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-wq96b" Oct 30 12:16:17.337360 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:17.337279 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8aaac2b2-a4e5-4736-96ec-30f545797512-istio-token\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-wq96b\" (UID: \"8aaac2b2-a4e5-4736-96ec-30f545797512\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-wq96b" Oct 30 12:16:17.338192 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:17.338150 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5mk6\" (UniqueName: \"kubernetes.io/projected/8aaac2b2-a4e5-4736-96ec-30f545797512-kube-api-access-l5mk6\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-wq96b\" (UID: \"8aaac2b2-a4e5-4736-96ec-30f545797512\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-wq96b" Oct 30 12:16:17.408267 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:17.408161 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-wq96b" Oct 30 12:16:17.743491 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:17.743441 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-wq96b"] Oct 30 12:16:17.748594 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:17.748550 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:16:17.748712 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:17.748645 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:16:17.748712 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:17.748689 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:16:18.072539 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:18.072432 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-wq96b" event={"ID":"8aaac2b2-a4e5-4736-96ec-30f545797512","Type":"ContainerStarted","Data":"67f327e201fe008df2005b78d98003d09f2872c93d4faded29e77d32594d65e1"} Oct 30 12:16:18.072539 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:18.072494 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-wq96b" event={"ID":"8aaac2b2-a4e5-4736-96ec-30f545797512","Type":"ContainerStarted","Data":"38c07fda28d85a100a06f14d7d7295706bad8bfab68329943afe1b43df0930fb"} Oct 30 12:16:18.117537 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:18.117473 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-wq96b" podStartSLOduration=1.117456355 podStartE2EDuration="1.117456355s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:16:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:16:18.115808346 +0000 UTC m=+1334.630237104" watchObservedRunningTime="2026-10-30 12:16:18.117456355 +0000 UTC m=+1334.631885102" Oct 30 12:16:18.361226 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:18.356887 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d554a235-x5ccx" Oct 30 12:16:18.413852 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:18.408857 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-wq96b" Oct 30 12:16:18.413852 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:18.412878 4908 patch_prober.go:29] interesting pod/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-wq96b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Oct 30 12:16:18.413852 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:18.412936 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-wq96b" podUID="8aaac2b2-a4e5-4736-96ec-30f545797512" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Oct 30 12:16:18.423752 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:18.423573 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b703a455-llp76" Oct 30 12:16:18.434436 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:18.434393 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dfeb8f6c-afd6-4dd2-99af-a4829db10a20-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfeb8f6c-afd6-4dd2-99af-a4829db10a20-maas-api-openshift-ca\") pod \"dfeb8f6c-afd6-4dd2-99af-a4829db10a20\" (UID: \"dfeb8f6c-afd6-4dd2-99af-a4829db10a20\") " Oct 30 12:16:18.434654 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:18.434486 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfeb8f6c-afd6-4dd2-99af-a4829db10a20-kube-api-access-bjr25\" (UniqueName: \"kubernetes.io/projected/dfeb8f6c-afd6-4dd2-99af-a4829db10a20-kube-api-access-bjr25\") pod \"dfeb8f6c-afd6-4dd2-99af-a4829db10a20\" (UID: \"dfeb8f6c-afd6-4dd2-99af-a4829db10a20\") " Oct 30 12:16:18.434654 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:18.434523 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dfeb8f6c-afd6-4dd2-99af-a4829db10a20-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfeb8f6c-afd6-4dd2-99af-a4829db10a20-maas-api-xks-ca\") pod \"dfeb8f6c-afd6-4dd2-99af-a4829db10a20\" (UID: \"dfeb8f6c-afd6-4dd2-99af-a4829db10a20\") " Oct 30 12:16:18.435708 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:18.435677 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfeb8f6c-afd6-4dd2-99af-a4829db10a20-maas-api-openshift-ca" pod "dfeb8f6c-afd6-4dd2-99af-a4829db10a20" (UID: "dfeb8f6c-afd6-4dd2-99af-a4829db10a20"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:16:18.439024 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:18.438981 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfeb8f6c-afd6-4dd2-99af-a4829db10a20-kube-api-access-bjr25" pod "dfeb8f6c-afd6-4dd2-99af-a4829db10a20" (UID: "dfeb8f6c-afd6-4dd2-99af-a4829db10a20"). InnerVolumeSpecName "kube-api-access-bjr25". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:16:18.439145 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:18.439118 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dfeb8f6c-afd6-4dd2-99af-a4829db10a20-maas-api-xks-ca" pod "dfeb8f6c-afd6-4dd2-99af-a4829db10a20" (UID: "dfeb8f6c-afd6-4dd2-99af-a4829db10a20"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:16:18.535663 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:18.535619 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/76c0e83d-ebe5-4e89-be6b-043568973731-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76c0e83d-ebe5-4e89-be6b-043568973731-maas-api-xks-ca\") pod \"76c0e83d-ebe5-4e89-be6b-043568973731\" (UID: \"76c0e83d-ebe5-4e89-be6b-043568973731\") " Oct 30 12:16:18.535880 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:18.535704 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76c0e83d-ebe5-4e89-be6b-043568973731-kube-api-access-lzwlt\" (UniqueName: \"kubernetes.io/projected/76c0e83d-ebe5-4e89-be6b-043568973731-kube-api-access-lzwlt\") pod \"76c0e83d-ebe5-4e89-be6b-043568973731\" (UID: \"76c0e83d-ebe5-4e89-be6b-043568973731\") " Oct 30 12:16:18.535963 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:18.535884 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76c0e83d-ebe5-4e89-be6b-043568973731-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76c0e83d-ebe5-4e89-be6b-043568973731-maas-api-openshift-ca\") pod \"76c0e83d-ebe5-4e89-be6b-043568973731\" (UID: \"76c0e83d-ebe5-4e89-be6b-043568973731\") " Oct 30 12:16:18.536287 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:18.536252 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76c0e83d-ebe5-4e89-be6b-043568973731-maas-api-openshift-ca" pod "76c0e83d-ebe5-4e89-be6b-043568973731" (UID: "76c0e83d-ebe5-4e89-be6b-043568973731"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:16:18.536405 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:18.536391 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76c0e83d-ebe5-4e89-be6b-043568973731-maas-api-openshift-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:16:18.536460 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:18.536411 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfeb8f6c-afd6-4dd2-99af-a4829db10a20-maas-api-openshift-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:16:18.536460 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:18.536420 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjr25\" (UniqueName: \"kubernetes.io/projected/dfeb8f6c-afd6-4dd2-99af-a4829db10a20-kube-api-access-bjr25\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:16:18.536460 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:18.536430 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfeb8f6c-afd6-4dd2-99af-a4829db10a20-maas-api-xks-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:16:18.538142 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:18.538109 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76c0e83d-ebe5-4e89-be6b-043568973731-maas-api-xks-ca" pod "76c0e83d-ebe5-4e89-be6b-043568973731" (UID: "76c0e83d-ebe5-4e89-be6b-043568973731"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:16:18.538484 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:18.538450 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76c0e83d-ebe5-4e89-be6b-043568973731-kube-api-access-lzwlt" pod "76c0e83d-ebe5-4e89-be6b-043568973731" (UID: "76c0e83d-ebe5-4e89-be6b-043568973731"). InnerVolumeSpecName "kube-api-access-lzwlt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:16:18.638023 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:18.637931 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76c0e83d-ebe5-4e89-be6b-043568973731-maas-api-xks-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:16:18.638023 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:18.637970 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lzwlt\" (UniqueName: \"kubernetes.io/projected/76c0e83d-ebe5-4e89-be6b-043568973731-kube-api-access-lzwlt\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:16:18.991852 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:18.987959 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71c0f7bc-nd54p"] Oct 30 12:16:18.993855 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:18.993562 4908 cpu_manager.go:414] "containerMap: removing container" podUID="76c0e83d-ebe5-4e89-be6b-043568973731" containerName="revoke-keys" Oct 30 12:16:18.993855 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:18.993595 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76c0e83d-ebe5-4e89-be6b-043568973731" containerName="revoke-keys" Oct 30 12:16:18.993855 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:18.993637 4908 cpu_manager.go:414] "containerMap: removing container" podUID="dfeb8f6c-afd6-4dd2-99af-a4829db10a20" containerName="revoke-keys" Oct 30 12:16:18.993855 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:18.993649 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfeb8f6c-afd6-4dd2-99af-a4829db10a20" containerName="revoke-keys" Oct 30 12:16:18.993855 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:18.993811 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfeb8f6c-afd6-4dd2-99af-a4829db10a20" containerName="revoke-keys" Oct 30 12:16:18.993855 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:18.993855 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="76c0e83d-ebe5-4e89-be6b-043568973731" containerName="revoke-keys" Oct 30 12:16:18.997874 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:18.997815 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71c0f7bc-nd54p" Oct 30 12:16:19.007799 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:19.007772 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71c0f7bc-nd54p"] Oct 30 12:16:19.042752 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:19.042717 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvztw\" (UniqueName: \"kubernetes.io/projected/af769e53-bc80-4f62-be1b-86c7eac896f3-kube-api-access-fvztw\") pod \"maas-api-revoke-sub-71c0f7bc-nd54p\" (UID: \"af769e53-bc80-4f62-be1b-86c7eac896f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71c0f7bc-nd54p" Oct 30 12:16:19.042947 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:19.042788 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af769e53-bc80-4f62-be1b-86c7eac896f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-71c0f7bc-nd54p\" (UID: \"af769e53-bc80-4f62-be1b-86c7eac896f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71c0f7bc-nd54p" Oct 30 12:16:19.043041 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:19.042979 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af769e53-bc80-4f62-be1b-86c7eac896f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-71c0f7bc-nd54p\" (UID: \"af769e53-bc80-4f62-be1b-86c7eac896f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71c0f7bc-nd54p" Oct 30 12:16:19.099975 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:19.099935 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b703a455-llp76" Oct 30 12:16:19.099975 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:19.099955 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b703a455-llp76" event={"ID":"76c0e83d-ebe5-4e89-be6b-043568973731","Type":"ContainerDied","Data":"0b47c1272148f80e91cde1abc58ae62092e06e6663c9cc709cd3c347bca2fc38"} Oct 30 12:16:19.103715 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:19.103517 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0b47c1272148f80e91cde1abc58ae62092e06e6663c9cc709cd3c347bca2fc38" Oct 30 12:16:19.115925 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:19.115861 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d554a235-x5ccx" event={"ID":"dfeb8f6c-afd6-4dd2-99af-a4829db10a20","Type":"ContainerDied","Data":"58dcfb37ffc41173596a2b72874d49acdd0090e4bcf85484b2a5d9625049e313"} Oct 30 12:16:19.116232 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:19.116210 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="58dcfb37ffc41173596a2b72874d49acdd0090e4bcf85484b2a5d9625049e313" Oct 30 12:16:19.116389 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:19.116375 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d554a235-x5ccx" Oct 30 12:16:19.144135 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:19.144095 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af769e53-bc80-4f62-be1b-86c7eac896f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-71c0f7bc-nd54p\" (UID: \"af769e53-bc80-4f62-be1b-86c7eac896f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71c0f7bc-nd54p" Oct 30 12:16:19.144777 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:19.144747 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af769e53-bc80-4f62-be1b-86c7eac896f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-71c0f7bc-nd54p\" (UID: \"af769e53-bc80-4f62-be1b-86c7eac896f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71c0f7bc-nd54p" Oct 30 12:16:19.144934 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:19.144794 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvztw\" (UniqueName: \"kubernetes.io/projected/af769e53-bc80-4f62-be1b-86c7eac896f3-kube-api-access-fvztw\") pod \"maas-api-revoke-sub-71c0f7bc-nd54p\" (UID: \"af769e53-bc80-4f62-be1b-86c7eac896f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71c0f7bc-nd54p" Oct 30 12:16:19.144934 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:19.144899 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af769e53-bc80-4f62-be1b-86c7eac896f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-71c0f7bc-nd54p\" (UID: \"af769e53-bc80-4f62-be1b-86c7eac896f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71c0f7bc-nd54p" Oct 30 12:16:19.147333 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:19.147308 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af769e53-bc80-4f62-be1b-86c7eac896f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-71c0f7bc-nd54p\" (UID: \"af769e53-bc80-4f62-be1b-86c7eac896f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71c0f7bc-nd54p" Oct 30 12:16:19.182692 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:19.182651 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvztw\" (UniqueName: \"kubernetes.io/projected/af769e53-bc80-4f62-be1b-86c7eac896f3-kube-api-access-fvztw\") pod \"maas-api-revoke-sub-71c0f7bc-nd54p\" (UID: \"af769e53-bc80-4f62-be1b-86c7eac896f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71c0f7bc-nd54p" Oct 30 12:16:19.317029 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:19.316932 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71c0f7bc-nd54p" Oct 30 12:16:19.414668 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:19.411311 4908 patch_prober.go:29] interesting pod/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-wq96b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Oct 30 12:16:19.414668 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:19.411375 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-wq96b" podUID="8aaac2b2-a4e5-4736-96ec-30f545797512" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Oct 30 12:16:19.648632 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:19.648532 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71c0f7bc-nd54p"] Oct 30 12:16:19.653048 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:16:19.653011 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf769e53_bc80_4f62_be1b_86c7eac896f3.slice/crio-ed75a86b97f0c772425da7aae51106681a8785f6fbc5f8cf9ab2d0a759a55d5d WatchSource:0}: Error finding container ed75a86b97f0c772425da7aae51106681a8785f6fbc5f8cf9ab2d0a759a55d5d: Status 404 returned error can't find the container with id ed75a86b97f0c772425da7aae51106681a8785f6fbc5f8cf9ab2d0a759a55d5d Oct 30 12:16:19.718536 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:16:19.718497 4908 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vn6vv" containerName="extractor" Oct 30 12:16:20.123960 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:20.123921 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71c0f7bc-nd54p" event={"ID":"af769e53-bc80-4f62-be1b-86c7eac896f3","Type":"ContainerStarted","Data":"10893ddb39916408c23cc3aa56971d59adb4d97407defbbd3cde1b9f21778c86"} Oct 30 12:16:20.124362 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:20.123978 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71c0f7bc-nd54p" event={"ID":"af769e53-bc80-4f62-be1b-86c7eac896f3","Type":"ContainerStarted","Data":"ed75a86b97f0c772425da7aae51106681a8785f6fbc5f8cf9ab2d0a759a55d5d"} Oct 30 12:16:20.409474 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:20.409439 4908 patch_prober.go:29] interesting pod/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-wq96b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Oct 30 12:16:20.409678 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:20.409503 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-wq96b" podUID="8aaac2b2-a4e5-4736-96ec-30f545797512" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Oct 30 12:16:21.134241 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:21.134139 4908 generic.go:356] "Generic (PLEG): container finished" podID="af769e53-bc80-4f62-be1b-86c7eac896f3" containerID="10893ddb39916408c23cc3aa56971d59adb4d97407defbbd3cde1b9f21778c86" exitCode=0 Oct 30 12:16:21.134241 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:21.134222 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71c0f7bc-nd54p" event={"ID":"af769e53-bc80-4f62-be1b-86c7eac896f3","Type":"ContainerDied","Data":"10893ddb39916408c23cc3aa56971d59adb4d97407defbbd3cde1b9f21778c86"} Oct 30 12:16:21.409499 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:21.409409 4908 patch_prober.go:29] interesting pod/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-wq96b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Oct 30 12:16:21.409499 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:21.409462 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-wq96b" podUID="8aaac2b2-a4e5-4736-96ec-30f545797512" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Oct 30 12:16:22.324251 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:22.324219 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71c0f7bc-nd54p" Oct 30 12:16:22.403554 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:22.403448 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af769e53-bc80-4f62-be1b-86c7eac896f3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af769e53-bc80-4f62-be1b-86c7eac896f3-maas-api-openshift-ca\") pod \"af769e53-bc80-4f62-be1b-86c7eac896f3\" (UID: \"af769e53-bc80-4f62-be1b-86c7eac896f3\") " Oct 30 12:16:22.403554 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:22.403499 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af769e53-bc80-4f62-be1b-86c7eac896f3-kube-api-access-fvztw\" (UniqueName: \"kubernetes.io/projected/af769e53-bc80-4f62-be1b-86c7eac896f3-kube-api-access-fvztw\") pod \"af769e53-bc80-4f62-be1b-86c7eac896f3\" (UID: \"af769e53-bc80-4f62-be1b-86c7eac896f3\") " Oct 30 12:16:22.403554 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:22.403551 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af769e53-bc80-4f62-be1b-86c7eac896f3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af769e53-bc80-4f62-be1b-86c7eac896f3-maas-api-xks-ca\") pod \"af769e53-bc80-4f62-be1b-86c7eac896f3\" (UID: \"af769e53-bc80-4f62-be1b-86c7eac896f3\") " Oct 30 12:16:22.404069 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:22.404033 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af769e53-bc80-4f62-be1b-86c7eac896f3-maas-api-openshift-ca" pod "af769e53-bc80-4f62-be1b-86c7eac896f3" (UID: "af769e53-bc80-4f62-be1b-86c7eac896f3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:16:22.406366 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:22.406337 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af769e53-bc80-4f62-be1b-86c7eac896f3-kube-api-access-fvztw" pod "af769e53-bc80-4f62-be1b-86c7eac896f3" (UID: "af769e53-bc80-4f62-be1b-86c7eac896f3"). InnerVolumeSpecName "kube-api-access-fvztw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:16:22.408767 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:22.408744 4908 patch_prober.go:29] interesting pod/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-wq96b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Oct 30 12:16:22.408925 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:22.408798 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-wq96b" podUID="8aaac2b2-a4e5-4736-96ec-30f545797512" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Oct 30 12:16:22.418212 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:22.418170 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af769e53-bc80-4f62-be1b-86c7eac896f3-maas-api-xks-ca" pod "af769e53-bc80-4f62-be1b-86c7eac896f3" (UID: "af769e53-bc80-4f62-be1b-86c7eac896f3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:16:22.505007 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:22.504960 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af769e53-bc80-4f62-be1b-86c7eac896f3-maas-api-xks-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:16:22.505007 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:22.504997 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af769e53-bc80-4f62-be1b-86c7eac896f3-maas-api-openshift-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:16:22.505007 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:22.505011 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvztw\" (UniqueName: \"kubernetes.io/projected/af769e53-bc80-4f62-be1b-86c7eac896f3-kube-api-access-fvztw\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:16:23.151781 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:23.151740 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71c0f7bc-nd54p" event={"ID":"af769e53-bc80-4f62-be1b-86c7eac896f3","Type":"ContainerDied","Data":"ed75a86b97f0c772425da7aae51106681a8785f6fbc5f8cf9ab2d0a759a55d5d"} Oct 30 12:16:23.151781 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:23.151761 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71c0f7bc-nd54p" Oct 30 12:16:23.152036 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:23.151824 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ed75a86b97f0c772425da7aae51106681a8785f6fbc5f8cf9ab2d0a759a55d5d" Oct 30 12:16:23.409094 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:23.409009 4908 patch_prober.go:29] interesting pod/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-wq96b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Oct 30 12:16:23.409483 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:23.409081 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-wq96b" podUID="8aaac2b2-a4e5-4736-96ec-30f545797512" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Oct 30 12:16:24.533811 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:24.533777 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-wq96b" Oct 30 12:16:24.534463 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:24.534435 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-wq96b" Oct 30 12:16:24.535585 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:24.535563 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-wq96b" Oct 30 12:16:33.314414 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:33.314373 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d8be2a15-88g2p"] Oct 30 12:16:33.314845 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:33.314817 4908 cpu_manager.go:414] "containerMap: removing container" podUID="af769e53-bc80-4f62-be1b-86c7eac896f3" containerName="revoke-keys" Oct 30 12:16:33.314894 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:33.314843 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af769e53-bc80-4f62-be1b-86c7eac896f3" containerName="revoke-keys" Oct 30 12:16:33.314951 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:33.314938 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="af769e53-bc80-4f62-be1b-86c7eac896f3" containerName="revoke-keys" Oct 30 12:16:33.318393 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:33.318375 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8be2a15-88g2p" Oct 30 12:16:33.345713 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:33.345669 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s75hq\"" Oct 30 12:16:33.345713 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:33.345674 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 12:16:33.345966 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:33.345751 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 12:16:33.345966 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:33.345764 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 12:16:33.348568 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:33.348541 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d8be2a15-88g2p"] Oct 30 12:16:33.423527 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:33.423480 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1608674-e935-4a04-87fa-d20ce21e7218-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d8be2a15-88g2p\" (UID: \"e1608674-e935-4a04-87fa-d20ce21e7218\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8be2a15-88g2p" Oct 30 12:16:33.423773 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:33.423625 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1608674-e935-4a04-87fa-d20ce21e7218-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d8be2a15-88g2p\" (UID: \"e1608674-e935-4a04-87fa-d20ce21e7218\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8be2a15-88g2p" Oct 30 12:16:33.423773 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:33.423668 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5z7r\" (UniqueName: \"kubernetes.io/projected/e1608674-e935-4a04-87fa-d20ce21e7218-kube-api-access-p5z7r\") pod \"maas-api-revoke-sub-d8be2a15-88g2p\" (UID: \"e1608674-e935-4a04-87fa-d20ce21e7218\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8be2a15-88g2p" Oct 30 12:16:33.524812 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:33.524771 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1608674-e935-4a04-87fa-d20ce21e7218-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d8be2a15-88g2p\" (UID: \"e1608674-e935-4a04-87fa-d20ce21e7218\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8be2a15-88g2p" Oct 30 12:16:33.524812 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:33.524812 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5z7r\" (UniqueName: \"kubernetes.io/projected/e1608674-e935-4a04-87fa-d20ce21e7218-kube-api-access-p5z7r\") pod \"maas-api-revoke-sub-d8be2a15-88g2p\" (UID: \"e1608674-e935-4a04-87fa-d20ce21e7218\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8be2a15-88g2p" Oct 30 12:16:33.525055 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:33.524932 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1608674-e935-4a04-87fa-d20ce21e7218-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d8be2a15-88g2p\" (UID: \"e1608674-e935-4a04-87fa-d20ce21e7218\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8be2a15-88g2p" Oct 30 12:16:33.525682 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:33.525659 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1608674-e935-4a04-87fa-d20ce21e7218-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d8be2a15-88g2p\" (UID: \"e1608674-e935-4a04-87fa-d20ce21e7218\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8be2a15-88g2p" Oct 30 12:16:33.527488 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:33.527462 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1608674-e935-4a04-87fa-d20ce21e7218-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d8be2a15-88g2p\" (UID: \"e1608674-e935-4a04-87fa-d20ce21e7218\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8be2a15-88g2p" Oct 30 12:16:33.547734 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:33.547693 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5z7r\" (UniqueName: \"kubernetes.io/projected/e1608674-e935-4a04-87fa-d20ce21e7218-kube-api-access-p5z7r\") pod \"maas-api-revoke-sub-d8be2a15-88g2p\" (UID: \"e1608674-e935-4a04-87fa-d20ce21e7218\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8be2a15-88g2p" Oct 30 12:16:33.632047 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:33.631939 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8be2a15-88g2p" Oct 30 12:16:33.943796 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:33.943711 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d8be2a15-88g2p"] Oct 30 12:16:33.947002 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:16:33.946962 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1608674_e935_4a04_87fa_d20ce21e7218.slice/crio-333e4ffd8ffcf64c83e309af42527b245f324a9ce9c92c4d558a8ec9db585cdf WatchSource:0}: Error finding container 333e4ffd8ffcf64c83e309af42527b245f324a9ce9c92c4d558a8ec9db585cdf: Status 404 returned error can't find the container with id 333e4ffd8ffcf64c83e309af42527b245f324a9ce9c92c4d558a8ec9db585cdf Oct 30 12:16:34.229132 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:34.229097 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8be2a15-88g2p" event={"ID":"e1608674-e935-4a04-87fa-d20ce21e7218","Type":"ContainerStarted","Data":"333e4ffd8ffcf64c83e309af42527b245f324a9ce9c92c4d558a8ec9db585cdf"} Oct 30 12:16:35.153206 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:35.153110 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9wwfc"] Oct 30 12:16:35.236915 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:35.236871 4908 generic.go:356] "Generic (PLEG): container finished" podID="e1608674-e935-4a04-87fa-d20ce21e7218" containerID="8d46ceeea9ff9ac6015e4cef6b044464282b5e0c28e1fb0bc43a5c2ce4880e11" exitCode=0 Oct 30 12:16:35.237122 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:35.236979 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8be2a15-88g2p" event={"ID":"e1608674-e935-4a04-87fa-d20ce21e7218","Type":"ContainerDied","Data":"8d46ceeea9ff9ac6015e4cef6b044464282b5e0c28e1fb0bc43a5c2ce4880e11"} Oct 30 12:16:36.157126 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:36.157085 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-8c5bs"] Oct 30 12:16:36.167144 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:36.167110 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-8c5bs" Oct 30 12:16:36.184760 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:36.184731 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-512ec5-openshift-default-dockercfg-ww4j8\"" Oct 30 12:16:36.219043 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:36.218896 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-8c5bs"] Oct 30 12:16:36.355360 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:36.355318 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07619fd1-c52a-45ce-851e-7436a3cb0723-workload-certs\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-8c5bs\" (UID: \"07619fd1-c52a-45ce-851e-7436a3cb0723\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-8c5bs" Oct 30 12:16:36.355529 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:36.355431 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07619fd1-c52a-45ce-851e-7436a3cb0723-credential-socket\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-8c5bs\" (UID: \"07619fd1-c52a-45ce-851e-7436a3cb0723\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-8c5bs" Oct 30 12:16:36.355529 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:36.355489 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07619fd1-c52a-45ce-851e-7436a3cb0723-istiod-ca-cert\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-8c5bs\" (UID: \"07619fd1-c52a-45ce-851e-7436a3cb0723\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-8c5bs" Oct 30 12:16:36.355603 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:36.355529 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07619fd1-c52a-45ce-851e-7436a3cb0723-istio-envoy\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-8c5bs\" (UID: \"07619fd1-c52a-45ce-851e-7436a3cb0723\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-8c5bs" Oct 30 12:16:36.355603 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:36.355547 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07619fd1-c52a-45ce-851e-7436a3cb0723-istio-data\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-8c5bs\" (UID: \"07619fd1-c52a-45ce-851e-7436a3cb0723\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-8c5bs" Oct 30 12:16:36.355603 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:36.355585 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07619fd1-c52a-45ce-851e-7436a3cb0723-workload-socket\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-8c5bs\" (UID: \"07619fd1-c52a-45ce-851e-7436a3cb0723\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-8c5bs" Oct 30 12:16:36.355748 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:36.355638 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07619fd1-c52a-45ce-851e-7436a3cb0723-istio-podinfo\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-8c5bs\" (UID: \"07619fd1-c52a-45ce-851e-7436a3cb0723\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-8c5bs" Oct 30 12:16:36.355748 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:36.355678 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07619fd1-c52a-45ce-851e-7436a3cb0723-istio-token\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-8c5bs\" (UID: \"07619fd1-c52a-45ce-851e-7436a3cb0723\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-8c5bs" Oct 30 12:16:36.355850 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:36.355791 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjkfp\" (UniqueName: \"kubernetes.io/projected/07619fd1-c52a-45ce-851e-7436a3cb0723-kube-api-access-jjkfp\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-8c5bs\" (UID: \"07619fd1-c52a-45ce-851e-7436a3cb0723\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-8c5bs" Oct 30 12:16:36.395274 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:36.395243 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8be2a15-88g2p" Oct 30 12:16:36.456641 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:36.456536 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07619fd1-c52a-45ce-851e-7436a3cb0723-istio-podinfo\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-8c5bs\" (UID: \"07619fd1-c52a-45ce-851e-7436a3cb0723\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-8c5bs" Oct 30 12:16:36.456641 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:36.456608 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07619fd1-c52a-45ce-851e-7436a3cb0723-istio-token\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-8c5bs\" (UID: \"07619fd1-c52a-45ce-851e-7436a3cb0723\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-8c5bs" Oct 30 12:16:36.456898 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:36.456642 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjkfp\" (UniqueName: \"kubernetes.io/projected/07619fd1-c52a-45ce-851e-7436a3cb0723-kube-api-access-jjkfp\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-8c5bs\" (UID: \"07619fd1-c52a-45ce-851e-7436a3cb0723\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-8c5bs" Oct 30 12:16:36.456898 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:36.456706 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07619fd1-c52a-45ce-851e-7436a3cb0723-workload-certs\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-8c5bs\" (UID: \"07619fd1-c52a-45ce-851e-7436a3cb0723\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-8c5bs" Oct 30 12:16:36.456898 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:36.456782 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07619fd1-c52a-45ce-851e-7436a3cb0723-credential-socket\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-8c5bs\" (UID: \"07619fd1-c52a-45ce-851e-7436a3cb0723\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-8c5bs" Oct 30 12:16:36.456898 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:36.456851 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07619fd1-c52a-45ce-851e-7436a3cb0723-istiod-ca-cert\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-8c5bs\" (UID: \"07619fd1-c52a-45ce-851e-7436a3cb0723\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-8c5bs" Oct 30 12:16:36.456898 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:36.456886 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07619fd1-c52a-45ce-851e-7436a3cb0723-istio-envoy\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-8c5bs\" (UID: \"07619fd1-c52a-45ce-851e-7436a3cb0723\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-8c5bs" Oct 30 12:16:36.457137 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:36.456910 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07619fd1-c52a-45ce-851e-7436a3cb0723-istio-data\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-8c5bs\" (UID: \"07619fd1-c52a-45ce-851e-7436a3cb0723\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-8c5bs" Oct 30 12:16:36.457137 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:36.456950 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07619fd1-c52a-45ce-851e-7436a3cb0723-workload-socket\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-8c5bs\" (UID: \"07619fd1-c52a-45ce-851e-7436a3cb0723\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-8c5bs" Oct 30 12:16:36.457387 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:36.457360 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07619fd1-c52a-45ce-851e-7436a3cb0723-workload-socket\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-8c5bs\" (UID: \"07619fd1-c52a-45ce-851e-7436a3cb0723\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-8c5bs" Oct 30 12:16:36.457462 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:36.457410 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07619fd1-c52a-45ce-851e-7436a3cb0723-credential-socket\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-8c5bs\" (UID: \"07619fd1-c52a-45ce-851e-7436a3cb0723\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-8c5bs" Oct 30 12:16:36.457664 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:36.457637 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07619fd1-c52a-45ce-851e-7436a3cb0723-istio-data\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-8c5bs\" (UID: \"07619fd1-c52a-45ce-851e-7436a3cb0723\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-8c5bs" Oct 30 12:16:36.457968 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:36.457942 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07619fd1-c52a-45ce-851e-7436a3cb0723-workload-certs\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-8c5bs\" (UID: \"07619fd1-c52a-45ce-851e-7436a3cb0723\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-8c5bs" Oct 30 12:16:36.458224 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:36.458198 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07619fd1-c52a-45ce-851e-7436a3cb0723-istiod-ca-cert\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-8c5bs\" (UID: \"07619fd1-c52a-45ce-851e-7436a3cb0723\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-8c5bs" Oct 30 12:16:36.459637 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:36.459612 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07619fd1-c52a-45ce-851e-7436a3cb0723-istio-podinfo\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-8c5bs\" (UID: \"07619fd1-c52a-45ce-851e-7436a3cb0723\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-8c5bs" Oct 30 12:16:36.459747 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:36.459617 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07619fd1-c52a-45ce-851e-7436a3cb0723-istio-envoy\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-8c5bs\" (UID: \"07619fd1-c52a-45ce-851e-7436a3cb0723\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-8c5bs" Oct 30 12:16:36.479404 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:36.479373 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjkfp\" (UniqueName: \"kubernetes.io/projected/07619fd1-c52a-45ce-851e-7436a3cb0723-kube-api-access-jjkfp\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-8c5bs\" (UID: \"07619fd1-c52a-45ce-851e-7436a3cb0723\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-8c5bs" Oct 30 12:16:36.479555 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:36.479370 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07619fd1-c52a-45ce-851e-7436a3cb0723-istio-token\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-8c5bs\" (UID: \"07619fd1-c52a-45ce-851e-7436a3cb0723\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-8c5bs" Oct 30 12:16:36.493851 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:36.493792 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-8c5bs" Oct 30 12:16:36.558325 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:36.558095 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e1608674-e935-4a04-87fa-d20ce21e7218-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1608674-e935-4a04-87fa-d20ce21e7218-maas-api-xks-ca\") pod \"e1608674-e935-4a04-87fa-d20ce21e7218\" (UID: \"e1608674-e935-4a04-87fa-d20ce21e7218\") " Oct 30 12:16:36.558325 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:36.558208 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1608674-e935-4a04-87fa-d20ce21e7218-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1608674-e935-4a04-87fa-d20ce21e7218-maas-api-openshift-ca\") pod \"e1608674-e935-4a04-87fa-d20ce21e7218\" (UID: \"e1608674-e935-4a04-87fa-d20ce21e7218\") " Oct 30 12:16:36.559887 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:36.558519 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1608674-e935-4a04-87fa-d20ce21e7218-kube-api-access-p5z7r\" (UniqueName: \"kubernetes.io/projected/e1608674-e935-4a04-87fa-d20ce21e7218-kube-api-access-p5z7r\") pod \"e1608674-e935-4a04-87fa-d20ce21e7218\" (UID: \"e1608674-e935-4a04-87fa-d20ce21e7218\") " Oct 30 12:16:36.559887 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:36.558666 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1608674-e935-4a04-87fa-d20ce21e7218-maas-api-openshift-ca" pod "e1608674-e935-4a04-87fa-d20ce21e7218" (UID: "e1608674-e935-4a04-87fa-d20ce21e7218"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:16:36.559887 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:36.558972 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1608674-e935-4a04-87fa-d20ce21e7218-maas-api-openshift-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:16:36.560797 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:36.560763 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1608674-e935-4a04-87fa-d20ce21e7218-maas-api-xks-ca" pod "e1608674-e935-4a04-87fa-d20ce21e7218" (UID: "e1608674-e935-4a04-87fa-d20ce21e7218"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:16:36.561994 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:36.561967 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1608674-e935-4a04-87fa-d20ce21e7218-kube-api-access-p5z7r" pod "e1608674-e935-4a04-87fa-d20ce21e7218" (UID: "e1608674-e935-4a04-87fa-d20ce21e7218"). InnerVolumeSpecName "kube-api-access-p5z7r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:16:36.668548 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:36.668510 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5z7r\" (UniqueName: \"kubernetes.io/projected/e1608674-e935-4a04-87fa-d20ce21e7218-kube-api-access-p5z7r\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:16:36.668548 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:36.668548 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1608674-e935-4a04-87fa-d20ce21e7218-maas-api-xks-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:16:36.817468 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:36.817423 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-8c5bs"] Oct 30 12:16:36.820568 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:16:36.820541 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07619fd1_c52a_45ce_851e_7436a3cb0723.slice/crio-31a9a3477177809d4aa10a5e0650ee9bdb24b0e064ac56ab040048e1a784fa4b WatchSource:0}: Error finding container 31a9a3477177809d4aa10a5e0650ee9bdb24b0e064ac56ab040048e1a784fa4b: Status 404 returned error can't find the container with id 31a9a3477177809d4aa10a5e0650ee9bdb24b0e064ac56ab040048e1a784fa4b Oct 30 12:16:36.823029 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:36.822992 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:16:36.823126 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:36.823081 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:16:36.823196 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:36.823160 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:16:37.253904 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:37.253860 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-8c5bs" event={"ID":"07619fd1-c52a-45ce-851e-7436a3cb0723","Type":"ContainerStarted","Data":"5acbfb21e58944ad079746c35dad9949744f021119be9224397bc2c0cc111c32"} Oct 30 12:16:37.254517 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:37.253929 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-8c5bs" event={"ID":"07619fd1-c52a-45ce-851e-7436a3cb0723","Type":"ContainerStarted","Data":"31a9a3477177809d4aa10a5e0650ee9bdb24b0e064ac56ab040048e1a784fa4b"} Oct 30 12:16:37.255454 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:37.255433 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8be2a15-88g2p" Oct 30 12:16:37.255578 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:37.255447 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8be2a15-88g2p" event={"ID":"e1608674-e935-4a04-87fa-d20ce21e7218","Type":"ContainerDied","Data":"333e4ffd8ffcf64c83e309af42527b245f324a9ce9c92c4d558a8ec9db585cdf"} Oct 30 12:16:37.255578 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:37.255486 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="333e4ffd8ffcf64c83e309af42527b245f324a9ce9c92c4d558a8ec9db585cdf" Oct 30 12:16:37.299242 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:37.298964 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-8c5bs" podStartSLOduration=1.2989437019999999 podStartE2EDuration="1.298943702s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:16:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:16:37.295864131 +0000 UTC m=+1353.810292894" watchObservedRunningTime="2026-10-30 12:16:37.298943702 +0000 UTC m=+1353.813372448" Oct 30 12:16:37.494371 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:37.494330 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-8c5bs" Oct 30 12:16:37.496260 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:37.496226 4908 patch_prober.go:29] interesting pod/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-8c5bs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Oct 30 12:16:37.496425 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:37.496283 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-8c5bs" podUID="07619fd1-c52a-45ce-851e-7436a3cb0723" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Oct 30 12:16:38.495851 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:38.495416 4908 patch_prober.go:29] interesting pod/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-8c5bs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Oct 30 12:16:38.495851 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:38.495494 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-8c5bs" podUID="07619fd1-c52a-45ce-851e-7436a3cb0723" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Oct 30 12:16:39.494846 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:39.494557 4908 patch_prober.go:29] interesting pod/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-8c5bs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Oct 30 12:16:39.494846 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:39.494636 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-8c5bs" podUID="07619fd1-c52a-45ce-851e-7436a3cb0723" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Oct 30 12:16:40.494964 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:40.494804 4908 patch_prober.go:29] interesting pod/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-8c5bs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Oct 30 12:16:40.494964 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:40.494896 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-8c5bs" podUID="07619fd1-c52a-45ce-851e-7436a3cb0723" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Oct 30 12:16:41.494128 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:41.494089 4908 patch_prober.go:29] interesting pod/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-8c5bs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Oct 30 12:16:41.494394 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:41.494364 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-8c5bs" podUID="07619fd1-c52a-45ce-851e-7436a3cb0723" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Oct 30 12:16:42.494353 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:42.494323 4908 patch_prober.go:29] interesting pod/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-8c5bs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Oct 30 12:16:42.494853 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:42.494810 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-8c5bs" podUID="07619fd1-c52a-45ce-851e-7436a3cb0723" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Oct 30 12:16:43.494531 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:43.494494 4908 patch_prober.go:29] interesting pod/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-8c5bs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Oct 30 12:16:43.494955 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:43.494550 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-8c5bs" podUID="07619fd1-c52a-45ce-851e-7436a3cb0723" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Oct 30 12:16:43.649214 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:43.649172 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-747839d7-thvml"] Oct 30 12:16:43.649846 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:43.649797 4908 cpu_manager.go:414] "containerMap: removing container" podUID="e1608674-e935-4a04-87fa-d20ce21e7218" containerName="revoke-keys" Oct 30 12:16:43.649846 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:43.649817 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1608674-e935-4a04-87fa-d20ce21e7218" containerName="revoke-keys" Oct 30 12:16:43.650037 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:43.649953 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1608674-e935-4a04-87fa-d20ce21e7218" containerName="revoke-keys" Oct 30 12:16:43.653751 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:43.653729 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-747839d7-thvml" Oct 30 12:16:43.670633 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:43.670607 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-747839d7-thvml"] Oct 30 12:16:43.677616 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:43.677577 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 12:16:43.677759 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:43.677649 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s75hq\"" Oct 30 12:16:43.681619 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:43.681595 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 12:16:43.681747 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:43.681728 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 12:16:43.761684 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:43.761566 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea24c359-cda6-4d11-a757-df67563772f2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-747839d7-thvml\" (UID: \"ea24c359-cda6-4d11-a757-df67563772f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-747839d7-thvml" Oct 30 12:16:43.761684 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:43.761626 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea24c359-cda6-4d11-a757-df67563772f2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-747839d7-thvml\" (UID: \"ea24c359-cda6-4d11-a757-df67563772f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-747839d7-thvml" Oct 30 12:16:43.762015 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:43.761899 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rl7vz\" (UniqueName: \"kubernetes.io/projected/ea24c359-cda6-4d11-a757-df67563772f2-kube-api-access-rl7vz\") pod \"maas-api-revoke-sub-747839d7-thvml\" (UID: \"ea24c359-cda6-4d11-a757-df67563772f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-747839d7-thvml" Oct 30 12:16:43.862554 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:43.862516 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rl7vz\" (UniqueName: \"kubernetes.io/projected/ea24c359-cda6-4d11-a757-df67563772f2-kube-api-access-rl7vz\") pod \"maas-api-revoke-sub-747839d7-thvml\" (UID: \"ea24c359-cda6-4d11-a757-df67563772f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-747839d7-thvml" Oct 30 12:16:43.862554 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:43.862573 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea24c359-cda6-4d11-a757-df67563772f2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-747839d7-thvml\" (UID: \"ea24c359-cda6-4d11-a757-df67563772f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-747839d7-thvml" Oct 30 12:16:43.862873 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:43.862597 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea24c359-cda6-4d11-a757-df67563772f2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-747839d7-thvml\" (UID: \"ea24c359-cda6-4d11-a757-df67563772f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-747839d7-thvml" Oct 30 12:16:43.863336 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:43.863308 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea24c359-cda6-4d11-a757-df67563772f2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-747839d7-thvml\" (UID: \"ea24c359-cda6-4d11-a757-df67563772f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-747839d7-thvml" Oct 30 12:16:43.865215 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:43.865186 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea24c359-cda6-4d11-a757-df67563772f2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-747839d7-thvml\" (UID: \"ea24c359-cda6-4d11-a757-df67563772f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-747839d7-thvml" Oct 30 12:16:43.885090 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:43.885059 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rl7vz\" (UniqueName: \"kubernetes.io/projected/ea24c359-cda6-4d11-a757-df67563772f2-kube-api-access-rl7vz\") pod \"maas-api-revoke-sub-747839d7-thvml\" (UID: \"ea24c359-cda6-4d11-a757-df67563772f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-747839d7-thvml" Oct 30 12:16:43.967459 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:43.967419 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-747839d7-thvml" Oct 30 12:16:44.289177 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:44.289104 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-747839d7-thvml"] Oct 30 12:16:44.290977 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:16:44.290946 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea24c359_cda6_4d11_a757_df67563772f2.slice/crio-4873bd0a6852c454f143485aea8adb6de5228f4549f921cc5c2f3872238967a0 WatchSource:0}: Error finding container 4873bd0a6852c454f143485aea8adb6de5228f4549f921cc5c2f3872238967a0: Status 404 returned error can't find the container with id 4873bd0a6852c454f143485aea8adb6de5228f4549f921cc5c2f3872238967a0 Oct 30 12:16:44.333137 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:44.333099 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-747839d7-thvml" event={"ID":"ea24c359-cda6-4d11-a757-df67563772f2","Type":"ContainerStarted","Data":"4873bd0a6852c454f143485aea8adb6de5228f4549f921cc5c2f3872238967a0"} Oct 30 12:16:44.499179 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:44.499136 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-8c5bs" Oct 30 12:16:44.499604 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:44.499468 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-8c5bs" Oct 30 12:16:44.500149 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:44.500126 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-8c5bs" Oct 30 12:16:45.342001 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:45.341899 4908 generic.go:356] "Generic (PLEG): container finished" podID="ea24c359-cda6-4d11-a757-df67563772f2" containerID="d3c5aa0a4c4902f4adb29f5ca37dfcc09f89602f6d1fc0e340082df9d1560af6" exitCode=0 Oct 30 12:16:45.342175 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:45.341984 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-747839d7-thvml" event={"ID":"ea24c359-cda6-4d11-a757-df67563772f2","Type":"ContainerDied","Data":"d3c5aa0a4c4902f4adb29f5ca37dfcc09f89602f6d1fc0e340082df9d1560af6"} Oct 30 12:16:46.503995 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:46.503967 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-747839d7-thvml" Oct 30 12:16:46.593886 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:46.593822 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea24c359-cda6-4d11-a757-df67563772f2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea24c359-cda6-4d11-a757-df67563772f2-maas-api-openshift-ca\") pod \"ea24c359-cda6-4d11-a757-df67563772f2\" (UID: \"ea24c359-cda6-4d11-a757-df67563772f2\") " Oct 30 12:16:46.594085 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:46.593905 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea24c359-cda6-4d11-a757-df67563772f2-kube-api-access-rl7vz\" (UniqueName: \"kubernetes.io/projected/ea24c359-cda6-4d11-a757-df67563772f2-kube-api-access-rl7vz\") pod \"ea24c359-cda6-4d11-a757-df67563772f2\" (UID: \"ea24c359-cda6-4d11-a757-df67563772f2\") " Oct 30 12:16:46.594085 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:46.594016 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ea24c359-cda6-4d11-a757-df67563772f2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea24c359-cda6-4d11-a757-df67563772f2-maas-api-xks-ca\") pod \"ea24c359-cda6-4d11-a757-df67563772f2\" (UID: \"ea24c359-cda6-4d11-a757-df67563772f2\") " Oct 30 12:16:46.594265 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:46.594226 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea24c359-cda6-4d11-a757-df67563772f2-maas-api-openshift-ca" pod "ea24c359-cda6-4d11-a757-df67563772f2" (UID: "ea24c359-cda6-4d11-a757-df67563772f2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:16:46.594445 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:46.594427 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea24c359-cda6-4d11-a757-df67563772f2-maas-api-openshift-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:16:46.596193 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:46.596157 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea24c359-cda6-4d11-a757-df67563772f2-maas-api-xks-ca" pod "ea24c359-cda6-4d11-a757-df67563772f2" (UID: "ea24c359-cda6-4d11-a757-df67563772f2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:16:46.596312 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:46.596266 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea24c359-cda6-4d11-a757-df67563772f2-kube-api-access-rl7vz" pod "ea24c359-cda6-4d11-a757-df67563772f2" (UID: "ea24c359-cda6-4d11-a757-df67563772f2"). InnerVolumeSpecName "kube-api-access-rl7vz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:16:46.695606 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:46.695566 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rl7vz\" (UniqueName: \"kubernetes.io/projected/ea24c359-cda6-4d11-a757-df67563772f2-kube-api-access-rl7vz\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:16:46.695606 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:46.695604 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea24c359-cda6-4d11-a757-df67563772f2-maas-api-xks-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:16:47.357612 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:47.357581 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-747839d7-thvml" Oct 30 12:16:47.357612 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:47.357592 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-747839d7-thvml" event={"ID":"ea24c359-cda6-4d11-a757-df67563772f2","Type":"ContainerDied","Data":"4873bd0a6852c454f143485aea8adb6de5228f4549f921cc5c2f3872238967a0"} Oct 30 12:16:47.357883 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:47.357651 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4873bd0a6852c454f143485aea8adb6de5228f4549f921cc5c2f3872238967a0" Oct 30 12:16:49.685904 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:16:49.685857 4908 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vn6vv" containerName="extractor" Oct 30 12:16:53.589785 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:53.589743 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9wwfc"] Oct 30 12:16:58.217298 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:58.217255 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-512ec5/e2e-conflict-b-2c88c229-kserve-545b7f4cdd-82qkd"] Oct 30 12:16:58.218048 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:58.218026 4908 cpu_manager.go:414] "containerMap: removing container" podUID="ea24c359-cda6-4d11-a757-df67563772f2" containerName="revoke-keys" Oct 30 12:16:58.218111 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:58.218052 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea24c359-cda6-4d11-a757-df67563772f2" containerName="revoke-keys" Oct 30 12:16:58.218201 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:58.218182 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea24c359-cda6-4d11-a757-df67563772f2" containerName="revoke-keys" Oct 30 12:16:58.222145 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:58.222118 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-512ec5/e2e-conflict-b-2c88c229-kserve-545b7f4cdd-82qkd" Oct 30 12:16:58.238950 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:58.238921 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-512ec5/e2e-conflict-b-2c88c229-kserve-545b7f4cdd-82qkd"] Oct 30 12:16:58.246495 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:58.246463 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-512ec5\"/\"e2e-conflict-b-2c88c229-kserve-self-signed-certs\"" Oct 30 12:16:58.325143 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:58.325110 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/acefe1ec-2c47-487a-aad9-61b0028827dd-home\") pod \"e2e-conflict-b-2c88c229-kserve-545b7f4cdd-82qkd\" (UID: \"acefe1ec-2c47-487a-aad9-61b0028827dd\") " pod="e2e-models-e2e-worker-w4-512ec5/e2e-conflict-b-2c88c229-kserve-545b7f4cdd-82qkd" Oct 30 12:16:58.325355 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:58.325171 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/acefe1ec-2c47-487a-aad9-61b0028827dd-tmp-dir\") pod \"e2e-conflict-b-2c88c229-kserve-545b7f4cdd-82qkd\" (UID: \"acefe1ec-2c47-487a-aad9-61b0028827dd\") " pod="e2e-models-e2e-worker-w4-512ec5/e2e-conflict-b-2c88c229-kserve-545b7f4cdd-82qkd" Oct 30 12:16:58.325355 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:58.325309 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/acefe1ec-2c47-487a-aad9-61b0028827dd-dshm\") pod \"e2e-conflict-b-2c88c229-kserve-545b7f4cdd-82qkd\" (UID: \"acefe1ec-2c47-487a-aad9-61b0028827dd\") " pod="e2e-models-e2e-worker-w4-512ec5/e2e-conflict-b-2c88c229-kserve-545b7f4cdd-82qkd" Oct 30 12:16:58.325471 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:58.325433 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/acefe1ec-2c47-487a-aad9-61b0028827dd-model-cache\") pod \"e2e-conflict-b-2c88c229-kserve-545b7f4cdd-82qkd\" (UID: \"acefe1ec-2c47-487a-aad9-61b0028827dd\") " pod="e2e-models-e2e-worker-w4-512ec5/e2e-conflict-b-2c88c229-kserve-545b7f4cdd-82qkd" Oct 30 12:16:58.325521 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:58.325474 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/acefe1ec-2c47-487a-aad9-61b0028827dd-tls-certs\") pod \"e2e-conflict-b-2c88c229-kserve-545b7f4cdd-82qkd\" (UID: \"acefe1ec-2c47-487a-aad9-61b0028827dd\") " pod="e2e-models-e2e-worker-w4-512ec5/e2e-conflict-b-2c88c229-kserve-545b7f4cdd-82qkd" Oct 30 12:16:58.325622 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:58.325594 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgxrs\" (UniqueName: \"kubernetes.io/projected/acefe1ec-2c47-487a-aad9-61b0028827dd-kube-api-access-rgxrs\") pod \"e2e-conflict-b-2c88c229-kserve-545b7f4cdd-82qkd\" (UID: \"acefe1ec-2c47-487a-aad9-61b0028827dd\") " pod="e2e-models-e2e-worker-w4-512ec5/e2e-conflict-b-2c88c229-kserve-545b7f4cdd-82qkd" Oct 30 12:16:58.430851 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:58.428366 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/acefe1ec-2c47-487a-aad9-61b0028827dd-model-cache\") pod \"e2e-conflict-b-2c88c229-kserve-545b7f4cdd-82qkd\" (UID: \"acefe1ec-2c47-487a-aad9-61b0028827dd\") " pod="e2e-models-e2e-worker-w4-512ec5/e2e-conflict-b-2c88c229-kserve-545b7f4cdd-82qkd" Oct 30 12:16:58.430851 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:58.428415 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/acefe1ec-2c47-487a-aad9-61b0028827dd-tls-certs\") pod \"e2e-conflict-b-2c88c229-kserve-545b7f4cdd-82qkd\" (UID: \"acefe1ec-2c47-487a-aad9-61b0028827dd\") " pod="e2e-models-e2e-worker-w4-512ec5/e2e-conflict-b-2c88c229-kserve-545b7f4cdd-82qkd" Oct 30 12:16:58.430851 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:58.428458 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgxrs\" (UniqueName: \"kubernetes.io/projected/acefe1ec-2c47-487a-aad9-61b0028827dd-kube-api-access-rgxrs\") pod \"e2e-conflict-b-2c88c229-kserve-545b7f4cdd-82qkd\" (UID: \"acefe1ec-2c47-487a-aad9-61b0028827dd\") " pod="e2e-models-e2e-worker-w4-512ec5/e2e-conflict-b-2c88c229-kserve-545b7f4cdd-82qkd" Oct 30 12:16:58.430851 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:58.428477 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/acefe1ec-2c47-487a-aad9-61b0028827dd-home\") pod \"e2e-conflict-b-2c88c229-kserve-545b7f4cdd-82qkd\" (UID: \"acefe1ec-2c47-487a-aad9-61b0028827dd\") " pod="e2e-models-e2e-worker-w4-512ec5/e2e-conflict-b-2c88c229-kserve-545b7f4cdd-82qkd" Oct 30 12:16:58.430851 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:58.428529 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/acefe1ec-2c47-487a-aad9-61b0028827dd-tmp-dir\") pod \"e2e-conflict-b-2c88c229-kserve-545b7f4cdd-82qkd\" (UID: \"acefe1ec-2c47-487a-aad9-61b0028827dd\") " pod="e2e-models-e2e-worker-w4-512ec5/e2e-conflict-b-2c88c229-kserve-545b7f4cdd-82qkd" Oct 30 12:16:58.430851 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:58.428569 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/acefe1ec-2c47-487a-aad9-61b0028827dd-dshm\") pod \"e2e-conflict-b-2c88c229-kserve-545b7f4cdd-82qkd\" (UID: \"acefe1ec-2c47-487a-aad9-61b0028827dd\") " pod="e2e-models-e2e-worker-w4-512ec5/e2e-conflict-b-2c88c229-kserve-545b7f4cdd-82qkd" Oct 30 12:16:58.430851 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:58.429389 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/acefe1ec-2c47-487a-aad9-61b0028827dd-tmp-dir\") pod \"e2e-conflict-b-2c88c229-kserve-545b7f4cdd-82qkd\" (UID: \"acefe1ec-2c47-487a-aad9-61b0028827dd\") " pod="e2e-models-e2e-worker-w4-512ec5/e2e-conflict-b-2c88c229-kserve-545b7f4cdd-82qkd" Oct 30 12:16:58.430851 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:58.430043 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/acefe1ec-2c47-487a-aad9-61b0028827dd-model-cache\") pod \"e2e-conflict-b-2c88c229-kserve-545b7f4cdd-82qkd\" (UID: \"acefe1ec-2c47-487a-aad9-61b0028827dd\") " pod="e2e-models-e2e-worker-w4-512ec5/e2e-conflict-b-2c88c229-kserve-545b7f4cdd-82qkd" Oct 30 12:16:58.430851 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:58.430306 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/acefe1ec-2c47-487a-aad9-61b0028827dd-home\") pod \"e2e-conflict-b-2c88c229-kserve-545b7f4cdd-82qkd\" (UID: \"acefe1ec-2c47-487a-aad9-61b0028827dd\") " pod="e2e-models-e2e-worker-w4-512ec5/e2e-conflict-b-2c88c229-kserve-545b7f4cdd-82qkd" Oct 30 12:16:58.440918 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:58.439775 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/acefe1ec-2c47-487a-aad9-61b0028827dd-tls-certs\") pod \"e2e-conflict-b-2c88c229-kserve-545b7f4cdd-82qkd\" (UID: \"acefe1ec-2c47-487a-aad9-61b0028827dd\") " pod="e2e-models-e2e-worker-w4-512ec5/e2e-conflict-b-2c88c229-kserve-545b7f4cdd-82qkd" Oct 30 12:16:58.450509 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:58.450473 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/acefe1ec-2c47-487a-aad9-61b0028827dd-dshm\") pod \"e2e-conflict-b-2c88c229-kserve-545b7f4cdd-82qkd\" (UID: \"acefe1ec-2c47-487a-aad9-61b0028827dd\") " pod="e2e-models-e2e-worker-w4-512ec5/e2e-conflict-b-2c88c229-kserve-545b7f4cdd-82qkd" Oct 30 12:16:58.462590 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:58.462555 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgxrs\" (UniqueName: \"kubernetes.io/projected/acefe1ec-2c47-487a-aad9-61b0028827dd-kube-api-access-rgxrs\") pod \"e2e-conflict-b-2c88c229-kserve-545b7f4cdd-82qkd\" (UID: \"acefe1ec-2c47-487a-aad9-61b0028827dd\") " pod="e2e-models-e2e-worker-w4-512ec5/e2e-conflict-b-2c88c229-kserve-545b7f4cdd-82qkd" Oct 30 12:16:58.536769 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:58.536733 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-512ec5/e2e-conflict-b-2c88c229-kserve-545b7f4cdd-82qkd" Oct 30 12:16:58.841754 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:58.841726 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-512ec5/e2e-conflict-b-2c88c229-kserve-545b7f4cdd-82qkd"] Oct 30 12:16:58.844481 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:16:58.844446 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podacefe1ec_2c47_487a_aad9_61b0028827dd.slice/crio-28688a757466d48993c82bf0d0cb0a56e6d9fce122c9eb392d2a87e4a729bc04 WatchSource:0}: Error finding container 28688a757466d48993c82bf0d0cb0a56e6d9fce122c9eb392d2a87e4a729bc04: Status 404 returned error can't find the container with id 28688a757466d48993c82bf0d0cb0a56e6d9fce122c9eb392d2a87e4a729bc04 Oct 30 12:16:59.476105 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:59.476062 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-512ec5/e2e-conflict-b-2c88c229-kserve-545b7f4cdd-82qkd" event={"ID":"acefe1ec-2c47-487a-aad9-61b0028827dd","Type":"ContainerStarted","Data":"00213e5699ce98b0f1141dfb145a8442362285b27c031af55ee43648808ffb3d"} Oct 30 12:16:59.476612 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:59.476119 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-512ec5/e2e-conflict-b-2c88c229-kserve-545b7f4cdd-82qkd" event={"ID":"acefe1ec-2c47-487a-aad9-61b0028827dd","Type":"ContainerStarted","Data":"28688a757466d48993c82bf0d0cb0a56e6d9fce122c9eb392d2a87e4a729bc04"} Oct 30 12:16:59.516207 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:59.516149 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-512ec5/e2e-conflict-b-2c88c229-kserve-545b7f4cdd-82qkd" podStartSLOduration=1.516135131 podStartE2EDuration="1.516135131s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:16:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:16:59.514493962 +0000 UTC m=+1376.028922708" watchObservedRunningTime="2026-10-30 12:16:59.516135131 +0000 UTC m=+1376.030563876" Oct 30 12:16:59.537510 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:16:59.537474 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-512ec5/e2e-conflict-b-2c88c229-kserve-545b7f4cdd-82qkd" Oct 30 12:17:08.537567 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:08.537535 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-512ec5/e2e-conflict-b-2c88c229-kserve-545b7f4cdd-82qkd" Oct 30 12:17:08.550408 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:08.550383 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-512ec5/e2e-conflict-b-2c88c229-kserve-545b7f4cdd-82qkd" Oct 30 12:17:09.553875 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:09.553816 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-512ec5/e2e-conflict-b-2c88c229-kserve-545b7f4cdd-82qkd" Oct 30 12:17:10.418290 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:10.418256 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-ljzls"] Oct 30 12:17:10.423511 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:10.423489 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-ljzls" Oct 30 12:17:10.437540 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:10.437512 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-f920f7-openshift-default-dockercfg-d89vp\"" Oct 30 12:17:10.467845 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:10.467642 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f0ed460-7572-4518-af84-589e8ac4ad29-workload-socket\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-ljzls\" (UID: \"6f0ed460-7572-4518-af84-589e8ac4ad29\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-ljzls" Oct 30 12:17:10.467845 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:10.467701 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f0ed460-7572-4518-af84-589e8ac4ad29-istio-data\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-ljzls\" (UID: \"6f0ed460-7572-4518-af84-589e8ac4ad29\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-ljzls" Oct 30 12:17:10.467845 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:10.467733 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfphq\" (UniqueName: \"kubernetes.io/projected/6f0ed460-7572-4518-af84-589e8ac4ad29-kube-api-access-pfphq\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-ljzls\" (UID: \"6f0ed460-7572-4518-af84-589e8ac4ad29\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-ljzls" Oct 30 12:17:10.468110 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:10.467858 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f0ed460-7572-4518-af84-589e8ac4ad29-istiod-ca-cert\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-ljzls\" (UID: \"6f0ed460-7572-4518-af84-589e8ac4ad29\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-ljzls" Oct 30 12:17:10.468110 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:10.467895 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f0ed460-7572-4518-af84-589e8ac4ad29-istio-podinfo\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-ljzls\" (UID: \"6f0ed460-7572-4518-af84-589e8ac4ad29\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-ljzls" Oct 30 12:17:10.468110 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:10.467931 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f0ed460-7572-4518-af84-589e8ac4ad29-istio-token\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-ljzls\" (UID: \"6f0ed460-7572-4518-af84-589e8ac4ad29\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-ljzls" Oct 30 12:17:10.468110 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:10.467984 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f0ed460-7572-4518-af84-589e8ac4ad29-credential-socket\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-ljzls\" (UID: \"6f0ed460-7572-4518-af84-589e8ac4ad29\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-ljzls" Oct 30 12:17:10.468110 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:10.468062 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f0ed460-7572-4518-af84-589e8ac4ad29-istio-envoy\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-ljzls\" (UID: \"6f0ed460-7572-4518-af84-589e8ac4ad29\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-ljzls" Oct 30 12:17:10.468110 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:10.468105 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f0ed460-7572-4518-af84-589e8ac4ad29-workload-certs\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-ljzls\" (UID: \"6f0ed460-7572-4518-af84-589e8ac4ad29\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-ljzls" Oct 30 12:17:10.477141 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:10.477110 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-ljzls"] Oct 30 12:17:10.569411 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:10.569379 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f0ed460-7572-4518-af84-589e8ac4ad29-credential-socket\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-ljzls\" (UID: \"6f0ed460-7572-4518-af84-589e8ac4ad29\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-ljzls" Oct 30 12:17:10.569855 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:10.569438 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f0ed460-7572-4518-af84-589e8ac4ad29-istio-envoy\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-ljzls\" (UID: \"6f0ed460-7572-4518-af84-589e8ac4ad29\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-ljzls" Oct 30 12:17:10.569855 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:10.569476 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f0ed460-7572-4518-af84-589e8ac4ad29-workload-certs\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-ljzls\" (UID: \"6f0ed460-7572-4518-af84-589e8ac4ad29\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-ljzls" Oct 30 12:17:10.569855 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:10.569519 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f0ed460-7572-4518-af84-589e8ac4ad29-workload-socket\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-ljzls\" (UID: \"6f0ed460-7572-4518-af84-589e8ac4ad29\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-ljzls" Oct 30 12:17:10.569855 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:10.569546 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f0ed460-7572-4518-af84-589e8ac4ad29-istio-data\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-ljzls\" (UID: \"6f0ed460-7572-4518-af84-589e8ac4ad29\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-ljzls" Oct 30 12:17:10.569855 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:10.569567 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfphq\" (UniqueName: \"kubernetes.io/projected/6f0ed460-7572-4518-af84-589e8ac4ad29-kube-api-access-pfphq\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-ljzls\" (UID: \"6f0ed460-7572-4518-af84-589e8ac4ad29\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-ljzls" Oct 30 12:17:10.569855 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:10.569677 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f0ed460-7572-4518-af84-589e8ac4ad29-istiod-ca-cert\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-ljzls\" (UID: \"6f0ed460-7572-4518-af84-589e8ac4ad29\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-ljzls" Oct 30 12:17:10.569855 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:10.569709 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f0ed460-7572-4518-af84-589e8ac4ad29-istio-podinfo\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-ljzls\" (UID: \"6f0ed460-7572-4518-af84-589e8ac4ad29\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-ljzls" Oct 30 12:17:10.569855 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:10.569743 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f0ed460-7572-4518-af84-589e8ac4ad29-istio-token\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-ljzls\" (UID: \"6f0ed460-7572-4518-af84-589e8ac4ad29\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-ljzls" Oct 30 12:17:10.570324 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:10.569933 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f0ed460-7572-4518-af84-589e8ac4ad29-credential-socket\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-ljzls\" (UID: \"6f0ed460-7572-4518-af84-589e8ac4ad29\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-ljzls" Oct 30 12:17:10.570324 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:10.570296 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f0ed460-7572-4518-af84-589e8ac4ad29-istio-data\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-ljzls\" (UID: \"6f0ed460-7572-4518-af84-589e8ac4ad29\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-ljzls" Oct 30 12:17:10.570502 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:10.570367 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f0ed460-7572-4518-af84-589e8ac4ad29-workload-certs\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-ljzls\" (UID: \"6f0ed460-7572-4518-af84-589e8ac4ad29\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-ljzls" Oct 30 12:17:10.570576 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:10.570556 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f0ed460-7572-4518-af84-589e8ac4ad29-workload-socket\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-ljzls\" (UID: \"6f0ed460-7572-4518-af84-589e8ac4ad29\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-ljzls" Oct 30 12:17:10.570674 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:10.570624 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f0ed460-7572-4518-af84-589e8ac4ad29-istiod-ca-cert\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-ljzls\" (UID: \"6f0ed460-7572-4518-af84-589e8ac4ad29\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-ljzls" Oct 30 12:17:10.572753 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:10.572721 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f0ed460-7572-4518-af84-589e8ac4ad29-istio-envoy\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-ljzls\" (UID: \"6f0ed460-7572-4518-af84-589e8ac4ad29\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-ljzls" Oct 30 12:17:10.572998 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:10.572975 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f0ed460-7572-4518-af84-589e8ac4ad29-istio-podinfo\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-ljzls\" (UID: \"6f0ed460-7572-4518-af84-589e8ac4ad29\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-ljzls" Oct 30 12:17:10.591969 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:10.591927 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f0ed460-7572-4518-af84-589e8ac4ad29-istio-token\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-ljzls\" (UID: \"6f0ed460-7572-4518-af84-589e8ac4ad29\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-ljzls" Oct 30 12:17:10.592931 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:10.592913 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfphq\" (UniqueName: \"kubernetes.io/projected/6f0ed460-7572-4518-af84-589e8ac4ad29-kube-api-access-pfphq\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-ljzls\" (UID: \"6f0ed460-7572-4518-af84-589e8ac4ad29\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-ljzls" Oct 30 12:17:10.741557 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:10.741517 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-ljzls" Oct 30 12:17:11.075940 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:11.075853 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-ljzls"] Oct 30 12:17:11.080254 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:17:11.080221 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f0ed460_7572_4518_af84_589e8ac4ad29.slice/crio-84a434992effc5097e4fe1485d6adbe254eee5b8ed2e24a6cbca95c0b8a8e7ad WatchSource:0}: Error finding container 84a434992effc5097e4fe1485d6adbe254eee5b8ed2e24a6cbca95c0b8a8e7ad: Status 404 returned error can't find the container with id 84a434992effc5097e4fe1485d6adbe254eee5b8ed2e24a6cbca95c0b8a8e7ad Oct 30 12:17:11.082729 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:11.082689 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:17:11.082850 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:11.082796 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:17:11.082897 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:11.082857 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:17:11.577985 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:11.577937 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-ljzls" event={"ID":"6f0ed460-7572-4518-af84-589e8ac4ad29","Type":"ContainerStarted","Data":"3da236f86a561f2d61503e62ccb9ed7e5586bb94851e79be1e2b35e011b9361c"} Oct 30 12:17:11.578597 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:11.578570 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-ljzls" event={"ID":"6f0ed460-7572-4518-af84-589e8ac4ad29","Type":"ContainerStarted","Data":"84a434992effc5097e4fe1485d6adbe254eee5b8ed2e24a6cbca95c0b8a8e7ad"} Oct 30 12:17:11.639240 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:11.639178 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-ljzls" podStartSLOduration=1.639160202 podStartE2EDuration="1.639160202s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:17:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:17:11.636540041 +0000 UTC m=+1388.150968788" watchObservedRunningTime="2026-10-30 12:17:11.639160202 +0000 UTC m=+1388.153588949" Oct 30 12:17:11.742430 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:11.742396 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-ljzls" Oct 30 12:17:12.743773 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:12.743735 4908 patch_prober.go:29] interesting pod/e2e-worker-w0-f920f7-openshift-default-7c9767c97-ljzls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Oct 30 12:17:12.744198 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:12.743782 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-ljzls" podUID="6f0ed460-7572-4518-af84-589e8ac4ad29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Oct 30 12:17:13.742206 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:13.742169 4908 patch_prober.go:29] interesting pod/e2e-worker-w0-f920f7-openshift-default-7c9767c97-ljzls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Oct 30 12:17:13.742392 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:13.742225 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-ljzls" podUID="6f0ed460-7572-4518-af84-589e8ac4ad29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Oct 30 12:17:14.742679 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:14.742640 4908 patch_prober.go:29] interesting pod/e2e-worker-w0-f920f7-openshift-default-7c9767c97-ljzls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Oct 30 12:17:14.743164 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:14.742711 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-ljzls" podUID="6f0ed460-7572-4518-af84-589e8ac4ad29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Oct 30 12:17:15.742554 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:15.742516 4908 patch_prober.go:29] interesting pod/e2e-worker-w0-f920f7-openshift-default-7c9767c97-ljzls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Oct 30 12:17:15.742761 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:15.742573 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-ljzls" podUID="6f0ed460-7572-4518-af84-589e8ac4ad29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Oct 30 12:17:16.742054 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:16.742011 4908 patch_prober.go:29] interesting pod/e2e-worker-w0-f920f7-openshift-default-7c9767c97-ljzls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Oct 30 12:17:16.742346 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:16.742309 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-ljzls" podUID="6f0ed460-7572-4518-af84-589e8ac4ad29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Oct 30 12:17:17.573963 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:17.573915 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9wwfc"] Oct 30 12:17:17.742787 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:17.742746 4908 patch_prober.go:29] interesting pod/e2e-worker-w0-f920f7-openshift-default-7c9767c97-ljzls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Oct 30 12:17:17.743000 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:17.742846 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-ljzls" podUID="6f0ed460-7572-4518-af84-589e8ac4ad29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Oct 30 12:17:18.747784 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:18.747745 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-ljzls" Oct 30 12:17:18.748395 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:18.748371 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-ljzls" Oct 30 12:17:18.749431 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:18.749401 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-ljzls" Oct 30 12:17:18.799528 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:18.799481 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-pzdcj"] Oct 30 12:17:18.804727 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:18.804701 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-pzdcj" Oct 30 12:17:18.830107 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:18.830075 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-pzdcj"] Oct 30 12:17:18.881597 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:18.881549 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd702b8b-1d62-41b9-8fbf-1cb4aebdaa71-workload-socket\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-pzdcj\" (UID: \"bd702b8b-1d62-41b9-8fbf-1cb4aebdaa71\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-pzdcj" Oct 30 12:17:18.881916 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:18.881619 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd702b8b-1d62-41b9-8fbf-1cb4aebdaa71-istio-podinfo\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-pzdcj\" (UID: \"bd702b8b-1d62-41b9-8fbf-1cb4aebdaa71\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-pzdcj" Oct 30 12:17:18.881916 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:18.881702 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd702b8b-1d62-41b9-8fbf-1cb4aebdaa71-workload-certs\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-pzdcj\" (UID: \"bd702b8b-1d62-41b9-8fbf-1cb4aebdaa71\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-pzdcj" Oct 30 12:17:18.881916 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:18.881766 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd702b8b-1d62-41b9-8fbf-1cb4aebdaa71-istio-envoy\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-pzdcj\" (UID: \"bd702b8b-1d62-41b9-8fbf-1cb4aebdaa71\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-pzdcj" Oct 30 12:17:18.881916 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:18.881794 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd702b8b-1d62-41b9-8fbf-1cb4aebdaa71-istiod-ca-cert\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-pzdcj\" (UID: \"bd702b8b-1d62-41b9-8fbf-1cb4aebdaa71\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-pzdcj" Oct 30 12:17:18.881916 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:18.881905 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7tzl\" (UniqueName: \"kubernetes.io/projected/bd702b8b-1d62-41b9-8fbf-1cb4aebdaa71-kube-api-access-s7tzl\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-pzdcj\" (UID: \"bd702b8b-1d62-41b9-8fbf-1cb4aebdaa71\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-pzdcj" Oct 30 12:17:18.882384 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:18.881975 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd702b8b-1d62-41b9-8fbf-1cb4aebdaa71-istio-data\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-pzdcj\" (UID: \"bd702b8b-1d62-41b9-8fbf-1cb4aebdaa71\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-pzdcj" Oct 30 12:17:18.882384 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:18.882066 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd702b8b-1d62-41b9-8fbf-1cb4aebdaa71-credential-socket\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-pzdcj\" (UID: \"bd702b8b-1d62-41b9-8fbf-1cb4aebdaa71\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-pzdcj" Oct 30 12:17:18.882384 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:18.882090 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd702b8b-1d62-41b9-8fbf-1cb4aebdaa71-istio-token\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-pzdcj\" (UID: \"bd702b8b-1d62-41b9-8fbf-1cb4aebdaa71\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-pzdcj" Oct 30 12:17:18.908549 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:18.907270 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-c1b1e8-openshift-default-dockercfg-k4lgw\"" Oct 30 12:17:18.982867 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:18.982813 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd702b8b-1d62-41b9-8fbf-1cb4aebdaa71-workload-certs\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-pzdcj\" (UID: \"bd702b8b-1d62-41b9-8fbf-1cb4aebdaa71\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-pzdcj" Oct 30 12:17:18.983198 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:18.983176 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd702b8b-1d62-41b9-8fbf-1cb4aebdaa71-istio-envoy\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-pzdcj\" (UID: \"bd702b8b-1d62-41b9-8fbf-1cb4aebdaa71\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-pzdcj" Oct 30 12:17:18.983349 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:18.983209 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd702b8b-1d62-41b9-8fbf-1cb4aebdaa71-istiod-ca-cert\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-pzdcj\" (UID: \"bd702b8b-1d62-41b9-8fbf-1cb4aebdaa71\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-pzdcj" Oct 30 12:17:18.983349 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:18.983293 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7tzl\" (UniqueName: \"kubernetes.io/projected/bd702b8b-1d62-41b9-8fbf-1cb4aebdaa71-kube-api-access-s7tzl\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-pzdcj\" (UID: \"bd702b8b-1d62-41b9-8fbf-1cb4aebdaa71\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-pzdcj" Oct 30 12:17:18.983349 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:18.983344 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd702b8b-1d62-41b9-8fbf-1cb4aebdaa71-istio-data\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-pzdcj\" (UID: \"bd702b8b-1d62-41b9-8fbf-1cb4aebdaa71\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-pzdcj" Oct 30 12:17:18.983530 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:18.983406 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd702b8b-1d62-41b9-8fbf-1cb4aebdaa71-credential-socket\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-pzdcj\" (UID: \"bd702b8b-1d62-41b9-8fbf-1cb4aebdaa71\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-pzdcj" Oct 30 12:17:18.983530 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:18.983431 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd702b8b-1d62-41b9-8fbf-1cb4aebdaa71-istio-token\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-pzdcj\" (UID: \"bd702b8b-1d62-41b9-8fbf-1cb4aebdaa71\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-pzdcj" Oct 30 12:17:18.983530 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:18.983477 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd702b8b-1d62-41b9-8fbf-1cb4aebdaa71-workload-socket\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-pzdcj\" (UID: \"bd702b8b-1d62-41b9-8fbf-1cb4aebdaa71\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-pzdcj" Oct 30 12:17:18.983530 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:18.983503 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd702b8b-1d62-41b9-8fbf-1cb4aebdaa71-workload-certs\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-pzdcj\" (UID: \"bd702b8b-1d62-41b9-8fbf-1cb4aebdaa71\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-pzdcj" Oct 30 12:17:18.983731 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:18.983512 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd702b8b-1d62-41b9-8fbf-1cb4aebdaa71-istio-podinfo\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-pzdcj\" (UID: \"bd702b8b-1d62-41b9-8fbf-1cb4aebdaa71\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-pzdcj" Oct 30 12:17:18.984425 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:18.983950 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd702b8b-1d62-41b9-8fbf-1cb4aebdaa71-istio-data\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-pzdcj\" (UID: \"bd702b8b-1d62-41b9-8fbf-1cb4aebdaa71\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-pzdcj" Oct 30 12:17:18.984425 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:18.984174 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd702b8b-1d62-41b9-8fbf-1cb4aebdaa71-credential-socket\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-pzdcj\" (UID: \"bd702b8b-1d62-41b9-8fbf-1cb4aebdaa71\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-pzdcj" Oct 30 12:17:18.984598 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:18.984469 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd702b8b-1d62-41b9-8fbf-1cb4aebdaa71-workload-socket\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-pzdcj\" (UID: \"bd702b8b-1d62-41b9-8fbf-1cb4aebdaa71\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-pzdcj" Oct 30 12:17:18.984660 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:18.984608 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd702b8b-1d62-41b9-8fbf-1cb4aebdaa71-istiod-ca-cert\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-pzdcj\" (UID: \"bd702b8b-1d62-41b9-8fbf-1cb4aebdaa71\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-pzdcj" Oct 30 12:17:18.987101 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:18.987079 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd702b8b-1d62-41b9-8fbf-1cb4aebdaa71-istio-envoy\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-pzdcj\" (UID: \"bd702b8b-1d62-41b9-8fbf-1cb4aebdaa71\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-pzdcj" Oct 30 12:17:18.987371 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:18.987348 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd702b8b-1d62-41b9-8fbf-1cb4aebdaa71-istio-podinfo\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-pzdcj\" (UID: \"bd702b8b-1d62-41b9-8fbf-1cb4aebdaa71\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-pzdcj" Oct 30 12:17:19.006723 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:19.006652 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7tzl\" (UniqueName: \"kubernetes.io/projected/bd702b8b-1d62-41b9-8fbf-1cb4aebdaa71-kube-api-access-s7tzl\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-pzdcj\" (UID: \"bd702b8b-1d62-41b9-8fbf-1cb4aebdaa71\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-pzdcj" Oct 30 12:17:19.006951 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:19.006664 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd702b8b-1d62-41b9-8fbf-1cb4aebdaa71-istio-token\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-pzdcj\" (UID: \"bd702b8b-1d62-41b9-8fbf-1cb4aebdaa71\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-pzdcj" Oct 30 12:17:19.129778 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:19.129734 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-pzdcj" Oct 30 12:17:19.479315 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:19.479220 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-pzdcj"] Oct 30 12:17:19.482247 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:17:19.482212 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd702b8b_1d62_41b9_8fbf_1cb4aebdaa71.slice/crio-6edff75f619ef4b909e91711444b0cbdd227a0cc28874c0142d60044b04e36a2 WatchSource:0}: Error finding container 6edff75f619ef4b909e91711444b0cbdd227a0cc28874c0142d60044b04e36a2: Status 404 returned error can't find the container with id 6edff75f619ef4b909e91711444b0cbdd227a0cc28874c0142d60044b04e36a2 Oct 30 12:17:19.485365 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:19.485326 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:17:19.485482 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:19.485415 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:17:19.485482 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:19.485457 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:17:19.650128 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:19.650082 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-pzdcj" event={"ID":"bd702b8b-1d62-41b9-8fbf-1cb4aebdaa71","Type":"ContainerStarted","Data":"1145f457ca80476d2590cbefa8c8c19e55242d855919b7bdfe576b5855a4495c"} Oct 30 12:17:19.650341 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:19.650158 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-pzdcj" event={"ID":"bd702b8b-1d62-41b9-8fbf-1cb4aebdaa71","Type":"ContainerStarted","Data":"6edff75f619ef4b909e91711444b0cbdd227a0cc28874c0142d60044b04e36a2"} Oct 30 12:17:19.689623 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:17:19.689551 4908 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vn6vv" containerName="extractor" Oct 30 12:17:19.693951 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:19.693897 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-pzdcj" podStartSLOduration=1.693877262 podStartE2EDuration="1.693877262s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:17:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:17:19.692801653 +0000 UTC m=+1396.207230404" watchObservedRunningTime="2026-10-30 12:17:19.693877262 +0000 UTC m=+1396.208306028" Oct 30 12:17:20.130430 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:20.130394 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-pzdcj" Oct 30 12:17:20.132112 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:20.132086 4908 patch_prober.go:29] interesting pod/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-pzdcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 30 12:17:20.132237 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:20.132143 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-pzdcj" podUID="bd702b8b-1d62-41b9-8fbf-1cb4aebdaa71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 30 12:17:21.131129 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:21.131085 4908 patch_prober.go:29] interesting pod/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-pzdcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 30 12:17:21.131656 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:21.131153 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-pzdcj" podUID="bd702b8b-1d62-41b9-8fbf-1cb4aebdaa71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 30 12:17:22.130134 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:22.130090 4908 patch_prober.go:29] interesting pod/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-pzdcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 30 12:17:22.130335 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:22.130152 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-pzdcj" podUID="bd702b8b-1d62-41b9-8fbf-1cb4aebdaa71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 30 12:17:23.131099 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:23.131058 4908 patch_prober.go:29] interesting pod/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-pzdcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 30 12:17:23.131606 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:23.131127 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-pzdcj" podUID="bd702b8b-1d62-41b9-8fbf-1cb4aebdaa71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 30 12:17:24.130106 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:24.130062 4908 patch_prober.go:29] interesting pod/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-pzdcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 30 12:17:24.130334 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:24.130122 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-pzdcj" podUID="bd702b8b-1d62-41b9-8fbf-1cb4aebdaa71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 30 12:17:25.130406 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:25.130369 4908 patch_prober.go:29] interesting pod/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-pzdcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 30 12:17:25.130881 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:25.130449 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-pzdcj" podUID="bd702b8b-1d62-41b9-8fbf-1cb4aebdaa71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 30 12:17:26.130682 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:26.130644 4908 patch_prober.go:29] interesting pod/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-pzdcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 30 12:17:26.131275 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:26.130699 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-pzdcj" podUID="bd702b8b-1d62-41b9-8fbf-1cb4aebdaa71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 30 12:17:27.134454 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:27.134419 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-pzdcj" Oct 30 12:17:27.134968 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:27.134713 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-pzdcj" Oct 30 12:17:27.135559 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:27.135538 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-pzdcj" Oct 30 12:17:32.100861 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:32.100793 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-512ec5/e2e-conflict-b-2c88c229-kserve-545b7f4cdd-82qkd"] Oct 30 12:17:40.133218 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:40.133178 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9wwfc"] Oct 30 12:17:44.900959 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:44.900922 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a164f939-sbn28"] Oct 30 12:17:44.904914 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:44.904888 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a164f939-sbn28" Oct 30 12:17:44.933256 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:44.933191 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 12:17:44.933256 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:44.933210 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 12:17:44.933540 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:44.933320 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 12:17:44.933540 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:44.933428 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s75hq\"" Oct 30 12:17:44.937063 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:44.937033 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a164f939-sbn28"] Oct 30 12:17:44.998267 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:44.998217 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87a04e72-f817-438e-890c-9e6843ff8c18-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a164f939-sbn28\" (UID: \"87a04e72-f817-438e-890c-9e6843ff8c18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a164f939-sbn28" Oct 30 12:17:44.998483 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:44.998398 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87a04e72-f817-438e-890c-9e6843ff8c18-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a164f939-sbn28\" (UID: \"87a04e72-f817-438e-890c-9e6843ff8c18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a164f939-sbn28" Oct 30 12:17:44.998568 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:44.998526 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5frwq\" (UniqueName: \"kubernetes.io/projected/87a04e72-f817-438e-890c-9e6843ff8c18-kube-api-access-5frwq\") pod \"maas-api-revoke-sub-a164f939-sbn28\" (UID: \"87a04e72-f817-438e-890c-9e6843ff8c18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a164f939-sbn28" Oct 30 12:17:45.099500 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:45.099461 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87a04e72-f817-438e-890c-9e6843ff8c18-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a164f939-sbn28\" (UID: \"87a04e72-f817-438e-890c-9e6843ff8c18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a164f939-sbn28" Oct 30 12:17:45.099704 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:45.099555 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87a04e72-f817-438e-890c-9e6843ff8c18-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a164f939-sbn28\" (UID: \"87a04e72-f817-438e-890c-9e6843ff8c18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a164f939-sbn28" Oct 30 12:17:45.099704 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:45.099616 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5frwq\" (UniqueName: \"kubernetes.io/projected/87a04e72-f817-438e-890c-9e6843ff8c18-kube-api-access-5frwq\") pod \"maas-api-revoke-sub-a164f939-sbn28\" (UID: \"87a04e72-f817-438e-890c-9e6843ff8c18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a164f939-sbn28" Oct 30 12:17:45.100305 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:45.100282 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87a04e72-f817-438e-890c-9e6843ff8c18-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a164f939-sbn28\" (UID: \"87a04e72-f817-438e-890c-9e6843ff8c18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a164f939-sbn28" Oct 30 12:17:45.102039 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:45.102021 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87a04e72-f817-438e-890c-9e6843ff8c18-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a164f939-sbn28\" (UID: \"87a04e72-f817-438e-890c-9e6843ff8c18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a164f939-sbn28" Oct 30 12:17:45.121684 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:45.121658 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5frwq\" (UniqueName: \"kubernetes.io/projected/87a04e72-f817-438e-890c-9e6843ff8c18-kube-api-access-5frwq\") pod \"maas-api-revoke-sub-a164f939-sbn28\" (UID: \"87a04e72-f817-438e-890c-9e6843ff8c18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a164f939-sbn28" Oct 30 12:17:45.221095 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:45.221003 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a164f939-sbn28" Oct 30 12:17:45.534319 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:45.534283 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a164f939-sbn28"] Oct 30 12:17:45.536432 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:17:45.536406 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87a04e72_f817_438e_890c_9e6843ff8c18.slice/crio-2487fd19ab21db777517b9c7ec1ec9a45dfe106de6d1506b4e0141988fe77677 WatchSource:0}: Error finding container 2487fd19ab21db777517b9c7ec1ec9a45dfe106de6d1506b4e0141988fe77677: Status 404 returned error can't find the container with id 2487fd19ab21db777517b9c7ec1ec9a45dfe106de6d1506b4e0141988fe77677 Oct 30 12:17:45.894496 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:45.894459 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a164f939-sbn28" event={"ID":"87a04e72-f817-438e-890c-9e6843ff8c18","Type":"ContainerStarted","Data":"b761ddeb3c274aa262e1ca9f2cb0e56b0c23c933494d282c6c244d81a33a6319"} Oct 30 12:17:45.894655 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:45.894513 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a164f939-sbn28" event={"ID":"87a04e72-f817-438e-890c-9e6843ff8c18","Type":"ContainerStarted","Data":"2487fd19ab21db777517b9c7ec1ec9a45dfe106de6d1506b4e0141988fe77677"} Oct 30 12:17:45.933647 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:45.933553 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a164f939-sbn28" podStartSLOduration=1.933534871 podStartE2EDuration="1.933534871s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:17:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:17:45.931707372 +0000 UTC m=+1422.446136117" watchObservedRunningTime="2026-10-30 12:17:45.933534871 +0000 UTC m=+1422.447963618" Oct 30 12:17:46.903446 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:46.903398 4908 generic.go:356] "Generic (PLEG): container finished" podID="87a04e72-f817-438e-890c-9e6843ff8c18" containerID="b761ddeb3c274aa262e1ca9f2cb0e56b0c23c933494d282c6c244d81a33a6319" exitCode=0 Oct 30 12:17:46.903681 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:46.903473 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a164f939-sbn28" event={"ID":"87a04e72-f817-438e-890c-9e6843ff8c18","Type":"ContainerDied","Data":"b761ddeb3c274aa262e1ca9f2cb0e56b0c23c933494d282c6c244d81a33a6319"} Oct 30 12:17:47.123034 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:47.122988 4908 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-512ec5/e2e-conflict-b-2c88c229-kserve-545b7f4cdd-82qkd" podUID="acefe1ec-2c47-487a-aad9-61b0028827dd" containerName="main" containerID="cri-o://00213e5699ce98b0f1141dfb145a8442362285b27c031af55ee43648808ffb3d" gracePeriod=45 Oct 30 12:17:47.409537 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:47.409505 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-512ec5/e2e-conflict-b-2c88c229-kserve-545b7f4cdd-82qkd" Oct 30 12:17:47.528891 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:47.528748 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/acefe1ec-2c47-487a-aad9-61b0028827dd-kube-api-access-rgxrs\" (UniqueName: \"kubernetes.io/projected/acefe1ec-2c47-487a-aad9-61b0028827dd-kube-api-access-rgxrs\") pod \"acefe1ec-2c47-487a-aad9-61b0028827dd\" (UID: \"acefe1ec-2c47-487a-aad9-61b0028827dd\") " Oct 30 12:17:47.528891 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:47.528819 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/acefe1ec-2c47-487a-aad9-61b0028827dd-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/acefe1ec-2c47-487a-aad9-61b0028827dd-model-cache\") pod \"acefe1ec-2c47-487a-aad9-61b0028827dd\" (UID: \"acefe1ec-2c47-487a-aad9-61b0028827dd\") " Oct 30 12:17:47.529151 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:47.528899 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/acefe1ec-2c47-487a-aad9-61b0028827dd-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/acefe1ec-2c47-487a-aad9-61b0028827dd-tmp-dir\") pod \"acefe1ec-2c47-487a-aad9-61b0028827dd\" (UID: \"acefe1ec-2c47-487a-aad9-61b0028827dd\") " Oct 30 12:17:47.529151 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:47.528940 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/acefe1ec-2c47-487a-aad9-61b0028827dd-dshm\" (UniqueName: \"kubernetes.io/empty-dir/acefe1ec-2c47-487a-aad9-61b0028827dd-dshm\") pod \"acefe1ec-2c47-487a-aad9-61b0028827dd\" (UID: \"acefe1ec-2c47-487a-aad9-61b0028827dd\") " Oct 30 12:17:47.529151 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:47.529087 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/acefe1ec-2c47-487a-aad9-61b0028827dd-tls-certs\" (UniqueName: \"kubernetes.io/secret/acefe1ec-2c47-487a-aad9-61b0028827dd-tls-certs\") pod \"acefe1ec-2c47-487a-aad9-61b0028827dd\" (UID: \"acefe1ec-2c47-487a-aad9-61b0028827dd\") " Oct 30 12:17:47.529151 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:47.529121 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/acefe1ec-2c47-487a-aad9-61b0028827dd-home\" (UniqueName: \"kubernetes.io/empty-dir/acefe1ec-2c47-487a-aad9-61b0028827dd-home\") pod \"acefe1ec-2c47-487a-aad9-61b0028827dd\" (UID: \"acefe1ec-2c47-487a-aad9-61b0028827dd\") " Oct 30 12:17:47.529151 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:47.529137 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acefe1ec-2c47-487a-aad9-61b0028827dd-model-cache" pod "acefe1ec-2c47-487a-aad9-61b0028827dd" (UID: "acefe1ec-2c47-487a-aad9-61b0028827dd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:17:47.529430 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:47.529194 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acefe1ec-2c47-487a-aad9-61b0028827dd-tmp-dir" pod "acefe1ec-2c47-487a-aad9-61b0028827dd" (UID: "acefe1ec-2c47-487a-aad9-61b0028827dd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:17:47.529485 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:47.529473 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acefe1ec-2c47-487a-aad9-61b0028827dd-home" pod "acefe1ec-2c47-487a-aad9-61b0028827dd" (UID: "acefe1ec-2c47-487a-aad9-61b0028827dd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:17:47.529609 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:47.529584 4908 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/acefe1ec-2c47-487a-aad9-61b0028827dd-home\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:17:47.529609 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:47.529603 4908 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/acefe1ec-2c47-487a-aad9-61b0028827dd-model-cache\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:17:47.529733 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:47.529618 4908 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/acefe1ec-2c47-487a-aad9-61b0028827dd-tmp-dir\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:17:47.531438 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:47.531406 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acefe1ec-2c47-487a-aad9-61b0028827dd-dshm" pod "acefe1ec-2c47-487a-aad9-61b0028827dd" (UID: "acefe1ec-2c47-487a-aad9-61b0028827dd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:17:47.531565 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:47.531466 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/acefe1ec-2c47-487a-aad9-61b0028827dd-tls-certs" pod "acefe1ec-2c47-487a-aad9-61b0028827dd" (UID: "acefe1ec-2c47-487a-aad9-61b0028827dd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:17:47.531565 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:47.531540 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acefe1ec-2c47-487a-aad9-61b0028827dd-kube-api-access-rgxrs" pod "acefe1ec-2c47-487a-aad9-61b0028827dd" (UID: "acefe1ec-2c47-487a-aad9-61b0028827dd"). InnerVolumeSpecName "kube-api-access-rgxrs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:17:47.630903 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:47.630868 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rgxrs\" (UniqueName: \"kubernetes.io/projected/acefe1ec-2c47-487a-aad9-61b0028827dd-kube-api-access-rgxrs\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:17:47.630903 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:47.630898 4908 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/acefe1ec-2c47-487a-aad9-61b0028827dd-dshm\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:17:47.630903 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:47.630907 4908 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/acefe1ec-2c47-487a-aad9-61b0028827dd-tls-certs\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:17:47.911979 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:47.911894 4908 generic.go:356] "Generic (PLEG): container finished" podID="acefe1ec-2c47-487a-aad9-61b0028827dd" containerID="00213e5699ce98b0f1141dfb145a8442362285b27c031af55ee43648808ffb3d" exitCode=0 Oct 30 12:17:47.911979 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:47.911939 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-512ec5/e2e-conflict-b-2c88c229-kserve-545b7f4cdd-82qkd" Oct 30 12:17:47.912184 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:47.911983 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-512ec5/e2e-conflict-b-2c88c229-kserve-545b7f4cdd-82qkd" event={"ID":"acefe1ec-2c47-487a-aad9-61b0028827dd","Type":"ContainerDied","Data":"00213e5699ce98b0f1141dfb145a8442362285b27c031af55ee43648808ffb3d"} Oct 30 12:17:47.912184 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:47.912029 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-512ec5/e2e-conflict-b-2c88c229-kserve-545b7f4cdd-82qkd" event={"ID":"acefe1ec-2c47-487a-aad9-61b0028827dd","Type":"ContainerDied","Data":"28688a757466d48993c82bf0d0cb0a56e6d9fce122c9eb392d2a87e4a729bc04"} Oct 30 12:17:47.912184 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:47.912052 4908 scope.go:122] "RemoveContainer" containerID="00213e5699ce98b0f1141dfb145a8442362285b27c031af55ee43648808ffb3d" Oct 30 12:17:47.922446 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:47.922420 4908 scope.go:122] "RemoveContainer" containerID="00213e5699ce98b0f1141dfb145a8442362285b27c031af55ee43648808ffb3d" Oct 30 12:17:47.922804 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:17:47.922782 4908 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00213e5699ce98b0f1141dfb145a8442362285b27c031af55ee43648808ffb3d\": container with ID starting with 00213e5699ce98b0f1141dfb145a8442362285b27c031af55ee43648808ffb3d not found: ID does not exist" containerID="00213e5699ce98b0f1141dfb145a8442362285b27c031af55ee43648808ffb3d" Oct 30 12:17:47.922892 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:47.922814 4908 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00213e5699ce98b0f1141dfb145a8442362285b27c031af55ee43648808ffb3d"} err="failed to get container status \"00213e5699ce98b0f1141dfb145a8442362285b27c031af55ee43648808ffb3d\": rpc error: code = NotFound desc = could not find container \"00213e5699ce98b0f1141dfb145a8442362285b27c031af55ee43648808ffb3d\": container with ID starting with 00213e5699ce98b0f1141dfb145a8442362285b27c031af55ee43648808ffb3d not found: ID does not exist" Oct 30 12:17:47.964585 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:47.964542 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-512ec5/e2e-conflict-b-2c88c229-kserve-545b7f4cdd-82qkd"] Oct 30 12:17:47.977004 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:47.973662 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-512ec5/e2e-conflict-b-2c88c229-kserve-545b7f4cdd-82qkd"] Oct 30 12:17:48.084903 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:48.084875 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a164f939-sbn28" Oct 30 12:17:48.135980 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:48.135938 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87a04e72-f817-438e-890c-9e6843ff8c18-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87a04e72-f817-438e-890c-9e6843ff8c18-maas-api-openshift-ca\") pod \"87a04e72-f817-438e-890c-9e6843ff8c18\" (UID: \"87a04e72-f817-438e-890c-9e6843ff8c18\") " Oct 30 12:17:48.136476 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:48.136014 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/87a04e72-f817-438e-890c-9e6843ff8c18-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87a04e72-f817-438e-890c-9e6843ff8c18-maas-api-xks-ca\") pod \"87a04e72-f817-438e-890c-9e6843ff8c18\" (UID: \"87a04e72-f817-438e-890c-9e6843ff8c18\") " Oct 30 12:17:48.136476 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:48.136326 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87a04e72-f817-438e-890c-9e6843ff8c18-maas-api-openshift-ca" pod "87a04e72-f817-438e-890c-9e6843ff8c18" (UID: "87a04e72-f817-438e-890c-9e6843ff8c18"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:17:48.136589 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:48.136575 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87a04e72-f817-438e-890c-9e6843ff8c18-maas-api-openshift-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:17:48.138308 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:48.138282 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87a04e72-f817-438e-890c-9e6843ff8c18-maas-api-xks-ca" pod "87a04e72-f817-438e-890c-9e6843ff8c18" (UID: "87a04e72-f817-438e-890c-9e6843ff8c18"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:17:48.237757 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:48.237718 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87a04e72-f817-438e-890c-9e6843ff8c18-kube-api-access-5frwq\" (UniqueName: \"kubernetes.io/projected/87a04e72-f817-438e-890c-9e6843ff8c18-kube-api-access-5frwq\") pod \"87a04e72-f817-438e-890c-9e6843ff8c18\" (UID: \"87a04e72-f817-438e-890c-9e6843ff8c18\") " Oct 30 12:17:48.237972 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:48.237959 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87a04e72-f817-438e-890c-9e6843ff8c18-maas-api-xks-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:17:48.239964 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:48.239940 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87a04e72-f817-438e-890c-9e6843ff8c18-kube-api-access-5frwq" pod "87a04e72-f817-438e-890c-9e6843ff8c18" (UID: "87a04e72-f817-438e-890c-9e6843ff8c18"). InnerVolumeSpecName "kube-api-access-5frwq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:17:48.339174 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:48.339143 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5frwq\" (UniqueName: \"kubernetes.io/projected/87a04e72-f817-438e-890c-9e6843ff8c18-kube-api-access-5frwq\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:17:48.920856 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:48.920799 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a164f939-sbn28" Oct 30 12:17:48.920856 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:48.920840 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a164f939-sbn28" event={"ID":"87a04e72-f817-438e-890c-9e6843ff8c18","Type":"ContainerDied","Data":"2487fd19ab21db777517b9c7ec1ec9a45dfe106de6d1506b4e0141988fe77677"} Oct 30 12:17:48.921116 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:48.920902 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2487fd19ab21db777517b9c7ec1ec9a45dfe106de6d1506b4e0141988fe77677" Oct 30 12:17:49.604992 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:49.604959 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="acefe1ec-2c47-487a-aad9-61b0028827dd" path="/var/lib/kubelet/pods/acefe1ec-2c47-487a-aad9-61b0028827dd/volumes" Oct 30 12:17:49.684771 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:17:49.684736 4908 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vn6vv" containerName="extractor" Oct 30 12:17:54.796332 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:54.796292 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-47fd99/distinct-2-w2-47fd99-kserve-77cf84b469-zxs8s"] Oct 30 12:17:54.797139 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:54.797115 4908 cpu_manager.go:414] "containerMap: removing container" podUID="87a04e72-f817-438e-890c-9e6843ff8c18" containerName="revoke-keys" Oct 30 12:17:54.797197 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:54.797144 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87a04e72-f817-438e-890c-9e6843ff8c18" containerName="revoke-keys" Oct 30 12:17:54.797197 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:54.797168 4908 cpu_manager.go:414] "containerMap: removing container" podUID="acefe1ec-2c47-487a-aad9-61b0028827dd" containerName="main" Oct 30 12:17:54.797197 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:54.797176 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="acefe1ec-2c47-487a-aad9-61b0028827dd" containerName="main" Oct 30 12:17:54.797345 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:54.797332 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="87a04e72-f817-438e-890c-9e6843ff8c18" containerName="revoke-keys" Oct 30 12:17:54.797391 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:54.797380 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="acefe1ec-2c47-487a-aad9-61b0028827dd" containerName="main" Oct 30 12:17:54.801561 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:54.801528 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-47fd99/distinct-2-w2-47fd99-kserve-77cf84b469-zxs8s" Oct 30 12:17:54.829119 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:54.829086 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-47fd99\"/\"distinct-2-w2-47fd99-kserve-self-signed-certs\"" Oct 30 12:17:54.834626 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:54.834596 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-47fd99/distinct-2-w2-47fd99-kserve-77cf84b469-zxs8s"] Oct 30 12:17:54.911217 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:54.911174 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/baf44d3d-42e6-43f7-a0ad-f0f9932320c1-dshm\") pod \"distinct-2-w2-47fd99-kserve-77cf84b469-zxs8s\" (UID: \"baf44d3d-42e6-43f7-a0ad-f0f9932320c1\") " pod="e2e-models-e2e-worker-w2-47fd99/distinct-2-w2-47fd99-kserve-77cf84b469-zxs8s" Oct 30 12:17:54.911444 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:54.911304 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cg89m\" (UniqueName: \"kubernetes.io/projected/baf44d3d-42e6-43f7-a0ad-f0f9932320c1-kube-api-access-cg89m\") pod \"distinct-2-w2-47fd99-kserve-77cf84b469-zxs8s\" (UID: \"baf44d3d-42e6-43f7-a0ad-f0f9932320c1\") " pod="e2e-models-e2e-worker-w2-47fd99/distinct-2-w2-47fd99-kserve-77cf84b469-zxs8s" Oct 30 12:17:54.911444 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:54.911371 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/baf44d3d-42e6-43f7-a0ad-f0f9932320c1-home\") pod \"distinct-2-w2-47fd99-kserve-77cf84b469-zxs8s\" (UID: \"baf44d3d-42e6-43f7-a0ad-f0f9932320c1\") " pod="e2e-models-e2e-worker-w2-47fd99/distinct-2-w2-47fd99-kserve-77cf84b469-zxs8s" Oct 30 12:17:54.911444 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:54.911411 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/baf44d3d-42e6-43f7-a0ad-f0f9932320c1-model-cache\") pod \"distinct-2-w2-47fd99-kserve-77cf84b469-zxs8s\" (UID: \"baf44d3d-42e6-43f7-a0ad-f0f9932320c1\") " pod="e2e-models-e2e-worker-w2-47fd99/distinct-2-w2-47fd99-kserve-77cf84b469-zxs8s" Oct 30 12:17:54.911444 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:54.911440 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/baf44d3d-42e6-43f7-a0ad-f0f9932320c1-tmp-dir\") pod \"distinct-2-w2-47fd99-kserve-77cf84b469-zxs8s\" (UID: \"baf44d3d-42e6-43f7-a0ad-f0f9932320c1\") " pod="e2e-models-e2e-worker-w2-47fd99/distinct-2-w2-47fd99-kserve-77cf84b469-zxs8s" Oct 30 12:17:54.911676 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:54.911481 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/baf44d3d-42e6-43f7-a0ad-f0f9932320c1-tls-certs\") pod \"distinct-2-w2-47fd99-kserve-77cf84b469-zxs8s\" (UID: \"baf44d3d-42e6-43f7-a0ad-f0f9932320c1\") " pod="e2e-models-e2e-worker-w2-47fd99/distinct-2-w2-47fd99-kserve-77cf84b469-zxs8s" Oct 30 12:17:55.013305 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:55.013261 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/baf44d3d-42e6-43f7-a0ad-f0f9932320c1-tls-certs\") pod \"distinct-2-w2-47fd99-kserve-77cf84b469-zxs8s\" (UID: \"baf44d3d-42e6-43f7-a0ad-f0f9932320c1\") " pod="e2e-models-e2e-worker-w2-47fd99/distinct-2-w2-47fd99-kserve-77cf84b469-zxs8s" Oct 30 12:17:55.013536 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:55.013375 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/baf44d3d-42e6-43f7-a0ad-f0f9932320c1-dshm\") pod \"distinct-2-w2-47fd99-kserve-77cf84b469-zxs8s\" (UID: \"baf44d3d-42e6-43f7-a0ad-f0f9932320c1\") " pod="e2e-models-e2e-worker-w2-47fd99/distinct-2-w2-47fd99-kserve-77cf84b469-zxs8s" Oct 30 12:17:55.013536 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:55.013489 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cg89m\" (UniqueName: \"kubernetes.io/projected/baf44d3d-42e6-43f7-a0ad-f0f9932320c1-kube-api-access-cg89m\") pod \"distinct-2-w2-47fd99-kserve-77cf84b469-zxs8s\" (UID: \"baf44d3d-42e6-43f7-a0ad-f0f9932320c1\") " pod="e2e-models-e2e-worker-w2-47fd99/distinct-2-w2-47fd99-kserve-77cf84b469-zxs8s" Oct 30 12:17:55.013665 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:55.013535 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/baf44d3d-42e6-43f7-a0ad-f0f9932320c1-home\") pod \"distinct-2-w2-47fd99-kserve-77cf84b469-zxs8s\" (UID: \"baf44d3d-42e6-43f7-a0ad-f0f9932320c1\") " pod="e2e-models-e2e-worker-w2-47fd99/distinct-2-w2-47fd99-kserve-77cf84b469-zxs8s" Oct 30 12:17:55.013665 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:55.013581 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/baf44d3d-42e6-43f7-a0ad-f0f9932320c1-model-cache\") pod \"distinct-2-w2-47fd99-kserve-77cf84b469-zxs8s\" (UID: \"baf44d3d-42e6-43f7-a0ad-f0f9932320c1\") " pod="e2e-models-e2e-worker-w2-47fd99/distinct-2-w2-47fd99-kserve-77cf84b469-zxs8s" Oct 30 12:17:55.013665 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:55.013619 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/baf44d3d-42e6-43f7-a0ad-f0f9932320c1-tmp-dir\") pod \"distinct-2-w2-47fd99-kserve-77cf84b469-zxs8s\" (UID: \"baf44d3d-42e6-43f7-a0ad-f0f9932320c1\") " pod="e2e-models-e2e-worker-w2-47fd99/distinct-2-w2-47fd99-kserve-77cf84b469-zxs8s" Oct 30 12:17:55.014799 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:55.013995 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/baf44d3d-42e6-43f7-a0ad-f0f9932320c1-tmp-dir\") pod \"distinct-2-w2-47fd99-kserve-77cf84b469-zxs8s\" (UID: \"baf44d3d-42e6-43f7-a0ad-f0f9932320c1\") " pod="e2e-models-e2e-worker-w2-47fd99/distinct-2-w2-47fd99-kserve-77cf84b469-zxs8s" Oct 30 12:17:55.014799 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:55.014170 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/baf44d3d-42e6-43f7-a0ad-f0f9932320c1-home\") pod \"distinct-2-w2-47fd99-kserve-77cf84b469-zxs8s\" (UID: \"baf44d3d-42e6-43f7-a0ad-f0f9932320c1\") " pod="e2e-models-e2e-worker-w2-47fd99/distinct-2-w2-47fd99-kserve-77cf84b469-zxs8s" Oct 30 12:17:55.014799 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:55.014752 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/baf44d3d-42e6-43f7-a0ad-f0f9932320c1-model-cache\") pod \"distinct-2-w2-47fd99-kserve-77cf84b469-zxs8s\" (UID: \"baf44d3d-42e6-43f7-a0ad-f0f9932320c1\") " pod="e2e-models-e2e-worker-w2-47fd99/distinct-2-w2-47fd99-kserve-77cf84b469-zxs8s" Oct 30 12:17:55.016446 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:55.016410 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/baf44d3d-42e6-43f7-a0ad-f0f9932320c1-tls-certs\") pod \"distinct-2-w2-47fd99-kserve-77cf84b469-zxs8s\" (UID: \"baf44d3d-42e6-43f7-a0ad-f0f9932320c1\") " pod="e2e-models-e2e-worker-w2-47fd99/distinct-2-w2-47fd99-kserve-77cf84b469-zxs8s" Oct 30 12:17:55.017035 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:55.017009 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/baf44d3d-42e6-43f7-a0ad-f0f9932320c1-dshm\") pod \"distinct-2-w2-47fd99-kserve-77cf84b469-zxs8s\" (UID: \"baf44d3d-42e6-43f7-a0ad-f0f9932320c1\") " pod="e2e-models-e2e-worker-w2-47fd99/distinct-2-w2-47fd99-kserve-77cf84b469-zxs8s" Oct 30 12:17:55.038071 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:55.038035 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cg89m\" (UniqueName: \"kubernetes.io/projected/baf44d3d-42e6-43f7-a0ad-f0f9932320c1-kube-api-access-cg89m\") pod \"distinct-2-w2-47fd99-kserve-77cf84b469-zxs8s\" (UID: \"baf44d3d-42e6-43f7-a0ad-f0f9932320c1\") " pod="e2e-models-e2e-worker-w2-47fd99/distinct-2-w2-47fd99-kserve-77cf84b469-zxs8s" Oct 30 12:17:55.118059 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:55.117967 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-47fd99/distinct-2-w2-47fd99-kserve-77cf84b469-zxs8s" Oct 30 12:17:55.257609 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:55.257570 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-504400d1-cktkq"] Oct 30 12:17:55.261713 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:55.261682 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-504400d1-cktkq" Oct 30 12:17:55.277092 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:55.277062 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 12:17:55.277092 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:55.277082 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 12:17:55.277381 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:55.277145 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s75hq\"" Oct 30 12:17:55.277381 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:55.277198 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 12:17:55.281892 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:55.281869 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-504400d1-cktkq"] Oct 30 12:17:55.317890 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:55.317849 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b2f3f3f-ca8b-4b15-a0e2-f3c1381664f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-504400d1-cktkq\" (UID: \"6b2f3f3f-ca8b-4b15-a0e2-f3c1381664f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-504400d1-cktkq" Oct 30 12:17:55.318066 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:55.318014 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b2f3f3f-ca8b-4b15-a0e2-f3c1381664f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-504400d1-cktkq\" (UID: \"6b2f3f3f-ca8b-4b15-a0e2-f3c1381664f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-504400d1-cktkq" Oct 30 12:17:55.318066 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:55.318042 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98fc6\" (UniqueName: \"kubernetes.io/projected/6b2f3f3f-ca8b-4b15-a0e2-f3c1381664f9-kube-api-access-98fc6\") pod \"maas-api-revoke-sub-504400d1-cktkq\" (UID: \"6b2f3f3f-ca8b-4b15-a0e2-f3c1381664f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-504400d1-cktkq" Oct 30 12:17:55.419235 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:55.419152 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b2f3f3f-ca8b-4b15-a0e2-f3c1381664f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-504400d1-cktkq\" (UID: \"6b2f3f3f-ca8b-4b15-a0e2-f3c1381664f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-504400d1-cktkq" Oct 30 12:17:55.419399 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:55.419252 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b2f3f3f-ca8b-4b15-a0e2-f3c1381664f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-504400d1-cktkq\" (UID: \"6b2f3f3f-ca8b-4b15-a0e2-f3c1381664f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-504400d1-cktkq" Oct 30 12:17:55.419399 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:55.419282 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98fc6\" (UniqueName: \"kubernetes.io/projected/6b2f3f3f-ca8b-4b15-a0e2-f3c1381664f9-kube-api-access-98fc6\") pod \"maas-api-revoke-sub-504400d1-cktkq\" (UID: \"6b2f3f3f-ca8b-4b15-a0e2-f3c1381664f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-504400d1-cktkq" Oct 30 12:17:55.419992 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:55.419966 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b2f3f3f-ca8b-4b15-a0e2-f3c1381664f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-504400d1-cktkq\" (UID: \"6b2f3f3f-ca8b-4b15-a0e2-f3c1381664f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-504400d1-cktkq" Oct 30 12:17:55.421711 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:55.421687 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b2f3f3f-ca8b-4b15-a0e2-f3c1381664f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-504400d1-cktkq\" (UID: \"6b2f3f3f-ca8b-4b15-a0e2-f3c1381664f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-504400d1-cktkq" Oct 30 12:17:55.428743 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:55.428721 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-47fd99/distinct-2-w2-47fd99-kserve-77cf84b469-zxs8s"] Oct 30 12:17:55.431119 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:17:55.431094 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbaf44d3d_42e6_43f7_a0ad_f0f9932320c1.slice/crio-28aa7de2180d2cc503528a49466f6b8bd98739b4cf69252cfcc9f4587501da4f WatchSource:0}: Error finding container 28aa7de2180d2cc503528a49466f6b8bd98739b4cf69252cfcc9f4587501da4f: Status 404 returned error can't find the container with id 28aa7de2180d2cc503528a49466f6b8bd98739b4cf69252cfcc9f4587501da4f Oct 30 12:17:55.441343 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:55.441313 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98fc6\" (UniqueName: \"kubernetes.io/projected/6b2f3f3f-ca8b-4b15-a0e2-f3c1381664f9-kube-api-access-98fc6\") pod \"maas-api-revoke-sub-504400d1-cktkq\" (UID: \"6b2f3f3f-ca8b-4b15-a0e2-f3c1381664f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-504400d1-cktkq" Oct 30 12:17:55.549783 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:55.549750 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-tsxml"] Oct 30 12:17:55.553694 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:55.553672 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-tsxml" Oct 30 12:17:55.575758 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:55.575724 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-tsxml"] Oct 30 12:17:55.578952 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:55.578926 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-504400d1-cktkq" Oct 30 12:17:55.620983 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:55.620941 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tn9dn\" (UniqueName: \"kubernetes.io/projected/c8dc0dee-2192-42cf-a386-1a02e5ceb4f0-kube-api-access-tn9dn\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-tsxml\" (UID: \"c8dc0dee-2192-42cf-a386-1a02e5ceb4f0\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-tsxml" Oct 30 12:17:55.621157 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:55.621013 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8dc0dee-2192-42cf-a386-1a02e5ceb4f0-istio-token\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-tsxml\" (UID: \"c8dc0dee-2192-42cf-a386-1a02e5ceb4f0\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-tsxml" Oct 30 12:17:55.621157 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:55.621051 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8dc0dee-2192-42cf-a386-1a02e5ceb4f0-istio-podinfo\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-tsxml\" (UID: \"c8dc0dee-2192-42cf-a386-1a02e5ceb4f0\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-tsxml" Oct 30 12:17:55.621157 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:55.621086 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8dc0dee-2192-42cf-a386-1a02e5ceb4f0-istio-envoy\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-tsxml\" (UID: \"c8dc0dee-2192-42cf-a386-1a02e5ceb4f0\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-tsxml" Oct 30 12:17:55.621157 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:55.621118 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8dc0dee-2192-42cf-a386-1a02e5ceb4f0-istio-data\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-tsxml\" (UID: \"c8dc0dee-2192-42cf-a386-1a02e5ceb4f0\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-tsxml" Oct 30 12:17:55.621382 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:55.621210 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8dc0dee-2192-42cf-a386-1a02e5ceb4f0-credential-socket\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-tsxml\" (UID: \"c8dc0dee-2192-42cf-a386-1a02e5ceb4f0\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-tsxml" Oct 30 12:17:55.621382 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:55.621249 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8dc0dee-2192-42cf-a386-1a02e5ceb4f0-workload-certs\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-tsxml\" (UID: \"c8dc0dee-2192-42cf-a386-1a02e5ceb4f0\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-tsxml" Oct 30 12:17:55.621382 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:55.621291 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8dc0dee-2192-42cf-a386-1a02e5ceb4f0-istiod-ca-cert\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-tsxml\" (UID: \"c8dc0dee-2192-42cf-a386-1a02e5ceb4f0\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-tsxml" Oct 30 12:17:55.621382 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:55.621320 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8dc0dee-2192-42cf-a386-1a02e5ceb4f0-workload-socket\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-tsxml\" (UID: \"c8dc0dee-2192-42cf-a386-1a02e5ceb4f0\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-tsxml" Oct 30 12:17:55.722612 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:55.722503 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8dc0dee-2192-42cf-a386-1a02e5ceb4f0-credential-socket\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-tsxml\" (UID: \"c8dc0dee-2192-42cf-a386-1a02e5ceb4f0\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-tsxml" Oct 30 12:17:55.722612 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:55.722569 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8dc0dee-2192-42cf-a386-1a02e5ceb4f0-workload-certs\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-tsxml\" (UID: \"c8dc0dee-2192-42cf-a386-1a02e5ceb4f0\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-tsxml" Oct 30 12:17:55.722890 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:55.722689 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8dc0dee-2192-42cf-a386-1a02e5ceb4f0-istiod-ca-cert\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-tsxml\" (UID: \"c8dc0dee-2192-42cf-a386-1a02e5ceb4f0\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-tsxml" Oct 30 12:17:55.722890 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:55.722735 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8dc0dee-2192-42cf-a386-1a02e5ceb4f0-workload-socket\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-tsxml\" (UID: \"c8dc0dee-2192-42cf-a386-1a02e5ceb4f0\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-tsxml" Oct 30 12:17:55.722996 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:55.722920 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tn9dn\" (UniqueName: \"kubernetes.io/projected/c8dc0dee-2192-42cf-a386-1a02e5ceb4f0-kube-api-access-tn9dn\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-tsxml\" (UID: \"c8dc0dee-2192-42cf-a386-1a02e5ceb4f0\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-tsxml" Oct 30 12:17:55.723050 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:55.722993 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8dc0dee-2192-42cf-a386-1a02e5ceb4f0-istio-token\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-tsxml\" (UID: \"c8dc0dee-2192-42cf-a386-1a02e5ceb4f0\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-tsxml" Oct 30 12:17:55.723050 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:55.723034 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8dc0dee-2192-42cf-a386-1a02e5ceb4f0-istio-podinfo\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-tsxml\" (UID: \"c8dc0dee-2192-42cf-a386-1a02e5ceb4f0\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-tsxml" Oct 30 12:17:55.723158 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:55.723075 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8dc0dee-2192-42cf-a386-1a02e5ceb4f0-istio-envoy\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-tsxml\" (UID: \"c8dc0dee-2192-42cf-a386-1a02e5ceb4f0\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-tsxml" Oct 30 12:17:55.723158 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:55.723092 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8dc0dee-2192-42cf-a386-1a02e5ceb4f0-credential-socket\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-tsxml\" (UID: \"c8dc0dee-2192-42cf-a386-1a02e5ceb4f0\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-tsxml" Oct 30 12:17:55.723158 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:55.723116 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8dc0dee-2192-42cf-a386-1a02e5ceb4f0-istio-data\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-tsxml\" (UID: \"c8dc0dee-2192-42cf-a386-1a02e5ceb4f0\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-tsxml" Oct 30 12:17:55.723410 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:55.723387 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8dc0dee-2192-42cf-a386-1a02e5ceb4f0-workload-socket\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-tsxml\" (UID: \"c8dc0dee-2192-42cf-a386-1a02e5ceb4f0\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-tsxml" Oct 30 12:17:55.723625 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:55.723600 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8dc0dee-2192-42cf-a386-1a02e5ceb4f0-istiod-ca-cert\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-tsxml\" (UID: \"c8dc0dee-2192-42cf-a386-1a02e5ceb4f0\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-tsxml" Oct 30 12:17:55.723733 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:55.723672 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8dc0dee-2192-42cf-a386-1a02e5ceb4f0-istio-data\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-tsxml\" (UID: \"c8dc0dee-2192-42cf-a386-1a02e5ceb4f0\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-tsxml" Oct 30 12:17:55.723807 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:55.723788 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8dc0dee-2192-42cf-a386-1a02e5ceb4f0-workload-certs\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-tsxml\" (UID: \"c8dc0dee-2192-42cf-a386-1a02e5ceb4f0\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-tsxml" Oct 30 12:17:55.725979 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:55.725957 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8dc0dee-2192-42cf-a386-1a02e5ceb4f0-istio-envoy\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-tsxml\" (UID: \"c8dc0dee-2192-42cf-a386-1a02e5ceb4f0\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-tsxml" Oct 30 12:17:55.726198 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:55.726180 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8dc0dee-2192-42cf-a386-1a02e5ceb4f0-istio-podinfo\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-tsxml\" (UID: \"c8dc0dee-2192-42cf-a386-1a02e5ceb4f0\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-tsxml" Oct 30 12:17:55.745561 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:55.745530 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8dc0dee-2192-42cf-a386-1a02e5ceb4f0-istio-token\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-tsxml\" (UID: \"c8dc0dee-2192-42cf-a386-1a02e5ceb4f0\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-tsxml" Oct 30 12:17:55.745674 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:55.745651 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tn9dn\" (UniqueName: \"kubernetes.io/projected/c8dc0dee-2192-42cf-a386-1a02e5ceb4f0-kube-api-access-tn9dn\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-tsxml\" (UID: \"c8dc0dee-2192-42cf-a386-1a02e5ceb4f0\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-tsxml" Oct 30 12:17:55.868587 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:55.868547 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-tsxml" Oct 30 12:17:55.941029 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:55.940943 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-504400d1-cktkq"] Oct 30 12:17:55.996528 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:55.994335 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-504400d1-cktkq" event={"ID":"6b2f3f3f-ca8b-4b15-a0e2-f3c1381664f9","Type":"ContainerStarted","Data":"4b7bf23e8ce3fdc810cdd09268cb58ecf58819612adf1ea2eda3530029928362"} Oct 30 12:17:55.996963 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:55.996926 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-47fd99/distinct-2-w2-47fd99-kserve-77cf84b469-zxs8s" event={"ID":"baf44d3d-42e6-43f7-a0ad-f0f9932320c1","Type":"ContainerStarted","Data":"cac60264305a95c80c5af5bb81022523e41a873fe60c98433574b0b96b20db41"} Oct 30 12:17:55.997068 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:55.996976 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-47fd99/distinct-2-w2-47fd99-kserve-77cf84b469-zxs8s" event={"ID":"baf44d3d-42e6-43f7-a0ad-f0f9932320c1","Type":"ContainerStarted","Data":"28aa7de2180d2cc503528a49466f6b8bd98739b4cf69252cfcc9f4587501da4f"} Oct 30 12:17:56.040785 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:56.040718 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-47fd99/distinct-2-w2-47fd99-kserve-77cf84b469-zxs8s" podStartSLOduration=2.040701001 podStartE2EDuration="2.040701001s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:17:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:17:56.037256832 +0000 UTC m=+1432.551685579" watchObservedRunningTime="2026-10-30 12:17:56.040701001 +0000 UTC m=+1432.555129792" Oct 30 12:17:56.118145 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:56.118107 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-47fd99/distinct-2-w2-47fd99-kserve-77cf84b469-zxs8s" Oct 30 12:17:56.276919 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:56.274720 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-tsxml"] Oct 30 12:17:56.359948 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:56.359888 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:17:56.360196 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:56.360171 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:17:56.360356 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:56.360336 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:17:57.009181 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:57.009131 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-504400d1-cktkq" event={"ID":"6b2f3f3f-ca8b-4b15-a0e2-f3c1381664f9","Type":"ContainerStarted","Data":"eb58bacec1879baa5a58f1d00649db8641536014bbbc9cd317dd1e536be5921a"} Oct 30 12:17:57.011929 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:57.011896 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-tsxml" event={"ID":"c8dc0dee-2192-42cf-a386-1a02e5ceb4f0","Type":"ContainerStarted","Data":"e2e87b2daf616b9c7457fd765c2d51f7100f211197cd0787125d80dda02bbcdd"} Oct 30 12:17:57.012178 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:57.011949 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-tsxml" event={"ID":"c8dc0dee-2192-42cf-a386-1a02e5ceb4f0","Type":"ContainerStarted","Data":"69088e1dee7a137067227771a770bd6ebc51c24d3ed0d1b3acff2174691c16cf"} Oct 30 12:17:57.048029 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:57.047970 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-504400d1-cktkq" podStartSLOduration=2.047952853 podStartE2EDuration="2.047952853s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:17:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:17:57.04738985 +0000 UTC m=+1433.561818600" watchObservedRunningTime="2026-10-30 12:17:57.047952853 +0000 UTC m=+1433.562381603" Oct 30 12:17:57.097954 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:57.097880 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-tsxml" podStartSLOduration=2.097854296 podStartE2EDuration="2.097854296s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:17:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:17:57.091160057 +0000 UTC m=+1433.605588804" watchObservedRunningTime="2026-10-30 12:17:57.097854296 +0000 UTC m=+1433.612283055" Oct 30 12:17:57.872305 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:57.872266 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-tsxml" Oct 30 12:17:57.879006 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:57.878967 4908 patch_prober.go:29] interesting pod/e2e-worker-w0-f920f7-openshift-default-7c9767c97-tsxml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 30 12:17:57.879285 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:57.879255 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-tsxml" podUID="c8dc0dee-2192-42cf-a386-1a02e5ceb4f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 30 12:17:58.020052 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:58.020015 4908 generic.go:356] "Generic (PLEG): container finished" podID="6b2f3f3f-ca8b-4b15-a0e2-f3c1381664f9" containerID="eb58bacec1879baa5a58f1d00649db8641536014bbbc9cd317dd1e536be5921a" exitCode=0 Oct 30 12:17:58.020508 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:58.020104 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-504400d1-cktkq" event={"ID":"6b2f3f3f-ca8b-4b15-a0e2-f3c1381664f9","Type":"ContainerDied","Data":"eb58bacec1879baa5a58f1d00649db8641536014bbbc9cd317dd1e536be5921a"} Oct 30 12:17:58.876984 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:58.876944 4908 patch_prober.go:29] interesting pod/e2e-worker-w0-f920f7-openshift-default-7c9767c97-tsxml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 30 12:17:58.877295 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:58.877261 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-tsxml" podUID="c8dc0dee-2192-42cf-a386-1a02e5ceb4f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 30 12:17:59.230980 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:59.230953 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-504400d1-cktkq" Oct 30 12:17:59.299257 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:59.299219 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b2f3f3f-ca8b-4b15-a0e2-f3c1381664f9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b2f3f3f-ca8b-4b15-a0e2-f3c1381664f9-maas-api-xks-ca\") pod \"6b2f3f3f-ca8b-4b15-a0e2-f3c1381664f9\" (UID: \"6b2f3f3f-ca8b-4b15-a0e2-f3c1381664f9\") " Oct 30 12:17:59.299465 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:59.299407 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b2f3f3f-ca8b-4b15-a0e2-f3c1381664f9-kube-api-access-98fc6\" (UniqueName: \"kubernetes.io/projected/6b2f3f3f-ca8b-4b15-a0e2-f3c1381664f9-kube-api-access-98fc6\") pod \"6b2f3f3f-ca8b-4b15-a0e2-f3c1381664f9\" (UID: \"6b2f3f3f-ca8b-4b15-a0e2-f3c1381664f9\") " Oct 30 12:17:59.299535 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:59.299520 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b2f3f3f-ca8b-4b15-a0e2-f3c1381664f9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b2f3f3f-ca8b-4b15-a0e2-f3c1381664f9-maas-api-openshift-ca\") pod \"6b2f3f3f-ca8b-4b15-a0e2-f3c1381664f9\" (UID: \"6b2f3f3f-ca8b-4b15-a0e2-f3c1381664f9\") " Oct 30 12:17:59.300342 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:59.300311 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b2f3f3f-ca8b-4b15-a0e2-f3c1381664f9-maas-api-openshift-ca" pod "6b2f3f3f-ca8b-4b15-a0e2-f3c1381664f9" (UID: "6b2f3f3f-ca8b-4b15-a0e2-f3c1381664f9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:17:59.302910 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:59.302866 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b2f3f3f-ca8b-4b15-a0e2-f3c1381664f9-maas-api-xks-ca" pod "6b2f3f3f-ca8b-4b15-a0e2-f3c1381664f9" (UID: "6b2f3f3f-ca8b-4b15-a0e2-f3c1381664f9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:17:59.303222 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:59.303201 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b2f3f3f-ca8b-4b15-a0e2-f3c1381664f9-kube-api-access-98fc6" pod "6b2f3f3f-ca8b-4b15-a0e2-f3c1381664f9" (UID: "6b2f3f3f-ca8b-4b15-a0e2-f3c1381664f9"). InnerVolumeSpecName "kube-api-access-98fc6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:17:59.401569 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:59.401532 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98fc6\" (UniqueName: \"kubernetes.io/projected/6b2f3f3f-ca8b-4b15-a0e2-f3c1381664f9-kube-api-access-98fc6\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:17:59.401569 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:59.401565 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b2f3f3f-ca8b-4b15-a0e2-f3c1381664f9-maas-api-openshift-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:17:59.401880 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:59.401588 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b2f3f3f-ca8b-4b15-a0e2-f3c1381664f9-maas-api-xks-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:17:59.876916 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:59.874217 4908 patch_prober.go:29] interesting pod/e2e-worker-w0-f920f7-openshift-default-7c9767c97-tsxml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 30 12:17:59.876916 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:17:59.874277 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-tsxml" podUID="c8dc0dee-2192-42cf-a386-1a02e5ceb4f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 30 12:18:00.034405 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:00.034377 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-504400d1-cktkq" Oct 30 12:18:00.034586 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:00.034408 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-504400d1-cktkq" event={"ID":"6b2f3f3f-ca8b-4b15-a0e2-f3c1381664f9","Type":"ContainerDied","Data":"4b7bf23e8ce3fdc810cdd09268cb58ecf58819612adf1ea2eda3530029928362"} Oct 30 12:18:00.034586 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:00.034456 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4b7bf23e8ce3fdc810cdd09268cb58ecf58819612adf1ea2eda3530029928362" Oct 30 12:18:00.876734 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:00.876253 4908 patch_prober.go:29] interesting pod/e2e-worker-w0-f920f7-openshift-default-7c9767c97-tsxml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 30 12:18:00.876734 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:00.876322 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-tsxml" podUID="c8dc0dee-2192-42cf-a386-1a02e5ceb4f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 30 12:18:01.897869 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:01.895995 4908 patch_prober.go:29] interesting pod/e2e-worker-w0-f920f7-openshift-default-7c9767c97-tsxml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 30 12:18:01.897869 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:01.896049 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-tsxml" podUID="c8dc0dee-2192-42cf-a386-1a02e5ceb4f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 30 12:18:02.871624 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:02.871576 4908 patch_prober.go:29] interesting pod/e2e-worker-w0-f920f7-openshift-default-7c9767c97-tsxml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 30 12:18:02.871814 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:02.871644 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-tsxml" podUID="c8dc0dee-2192-42cf-a386-1a02e5ceb4f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 30 12:18:04.075313 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:04.075278 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-tsxml" Oct 30 12:18:04.076370 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:04.076336 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-tsxml" Oct 30 12:18:04.077516 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:04.077483 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-tsxml" Oct 30 12:18:05.118672 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:05.118632 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-47fd99/distinct-2-w2-47fd99-kserve-77cf84b469-zxs8s" Oct 30 12:18:05.132766 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:05.132734 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-47fd99/distinct-2-w2-47fd99-kserve-77cf84b469-zxs8s" Oct 30 12:18:06.088059 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:06.088030 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-47fd99/distinct-2-w2-47fd99-kserve-77cf84b469-zxs8s" Oct 30 12:18:08.288906 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:08.288865 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-66838cc1-wwvtr"] Oct 30 12:18:08.289365 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:08.289351 4908 cpu_manager.go:414] "containerMap: removing container" podUID="6b2f3f3f-ca8b-4b15-a0e2-f3c1381664f9" containerName="revoke-keys" Oct 30 12:18:08.289426 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:08.289366 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b2f3f3f-ca8b-4b15-a0e2-f3c1381664f9" containerName="revoke-keys" Oct 30 12:18:08.289490 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:08.289481 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b2f3f3f-ca8b-4b15-a0e2-f3c1381664f9" containerName="revoke-keys" Oct 30 12:18:08.294736 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:08.294713 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66838cc1-wwvtr" Oct 30 12:18:08.310166 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:08.310135 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 12:18:08.321341 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:08.321307 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 12:18:08.321341 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:08.321314 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s75hq\"" Oct 30 12:18:08.321540 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:08.321354 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 12:18:08.325993 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:08.325969 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-66838cc1-wwvtr"] Oct 30 12:18:08.417324 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:08.417271 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/590ad346-4bdd-4519-bfb8-adeefa86ba38-maas-api-xks-ca\") pod \"maas-api-revoke-sub-66838cc1-wwvtr\" (UID: \"590ad346-4bdd-4519-bfb8-adeefa86ba38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66838cc1-wwvtr" Oct 30 12:18:08.417522 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:08.417450 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/590ad346-4bdd-4519-bfb8-adeefa86ba38-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-66838cc1-wwvtr\" (UID: \"590ad346-4bdd-4519-bfb8-adeefa86ba38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66838cc1-wwvtr" Oct 30 12:18:08.417522 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:08.417511 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5qxx\" (UniqueName: \"kubernetes.io/projected/590ad346-4bdd-4519-bfb8-adeefa86ba38-kube-api-access-g5qxx\") pod \"maas-api-revoke-sub-66838cc1-wwvtr\" (UID: \"590ad346-4bdd-4519-bfb8-adeefa86ba38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66838cc1-wwvtr" Oct 30 12:18:08.519036 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:08.518996 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/590ad346-4bdd-4519-bfb8-adeefa86ba38-maas-api-xks-ca\") pod \"maas-api-revoke-sub-66838cc1-wwvtr\" (UID: \"590ad346-4bdd-4519-bfb8-adeefa86ba38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66838cc1-wwvtr" Oct 30 12:18:08.519236 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:08.519073 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/590ad346-4bdd-4519-bfb8-adeefa86ba38-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-66838cc1-wwvtr\" (UID: \"590ad346-4bdd-4519-bfb8-adeefa86ba38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66838cc1-wwvtr" Oct 30 12:18:08.519236 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:08.519108 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5qxx\" (UniqueName: \"kubernetes.io/projected/590ad346-4bdd-4519-bfb8-adeefa86ba38-kube-api-access-g5qxx\") pod \"maas-api-revoke-sub-66838cc1-wwvtr\" (UID: \"590ad346-4bdd-4519-bfb8-adeefa86ba38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66838cc1-wwvtr" Oct 30 12:18:08.519876 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:08.519838 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/590ad346-4bdd-4519-bfb8-adeefa86ba38-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-66838cc1-wwvtr\" (UID: \"590ad346-4bdd-4519-bfb8-adeefa86ba38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66838cc1-wwvtr" Oct 30 12:18:08.521476 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:08.521447 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/590ad346-4bdd-4519-bfb8-adeefa86ba38-maas-api-xks-ca\") pod \"maas-api-revoke-sub-66838cc1-wwvtr\" (UID: \"590ad346-4bdd-4519-bfb8-adeefa86ba38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66838cc1-wwvtr" Oct 30 12:18:08.541872 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:08.541771 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5qxx\" (UniqueName: \"kubernetes.io/projected/590ad346-4bdd-4519-bfb8-adeefa86ba38-kube-api-access-g5qxx\") pod \"maas-api-revoke-sub-66838cc1-wwvtr\" (UID: \"590ad346-4bdd-4519-bfb8-adeefa86ba38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66838cc1-wwvtr" Oct 30 12:18:08.611288 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:08.611251 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66838cc1-wwvtr" Oct 30 12:18:08.919027 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:08.919000 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-66838cc1-wwvtr"] Oct 30 12:18:08.921901 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:18:08.921870 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod590ad346_4bdd_4519_bfb8_adeefa86ba38.slice/crio-25f64d77f7f9d9a386416ebbe910a4198ea8344809fbc603ce1dea9dc030f2f9 WatchSource:0}: Error finding container 25f64d77f7f9d9a386416ebbe910a4198ea8344809fbc603ce1dea9dc030f2f9: Status 404 returned error can't find the container with id 25f64d77f7f9d9a386416ebbe910a4198ea8344809fbc603ce1dea9dc030f2f9 Oct 30 12:18:09.101059 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:09.101018 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66838cc1-wwvtr" event={"ID":"590ad346-4bdd-4519-bfb8-adeefa86ba38","Type":"ContainerStarted","Data":"25f64d77f7f9d9a386416ebbe910a4198ea8344809fbc603ce1dea9dc030f2f9"} Oct 30 12:18:09.316966 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:09.316931 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d325dcc2-l845t"] Oct 30 12:18:09.322968 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:09.322947 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d325dcc2-l845t" Oct 30 12:18:09.339691 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:09.339668 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d325dcc2-l845t"] Oct 30 12:18:09.428869 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:09.428843 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5685d\" (UniqueName: \"kubernetes.io/projected/4bfb80b4-66c6-42a0-90c2-9ddd522c2811-kube-api-access-5685d\") pod \"maas-api-revoke-sub-d325dcc2-l845t\" (UID: \"4bfb80b4-66c6-42a0-90c2-9ddd522c2811\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d325dcc2-l845t" Oct 30 12:18:09.429029 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:09.429007 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4bfb80b4-66c6-42a0-90c2-9ddd522c2811-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d325dcc2-l845t\" (UID: \"4bfb80b4-66c6-42a0-90c2-9ddd522c2811\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d325dcc2-l845t" Oct 30 12:18:09.429131 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:09.429115 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4bfb80b4-66c6-42a0-90c2-9ddd522c2811-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d325dcc2-l845t\" (UID: \"4bfb80b4-66c6-42a0-90c2-9ddd522c2811\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d325dcc2-l845t" Oct 30 12:18:09.530307 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:09.530266 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4bfb80b4-66c6-42a0-90c2-9ddd522c2811-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d325dcc2-l845t\" (UID: \"4bfb80b4-66c6-42a0-90c2-9ddd522c2811\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d325dcc2-l845t" Oct 30 12:18:09.530446 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:09.530348 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5685d\" (UniqueName: \"kubernetes.io/projected/4bfb80b4-66c6-42a0-90c2-9ddd522c2811-kube-api-access-5685d\") pod \"maas-api-revoke-sub-d325dcc2-l845t\" (UID: \"4bfb80b4-66c6-42a0-90c2-9ddd522c2811\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d325dcc2-l845t" Oct 30 12:18:09.530446 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:09.530437 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4bfb80b4-66c6-42a0-90c2-9ddd522c2811-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d325dcc2-l845t\" (UID: \"4bfb80b4-66c6-42a0-90c2-9ddd522c2811\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d325dcc2-l845t" Oct 30 12:18:09.531034 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:09.531015 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4bfb80b4-66c6-42a0-90c2-9ddd522c2811-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d325dcc2-l845t\" (UID: \"4bfb80b4-66c6-42a0-90c2-9ddd522c2811\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d325dcc2-l845t" Oct 30 12:18:09.532623 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:09.532599 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4bfb80b4-66c6-42a0-90c2-9ddd522c2811-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d325dcc2-l845t\" (UID: \"4bfb80b4-66c6-42a0-90c2-9ddd522c2811\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d325dcc2-l845t" Oct 30 12:18:09.552685 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:09.552657 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5685d\" (UniqueName: \"kubernetes.io/projected/4bfb80b4-66c6-42a0-90c2-9ddd522c2811-kube-api-access-5685d\") pod \"maas-api-revoke-sub-d325dcc2-l845t\" (UID: \"4bfb80b4-66c6-42a0-90c2-9ddd522c2811\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d325dcc2-l845t" Oct 30 12:18:09.636652 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:09.636625 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d325dcc2-l845t" Oct 30 12:18:09.950980 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:09.950953 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d325dcc2-l845t"] Oct 30 12:18:09.953322 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:18:09.953289 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bfb80b4_66c6_42a0_90c2_9ddd522c2811.slice/crio-9421f133a0b5aaa2f2a83758a3d4e337e150fe845dba87d9fbdb70d54acc3842 WatchSource:0}: Error finding container 9421f133a0b5aaa2f2a83758a3d4e337e150fe845dba87d9fbdb70d54acc3842: Status 404 returned error can't find the container with id 9421f133a0b5aaa2f2a83758a3d4e337e150fe845dba87d9fbdb70d54acc3842 Oct 30 12:18:10.111688 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:10.111649 4908 generic.go:356] "Generic (PLEG): container finished" podID="590ad346-4bdd-4519-bfb8-adeefa86ba38" containerID="6439df20b81d3959712b21227dfdb0cfc0afb68492590eb1382a29fb774fff9d" exitCode=0 Oct 30 12:18:10.111823 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:10.111730 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66838cc1-wwvtr" event={"ID":"590ad346-4bdd-4519-bfb8-adeefa86ba38","Type":"ContainerDied","Data":"6439df20b81d3959712b21227dfdb0cfc0afb68492590eb1382a29fb774fff9d"} Oct 30 12:18:10.113197 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:10.113175 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d325dcc2-l845t" event={"ID":"4bfb80b4-66c6-42a0-90c2-9ddd522c2811","Type":"ContainerStarted","Data":"9421f133a0b5aaa2f2a83758a3d4e337e150fe845dba87d9fbdb70d54acc3842"} Oct 30 12:18:11.122130 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:11.122049 4908 generic.go:356] "Generic (PLEG): container finished" podID="4bfb80b4-66c6-42a0-90c2-9ddd522c2811" containerID="2c06cc1e52f1692f36a5dab13acb200cb1eea4f00385b5e76eabd9b49b6b93e7" exitCode=0 Oct 30 12:18:11.122494 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:11.122136 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d325dcc2-l845t" event={"ID":"4bfb80b4-66c6-42a0-90c2-9ddd522c2811","Type":"ContainerDied","Data":"2c06cc1e52f1692f36a5dab13acb200cb1eea4f00385b5e76eabd9b49b6b93e7"} Oct 30 12:18:11.268241 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:11.268217 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66838cc1-wwvtr" Oct 30 12:18:11.354501 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:11.354472 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/590ad346-4bdd-4519-bfb8-adeefa86ba38-kube-api-access-g5qxx\" (UniqueName: \"kubernetes.io/projected/590ad346-4bdd-4519-bfb8-adeefa86ba38-kube-api-access-g5qxx\") pod \"590ad346-4bdd-4519-bfb8-adeefa86ba38\" (UID: \"590ad346-4bdd-4519-bfb8-adeefa86ba38\") " Oct 30 12:18:11.354717 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:11.354560 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/590ad346-4bdd-4519-bfb8-adeefa86ba38-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/590ad346-4bdd-4519-bfb8-adeefa86ba38-maas-api-openshift-ca\") pod \"590ad346-4bdd-4519-bfb8-adeefa86ba38\" (UID: \"590ad346-4bdd-4519-bfb8-adeefa86ba38\") " Oct 30 12:18:11.354717 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:11.354619 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/590ad346-4bdd-4519-bfb8-adeefa86ba38-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/590ad346-4bdd-4519-bfb8-adeefa86ba38-maas-api-xks-ca\") pod \"590ad346-4bdd-4519-bfb8-adeefa86ba38\" (UID: \"590ad346-4bdd-4519-bfb8-adeefa86ba38\") " Oct 30 12:18:11.354948 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:11.354927 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/590ad346-4bdd-4519-bfb8-adeefa86ba38-maas-api-openshift-ca" pod "590ad346-4bdd-4519-bfb8-adeefa86ba38" (UID: "590ad346-4bdd-4519-bfb8-adeefa86ba38"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:18:11.356654 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:11.356628 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/590ad346-4bdd-4519-bfb8-adeefa86ba38-kube-api-access-g5qxx" pod "590ad346-4bdd-4519-bfb8-adeefa86ba38" (UID: "590ad346-4bdd-4519-bfb8-adeefa86ba38"). InnerVolumeSpecName "kube-api-access-g5qxx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:18:11.356654 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:11.356637 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/590ad346-4bdd-4519-bfb8-adeefa86ba38-maas-api-xks-ca" pod "590ad346-4bdd-4519-bfb8-adeefa86ba38" (UID: "590ad346-4bdd-4519-bfb8-adeefa86ba38"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:18:11.455803 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:11.455735 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g5qxx\" (UniqueName: \"kubernetes.io/projected/590ad346-4bdd-4519-bfb8-adeefa86ba38-kube-api-access-g5qxx\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:18:11.455803 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:11.455764 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/590ad346-4bdd-4519-bfb8-adeefa86ba38-maas-api-openshift-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:18:11.455803 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:11.455774 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/590ad346-4bdd-4519-bfb8-adeefa86ba38-maas-api-xks-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:18:11.485371 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:11.485340 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9wwfc"] Oct 30 12:18:12.131681 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:12.131641 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66838cc1-wwvtr" Oct 30 12:18:12.132130 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:12.131682 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66838cc1-wwvtr" event={"ID":"590ad346-4bdd-4519-bfb8-adeefa86ba38","Type":"ContainerDied","Data":"25f64d77f7f9d9a386416ebbe910a4198ea8344809fbc603ce1dea9dc030f2f9"} Oct 30 12:18:12.132130 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:12.131740 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="25f64d77f7f9d9a386416ebbe910a4198ea8344809fbc603ce1dea9dc030f2f9" Oct 30 12:18:12.345083 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:12.345059 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d325dcc2-l845t" Oct 30 12:18:12.474316 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:12.474225 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4bfb80b4-66c6-42a0-90c2-9ddd522c2811-kube-api-access-5685d\" (UniqueName: \"kubernetes.io/projected/4bfb80b4-66c6-42a0-90c2-9ddd522c2811-kube-api-access-5685d\") pod \"4bfb80b4-66c6-42a0-90c2-9ddd522c2811\" (UID: \"4bfb80b4-66c6-42a0-90c2-9ddd522c2811\") " Oct 30 12:18:12.474316 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:12.474273 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4bfb80b4-66c6-42a0-90c2-9ddd522c2811-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4bfb80b4-66c6-42a0-90c2-9ddd522c2811-maas-api-openshift-ca\") pod \"4bfb80b4-66c6-42a0-90c2-9ddd522c2811\" (UID: \"4bfb80b4-66c6-42a0-90c2-9ddd522c2811\") " Oct 30 12:18:12.474537 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:12.474428 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4bfb80b4-66c6-42a0-90c2-9ddd522c2811-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4bfb80b4-66c6-42a0-90c2-9ddd522c2811-maas-api-xks-ca\") pod \"4bfb80b4-66c6-42a0-90c2-9ddd522c2811\" (UID: \"4bfb80b4-66c6-42a0-90c2-9ddd522c2811\") " Oct 30 12:18:12.474750 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:12.474721 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4bfb80b4-66c6-42a0-90c2-9ddd522c2811-maas-api-openshift-ca" pod "4bfb80b4-66c6-42a0-90c2-9ddd522c2811" (UID: "4bfb80b4-66c6-42a0-90c2-9ddd522c2811"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:18:12.474928 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:12.474900 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4bfb80b4-66c6-42a0-90c2-9ddd522c2811-maas-api-openshift-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:18:12.476436 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:12.476408 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bfb80b4-66c6-42a0-90c2-9ddd522c2811-kube-api-access-5685d" pod "4bfb80b4-66c6-42a0-90c2-9ddd522c2811" (UID: "4bfb80b4-66c6-42a0-90c2-9ddd522c2811"). InnerVolumeSpecName "kube-api-access-5685d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:18:12.476513 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:12.476471 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4bfb80b4-66c6-42a0-90c2-9ddd522c2811-maas-api-xks-ca" pod "4bfb80b4-66c6-42a0-90c2-9ddd522c2811" (UID: "4bfb80b4-66c6-42a0-90c2-9ddd522c2811"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:18:12.576148 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:12.576109 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4bfb80b4-66c6-42a0-90c2-9ddd522c2811-maas-api-xks-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:18:12.576148 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:12.576150 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5685d\" (UniqueName: \"kubernetes.io/projected/4bfb80b4-66c6-42a0-90c2-9ddd522c2811-kube-api-access-5685d\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:18:13.140505 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:13.140474 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d325dcc2-l845t" Oct 30 12:18:13.140945 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:13.140468 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d325dcc2-l845t" event={"ID":"4bfb80b4-66c6-42a0-90c2-9ddd522c2811","Type":"ContainerDied","Data":"9421f133a0b5aaa2f2a83758a3d4e337e150fe845dba87d9fbdb70d54acc3842"} Oct 30 12:18:13.140945 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:13.140618 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9421f133a0b5aaa2f2a83758a3d4e337e150fe845dba87d9fbdb70d54acc3842" Oct 30 12:18:19.690142 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:18:19.690095 4908 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vn6vv" containerName="extractor" Oct 30 12:18:20.807424 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:20.807339 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4988034a-6w927"] Oct 30 12:18:20.808216 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:20.808188 4908 cpu_manager.go:414] "containerMap: removing container" podUID="590ad346-4bdd-4519-bfb8-adeefa86ba38" containerName="revoke-keys" Oct 30 12:18:20.808278 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:20.808216 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="590ad346-4bdd-4519-bfb8-adeefa86ba38" containerName="revoke-keys" Oct 30 12:18:20.808278 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:20.808246 4908 cpu_manager.go:414] "containerMap: removing container" podUID="4bfb80b4-66c6-42a0-90c2-9ddd522c2811" containerName="revoke-keys" Oct 30 12:18:20.808278 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:20.808253 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bfb80b4-66c6-42a0-90c2-9ddd522c2811" containerName="revoke-keys" Oct 30 12:18:20.808389 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:20.808378 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="590ad346-4bdd-4519-bfb8-adeefa86ba38" containerName="revoke-keys" Oct 30 12:18:20.808425 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:20.808392 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="4bfb80b4-66c6-42a0-90c2-9ddd522c2811" containerName="revoke-keys" Oct 30 12:18:20.812481 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:20.812461 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4988034a-6w927" Oct 30 12:18:20.829077 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:20.829047 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4988034a-6w927"] Oct 30 12:18:20.835802 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:20.835773 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 12:18:20.835802 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:20.835785 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 12:18:20.836878 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:20.836852 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 12:18:20.837034 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:20.836906 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s75hq\"" Oct 30 12:18:20.886661 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:20.886531 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/261a158f-13b4-4c82-883d-46f2f71cc6e8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4988034a-6w927\" (UID: \"261a158f-13b4-4c82-883d-46f2f71cc6e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4988034a-6w927" Oct 30 12:18:20.886661 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:20.886652 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwddd\" (UniqueName: \"kubernetes.io/projected/261a158f-13b4-4c82-883d-46f2f71cc6e8-kube-api-access-zwddd\") pod \"maas-api-revoke-sub-4988034a-6w927\" (UID: \"261a158f-13b4-4c82-883d-46f2f71cc6e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4988034a-6w927" Oct 30 12:18:20.886970 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:20.886903 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/261a158f-13b4-4c82-883d-46f2f71cc6e8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4988034a-6w927\" (UID: \"261a158f-13b4-4c82-883d-46f2f71cc6e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4988034a-6w927" Oct 30 12:18:20.988601 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:20.988556 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/261a158f-13b4-4c82-883d-46f2f71cc6e8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4988034a-6w927\" (UID: \"261a158f-13b4-4c82-883d-46f2f71cc6e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4988034a-6w927" Oct 30 12:18:20.988780 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:20.988648 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwddd\" (UniqueName: \"kubernetes.io/projected/261a158f-13b4-4c82-883d-46f2f71cc6e8-kube-api-access-zwddd\") pod \"maas-api-revoke-sub-4988034a-6w927\" (UID: \"261a158f-13b4-4c82-883d-46f2f71cc6e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4988034a-6w927" Oct 30 12:18:20.988780 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:20.988712 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/261a158f-13b4-4c82-883d-46f2f71cc6e8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4988034a-6w927\" (UID: \"261a158f-13b4-4c82-883d-46f2f71cc6e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4988034a-6w927" Oct 30 12:18:20.989245 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:20.989216 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/261a158f-13b4-4c82-883d-46f2f71cc6e8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4988034a-6w927\" (UID: \"261a158f-13b4-4c82-883d-46f2f71cc6e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4988034a-6w927" Oct 30 12:18:20.991098 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:20.991066 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/261a158f-13b4-4c82-883d-46f2f71cc6e8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4988034a-6w927\" (UID: \"261a158f-13b4-4c82-883d-46f2f71cc6e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4988034a-6w927" Oct 30 12:18:21.010999 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:21.010970 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwddd\" (UniqueName: \"kubernetes.io/projected/261a158f-13b4-4c82-883d-46f2f71cc6e8-kube-api-access-zwddd\") pod \"maas-api-revoke-sub-4988034a-6w927\" (UID: \"261a158f-13b4-4c82-883d-46f2f71cc6e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4988034a-6w927" Oct 30 12:18:21.127578 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:21.127481 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4988034a-6w927" Oct 30 12:18:21.439721 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:21.439647 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4988034a-6w927"] Oct 30 12:18:21.443912 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:18:21.443875 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod261a158f_13b4_4c82_883d_46f2f71cc6e8.slice/crio-2809745c2abd7194b28f24cd53ddbc2c6b61d4381f63689fb28abdf9b716204e WatchSource:0}: Error finding container 2809745c2abd7194b28f24cd53ddbc2c6b61d4381f63689fb28abdf9b716204e: Status 404 returned error can't find the container with id 2809745c2abd7194b28f24cd53ddbc2c6b61d4381f63689fb28abdf9b716204e Oct 30 12:18:22.217340 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:22.217228 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4988034a-6w927" event={"ID":"261a158f-13b4-4c82-883d-46f2f71cc6e8","Type":"ContainerStarted","Data":"7f50075bac1a3f4ace0e8eea604ad0390fa4a214083d11cba98917a12221c6fa"} Oct 30 12:18:22.217340 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:22.217295 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4988034a-6w927" event={"ID":"261a158f-13b4-4c82-883d-46f2f71cc6e8","Type":"ContainerStarted","Data":"2809745c2abd7194b28f24cd53ddbc2c6b61d4381f63689fb28abdf9b716204e"} Oct 30 12:18:22.258933 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:22.258752 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4988034a-6w927" podStartSLOduration=2.258717467 podStartE2EDuration="2.258717467s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:18:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:18:22.254442101 +0000 UTC m=+1458.768870847" watchObservedRunningTime="2026-10-30 12:18:22.258717467 +0000 UTC m=+1458.773146212" Oct 30 12:18:23.226283 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:23.226246 4908 generic.go:356] "Generic (PLEG): container finished" podID="261a158f-13b4-4c82-883d-46f2f71cc6e8" containerID="7f50075bac1a3f4ace0e8eea604ad0390fa4a214083d11cba98917a12221c6fa" exitCode=0 Oct 30 12:18:23.226778 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:23.226317 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4988034a-6w927" event={"ID":"261a158f-13b4-4c82-883d-46f2f71cc6e8","Type":"ContainerDied","Data":"7f50075bac1a3f4ace0e8eea604ad0390fa4a214083d11cba98917a12221c6fa"} Oct 30 12:18:24.232999 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:24.232964 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89356cf2-wzcrk"] Oct 30 12:18:24.246874 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:24.246841 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89356cf2-wzcrk" Oct 30 12:18:24.258895 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:24.256187 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89356cf2-wzcrk"] Oct 30 12:18:24.325747 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:24.325705 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2c42610-d2fb-467b-9c70-a7afc6e3047b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-89356cf2-wzcrk\" (UID: \"a2c42610-d2fb-467b-9c70-a7afc6e3047b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89356cf2-wzcrk" Oct 30 12:18:24.325929 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:24.325903 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c87d9\" (UniqueName: \"kubernetes.io/projected/a2c42610-d2fb-467b-9c70-a7afc6e3047b-kube-api-access-c87d9\") pod \"maas-api-revoke-sub-89356cf2-wzcrk\" (UID: \"a2c42610-d2fb-467b-9c70-a7afc6e3047b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89356cf2-wzcrk" Oct 30 12:18:24.326030 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:24.326013 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2c42610-d2fb-467b-9c70-a7afc6e3047b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-89356cf2-wzcrk\" (UID: \"a2c42610-d2fb-467b-9c70-a7afc6e3047b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89356cf2-wzcrk" Oct 30 12:18:24.427319 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:24.427286 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c87d9\" (UniqueName: \"kubernetes.io/projected/a2c42610-d2fb-467b-9c70-a7afc6e3047b-kube-api-access-c87d9\") pod \"maas-api-revoke-sub-89356cf2-wzcrk\" (UID: \"a2c42610-d2fb-467b-9c70-a7afc6e3047b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89356cf2-wzcrk" Oct 30 12:18:24.427488 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:24.427354 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2c42610-d2fb-467b-9c70-a7afc6e3047b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-89356cf2-wzcrk\" (UID: \"a2c42610-d2fb-467b-9c70-a7afc6e3047b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89356cf2-wzcrk" Oct 30 12:18:24.427488 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:24.427460 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2c42610-d2fb-467b-9c70-a7afc6e3047b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-89356cf2-wzcrk\" (UID: \"a2c42610-d2fb-467b-9c70-a7afc6e3047b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89356cf2-wzcrk" Oct 30 12:18:24.428188 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:24.428161 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2c42610-d2fb-467b-9c70-a7afc6e3047b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-89356cf2-wzcrk\" (UID: \"a2c42610-d2fb-467b-9c70-a7afc6e3047b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89356cf2-wzcrk" Oct 30 12:18:24.429816 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:24.429786 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2c42610-d2fb-467b-9c70-a7afc6e3047b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-89356cf2-wzcrk\" (UID: \"a2c42610-d2fb-467b-9c70-a7afc6e3047b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89356cf2-wzcrk" Oct 30 12:18:24.433394 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:24.433372 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4988034a-6w927" Oct 30 12:18:24.449416 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:24.449384 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c87d9\" (UniqueName: \"kubernetes.io/projected/a2c42610-d2fb-467b-9c70-a7afc6e3047b-kube-api-access-c87d9\") pod \"maas-api-revoke-sub-89356cf2-wzcrk\" (UID: \"a2c42610-d2fb-467b-9c70-a7afc6e3047b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89356cf2-wzcrk" Oct 30 12:18:24.515447 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:24.515355 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zvsx7"] Oct 30 12:18:24.516170 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:24.516147 4908 cpu_manager.go:414] "containerMap: removing container" podUID="261a158f-13b4-4c82-883d-46f2f71cc6e8" containerName="revoke-keys" Oct 30 12:18:24.516224 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:24.516174 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="261a158f-13b4-4c82-883d-46f2f71cc6e8" containerName="revoke-keys" Oct 30 12:18:24.516335 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:24.516322 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="261a158f-13b4-4c82-883d-46f2f71cc6e8" containerName="revoke-keys" Oct 30 12:18:24.520349 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:24.520323 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zvsx7" Oct 30 12:18:24.528368 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:24.528340 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/261a158f-13b4-4c82-883d-46f2f71cc6e8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/261a158f-13b4-4c82-883d-46f2f71cc6e8-maas-api-openshift-ca\") pod \"261a158f-13b4-4c82-883d-46f2f71cc6e8\" (UID: \"261a158f-13b4-4c82-883d-46f2f71cc6e8\") " Oct 30 12:18:24.528528 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:24.528442 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/261a158f-13b4-4c82-883d-46f2f71cc6e8-kube-api-access-zwddd\" (UniqueName: \"kubernetes.io/projected/261a158f-13b4-4c82-883d-46f2f71cc6e8-kube-api-access-zwddd\") pod \"261a158f-13b4-4c82-883d-46f2f71cc6e8\" (UID: \"261a158f-13b4-4c82-883d-46f2f71cc6e8\") " Oct 30 12:18:24.528761 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:24.528543 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/261a158f-13b4-4c82-883d-46f2f71cc6e8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/261a158f-13b4-4c82-883d-46f2f71cc6e8-maas-api-xks-ca\") pod \"261a158f-13b4-4c82-883d-46f2f71cc6e8\" (UID: \"261a158f-13b4-4c82-883d-46f2f71cc6e8\") " Oct 30 12:18:24.528761 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:24.528728 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/261a158f-13b4-4c82-883d-46f2f71cc6e8-maas-api-openshift-ca" pod "261a158f-13b4-4c82-883d-46f2f71cc6e8" (UID: "261a158f-13b4-4c82-883d-46f2f71cc6e8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:18:24.529072 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:24.529045 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/261a158f-13b4-4c82-883d-46f2f71cc6e8-maas-api-openshift-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:18:24.530712 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:24.530684 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/261a158f-13b4-4c82-883d-46f2f71cc6e8-maas-api-xks-ca" pod "261a158f-13b4-4c82-883d-46f2f71cc6e8" (UID: "261a158f-13b4-4c82-883d-46f2f71cc6e8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:18:24.530805 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:24.530721 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/261a158f-13b4-4c82-883d-46f2f71cc6e8-kube-api-access-zwddd" pod "261a158f-13b4-4c82-883d-46f2f71cc6e8" (UID: "261a158f-13b4-4c82-883d-46f2f71cc6e8"). InnerVolumeSpecName "kube-api-access-zwddd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:18:24.541015 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:24.540982 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zvsx7"] Oct 30 12:18:24.567923 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:24.567888 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89356cf2-wzcrk" Oct 30 12:18:24.633984 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:24.630852 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zstw5\" (UniqueName: \"kubernetes.io/projected/06ac4d9d-7718-4f9d-a5b0-99047a324a3c-kube-api-access-zstw5\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zvsx7\" (UID: \"06ac4d9d-7718-4f9d-a5b0-99047a324a3c\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zvsx7" Oct 30 12:18:24.633984 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:24.630927 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06ac4d9d-7718-4f9d-a5b0-99047a324a3c-credential-socket\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zvsx7\" (UID: \"06ac4d9d-7718-4f9d-a5b0-99047a324a3c\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zvsx7" Oct 30 12:18:24.633984 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:24.631035 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06ac4d9d-7718-4f9d-a5b0-99047a324a3c-istio-envoy\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zvsx7\" (UID: \"06ac4d9d-7718-4f9d-a5b0-99047a324a3c\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zvsx7" Oct 30 12:18:24.633984 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:24.631067 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06ac4d9d-7718-4f9d-a5b0-99047a324a3c-istio-token\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zvsx7\" (UID: \"06ac4d9d-7718-4f9d-a5b0-99047a324a3c\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zvsx7" Oct 30 12:18:24.633984 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:24.631121 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06ac4d9d-7718-4f9d-a5b0-99047a324a3c-workload-certs\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zvsx7\" (UID: \"06ac4d9d-7718-4f9d-a5b0-99047a324a3c\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zvsx7" Oct 30 12:18:24.633984 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:24.631175 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06ac4d9d-7718-4f9d-a5b0-99047a324a3c-istio-podinfo\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zvsx7\" (UID: \"06ac4d9d-7718-4f9d-a5b0-99047a324a3c\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zvsx7" Oct 30 12:18:24.633984 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:24.631222 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06ac4d9d-7718-4f9d-a5b0-99047a324a3c-istio-data\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zvsx7\" (UID: \"06ac4d9d-7718-4f9d-a5b0-99047a324a3c\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zvsx7" Oct 30 12:18:24.633984 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:24.631256 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06ac4d9d-7718-4f9d-a5b0-99047a324a3c-workload-socket\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zvsx7\" (UID: \"06ac4d9d-7718-4f9d-a5b0-99047a324a3c\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zvsx7" Oct 30 12:18:24.633984 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:24.631283 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06ac4d9d-7718-4f9d-a5b0-99047a324a3c-istiod-ca-cert\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zvsx7\" (UID: \"06ac4d9d-7718-4f9d-a5b0-99047a324a3c\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zvsx7" Oct 30 12:18:24.633984 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:24.631368 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwddd\" (UniqueName: \"kubernetes.io/projected/261a158f-13b4-4c82-883d-46f2f71cc6e8-kube-api-access-zwddd\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:18:24.633984 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:24.631380 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/261a158f-13b4-4c82-883d-46f2f71cc6e8-maas-api-xks-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:18:24.732368 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:24.732326 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06ac4d9d-7718-4f9d-a5b0-99047a324a3c-workload-socket\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zvsx7\" (UID: \"06ac4d9d-7718-4f9d-a5b0-99047a324a3c\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zvsx7" Oct 30 12:18:24.732368 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:24.732372 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06ac4d9d-7718-4f9d-a5b0-99047a324a3c-istiod-ca-cert\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zvsx7\" (UID: \"06ac4d9d-7718-4f9d-a5b0-99047a324a3c\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zvsx7" Oct 30 12:18:24.732620 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:24.732441 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zstw5\" (UniqueName: \"kubernetes.io/projected/06ac4d9d-7718-4f9d-a5b0-99047a324a3c-kube-api-access-zstw5\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zvsx7\" (UID: \"06ac4d9d-7718-4f9d-a5b0-99047a324a3c\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zvsx7" Oct 30 12:18:24.732620 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:24.732496 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06ac4d9d-7718-4f9d-a5b0-99047a324a3c-credential-socket\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zvsx7\" (UID: \"06ac4d9d-7718-4f9d-a5b0-99047a324a3c\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zvsx7" Oct 30 12:18:24.732620 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:24.732522 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06ac4d9d-7718-4f9d-a5b0-99047a324a3c-istio-envoy\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zvsx7\" (UID: \"06ac4d9d-7718-4f9d-a5b0-99047a324a3c\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zvsx7" Oct 30 12:18:24.732620 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:24.732550 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06ac4d9d-7718-4f9d-a5b0-99047a324a3c-istio-token\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zvsx7\" (UID: \"06ac4d9d-7718-4f9d-a5b0-99047a324a3c\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zvsx7" Oct 30 12:18:24.732620 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:24.732585 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06ac4d9d-7718-4f9d-a5b0-99047a324a3c-workload-certs\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zvsx7\" (UID: \"06ac4d9d-7718-4f9d-a5b0-99047a324a3c\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zvsx7" Oct 30 12:18:24.732903 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:24.732657 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06ac4d9d-7718-4f9d-a5b0-99047a324a3c-istio-podinfo\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zvsx7\" (UID: \"06ac4d9d-7718-4f9d-a5b0-99047a324a3c\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zvsx7" Oct 30 12:18:24.732903 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:24.732737 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06ac4d9d-7718-4f9d-a5b0-99047a324a3c-istio-data\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zvsx7\" (UID: \"06ac4d9d-7718-4f9d-a5b0-99047a324a3c\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zvsx7" Oct 30 12:18:24.732903 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:24.732762 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06ac4d9d-7718-4f9d-a5b0-99047a324a3c-workload-socket\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zvsx7\" (UID: \"06ac4d9d-7718-4f9d-a5b0-99047a324a3c\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zvsx7" Oct 30 12:18:24.733046 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:24.732931 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06ac4d9d-7718-4f9d-a5b0-99047a324a3c-credential-socket\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zvsx7\" (UID: \"06ac4d9d-7718-4f9d-a5b0-99047a324a3c\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zvsx7" Oct 30 12:18:24.733101 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:24.733079 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06ac4d9d-7718-4f9d-a5b0-99047a324a3c-istio-data\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zvsx7\" (UID: \"06ac4d9d-7718-4f9d-a5b0-99047a324a3c\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zvsx7" Oct 30 12:18:24.733171 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:24.733143 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06ac4d9d-7718-4f9d-a5b0-99047a324a3c-workload-certs\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zvsx7\" (UID: \"06ac4d9d-7718-4f9d-a5b0-99047a324a3c\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zvsx7" Oct 30 12:18:24.733300 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:24.733207 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06ac4d9d-7718-4f9d-a5b0-99047a324a3c-istiod-ca-cert\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zvsx7\" (UID: \"06ac4d9d-7718-4f9d-a5b0-99047a324a3c\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zvsx7" Oct 30 12:18:24.735023 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:24.735001 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06ac4d9d-7718-4f9d-a5b0-99047a324a3c-istio-envoy\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zvsx7\" (UID: \"06ac4d9d-7718-4f9d-a5b0-99047a324a3c\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zvsx7" Oct 30 12:18:24.735150 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:24.735135 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06ac4d9d-7718-4f9d-a5b0-99047a324a3c-istio-podinfo\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zvsx7\" (UID: \"06ac4d9d-7718-4f9d-a5b0-99047a324a3c\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zvsx7" Oct 30 12:18:24.754899 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:24.754817 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06ac4d9d-7718-4f9d-a5b0-99047a324a3c-istio-token\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zvsx7\" (UID: \"06ac4d9d-7718-4f9d-a5b0-99047a324a3c\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zvsx7" Oct 30 12:18:24.755033 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:24.754949 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zstw5\" (UniqueName: \"kubernetes.io/projected/06ac4d9d-7718-4f9d-a5b0-99047a324a3c-kube-api-access-zstw5\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zvsx7\" (UID: \"06ac4d9d-7718-4f9d-a5b0-99047a324a3c\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zvsx7" Oct 30 12:18:24.836239 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:24.836161 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zvsx7" Oct 30 12:18:24.871676 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:24.871650 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89356cf2-wzcrk"] Oct 30 12:18:24.874899 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:18:24.874865 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2c42610_d2fb_467b_9c70_a7afc6e3047b.slice/crio-be4effdb4b0cced8dde10d93428a75532b61c9eb0cc7e34d126a2771919b47ad WatchSource:0}: Error finding container be4effdb4b0cced8dde10d93428a75532b61c9eb0cc7e34d126a2771919b47ad: Status 404 returned error can't find the container with id be4effdb4b0cced8dde10d93428a75532b61c9eb0cc7e34d126a2771919b47ad Oct 30 12:18:25.170392 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:18:25.170247 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06ac4d9d_7718_4f9d_a5b0_99047a324a3c.slice/crio-bf928e7e5d64fbb2f220adfb0118b55e78aa3786871480eaded6bdb6500d2ece WatchSource:0}: Error finding container bf928e7e5d64fbb2f220adfb0118b55e78aa3786871480eaded6bdb6500d2ece: Status 404 returned error can't find the container with id bf928e7e5d64fbb2f220adfb0118b55e78aa3786871480eaded6bdb6500d2ece Oct 30 12:18:25.173249 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:25.173217 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zvsx7"] Oct 30 12:18:25.173706 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:25.173673 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:18:25.173793 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:25.173738 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:18:25.173793 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:25.173769 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:18:25.245193 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:25.245159 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89356cf2-wzcrk" event={"ID":"a2c42610-d2fb-467b-9c70-a7afc6e3047b","Type":"ContainerStarted","Data":"95a5994b61fc34f72c35f50c4385b0222fcb8847253b68a61d85da07f7c408eb"} Oct 30 12:18:25.245529 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:25.245222 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89356cf2-wzcrk" event={"ID":"a2c42610-d2fb-467b-9c70-a7afc6e3047b","Type":"ContainerStarted","Data":"be4effdb4b0cced8dde10d93428a75532b61c9eb0cc7e34d126a2771919b47ad"} Oct 30 12:18:25.246929 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:25.246797 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4988034a-6w927" Oct 30 12:18:25.246929 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:25.246816 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4988034a-6w927" event={"ID":"261a158f-13b4-4c82-883d-46f2f71cc6e8","Type":"ContainerDied","Data":"2809745c2abd7194b28f24cd53ddbc2c6b61d4381f63689fb28abdf9b716204e"} Oct 30 12:18:25.246929 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:25.246877 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2809745c2abd7194b28f24cd53ddbc2c6b61d4381f63689fb28abdf9b716204e" Oct 30 12:18:25.248752 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:25.248725 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zvsx7" event={"ID":"06ac4d9d-7718-4f9d-a5b0-99047a324a3c","Type":"ContainerStarted","Data":"bf928e7e5d64fbb2f220adfb0118b55e78aa3786871480eaded6bdb6500d2ece"} Oct 30 12:18:25.286632 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:25.286562 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89356cf2-wzcrk" podStartSLOduration=1.286542101 podStartE2EDuration="1.286542101s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:18:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:18:25.282927811 +0000 UTC m=+1461.797356597" watchObservedRunningTime="2026-10-30 12:18:25.286542101 +0000 UTC m=+1461.800970849" Oct 30 12:18:25.351176 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:25.351121 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zvsx7" podStartSLOduration=1.3511049769999999 podStartE2EDuration="1.351104977s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:18:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:18:25.348190457 +0000 UTC m=+1461.862619219" watchObservedRunningTime="2026-10-30 12:18:25.351104977 +0000 UTC m=+1461.865533793" Oct 30 12:18:25.836966 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:25.836917 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zvsx7" Oct 30 12:18:26.257703 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:26.257664 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zvsx7" event={"ID":"06ac4d9d-7718-4f9d-a5b0-99047a324a3c","Type":"ContainerStarted","Data":"b3332897c0fc28be62327823a761e65ea6d776c719ba72bdfc4b5a4975da7baa"} Oct 30 12:18:26.263850 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:26.259540 4908 generic.go:356] "Generic (PLEG): container finished" podID="a2c42610-d2fb-467b-9c70-a7afc6e3047b" containerID="95a5994b61fc34f72c35f50c4385b0222fcb8847253b68a61d85da07f7c408eb" exitCode=0 Oct 30 12:18:26.263850 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:26.259691 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89356cf2-wzcrk" event={"ID":"a2c42610-d2fb-467b-9c70-a7afc6e3047b","Type":"ContainerDied","Data":"95a5994b61fc34f72c35f50c4385b0222fcb8847253b68a61d85da07f7c408eb"} Oct 30 12:18:26.838450 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:26.838412 4908 patch_prober.go:29] interesting pod/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zvsx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 30 12:18:26.838640 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:26.838470 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zvsx7" podUID="06ac4d9d-7718-4f9d-a5b0-99047a324a3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 30 12:18:27.539037 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:27.538971 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89356cf2-wzcrk" Oct 30 12:18:27.700140 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:27.700108 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2c42610-d2fb-467b-9c70-a7afc6e3047b-kube-api-access-c87d9\" (UniqueName: \"kubernetes.io/projected/a2c42610-d2fb-467b-9c70-a7afc6e3047b-kube-api-access-c87d9\") pod \"a2c42610-d2fb-467b-9c70-a7afc6e3047b\" (UID: \"a2c42610-d2fb-467b-9c70-a7afc6e3047b\") " Oct 30 12:18:27.700673 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:27.700646 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a2c42610-d2fb-467b-9c70-a7afc6e3047b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2c42610-d2fb-467b-9c70-a7afc6e3047b-maas-api-xks-ca\") pod \"a2c42610-d2fb-467b-9c70-a7afc6e3047b\" (UID: \"a2c42610-d2fb-467b-9c70-a7afc6e3047b\") " Oct 30 12:18:27.700912 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:27.700890 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2c42610-d2fb-467b-9c70-a7afc6e3047b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2c42610-d2fb-467b-9c70-a7afc6e3047b-maas-api-openshift-ca\") pod \"a2c42610-d2fb-467b-9c70-a7afc6e3047b\" (UID: \"a2c42610-d2fb-467b-9c70-a7afc6e3047b\") " Oct 30 12:18:27.701428 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:27.701386 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2c42610-d2fb-467b-9c70-a7afc6e3047b-maas-api-openshift-ca" pod "a2c42610-d2fb-467b-9c70-a7afc6e3047b" (UID: "a2c42610-d2fb-467b-9c70-a7afc6e3047b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:18:27.702204 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:27.701840 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2c42610-d2fb-467b-9c70-a7afc6e3047b-maas-api-openshift-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:18:27.704423 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:27.704399 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2c42610-d2fb-467b-9c70-a7afc6e3047b-kube-api-access-c87d9" pod "a2c42610-d2fb-467b-9c70-a7afc6e3047b" (UID: "a2c42610-d2fb-467b-9c70-a7afc6e3047b"). InnerVolumeSpecName "kube-api-access-c87d9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:18:27.704516 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:27.704441 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2c42610-d2fb-467b-9c70-a7afc6e3047b-maas-api-xks-ca" pod "a2c42610-d2fb-467b-9c70-a7afc6e3047b" (UID: "a2c42610-d2fb-467b-9c70-a7afc6e3047b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:18:27.802688 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:27.802587 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c87d9\" (UniqueName: \"kubernetes.io/projected/a2c42610-d2fb-467b-9c70-a7afc6e3047b-kube-api-access-c87d9\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:18:27.802688 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:27.802628 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2c42610-d2fb-467b-9c70-a7afc6e3047b-maas-api-xks-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:18:27.837187 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:27.837150 4908 patch_prober.go:29] interesting pod/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zvsx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 30 12:18:27.837376 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:27.837222 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zvsx7" podUID="06ac4d9d-7718-4f9d-a5b0-99047a324a3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 30 12:18:28.297088 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:28.297051 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89356cf2-wzcrk" event={"ID":"a2c42610-d2fb-467b-9c70-a7afc6e3047b","Type":"ContainerDied","Data":"be4effdb4b0cced8dde10d93428a75532b61c9eb0cc7e34d126a2771919b47ad"} Oct 30 12:18:28.297244 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:28.297101 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="be4effdb4b0cced8dde10d93428a75532b61c9eb0cc7e34d126a2771919b47ad" Oct 30 12:18:28.297244 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:28.297105 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89356cf2-wzcrk" Oct 30 12:18:28.837142 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:28.837103 4908 patch_prober.go:29] interesting pod/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zvsx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 30 12:18:28.837660 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:28.837164 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zvsx7" podUID="06ac4d9d-7718-4f9d-a5b0-99047a324a3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 30 12:18:29.840320 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:29.840277 4908 patch_prober.go:29] interesting pod/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zvsx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 30 12:18:29.840998 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:29.840959 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zvsx7" podUID="06ac4d9d-7718-4f9d-a5b0-99047a324a3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 30 12:18:30.837398 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:30.837358 4908 patch_prober.go:29] interesting pod/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zvsx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 30 12:18:30.837701 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:30.837668 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zvsx7" podUID="06ac4d9d-7718-4f9d-a5b0-99047a324a3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 30 12:18:31.348993 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:31.348932 4908 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-512ec5/simulator-w4-512ec5-kserve-7df7674f8-8rqwb" podUID="8a5dfcc9-dec3-4bc6-b0a8-c928350b3596" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:18:31.550935 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:31.546069 4908 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f920f7/simulator-w0-f920f7-kserve-599b7ff45f-g79d7" podUID="a85e562f-d655-4c22-b4f4-a299a4e494a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 12:18:31.655909 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:31.654805 4908 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47fd99/premium-w2-47fd99-kserve-5d85f9bb95-hqkmf" podUID="fd2cbeda-dbc4-41f6-a730-e1c6f9343c5e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:18:31.778850 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:31.772913 4908 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6a7148/premium-w3-6a7148-kserve-67b976c888-tgd6c" podUID="13de4441-e7f6-4cbd-97d9-0f079d11cb8e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:18:31.934483 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:31.934378 4908 patch_prober.go:29] interesting pod/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zvsx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 30 12:18:31.935172 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:31.935137 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zvsx7" podUID="06ac4d9d-7718-4f9d-a5b0-99047a324a3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 30 12:18:32.217556 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:32.214986 4908 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-f920f7/simulator-w0-f920f7-kserve-599b7ff45f-g79d7" podUID="a85e562f-d655-4c22-b4f4-a299a4e494a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:18:32.568153 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:32.568105 4908 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f920f7/simulator-w0-f920f7-kserve-599b7ff45f-g79d7" podUID="a85e562f-d655-4c22-b4f4-a299a4e494a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Oct 30 12:18:32.569504 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:32.569370 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f920f7/simulator-w0-f920f7-kserve-599b7ff45f-g79d7" Oct 30 12:18:32.635021 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:32.634991 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f920f7/simulator-w0-f920f7-kserve-599b7ff45f-g79d7" Oct 30 12:18:32.655090 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:32.655046 4908 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47fd99/premium-w2-47fd99-kserve-5d85f9bb95-hqkmf" podUID="fd2cbeda-dbc4-41f6-a730-e1c6f9343c5e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:18:32.655981 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:32.655958 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-47fd99/premium-w2-47fd99-kserve-5d85f9bb95-hqkmf" Oct 30 12:18:32.678010 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:32.677981 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-47fd99/premium-w2-47fd99-kserve-5d85f9bb95-hqkmf" Oct 30 12:18:32.837472 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:32.837380 4908 patch_prober.go:29] interesting pod/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zvsx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 30 12:18:32.837472 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:32.837446 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zvsx7" podUID="06ac4d9d-7718-4f9d-a5b0-99047a324a3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 30 12:18:34.065597 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:34.065561 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zvsx7" Oct 30 12:18:34.066174 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:34.066152 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zvsx7" Oct 30 12:18:34.066944 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:34.066921 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zvsx7" Oct 30 12:18:34.445162 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:34.445077 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a08284d4-jrgdb"] Oct 30 12:18:34.445561 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:34.445548 4908 cpu_manager.go:414] "containerMap: removing container" podUID="a2c42610-d2fb-467b-9c70-a7afc6e3047b" containerName="revoke-keys" Oct 30 12:18:34.445608 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:34.445563 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2c42610-d2fb-467b-9c70-a7afc6e3047b" containerName="revoke-keys" Oct 30 12:18:34.445684 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:34.445675 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2c42610-d2fb-467b-9c70-a7afc6e3047b" containerName="revoke-keys" Oct 30 12:18:34.450199 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:34.450178 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a08284d4-jrgdb" Oct 30 12:18:34.464302 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:34.464245 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 12:18:34.464761 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:34.464715 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 12:18:34.466536 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:34.466516 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 12:18:34.466927 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:34.466909 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s75hq\"" Oct 30 12:18:34.469447 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:34.469420 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a08284d4-jrgdb"] Oct 30 12:18:34.531348 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:34.531313 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnnsv\" (UniqueName: \"kubernetes.io/projected/bec895b3-a116-43e4-8465-735a56c63b51-kube-api-access-vnnsv\") pod \"maas-api-revoke-sub-a08284d4-jrgdb\" (UID: \"bec895b3-a116-43e4-8465-735a56c63b51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a08284d4-jrgdb" Oct 30 12:18:34.531524 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:34.531363 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bec895b3-a116-43e4-8465-735a56c63b51-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a08284d4-jrgdb\" (UID: \"bec895b3-a116-43e4-8465-735a56c63b51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a08284d4-jrgdb" Oct 30 12:18:34.531524 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:34.531501 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bec895b3-a116-43e4-8465-735a56c63b51-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a08284d4-jrgdb\" (UID: \"bec895b3-a116-43e4-8465-735a56c63b51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a08284d4-jrgdb" Oct 30 12:18:34.632777 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:34.632740 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bec895b3-a116-43e4-8465-735a56c63b51-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a08284d4-jrgdb\" (UID: \"bec895b3-a116-43e4-8465-735a56c63b51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a08284d4-jrgdb" Oct 30 12:18:34.633023 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:34.632876 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnnsv\" (UniqueName: \"kubernetes.io/projected/bec895b3-a116-43e4-8465-735a56c63b51-kube-api-access-vnnsv\") pod \"maas-api-revoke-sub-a08284d4-jrgdb\" (UID: \"bec895b3-a116-43e4-8465-735a56c63b51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a08284d4-jrgdb" Oct 30 12:18:34.633023 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:34.632918 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bec895b3-a116-43e4-8465-735a56c63b51-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a08284d4-jrgdb\" (UID: \"bec895b3-a116-43e4-8465-735a56c63b51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a08284d4-jrgdb" Oct 30 12:18:34.633487 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:34.633460 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bec895b3-a116-43e4-8465-735a56c63b51-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a08284d4-jrgdb\" (UID: \"bec895b3-a116-43e4-8465-735a56c63b51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a08284d4-jrgdb" Oct 30 12:18:34.635472 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:34.635452 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bec895b3-a116-43e4-8465-735a56c63b51-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a08284d4-jrgdb\" (UID: \"bec895b3-a116-43e4-8465-735a56c63b51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a08284d4-jrgdb" Oct 30 12:18:34.657816 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:34.657784 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnnsv\" (UniqueName: \"kubernetes.io/projected/bec895b3-a116-43e4-8465-735a56c63b51-kube-api-access-vnnsv\") pod \"maas-api-revoke-sub-a08284d4-jrgdb\" (UID: \"bec895b3-a116-43e4-8465-735a56c63b51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a08284d4-jrgdb" Oct 30 12:18:34.771823 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:34.771515 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a08284d4-jrgdb" Oct 30 12:18:35.092254 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:35.092228 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a08284d4-jrgdb"] Oct 30 12:18:35.627372 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:35.627326 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a08284d4-jrgdb" event={"ID":"bec895b3-a116-43e4-8465-735a56c63b51","Type":"ContainerStarted","Data":"868234781b4bc2ab281698b7268bb2addf2ff4f61df2054615cb08783be6850f"} Oct 30 12:18:35.627582 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:35.627393 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a08284d4-jrgdb" event={"ID":"bec895b3-a116-43e4-8465-735a56c63b51","Type":"ContainerStarted","Data":"115123d8eb687dfa11b4ee00335412d11437884b462d4273888ef29c4afb0119"} Oct 30 12:18:35.666593 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:35.666538 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a08284d4-jrgdb" podStartSLOduration=1.666523429 podStartE2EDuration="1.666523429s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:18:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:18:35.664712355 +0000 UTC m=+1472.179141104" watchObservedRunningTime="2026-10-30 12:18:35.666523429 +0000 UTC m=+1472.180952203" Oct 30 12:18:36.636815 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:36.636769 4908 generic.go:356] "Generic (PLEG): container finished" podID="bec895b3-a116-43e4-8465-735a56c63b51" containerID="868234781b4bc2ab281698b7268bb2addf2ff4f61df2054615cb08783be6850f" exitCode=0 Oct 30 12:18:36.637345 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:36.636860 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a08284d4-jrgdb" event={"ID":"bec895b3-a116-43e4-8465-735a56c63b51","Type":"ContainerDied","Data":"868234781b4bc2ab281698b7268bb2addf2ff4f61df2054615cb08783be6850f"} Oct 30 12:18:37.807753 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:37.807727 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a08284d4-jrgdb" Oct 30 12:18:37.869922 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:37.869888 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bec895b3-a116-43e4-8465-735a56c63b51-kube-api-access-vnnsv\" (UniqueName: \"kubernetes.io/projected/bec895b3-a116-43e4-8465-735a56c63b51-kube-api-access-vnnsv\") pod \"bec895b3-a116-43e4-8465-735a56c63b51\" (UID: \"bec895b3-a116-43e4-8465-735a56c63b51\") " Oct 30 12:18:37.870104 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:37.869977 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bec895b3-a116-43e4-8465-735a56c63b51-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bec895b3-a116-43e4-8465-735a56c63b51-maas-api-openshift-ca\") pod \"bec895b3-a116-43e4-8465-735a56c63b51\" (UID: \"bec895b3-a116-43e4-8465-735a56c63b51\") " Oct 30 12:18:37.870104 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:37.870026 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bec895b3-a116-43e4-8465-735a56c63b51-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bec895b3-a116-43e4-8465-735a56c63b51-maas-api-xks-ca\") pod \"bec895b3-a116-43e4-8465-735a56c63b51\" (UID: \"bec895b3-a116-43e4-8465-735a56c63b51\") " Oct 30 12:18:37.870403 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:37.870375 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bec895b3-a116-43e4-8465-735a56c63b51-maas-api-openshift-ca" pod "bec895b3-a116-43e4-8465-735a56c63b51" (UID: "bec895b3-a116-43e4-8465-735a56c63b51"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:18:37.872166 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:37.872133 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bec895b3-a116-43e4-8465-735a56c63b51-maas-api-xks-ca" pod "bec895b3-a116-43e4-8465-735a56c63b51" (UID: "bec895b3-a116-43e4-8465-735a56c63b51"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:18:37.872553 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:37.872535 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bec895b3-a116-43e4-8465-735a56c63b51-kube-api-access-vnnsv" pod "bec895b3-a116-43e4-8465-735a56c63b51" (UID: "bec895b3-a116-43e4-8465-735a56c63b51"). InnerVolumeSpecName "kube-api-access-vnnsv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:18:37.971850 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:37.971751 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bec895b3-a116-43e4-8465-735a56c63b51-maas-api-openshift-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:18:37.971850 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:37.971788 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bec895b3-a116-43e4-8465-735a56c63b51-maas-api-xks-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:18:37.971850 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:37.971802 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vnnsv\" (UniqueName: \"kubernetes.io/projected/bec895b3-a116-43e4-8465-735a56c63b51-kube-api-access-vnnsv\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:18:38.027384 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:38.027346 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0452683a-mwvss"] Oct 30 12:18:38.027878 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:38.027862 4908 cpu_manager.go:414] "containerMap: removing container" podUID="bec895b3-a116-43e4-8465-735a56c63b51" containerName="revoke-keys" Oct 30 12:18:38.027878 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:38.027878 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bec895b3-a116-43e4-8465-735a56c63b51" containerName="revoke-keys" Oct 30 12:18:38.028036 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:38.027971 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="bec895b3-a116-43e4-8465-735a56c63b51" containerName="revoke-keys" Oct 30 12:18:38.031538 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:38.031507 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0452683a-mwvss" Oct 30 12:18:38.049013 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:38.048984 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0452683a-mwvss"] Oct 30 12:18:38.073045 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:38.073006 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e059d9e-06a0-4f7b-9b54-faa87582e184-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0452683a-mwvss\" (UID: \"4e059d9e-06a0-4f7b-9b54-faa87582e184\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0452683a-mwvss" Oct 30 12:18:38.073219 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:38.073149 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e059d9e-06a0-4f7b-9b54-faa87582e184-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0452683a-mwvss\" (UID: \"4e059d9e-06a0-4f7b-9b54-faa87582e184\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0452683a-mwvss" Oct 30 12:18:38.073297 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:38.073240 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdhsk\" (UniqueName: \"kubernetes.io/projected/4e059d9e-06a0-4f7b-9b54-faa87582e184-kube-api-access-bdhsk\") pod \"maas-api-revoke-sub-0452683a-mwvss\" (UID: \"4e059d9e-06a0-4f7b-9b54-faa87582e184\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0452683a-mwvss" Oct 30 12:18:38.175011 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:38.174968 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e059d9e-06a0-4f7b-9b54-faa87582e184-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0452683a-mwvss\" (UID: \"4e059d9e-06a0-4f7b-9b54-faa87582e184\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0452683a-mwvss" Oct 30 12:18:38.175197 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:38.175054 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e059d9e-06a0-4f7b-9b54-faa87582e184-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0452683a-mwvss\" (UID: \"4e059d9e-06a0-4f7b-9b54-faa87582e184\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0452683a-mwvss" Oct 30 12:18:38.175197 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:38.175117 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdhsk\" (UniqueName: \"kubernetes.io/projected/4e059d9e-06a0-4f7b-9b54-faa87582e184-kube-api-access-bdhsk\") pod \"maas-api-revoke-sub-0452683a-mwvss\" (UID: \"4e059d9e-06a0-4f7b-9b54-faa87582e184\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0452683a-mwvss" Oct 30 12:18:38.175798 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:38.175772 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e059d9e-06a0-4f7b-9b54-faa87582e184-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0452683a-mwvss\" (UID: \"4e059d9e-06a0-4f7b-9b54-faa87582e184\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0452683a-mwvss" Oct 30 12:18:38.177452 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:38.177418 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e059d9e-06a0-4f7b-9b54-faa87582e184-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0452683a-mwvss\" (UID: \"4e059d9e-06a0-4f7b-9b54-faa87582e184\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0452683a-mwvss" Oct 30 12:18:38.197390 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:38.197362 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdhsk\" (UniqueName: \"kubernetes.io/projected/4e059d9e-06a0-4f7b-9b54-faa87582e184-kube-api-access-bdhsk\") pod \"maas-api-revoke-sub-0452683a-mwvss\" (UID: \"4e059d9e-06a0-4f7b-9b54-faa87582e184\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0452683a-mwvss" Oct 30 12:18:38.357283 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:38.357250 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0452683a-mwvss" Oct 30 12:18:38.655577 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:38.655478 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a08284d4-jrgdb" Oct 30 12:18:38.655577 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:38.655489 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a08284d4-jrgdb" event={"ID":"bec895b3-a116-43e4-8465-735a56c63b51","Type":"ContainerDied","Data":"115123d8eb687dfa11b4ee00335412d11437884b462d4273888ef29c4afb0119"} Oct 30 12:18:38.655577 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:38.655541 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="115123d8eb687dfa11b4ee00335412d11437884b462d4273888ef29c4afb0119" Oct 30 12:18:38.667943 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:38.667915 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0452683a-mwvss"] Oct 30 12:18:38.668277 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:18:38.668253 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e059d9e_06a0_4f7b_9b54_faa87582e184.slice/crio-df85d37bf63eacde1e159072f26954e72a75c74e219c1deb201ac9a81ef7c414 WatchSource:0}: Error finding container df85d37bf63eacde1e159072f26954e72a75c74e219c1deb201ac9a81ef7c414: Status 404 returned error can't find the container with id df85d37bf63eacde1e159072f26954e72a75c74e219c1deb201ac9a81ef7c414 Oct 30 12:18:39.571485 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:39.571448 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-j8kfd"] Oct 30 12:18:39.575624 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:39.575593 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-j8kfd" Oct 30 12:18:39.609520 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:39.609484 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-j8kfd"] Oct 30 12:18:39.664753 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:39.664669 4908 generic.go:356] "Generic (PLEG): container finished" podID="4e059d9e-06a0-4f7b-9b54-faa87582e184" containerID="a997ecd54d51424760c9b9401b2efe6c09f44bd4377b72bbd5f09adadd20b9ee" exitCode=0 Oct 30 12:18:39.665058 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:39.664757 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0452683a-mwvss" event={"ID":"4e059d9e-06a0-4f7b-9b54-faa87582e184","Type":"ContainerDied","Data":"a997ecd54d51424760c9b9401b2efe6c09f44bd4377b72bbd5f09adadd20b9ee"} Oct 30 12:18:39.665058 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:39.664820 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0452683a-mwvss" event={"ID":"4e059d9e-06a0-4f7b-9b54-faa87582e184","Type":"ContainerStarted","Data":"df85d37bf63eacde1e159072f26954e72a75c74e219c1deb201ac9a81ef7c414"} Oct 30 12:18:39.698451 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:39.698414 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2068d5c3-6f62-4bd1-bbf1-20a12a84725c-workload-certs\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-j8kfd\" (UID: \"2068d5c3-6f62-4bd1-bbf1-20a12a84725c\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-j8kfd" Oct 30 12:18:39.698660 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:39.698467 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2068d5c3-6f62-4bd1-bbf1-20a12a84725c-istio-envoy\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-j8kfd\" (UID: \"2068d5c3-6f62-4bd1-bbf1-20a12a84725c\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-j8kfd" Oct 30 12:18:39.698660 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:39.698570 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2068d5c3-6f62-4bd1-bbf1-20a12a84725c-istiod-ca-cert\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-j8kfd\" (UID: \"2068d5c3-6f62-4bd1-bbf1-20a12a84725c\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-j8kfd" Oct 30 12:18:39.698660 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:39.698621 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2068d5c3-6f62-4bd1-bbf1-20a12a84725c-istio-data\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-j8kfd\" (UID: \"2068d5c3-6f62-4bd1-bbf1-20a12a84725c\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-j8kfd" Oct 30 12:18:39.698660 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:39.698656 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2068d5c3-6f62-4bd1-bbf1-20a12a84725c-credential-socket\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-j8kfd\" (UID: \"2068d5c3-6f62-4bd1-bbf1-20a12a84725c\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-j8kfd" Oct 30 12:18:39.698858 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:39.698718 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2068d5c3-6f62-4bd1-bbf1-20a12a84725c-istio-podinfo\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-j8kfd\" (UID: \"2068d5c3-6f62-4bd1-bbf1-20a12a84725c\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-j8kfd" Oct 30 12:18:39.698980 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:39.698951 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tb68\" (UniqueName: \"kubernetes.io/projected/2068d5c3-6f62-4bd1-bbf1-20a12a84725c-kube-api-access-6tb68\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-j8kfd\" (UID: \"2068d5c3-6f62-4bd1-bbf1-20a12a84725c\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-j8kfd" Oct 30 12:18:39.699127 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:39.699014 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2068d5c3-6f62-4bd1-bbf1-20a12a84725c-workload-socket\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-j8kfd\" (UID: \"2068d5c3-6f62-4bd1-bbf1-20a12a84725c\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-j8kfd" Oct 30 12:18:39.699191 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:39.699136 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2068d5c3-6f62-4bd1-bbf1-20a12a84725c-istio-token\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-j8kfd\" (UID: \"2068d5c3-6f62-4bd1-bbf1-20a12a84725c\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-j8kfd" Oct 30 12:18:39.710528 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:39.710501 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-779a2a29-xbdqx"] Oct 30 12:18:39.714662 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:39.714641 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-779a2a29-xbdqx" Oct 30 12:18:39.733141 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:39.733065 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-779a2a29-xbdqx"] Oct 30 12:18:39.800156 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:39.800115 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2068d5c3-6f62-4bd1-bbf1-20a12a84725c-workload-socket\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-j8kfd\" (UID: \"2068d5c3-6f62-4bd1-bbf1-20a12a84725c\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-j8kfd" Oct 30 12:18:39.800337 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:39.800276 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2068d5c3-6f62-4bd1-bbf1-20a12a84725c-istio-token\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-j8kfd\" (UID: \"2068d5c3-6f62-4bd1-bbf1-20a12a84725c\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-j8kfd" Oct 30 12:18:39.800380 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:39.800335 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2068d5c3-6f62-4bd1-bbf1-20a12a84725c-workload-certs\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-j8kfd\" (UID: \"2068d5c3-6f62-4bd1-bbf1-20a12a84725c\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-j8kfd" Oct 30 12:18:39.800380 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:39.800362 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99920224-1d67-4af4-90a0-ebd548e4f618-maas-api-xks-ca\") pod \"maas-api-revoke-sub-779a2a29-xbdqx\" (UID: \"99920224-1d67-4af4-90a0-ebd548e4f618\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-779a2a29-xbdqx" Oct 30 12:18:39.800463 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:39.800396 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2068d5c3-6f62-4bd1-bbf1-20a12a84725c-istio-envoy\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-j8kfd\" (UID: \"2068d5c3-6f62-4bd1-bbf1-20a12a84725c\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-j8kfd" Oct 30 12:18:39.800463 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:39.800419 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2068d5c3-6f62-4bd1-bbf1-20a12a84725c-istiod-ca-cert\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-j8kfd\" (UID: \"2068d5c3-6f62-4bd1-bbf1-20a12a84725c\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-j8kfd" Oct 30 12:18:39.800463 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:39.800444 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99920224-1d67-4af4-90a0-ebd548e4f618-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-779a2a29-xbdqx\" (UID: \"99920224-1d67-4af4-90a0-ebd548e4f618\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-779a2a29-xbdqx" Oct 30 12:18:39.800607 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:39.800543 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2068d5c3-6f62-4bd1-bbf1-20a12a84725c-workload-socket\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-j8kfd\" (UID: \"2068d5c3-6f62-4bd1-bbf1-20a12a84725c\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-j8kfd" Oct 30 12:18:39.800692 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:39.800658 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2068d5c3-6f62-4bd1-bbf1-20a12a84725c-istio-data\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-j8kfd\" (UID: \"2068d5c3-6f62-4bd1-bbf1-20a12a84725c\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-j8kfd" Oct 30 12:18:39.800745 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:39.800701 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2068d5c3-6f62-4bd1-bbf1-20a12a84725c-credential-socket\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-j8kfd\" (UID: \"2068d5c3-6f62-4bd1-bbf1-20a12a84725c\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-j8kfd" Oct 30 12:18:39.800855 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:39.800801 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2068d5c3-6f62-4bd1-bbf1-20a12a84725c-workload-certs\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-j8kfd\" (UID: \"2068d5c3-6f62-4bd1-bbf1-20a12a84725c\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-j8kfd" Oct 30 12:18:39.800916 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:39.800848 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgpm5\" (UniqueName: \"kubernetes.io/projected/99920224-1d67-4af4-90a0-ebd548e4f618-kube-api-access-xgpm5\") pod \"maas-api-revoke-sub-779a2a29-xbdqx\" (UID: \"99920224-1d67-4af4-90a0-ebd548e4f618\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-779a2a29-xbdqx" Oct 30 12:18:39.800976 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:39.800952 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2068d5c3-6f62-4bd1-bbf1-20a12a84725c-istio-podinfo\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-j8kfd\" (UID: \"2068d5c3-6f62-4bd1-bbf1-20a12a84725c\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-j8kfd" Oct 30 12:18:39.801114 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:39.800986 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2068d5c3-6f62-4bd1-bbf1-20a12a84725c-credential-socket\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-j8kfd\" (UID: \"2068d5c3-6f62-4bd1-bbf1-20a12a84725c\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-j8kfd" Oct 30 12:18:39.801114 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:39.801056 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2068d5c3-6f62-4bd1-bbf1-20a12a84725c-istio-data\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-j8kfd\" (UID: \"2068d5c3-6f62-4bd1-bbf1-20a12a84725c\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-j8kfd" Oct 30 12:18:39.801183 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:39.801164 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tb68\" (UniqueName: \"kubernetes.io/projected/2068d5c3-6f62-4bd1-bbf1-20a12a84725c-kube-api-access-6tb68\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-j8kfd\" (UID: \"2068d5c3-6f62-4bd1-bbf1-20a12a84725c\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-j8kfd" Oct 30 12:18:39.801241 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:39.801214 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2068d5c3-6f62-4bd1-bbf1-20a12a84725c-istiod-ca-cert\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-j8kfd\" (UID: \"2068d5c3-6f62-4bd1-bbf1-20a12a84725c\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-j8kfd" Oct 30 12:18:39.802894 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:39.802866 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2068d5c3-6f62-4bd1-bbf1-20a12a84725c-istio-envoy\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-j8kfd\" (UID: \"2068d5c3-6f62-4bd1-bbf1-20a12a84725c\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-j8kfd" Oct 30 12:18:39.803244 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:39.803226 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2068d5c3-6f62-4bd1-bbf1-20a12a84725c-istio-podinfo\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-j8kfd\" (UID: \"2068d5c3-6f62-4bd1-bbf1-20a12a84725c\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-j8kfd" Oct 30 12:18:39.823230 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:39.823199 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2068d5c3-6f62-4bd1-bbf1-20a12a84725c-istio-token\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-j8kfd\" (UID: \"2068d5c3-6f62-4bd1-bbf1-20a12a84725c\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-j8kfd" Oct 30 12:18:39.823521 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:39.823496 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tb68\" (UniqueName: \"kubernetes.io/projected/2068d5c3-6f62-4bd1-bbf1-20a12a84725c-kube-api-access-6tb68\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-j8kfd\" (UID: \"2068d5c3-6f62-4bd1-bbf1-20a12a84725c\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-j8kfd" Oct 30 12:18:39.891384 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:39.891332 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-j8kfd" Oct 30 12:18:39.902155 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:39.902122 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99920224-1d67-4af4-90a0-ebd548e4f618-maas-api-xks-ca\") pod \"maas-api-revoke-sub-779a2a29-xbdqx\" (UID: \"99920224-1d67-4af4-90a0-ebd548e4f618\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-779a2a29-xbdqx" Oct 30 12:18:39.902299 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:39.902178 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99920224-1d67-4af4-90a0-ebd548e4f618-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-779a2a29-xbdqx\" (UID: \"99920224-1d67-4af4-90a0-ebd548e4f618\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-779a2a29-xbdqx" Oct 30 12:18:39.902299 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:39.902213 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgpm5\" (UniqueName: \"kubernetes.io/projected/99920224-1d67-4af4-90a0-ebd548e4f618-kube-api-access-xgpm5\") pod \"maas-api-revoke-sub-779a2a29-xbdqx\" (UID: \"99920224-1d67-4af4-90a0-ebd548e4f618\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-779a2a29-xbdqx" Oct 30 12:18:39.903063 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:39.903036 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99920224-1d67-4af4-90a0-ebd548e4f618-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-779a2a29-xbdqx\" (UID: \"99920224-1d67-4af4-90a0-ebd548e4f618\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-779a2a29-xbdqx" Oct 30 12:18:39.904560 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:39.904537 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99920224-1d67-4af4-90a0-ebd548e4f618-maas-api-xks-ca\") pod \"maas-api-revoke-sub-779a2a29-xbdqx\" (UID: \"99920224-1d67-4af4-90a0-ebd548e4f618\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-779a2a29-xbdqx" Oct 30 12:18:39.925016 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:39.924985 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgpm5\" (UniqueName: \"kubernetes.io/projected/99920224-1d67-4af4-90a0-ebd548e4f618-kube-api-access-xgpm5\") pod \"maas-api-revoke-sub-779a2a29-xbdqx\" (UID: \"99920224-1d67-4af4-90a0-ebd548e4f618\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-779a2a29-xbdqx" Oct 30 12:18:40.030693 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:40.030593 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-779a2a29-xbdqx" Oct 30 12:18:40.227283 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:40.227253 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-j8kfd"] Oct 30 12:18:40.227389 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:18:40.227354 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2068d5c3_6f62_4bd1_bbf1_20a12a84725c.slice/crio-d8ccd10036614d2b5ed3d939aacf9d051f9daf01e2eb1e50b51b034551664818 WatchSource:0}: Error finding container d8ccd10036614d2b5ed3d939aacf9d051f9daf01e2eb1e50b51b034551664818: Status 404 returned error can't find the container with id d8ccd10036614d2b5ed3d939aacf9d051f9daf01e2eb1e50b51b034551664818 Oct 30 12:18:40.230035 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:40.230004 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:18:40.230120 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:40.230071 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:18:40.230120 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:40.230108 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:18:40.340931 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:40.339709 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-779a2a29-xbdqx"] Oct 30 12:18:40.344789 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:18:40.344760 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99920224_1d67_4af4_90a0_ebd548e4f618.slice/crio-5f2b58f7211d5ef136a0a84fede165873549459ea41956a8451126dd089ad826 WatchSource:0}: Error finding container 5f2b58f7211d5ef136a0a84fede165873549459ea41956a8451126dd089ad826: Status 404 returned error can't find the container with id 5f2b58f7211d5ef136a0a84fede165873549459ea41956a8451126dd089ad826 Oct 30 12:18:40.676444 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:40.676404 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-779a2a29-xbdqx" event={"ID":"99920224-1d67-4af4-90a0-ebd548e4f618","Type":"ContainerStarted","Data":"5f2b58f7211d5ef136a0a84fede165873549459ea41956a8451126dd089ad826"} Oct 30 12:18:40.678711 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:40.678669 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-j8kfd" event={"ID":"2068d5c3-6f62-4bd1-bbf1-20a12a84725c","Type":"ContainerStarted","Data":"24e4b5280446741f81b6df501d4d1842ba6b2f8b6709fb1c93dc223276414cb8"} Oct 30 12:18:40.679248 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:40.679124 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-j8kfd" event={"ID":"2068d5c3-6f62-4bd1-bbf1-20a12a84725c","Type":"ContainerStarted","Data":"d8ccd10036614d2b5ed3d939aacf9d051f9daf01e2eb1e50b51b034551664818"} Oct 30 12:18:40.695571 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:40.695539 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-6p2qs"] Oct 30 12:18:40.699866 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:40.699840 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-6p2qs" Oct 30 12:18:40.727841 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:40.727791 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-6p2qs"] Oct 30 12:18:40.740740 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:40.740321 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-j8kfd" podStartSLOduration=1.7398224359999999 podStartE2EDuration="1.739822436s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:18:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:18:40.730322437 +0000 UTC m=+1477.244751186" watchObservedRunningTime="2026-10-30 12:18:40.739822436 +0000 UTC m=+1477.254251184" Oct 30 12:18:40.824789 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:40.824748 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d8040e9-df95-42d1-8870-603107c66560-istio-data\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-6p2qs\" (UID: \"1d8040e9-df95-42d1-8870-603107c66560\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-6p2qs" Oct 30 12:18:40.824974 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:40.824869 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d8040e9-df95-42d1-8870-603107c66560-istio-token\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-6p2qs\" (UID: \"1d8040e9-df95-42d1-8870-603107c66560\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-6p2qs" Oct 30 12:18:40.824974 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:40.824942 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d8040e9-df95-42d1-8870-603107c66560-workload-certs\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-6p2qs\" (UID: \"1d8040e9-df95-42d1-8870-603107c66560\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-6p2qs" Oct 30 12:18:40.825105 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:40.825038 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d8040e9-df95-42d1-8870-603107c66560-workload-socket\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-6p2qs\" (UID: \"1d8040e9-df95-42d1-8870-603107c66560\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-6p2qs" Oct 30 12:18:40.825105 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:40.825063 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d8040e9-df95-42d1-8870-603107c66560-istio-podinfo\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-6p2qs\" (UID: \"1d8040e9-df95-42d1-8870-603107c66560\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-6p2qs" Oct 30 12:18:40.825211 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:40.825136 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d8040e9-df95-42d1-8870-603107c66560-istio-envoy\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-6p2qs\" (UID: \"1d8040e9-df95-42d1-8870-603107c66560\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-6p2qs" Oct 30 12:18:40.825211 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:40.825186 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pgsj\" (UniqueName: \"kubernetes.io/projected/1d8040e9-df95-42d1-8870-603107c66560-kube-api-access-8pgsj\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-6p2qs\" (UID: \"1d8040e9-df95-42d1-8870-603107c66560\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-6p2qs" Oct 30 12:18:40.825321 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:40.825235 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d8040e9-df95-42d1-8870-603107c66560-credential-socket\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-6p2qs\" (UID: \"1d8040e9-df95-42d1-8870-603107c66560\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-6p2qs" Oct 30 12:18:40.825321 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:40.825293 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d8040e9-df95-42d1-8870-603107c66560-istiod-ca-cert\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-6p2qs\" (UID: \"1d8040e9-df95-42d1-8870-603107c66560\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-6p2qs" Oct 30 12:18:40.904633 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:40.902996 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-j8kfd" Oct 30 12:18:40.926787 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:40.926667 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d8040e9-df95-42d1-8870-603107c66560-istio-envoy\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-6p2qs\" (UID: \"1d8040e9-df95-42d1-8870-603107c66560\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-6p2qs" Oct 30 12:18:40.926787 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:40.926740 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pgsj\" (UniqueName: \"kubernetes.io/projected/1d8040e9-df95-42d1-8870-603107c66560-kube-api-access-8pgsj\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-6p2qs\" (UID: \"1d8040e9-df95-42d1-8870-603107c66560\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-6p2qs" Oct 30 12:18:40.926787 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:40.926780 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d8040e9-df95-42d1-8870-603107c66560-credential-socket\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-6p2qs\" (UID: \"1d8040e9-df95-42d1-8870-603107c66560\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-6p2qs" Oct 30 12:18:40.926973 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:40.926865 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d8040e9-df95-42d1-8870-603107c66560-istiod-ca-cert\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-6p2qs\" (UID: \"1d8040e9-df95-42d1-8870-603107c66560\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-6p2qs" Oct 30 12:18:40.926973 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:40.926924 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d8040e9-df95-42d1-8870-603107c66560-istio-data\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-6p2qs\" (UID: \"1d8040e9-df95-42d1-8870-603107c66560\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-6p2qs" Oct 30 12:18:40.926973 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:40.926951 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d8040e9-df95-42d1-8870-603107c66560-istio-token\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-6p2qs\" (UID: \"1d8040e9-df95-42d1-8870-603107c66560\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-6p2qs" Oct 30 12:18:40.927070 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:40.927012 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d8040e9-df95-42d1-8870-603107c66560-workload-certs\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-6p2qs\" (UID: \"1d8040e9-df95-42d1-8870-603107c66560\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-6p2qs" Oct 30 12:18:40.927109 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:40.927087 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d8040e9-df95-42d1-8870-603107c66560-workload-socket\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-6p2qs\" (UID: \"1d8040e9-df95-42d1-8870-603107c66560\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-6p2qs" Oct 30 12:18:40.927161 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:40.927113 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d8040e9-df95-42d1-8870-603107c66560-istio-podinfo\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-6p2qs\" (UID: \"1d8040e9-df95-42d1-8870-603107c66560\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-6p2qs" Oct 30 12:18:40.927562 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:40.927485 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d8040e9-df95-42d1-8870-603107c66560-istio-data\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-6p2qs\" (UID: \"1d8040e9-df95-42d1-8870-603107c66560\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-6p2qs" Oct 30 12:18:40.927869 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:40.927797 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d8040e9-df95-42d1-8870-603107c66560-workload-certs\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-6p2qs\" (UID: \"1d8040e9-df95-42d1-8870-603107c66560\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-6p2qs" Oct 30 12:18:40.927964 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:40.927823 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d8040e9-df95-42d1-8870-603107c66560-credential-socket\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-6p2qs\" (UID: \"1d8040e9-df95-42d1-8870-603107c66560\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-6p2qs" Oct 30 12:18:40.928139 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:40.928114 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d8040e9-df95-42d1-8870-603107c66560-istiod-ca-cert\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-6p2qs\" (UID: \"1d8040e9-df95-42d1-8870-603107c66560\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-6p2qs" Oct 30 12:18:40.929083 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:40.929060 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d8040e9-df95-42d1-8870-603107c66560-workload-socket\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-6p2qs\" (UID: \"1d8040e9-df95-42d1-8870-603107c66560\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-6p2qs" Oct 30 12:18:40.929790 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:40.929768 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d8040e9-df95-42d1-8870-603107c66560-istio-envoy\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-6p2qs\" (UID: \"1d8040e9-df95-42d1-8870-603107c66560\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-6p2qs" Oct 30 12:18:40.930994 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:40.930976 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d8040e9-df95-42d1-8870-603107c66560-istio-podinfo\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-6p2qs\" (UID: \"1d8040e9-df95-42d1-8870-603107c66560\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-6p2qs" Oct 30 12:18:40.949611 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:40.949574 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d8040e9-df95-42d1-8870-603107c66560-istio-token\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-6p2qs\" (UID: \"1d8040e9-df95-42d1-8870-603107c66560\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-6p2qs" Oct 30 12:18:40.949764 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:40.949634 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pgsj\" (UniqueName: \"kubernetes.io/projected/1d8040e9-df95-42d1-8870-603107c66560-kube-api-access-8pgsj\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-6p2qs\" (UID: \"1d8040e9-df95-42d1-8870-603107c66560\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-6p2qs" Oct 30 12:18:40.964208 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:40.964165 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0452683a-mwvss" Oct 30 12:18:41.027787 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:41.027736 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e059d9e-06a0-4f7b-9b54-faa87582e184-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e059d9e-06a0-4f7b-9b54-faa87582e184-maas-api-xks-ca\") pod \"4e059d9e-06a0-4f7b-9b54-faa87582e184\" (UID: \"4e059d9e-06a0-4f7b-9b54-faa87582e184\") " Oct 30 12:18:41.028007 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:41.027805 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e059d9e-06a0-4f7b-9b54-faa87582e184-kube-api-access-bdhsk\" (UniqueName: \"kubernetes.io/projected/4e059d9e-06a0-4f7b-9b54-faa87582e184-kube-api-access-bdhsk\") pod \"4e059d9e-06a0-4f7b-9b54-faa87582e184\" (UID: \"4e059d9e-06a0-4f7b-9b54-faa87582e184\") " Oct 30 12:18:41.028007 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:41.027922 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e059d9e-06a0-4f7b-9b54-faa87582e184-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e059d9e-06a0-4f7b-9b54-faa87582e184-maas-api-openshift-ca\") pod \"4e059d9e-06a0-4f7b-9b54-faa87582e184\" (UID: \"4e059d9e-06a0-4f7b-9b54-faa87582e184\") " Oct 30 12:18:41.028340 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:41.028310 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e059d9e-06a0-4f7b-9b54-faa87582e184-maas-api-openshift-ca" pod "4e059d9e-06a0-4f7b-9b54-faa87582e184" (UID: "4e059d9e-06a0-4f7b-9b54-faa87582e184"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:18:41.029139 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:41.029115 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e059d9e-06a0-4f7b-9b54-faa87582e184-maas-api-openshift-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:18:41.031047 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:41.031022 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e059d9e-06a0-4f7b-9b54-faa87582e184-maas-api-xks-ca" pod "4e059d9e-06a0-4f7b-9b54-faa87582e184" (UID: "4e059d9e-06a0-4f7b-9b54-faa87582e184"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:18:41.031607 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:41.031582 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e059d9e-06a0-4f7b-9b54-faa87582e184-kube-api-access-bdhsk" pod "4e059d9e-06a0-4f7b-9b54-faa87582e184" (UID: "4e059d9e-06a0-4f7b-9b54-faa87582e184"). InnerVolumeSpecName "kube-api-access-bdhsk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:18:41.040958 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:41.040934 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-6p2qs" Oct 30 12:18:41.131867 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:41.131410 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e059d9e-06a0-4f7b-9b54-faa87582e184-maas-api-xks-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:18:41.131867 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:41.131439 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bdhsk\" (UniqueName: \"kubernetes.io/projected/4e059d9e-06a0-4f7b-9b54-faa87582e184-kube-api-access-bdhsk\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:18:41.399062 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:41.399031 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-6p2qs"] Oct 30 12:18:41.402259 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:18:41.402225 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d8040e9_df95_42d1_8870_603107c66560.slice/crio-4c07cb9fcd515061bb4fd58156fe7aa0daabca6705a1b4212471617cd3aad768 WatchSource:0}: Error finding container 4c07cb9fcd515061bb4fd58156fe7aa0daabca6705a1b4212471617cd3aad768: Status 404 returned error can't find the container with id 4c07cb9fcd515061bb4fd58156fe7aa0daabca6705a1b4212471617cd3aad768 Oct 30 12:18:41.405082 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:41.405043 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:18:41.405164 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:41.405126 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:18:41.405204 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:41.405170 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:18:41.716886 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:41.716681 4908 generic.go:356] "Generic (PLEG): container finished" podID="99920224-1d67-4af4-90a0-ebd548e4f618" containerID="4d7d2a67f0ecf7f482747df45b6ba4027a052def3d7f387574b2f04505d348e3" exitCode=0 Oct 30 12:18:41.717612 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:41.717080 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-779a2a29-xbdqx" event={"ID":"99920224-1d67-4af4-90a0-ebd548e4f618","Type":"ContainerDied","Data":"4d7d2a67f0ecf7f482747df45b6ba4027a052def3d7f387574b2f04505d348e3"} Oct 30 12:18:41.721872 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:41.721582 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-6p2qs" event={"ID":"1d8040e9-df95-42d1-8870-603107c66560","Type":"ContainerStarted","Data":"f0bc49434d657b8c2135dd865a0755c97d42fd1eff1fd8def32a3324fc0a45d0"} Oct 30 12:18:41.721872 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:41.721691 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-6p2qs" event={"ID":"1d8040e9-df95-42d1-8870-603107c66560","Type":"ContainerStarted","Data":"4c07cb9fcd515061bb4fd58156fe7aa0daabca6705a1b4212471617cd3aad768"} Oct 30 12:18:41.726878 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:41.726419 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0452683a-mwvss" Oct 30 12:18:41.726878 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:41.726412 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0452683a-mwvss" event={"ID":"4e059d9e-06a0-4f7b-9b54-faa87582e184","Type":"ContainerDied","Data":"df85d37bf63eacde1e159072f26954e72a75c74e219c1deb201ac9a81ef7c414"} Oct 30 12:18:41.726878 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:41.726593 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="df85d37bf63eacde1e159072f26954e72a75c74e219c1deb201ac9a81ef7c414" Oct 30 12:18:41.817964 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:41.817894 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-6p2qs" podStartSLOduration=1.8178732210000002 podStartE2EDuration="1.817873221s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:18:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:18:41.815572138 +0000 UTC m=+1478.330000885" watchObservedRunningTime="2026-10-30 12:18:41.817873221 +0000 UTC m=+1478.332301968" Oct 30 12:18:41.898989 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:41.898948 4908 patch_prober.go:29] interesting pod/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-j8kfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Oct 30 12:18:41.899170 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:41.899014 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-j8kfd" podUID="2068d5c3-6f62-4bd1-bbf1-20a12a84725c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Oct 30 12:18:42.041312 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:42.041277 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-6p2qs" Oct 30 12:18:42.043019 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:42.042996 4908 patch_prober.go:29] interesting pod/e2e-worker-w0-f920f7-openshift-default-7c9767c97-6p2qs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Oct 30 12:18:42.043141 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:42.043050 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-6p2qs" podUID="1d8040e9-df95-42d1-8870-603107c66560" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Oct 30 12:18:42.892849 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:42.892243 4908 patch_prober.go:29] interesting pod/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-j8kfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Oct 30 12:18:42.892849 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:42.892309 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-j8kfd" podUID="2068d5c3-6f62-4bd1-bbf1-20a12a84725c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Oct 30 12:18:43.042386 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:43.042301 4908 patch_prober.go:29] interesting pod/e2e-worker-w0-f920f7-openshift-default-7c9767c97-6p2qs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Oct 30 12:18:43.042386 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:43.042352 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-6p2qs" podUID="1d8040e9-df95-42d1-8870-603107c66560" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Oct 30 12:18:43.145149 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:43.145124 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-779a2a29-xbdqx" Oct 30 12:18:43.203140 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:43.203060 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99920224-1d67-4af4-90a0-ebd548e4f618-maas-api-openshift-ca" pod "99920224-1d67-4af4-90a0-ebd548e4f618" (UID: "99920224-1d67-4af4-90a0-ebd548e4f618"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:18:43.204815 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:43.203377 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99920224-1d67-4af4-90a0-ebd548e4f618-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99920224-1d67-4af4-90a0-ebd548e4f618-maas-api-openshift-ca\") pod \"99920224-1d67-4af4-90a0-ebd548e4f618\" (UID: \"99920224-1d67-4af4-90a0-ebd548e4f618\") " Oct 30 12:18:43.204815 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:43.203476 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99920224-1d67-4af4-90a0-ebd548e4f618-kube-api-access-xgpm5\" (UniqueName: \"kubernetes.io/projected/99920224-1d67-4af4-90a0-ebd548e4f618-kube-api-access-xgpm5\") pod \"99920224-1d67-4af4-90a0-ebd548e4f618\" (UID: \"99920224-1d67-4af4-90a0-ebd548e4f618\") " Oct 30 12:18:43.204815 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:43.203510 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/99920224-1d67-4af4-90a0-ebd548e4f618-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99920224-1d67-4af4-90a0-ebd548e4f618-maas-api-xks-ca\") pod \"99920224-1d67-4af4-90a0-ebd548e4f618\" (UID: \"99920224-1d67-4af4-90a0-ebd548e4f618\") " Oct 30 12:18:43.204815 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:43.204118 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99920224-1d67-4af4-90a0-ebd548e4f618-maas-api-openshift-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:18:43.230411 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:43.230366 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99920224-1d67-4af4-90a0-ebd548e4f618-maas-api-xks-ca" pod "99920224-1d67-4af4-90a0-ebd548e4f618" (UID: "99920224-1d67-4af4-90a0-ebd548e4f618"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:18:43.247810 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:43.247756 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99920224-1d67-4af4-90a0-ebd548e4f618-kube-api-access-xgpm5" pod "99920224-1d67-4af4-90a0-ebd548e4f618" (UID: "99920224-1d67-4af4-90a0-ebd548e4f618"). InnerVolumeSpecName "kube-api-access-xgpm5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:18:43.322932 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:43.322759 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgpm5\" (UniqueName: \"kubernetes.io/projected/99920224-1d67-4af4-90a0-ebd548e4f618-kube-api-access-xgpm5\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:18:43.322932 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:43.322809 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99920224-1d67-4af4-90a0-ebd548e4f618-maas-api-xks-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:18:43.786522 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:43.786488 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-779a2a29-xbdqx" Oct 30 12:18:43.789964 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:43.789933 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-779a2a29-xbdqx" event={"ID":"99920224-1d67-4af4-90a0-ebd548e4f618","Type":"ContainerDied","Data":"5f2b58f7211d5ef136a0a84fede165873549459ea41956a8451126dd089ad826"} Oct 30 12:18:43.796847 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:43.796803 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5f2b58f7211d5ef136a0a84fede165873549459ea41956a8451126dd089ad826" Oct 30 12:18:43.892511 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:43.892402 4908 patch_prober.go:29] interesting pod/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-j8kfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Oct 30 12:18:43.892814 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:43.892763 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-j8kfd" podUID="2068d5c3-6f62-4bd1-bbf1-20a12a84725c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Oct 30 12:18:44.041548 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:44.041515 4908 patch_prober.go:29] interesting pod/e2e-worker-w0-f920f7-openshift-default-7c9767c97-6p2qs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Oct 30 12:18:44.042002 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:44.041571 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-6p2qs" podUID="1d8040e9-df95-42d1-8870-603107c66560" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Oct 30 12:18:44.836898 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:44.836820 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20f2699e-ljxv4"] Oct 30 12:18:44.837708 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:44.837682 4908 cpu_manager.go:414] "containerMap: removing container" podUID="99920224-1d67-4af4-90a0-ebd548e4f618" containerName="revoke-keys" Oct 30 12:18:44.837894 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:44.837878 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99920224-1d67-4af4-90a0-ebd548e4f618" containerName="revoke-keys" Oct 30 12:18:44.838020 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:44.838007 4908 cpu_manager.go:414] "containerMap: removing container" podUID="4e059d9e-06a0-4f7b-9b54-faa87582e184" containerName="revoke-keys" Oct 30 12:18:44.838109 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:44.838098 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e059d9e-06a0-4f7b-9b54-faa87582e184" containerName="revoke-keys" Oct 30 12:18:44.838397 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:44.838383 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e059d9e-06a0-4f7b-9b54-faa87582e184" containerName="revoke-keys" Oct 30 12:18:44.838491 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:44.838484 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="99920224-1d67-4af4-90a0-ebd548e4f618" containerName="revoke-keys" Oct 30 12:18:44.841956 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:44.841941 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20f2699e-ljxv4" Oct 30 12:18:44.848758 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:44.848725 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20f2699e-ljxv4"] Oct 30 12:18:44.875106 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:44.875071 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s75hq\"" Oct 30 12:18:44.875384 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:44.875369 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 12:18:44.875498 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:44.875487 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 12:18:44.875681 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:44.875665 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 12:18:44.916858 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:44.911956 4908 patch_prober.go:29] interesting pod/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-j8kfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Oct 30 12:18:44.916858 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:44.912022 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-j8kfd" podUID="2068d5c3-6f62-4bd1-bbf1-20a12a84725c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Oct 30 12:18:44.950539 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:44.950453 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1122f4cb-4c3f-41e6-b9fb-c2079a9e208c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-20f2699e-ljxv4\" (UID: \"1122f4cb-4c3f-41e6-b9fb-c2079a9e208c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20f2699e-ljxv4" Oct 30 12:18:44.950866 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:44.950838 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmmk5\" (UniqueName: \"kubernetes.io/projected/1122f4cb-4c3f-41e6-b9fb-c2079a9e208c-kube-api-access-vmmk5\") pod \"maas-api-revoke-sub-20f2699e-ljxv4\" (UID: \"1122f4cb-4c3f-41e6-b9fb-c2079a9e208c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20f2699e-ljxv4" Oct 30 12:18:44.951052 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:44.951037 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1122f4cb-4c3f-41e6-b9fb-c2079a9e208c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-20f2699e-ljxv4\" (UID: \"1122f4cb-4c3f-41e6-b9fb-c2079a9e208c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20f2699e-ljxv4" Oct 30 12:18:45.042028 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:45.041998 4908 patch_prober.go:29] interesting pod/e2e-worker-w0-f920f7-openshift-default-7c9767c97-6p2qs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Oct 30 12:18:45.042517 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:45.042491 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-6p2qs" podUID="1d8040e9-df95-42d1-8870-603107c66560" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Oct 30 12:18:45.052812 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:45.052776 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1122f4cb-4c3f-41e6-b9fb-c2079a9e208c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-20f2699e-ljxv4\" (UID: \"1122f4cb-4c3f-41e6-b9fb-c2079a9e208c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20f2699e-ljxv4" Oct 30 12:18:45.052970 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:45.052871 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmmk5\" (UniqueName: \"kubernetes.io/projected/1122f4cb-4c3f-41e6-b9fb-c2079a9e208c-kube-api-access-vmmk5\") pod \"maas-api-revoke-sub-20f2699e-ljxv4\" (UID: \"1122f4cb-4c3f-41e6-b9fb-c2079a9e208c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20f2699e-ljxv4" Oct 30 12:18:45.052970 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:45.052962 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1122f4cb-4c3f-41e6-b9fb-c2079a9e208c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-20f2699e-ljxv4\" (UID: \"1122f4cb-4c3f-41e6-b9fb-c2079a9e208c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20f2699e-ljxv4" Oct 30 12:18:45.053717 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:45.053692 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1122f4cb-4c3f-41e6-b9fb-c2079a9e208c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-20f2699e-ljxv4\" (UID: \"1122f4cb-4c3f-41e6-b9fb-c2079a9e208c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20f2699e-ljxv4" Oct 30 12:18:45.055227 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:45.055207 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1122f4cb-4c3f-41e6-b9fb-c2079a9e208c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-20f2699e-ljxv4\" (UID: \"1122f4cb-4c3f-41e6-b9fb-c2079a9e208c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20f2699e-ljxv4" Oct 30 12:18:45.089971 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:45.089888 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmmk5\" (UniqueName: \"kubernetes.io/projected/1122f4cb-4c3f-41e6-b9fb-c2079a9e208c-kube-api-access-vmmk5\") pod \"maas-api-revoke-sub-20f2699e-ljxv4\" (UID: \"1122f4cb-4c3f-41e6-b9fb-c2079a9e208c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20f2699e-ljxv4" Oct 30 12:18:45.162857 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:45.161007 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20f2699e-ljxv4" Oct 30 12:18:45.648329 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:45.648302 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20f2699e-ljxv4"] Oct 30 12:18:45.850912 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:45.850102 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20f2699e-ljxv4" event={"ID":"1122f4cb-4c3f-41e6-b9fb-c2079a9e208c","Type":"ContainerStarted","Data":"f8072b8ca0399d3b1b8bbc9dde0618e3a0e9cbc8ac3127b42c5cc8fee183ebdb"} Oct 30 12:18:45.892928 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:45.892270 4908 patch_prober.go:29] interesting pod/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-j8kfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Oct 30 12:18:45.892928 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:45.892345 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-j8kfd" podUID="2068d5c3-6f62-4bd1-bbf1-20a12a84725c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Oct 30 12:18:46.041671 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:46.041519 4908 patch_prober.go:29] interesting pod/e2e-worker-w0-f920f7-openshift-default-7c9767c97-6p2qs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Oct 30 12:18:46.041671 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:46.041571 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-6p2qs" podUID="1d8040e9-df95-42d1-8870-603107c66560" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Oct 30 12:18:46.905860 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:46.892232 4908 patch_prober.go:29] interesting pod/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-j8kfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Oct 30 12:18:46.905860 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:46.892300 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-j8kfd" podUID="2068d5c3-6f62-4bd1-bbf1-20a12a84725c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Oct 30 12:18:46.909243 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:46.909136 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20f2699e-ljxv4" event={"ID":"1122f4cb-4c3f-41e6-b9fb-c2079a9e208c","Type":"ContainerStarted","Data":"d54656eee5ee5e4a34afa9ea69ecabd4b0dddc1e3c276194982eb98bda987926"} Oct 30 12:18:46.912626 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:46.912598 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9wwfc"] Oct 30 12:18:46.952551 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:46.952493 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20f2699e-ljxv4" podStartSLOduration=2.952477668 podStartE2EDuration="2.952477668s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:18:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:18:46.950665994 +0000 UTC m=+1483.465094760" watchObservedRunningTime="2026-10-30 12:18:46.952477668 +0000 UTC m=+1483.466906413" Oct 30 12:18:47.042009 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:47.041954 4908 patch_prober.go:29] interesting pod/e2e-worker-w0-f920f7-openshift-default-7c9767c97-6p2qs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Oct 30 12:18:47.042206 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:47.042042 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-6p2qs" podUID="1d8040e9-df95-42d1-8870-603107c66560" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Oct 30 12:18:47.934578 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:47.934450 4908 generic.go:356] "Generic (PLEG): container finished" podID="1122f4cb-4c3f-41e6-b9fb-c2079a9e208c" containerID="d54656eee5ee5e4a34afa9ea69ecabd4b0dddc1e3c276194982eb98bda987926" exitCode=0 Oct 30 12:18:47.934578 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:47.934503 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20f2699e-ljxv4" event={"ID":"1122f4cb-4c3f-41e6-b9fb-c2079a9e208c","Type":"ContainerDied","Data":"d54656eee5ee5e4a34afa9ea69ecabd4b0dddc1e3c276194982eb98bda987926"} Oct 30 12:18:48.042498 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:48.042459 4908 patch_prober.go:29] interesting pod/e2e-worker-w0-f920f7-openshift-default-7c9767c97-6p2qs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Oct 30 12:18:48.042792 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:48.042760 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-6p2qs" podUID="1d8040e9-df95-42d1-8870-603107c66560" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Oct 30 12:18:48.463420 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:48.463381 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-j8kfd" Oct 30 12:18:48.464711 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:48.464684 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-j8kfd" Oct 30 12:18:48.466387 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:48.466368 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-j8kfd" Oct 30 12:18:48.574853 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:48.574622 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de045378-pcjbr"] Oct 30 12:18:48.594853 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:48.590333 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de045378-pcjbr" Oct 30 12:18:48.679553 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:48.679499 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3054913-cac1-435a-ac9a-1b06e9434c35-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-de045378-pcjbr\" (UID: \"b3054913-cac1-435a-ac9a-1b06e9434c35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de045378-pcjbr" Oct 30 12:18:48.683570 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:48.683072 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3054913-cac1-435a-ac9a-1b06e9434c35-maas-api-xks-ca\") pod \"maas-api-revoke-sub-de045378-pcjbr\" (UID: \"b3054913-cac1-435a-ac9a-1b06e9434c35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de045378-pcjbr" Oct 30 12:18:48.728914 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:48.726856 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdnb2\" (UniqueName: \"kubernetes.io/projected/b3054913-cac1-435a-ac9a-1b06e9434c35-kube-api-access-hdnb2\") pod \"maas-api-revoke-sub-de045378-pcjbr\" (UID: \"b3054913-cac1-435a-ac9a-1b06e9434c35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de045378-pcjbr" Oct 30 12:18:48.740066 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:48.740017 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de045378-pcjbr"] Oct 30 12:18:48.879882 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:48.875985 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3054913-cac1-435a-ac9a-1b06e9434c35-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-de045378-pcjbr\" (UID: \"b3054913-cac1-435a-ac9a-1b06e9434c35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de045378-pcjbr" Oct 30 12:18:48.904250 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:48.904197 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3054913-cac1-435a-ac9a-1b06e9434c35-maas-api-xks-ca\") pod \"maas-api-revoke-sub-de045378-pcjbr\" (UID: \"b3054913-cac1-435a-ac9a-1b06e9434c35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de045378-pcjbr" Oct 30 12:18:48.904735 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:48.904708 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdnb2\" (UniqueName: \"kubernetes.io/projected/b3054913-cac1-435a-ac9a-1b06e9434c35-kube-api-access-hdnb2\") pod \"maas-api-revoke-sub-de045378-pcjbr\" (UID: \"b3054913-cac1-435a-ac9a-1b06e9434c35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de045378-pcjbr" Oct 30 12:18:48.925854 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:48.921016 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3054913-cac1-435a-ac9a-1b06e9434c35-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-de045378-pcjbr\" (UID: \"b3054913-cac1-435a-ac9a-1b06e9434c35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de045378-pcjbr" Oct 30 12:18:48.949422 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:48.949382 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3054913-cac1-435a-ac9a-1b06e9434c35-maas-api-xks-ca\") pod \"maas-api-revoke-sub-de045378-pcjbr\" (UID: \"b3054913-cac1-435a-ac9a-1b06e9434c35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de045378-pcjbr" Oct 30 12:18:48.961870 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:48.961641 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdnb2\" (UniqueName: \"kubernetes.io/projected/b3054913-cac1-435a-ac9a-1b06e9434c35-kube-api-access-hdnb2\") pod \"maas-api-revoke-sub-de045378-pcjbr\" (UID: \"b3054913-cac1-435a-ac9a-1b06e9434c35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de045378-pcjbr" Oct 30 12:18:49.168047 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:49.167950 4908 patch_prober.go:29] interesting pod/e2e-worker-w0-f920f7-openshift-default-7c9767c97-6p2qs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Oct 30 12:18:49.168341 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:49.168300 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-6p2qs" podUID="1d8040e9-df95-42d1-8870-603107c66560" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Oct 30 12:18:49.199986 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:49.199937 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rx4tf"] Oct 30 12:18:49.218851 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:49.205927 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rx4tf" Oct 30 12:18:49.254796 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:49.254588 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76c80954-caf1-42b7-a812-fa52488f2e1b-workload-socket\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rx4tf\" (UID: \"76c80954-caf1-42b7-a812-fa52488f2e1b\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rx4tf" Oct 30 12:18:49.254796 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:49.254662 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76c80954-caf1-42b7-a812-fa52488f2e1b-credential-socket\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rx4tf\" (UID: \"76c80954-caf1-42b7-a812-fa52488f2e1b\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rx4tf" Oct 30 12:18:49.254796 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:49.254788 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76c80954-caf1-42b7-a812-fa52488f2e1b-workload-certs\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rx4tf\" (UID: \"76c80954-caf1-42b7-a812-fa52488f2e1b\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rx4tf" Oct 30 12:18:49.256540 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:49.256497 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de045378-pcjbr" Oct 30 12:18:49.309379 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:49.309348 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rx4tf"] Oct 30 12:18:49.373489 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:49.373448 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76c80954-caf1-42b7-a812-fa52488f2e1b-workload-socket\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rx4tf\" (UID: \"76c80954-caf1-42b7-a812-fa52488f2e1b\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rx4tf" Oct 30 12:18:49.373804 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:49.373755 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dm5l\" (UniqueName: \"kubernetes.io/projected/76c80954-caf1-42b7-a812-fa52488f2e1b-kube-api-access-4dm5l\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rx4tf\" (UID: \"76c80954-caf1-42b7-a812-fa52488f2e1b\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rx4tf" Oct 30 12:18:49.374039 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:49.374016 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76c80954-caf1-42b7-a812-fa52488f2e1b-istio-envoy\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rx4tf\" (UID: \"76c80954-caf1-42b7-a812-fa52488f2e1b\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rx4tf" Oct 30 12:18:49.374188 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:49.374170 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76c80954-caf1-42b7-a812-fa52488f2e1b-credential-socket\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rx4tf\" (UID: \"76c80954-caf1-42b7-a812-fa52488f2e1b\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rx4tf" Oct 30 12:18:49.374394 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:49.374374 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76c80954-caf1-42b7-a812-fa52488f2e1b-istiod-ca-cert\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rx4tf\" (UID: \"76c80954-caf1-42b7-a812-fa52488f2e1b\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rx4tf" Oct 30 12:18:49.389213 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:49.389163 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76c80954-caf1-42b7-a812-fa52488f2e1b-istio-data\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rx4tf\" (UID: \"76c80954-caf1-42b7-a812-fa52488f2e1b\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rx4tf" Oct 30 12:18:49.389476 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:49.389451 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76c80954-caf1-42b7-a812-fa52488f2e1b-istio-podinfo\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rx4tf\" (UID: \"76c80954-caf1-42b7-a812-fa52488f2e1b\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rx4tf" Oct 30 12:18:49.389712 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:49.389687 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76c80954-caf1-42b7-a812-fa52488f2e1b-workload-certs\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rx4tf\" (UID: \"76c80954-caf1-42b7-a812-fa52488f2e1b\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rx4tf" Oct 30 12:18:49.389940 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:49.389923 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76c80954-caf1-42b7-a812-fa52488f2e1b-istio-token\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rx4tf\" (UID: \"76c80954-caf1-42b7-a812-fa52488f2e1b\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rx4tf" Oct 30 12:18:49.390977 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:49.390956 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76c80954-caf1-42b7-a812-fa52488f2e1b-workload-certs\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rx4tf\" (UID: \"76c80954-caf1-42b7-a812-fa52488f2e1b\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rx4tf" Oct 30 12:18:49.391546 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:49.391518 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76c80954-caf1-42b7-a812-fa52488f2e1b-workload-socket\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rx4tf\" (UID: \"76c80954-caf1-42b7-a812-fa52488f2e1b\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rx4tf" Oct 30 12:18:49.408738 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:49.408713 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76c80954-caf1-42b7-a812-fa52488f2e1b-credential-socket\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rx4tf\" (UID: \"76c80954-caf1-42b7-a812-fa52488f2e1b\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rx4tf" Oct 30 12:18:49.517899 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:49.496714 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76c80954-caf1-42b7-a812-fa52488f2e1b-istiod-ca-cert\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rx4tf\" (UID: \"76c80954-caf1-42b7-a812-fa52488f2e1b\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rx4tf" Oct 30 12:18:49.517899 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:49.496782 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76c80954-caf1-42b7-a812-fa52488f2e1b-istio-data\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rx4tf\" (UID: \"76c80954-caf1-42b7-a812-fa52488f2e1b\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rx4tf" Oct 30 12:18:49.517899 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:49.496849 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76c80954-caf1-42b7-a812-fa52488f2e1b-istio-podinfo\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rx4tf\" (UID: \"76c80954-caf1-42b7-a812-fa52488f2e1b\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rx4tf" Oct 30 12:18:49.517899 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:49.496976 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76c80954-caf1-42b7-a812-fa52488f2e1b-istio-token\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rx4tf\" (UID: \"76c80954-caf1-42b7-a812-fa52488f2e1b\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rx4tf" Oct 30 12:18:49.517899 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:49.497069 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dm5l\" (UniqueName: \"kubernetes.io/projected/76c80954-caf1-42b7-a812-fa52488f2e1b-kube-api-access-4dm5l\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rx4tf\" (UID: \"76c80954-caf1-42b7-a812-fa52488f2e1b\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rx4tf" Oct 30 12:18:49.517899 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:49.497160 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76c80954-caf1-42b7-a812-fa52488f2e1b-istio-envoy\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rx4tf\" (UID: \"76c80954-caf1-42b7-a812-fa52488f2e1b\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rx4tf" Oct 30 12:18:49.517899 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:49.514189 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76c80954-caf1-42b7-a812-fa52488f2e1b-istio-envoy\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rx4tf\" (UID: \"76c80954-caf1-42b7-a812-fa52488f2e1b\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rx4tf" Oct 30 12:18:49.519893 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:49.519863 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76c80954-caf1-42b7-a812-fa52488f2e1b-istio-data\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rx4tf\" (UID: \"76c80954-caf1-42b7-a812-fa52488f2e1b\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rx4tf" Oct 30 12:18:49.525849 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:49.524066 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76c80954-caf1-42b7-a812-fa52488f2e1b-istiod-ca-cert\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rx4tf\" (UID: \"76c80954-caf1-42b7-a812-fa52488f2e1b\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rx4tf" Oct 30 12:18:49.564021 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:49.563980 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76c80954-caf1-42b7-a812-fa52488f2e1b-istio-podinfo\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rx4tf\" (UID: \"76c80954-caf1-42b7-a812-fa52488f2e1b\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rx4tf" Oct 30 12:18:49.690794 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:49.690725 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76c80954-caf1-42b7-a812-fa52488f2e1b-istio-token\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rx4tf\" (UID: \"76c80954-caf1-42b7-a812-fa52488f2e1b\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rx4tf" Oct 30 12:18:49.693885 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:49.693782 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dm5l\" (UniqueName: \"kubernetes.io/projected/76c80954-caf1-42b7-a812-fa52488f2e1b-kube-api-access-4dm5l\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rx4tf\" (UID: \"76c80954-caf1-42b7-a812-fa52488f2e1b\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rx4tf" Oct 30 12:18:49.728329 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:49.727989 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rx4tf" Oct 30 12:18:49.753607 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:49.753254 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20f2699e-ljxv4" Oct 30 12:18:49.810243 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:18:49.808574 4908 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vn6vv" containerName="extractor" Oct 30 12:18:49.866323 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:49.866286 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1122f4cb-4c3f-41e6-b9fb-c2079a9e208c-kube-api-access-vmmk5\" (UniqueName: \"kubernetes.io/projected/1122f4cb-4c3f-41e6-b9fb-c2079a9e208c-kube-api-access-vmmk5\") pod \"1122f4cb-4c3f-41e6-b9fb-c2079a9e208c\" (UID: \"1122f4cb-4c3f-41e6-b9fb-c2079a9e208c\") " Oct 30 12:18:49.866717 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:49.866689 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1122f4cb-4c3f-41e6-b9fb-c2079a9e208c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1122f4cb-4c3f-41e6-b9fb-c2079a9e208c-maas-api-openshift-ca\") pod \"1122f4cb-4c3f-41e6-b9fb-c2079a9e208c\" (UID: \"1122f4cb-4c3f-41e6-b9fb-c2079a9e208c\") " Oct 30 12:18:49.867001 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:49.866983 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1122f4cb-4c3f-41e6-b9fb-c2079a9e208c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1122f4cb-4c3f-41e6-b9fb-c2079a9e208c-maas-api-xks-ca\") pod \"1122f4cb-4c3f-41e6-b9fb-c2079a9e208c\" (UID: \"1122f4cb-4c3f-41e6-b9fb-c2079a9e208c\") " Oct 30 12:18:49.868745 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:49.868717 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1122f4cb-4c3f-41e6-b9fb-c2079a9e208c-maas-api-openshift-ca" pod "1122f4cb-4c3f-41e6-b9fb-c2079a9e208c" (UID: "1122f4cb-4c3f-41e6-b9fb-c2079a9e208c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:18:49.894587 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:49.894527 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1122f4cb-4c3f-41e6-b9fb-c2079a9e208c-maas-api-xks-ca" pod "1122f4cb-4c3f-41e6-b9fb-c2079a9e208c" (UID: "1122f4cb-4c3f-41e6-b9fb-c2079a9e208c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:18:49.897058 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:49.897010 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1122f4cb-4c3f-41e6-b9fb-c2079a9e208c-kube-api-access-vmmk5" pod "1122f4cb-4c3f-41e6-b9fb-c2079a9e208c" (UID: "1122f4cb-4c3f-41e6-b9fb-c2079a9e208c"). InnerVolumeSpecName "kube-api-access-vmmk5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:18:49.986816 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:49.986693 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1122f4cb-4c3f-41e6-b9fb-c2079a9e208c-maas-api-xks-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:18:49.986816 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:49.986734 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmmk5\" (UniqueName: \"kubernetes.io/projected/1122f4cb-4c3f-41e6-b9fb-c2079a9e208c-kube-api-access-vmmk5\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:18:49.986816 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:49.986751 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1122f4cb-4c3f-41e6-b9fb-c2079a9e208c-maas-api-openshift-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:18:50.018956 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:50.018859 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20f2699e-ljxv4" event={"ID":"1122f4cb-4c3f-41e6-b9fb-c2079a9e208c","Type":"ContainerDied","Data":"f8072b8ca0399d3b1b8bbc9dde0618e3a0e9cbc8ac3127b42c5cc8fee183ebdb"} Oct 30 12:18:50.018956 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:50.018928 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20f2699e-ljxv4" Oct 30 12:18:50.018956 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:50.018941 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f8072b8ca0399d3b1b8bbc9dde0618e3a0e9cbc8ac3127b42c5cc8fee183ebdb" Oct 30 12:18:50.072078 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:50.072047 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de045378-pcjbr"] Oct 30 12:18:50.073971 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:18:50.073942 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3054913_cac1_435a_ac9a_1b06e9434c35.slice/crio-d6e7591542c43d578ef6470405ddf70a5cfb216266847ede67dfcd65850ff2c8 WatchSource:0}: Error finding container d6e7591542c43d578ef6470405ddf70a5cfb216266847ede67dfcd65850ff2c8: Status 404 returned error can't find the container with id d6e7591542c43d578ef6470405ddf70a5cfb216266847ede67dfcd65850ff2c8 Oct 30 12:18:50.170359 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:50.170326 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rx4tf"] Oct 30 12:18:50.170626 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:18:50.170599 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76c80954_caf1_42b7_a812_fa52488f2e1b.slice/crio-3bc88771ac88beba4f4b4f3f3f7e88f8a6c5c64c86d01c364d1ec73930d64e79 WatchSource:0}: Error finding container 3bc88771ac88beba4f4b4f3f3f7e88f8a6c5c64c86d01c364d1ec73930d64e79: Status 404 returned error can't find the container with id 3bc88771ac88beba4f4b4f3f3f7e88f8a6c5c64c86d01c364d1ec73930d64e79 Oct 30 12:18:50.173894 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:50.173855 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:18:50.174000 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:50.173934 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:18:50.174000 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:50.173968 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:18:50.213807 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:50.213775 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-6p2qs" Oct 30 12:18:50.215171 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:50.215149 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-6p2qs" Oct 30 12:18:50.215941 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:50.215921 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-6p2qs" Oct 30 12:18:51.032662 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:51.032618 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de045378-pcjbr" event={"ID":"b3054913-cac1-435a-ac9a-1b06e9434c35","Type":"ContainerStarted","Data":"e2d5fa0c2c585d90d93e16fc54b74572f523df0ff806e81cda2b3ed654358617"} Oct 30 12:18:51.033212 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:51.032693 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de045378-pcjbr" event={"ID":"b3054913-cac1-435a-ac9a-1b06e9434c35","Type":"ContainerStarted","Data":"d6e7591542c43d578ef6470405ddf70a5cfb216266847ede67dfcd65850ff2c8"} Oct 30 12:18:51.037213 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:51.037180 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rx4tf" event={"ID":"76c80954-caf1-42b7-a812-fa52488f2e1b","Type":"ContainerStarted","Data":"9fc1dc78272c01ea0b0a3ea409879f085f1c4fe92405803a6c3e0879507206b2"} Oct 30 12:18:51.037364 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:51.037237 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rx4tf" event={"ID":"76c80954-caf1-42b7-a812-fa52488f2e1b","Type":"ContainerStarted","Data":"3bc88771ac88beba4f4b4f3f3f7e88f8a6c5c64c86d01c364d1ec73930d64e79"} Oct 30 12:18:51.083854 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:51.080378 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de045378-pcjbr" podStartSLOduration=3.080356157 podStartE2EDuration="3.080356157s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:18:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:18:51.078126372 +0000 UTC m=+1487.592555119" watchObservedRunningTime="2026-10-30 12:18:51.080356157 +0000 UTC m=+1487.594784904" Oct 30 12:18:51.134865 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:51.134781 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rx4tf" podStartSLOduration=2.134759758 podStartE2EDuration="2.134759758s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:18:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:18:51.127022678 +0000 UTC m=+1487.641451455" watchObservedRunningTime="2026-10-30 12:18:51.134759758 +0000 UTC m=+1487.649188505" Oct 30 12:18:51.728915 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:51.728874 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rx4tf" Oct 30 12:18:51.730619 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:51.730594 4908 patch_prober.go:29] interesting pod/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rx4tf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 30 12:18:51.730727 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:51.730644 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rx4tf" podUID="76c80954-caf1-42b7-a812-fa52488f2e1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 30 12:18:52.050274 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:52.050117 4908 generic.go:356] "Generic (PLEG): container finished" podID="b3054913-cac1-435a-ac9a-1b06e9434c35" containerID="e2d5fa0c2c585d90d93e16fc54b74572f523df0ff806e81cda2b3ed654358617" exitCode=0 Oct 30 12:18:52.050274 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:52.050210 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de045378-pcjbr" event={"ID":"b3054913-cac1-435a-ac9a-1b06e9434c35","Type":"ContainerDied","Data":"e2d5fa0c2c585d90d93e16fc54b74572f523df0ff806e81cda2b3ed654358617"} Oct 30 12:18:52.729322 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:52.729281 4908 patch_prober.go:29] interesting pod/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rx4tf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 30 12:18:52.729509 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:52.729342 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rx4tf" podUID="76c80954-caf1-42b7-a812-fa52488f2e1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 30 12:18:52.926523 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:52.926484 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9b22a7d-dtk4r"] Oct 30 12:18:52.927081 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:52.927065 4908 cpu_manager.go:414] "containerMap: removing container" podUID="1122f4cb-4c3f-41e6-b9fb-c2079a9e208c" containerName="revoke-keys" Oct 30 12:18:52.927144 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:52.927081 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1122f4cb-4c3f-41e6-b9fb-c2079a9e208c" containerName="revoke-keys" Oct 30 12:18:52.927208 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:52.927198 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="1122f4cb-4c3f-41e6-b9fb-c2079a9e208c" containerName="revoke-keys" Oct 30 12:18:52.933537 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:52.931439 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9b22a7d-dtk4r" Oct 30 12:18:52.952145 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:52.952088 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9b22a7d-dtk4r"] Oct 30 12:18:53.036130 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:53.036029 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzn9g\" (UniqueName: \"kubernetes.io/projected/5d5b74c2-a2b1-4e10-a9e2-7690fd383724-kube-api-access-qzn9g\") pod \"maas-api-revoke-sub-d9b22a7d-dtk4r\" (UID: \"5d5b74c2-a2b1-4e10-a9e2-7690fd383724\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9b22a7d-dtk4r" Oct 30 12:18:53.036130 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:53.036085 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d5b74c2-a2b1-4e10-a9e2-7690fd383724-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d9b22a7d-dtk4r\" (UID: \"5d5b74c2-a2b1-4e10-a9e2-7690fd383724\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9b22a7d-dtk4r" Oct 30 12:18:53.036130 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:53.036117 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d5b74c2-a2b1-4e10-a9e2-7690fd383724-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d9b22a7d-dtk4r\" (UID: \"5d5b74c2-a2b1-4e10-a9e2-7690fd383724\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9b22a7d-dtk4r" Oct 30 12:18:53.138886 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:53.138837 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzn9g\" (UniqueName: \"kubernetes.io/projected/5d5b74c2-a2b1-4e10-a9e2-7690fd383724-kube-api-access-qzn9g\") pod \"maas-api-revoke-sub-d9b22a7d-dtk4r\" (UID: \"5d5b74c2-a2b1-4e10-a9e2-7690fd383724\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9b22a7d-dtk4r" Oct 30 12:18:53.139374 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:53.139201 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d5b74c2-a2b1-4e10-a9e2-7690fd383724-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d9b22a7d-dtk4r\" (UID: \"5d5b74c2-a2b1-4e10-a9e2-7690fd383724\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9b22a7d-dtk4r" Oct 30 12:18:53.139374 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:53.139260 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d5b74c2-a2b1-4e10-a9e2-7690fd383724-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d9b22a7d-dtk4r\" (UID: \"5d5b74c2-a2b1-4e10-a9e2-7690fd383724\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9b22a7d-dtk4r" Oct 30 12:18:53.139986 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:53.139962 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d5b74c2-a2b1-4e10-a9e2-7690fd383724-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d9b22a7d-dtk4r\" (UID: \"5d5b74c2-a2b1-4e10-a9e2-7690fd383724\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9b22a7d-dtk4r" Oct 30 12:18:53.142248 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:53.142201 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d5b74c2-a2b1-4e10-a9e2-7690fd383724-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d9b22a7d-dtk4r\" (UID: \"5d5b74c2-a2b1-4e10-a9e2-7690fd383724\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9b22a7d-dtk4r" Oct 30 12:18:53.207058 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:53.206821 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzn9g\" (UniqueName: \"kubernetes.io/projected/5d5b74c2-a2b1-4e10-a9e2-7690fd383724-kube-api-access-qzn9g\") pod \"maas-api-revoke-sub-d9b22a7d-dtk4r\" (UID: \"5d5b74c2-a2b1-4e10-a9e2-7690fd383724\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9b22a7d-dtk4r" Oct 30 12:18:53.252851 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:53.249660 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9b22a7d-dtk4r" Oct 30 12:18:53.392480 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:53.392379 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de045378-pcjbr" Oct 30 12:18:53.452401 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:53.451292 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3054913-cac1-435a-ac9a-1b06e9434c35-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3054913-cac1-435a-ac9a-1b06e9434c35-maas-api-openshift-ca\") pod \"b3054913-cac1-435a-ac9a-1b06e9434c35\" (UID: \"b3054913-cac1-435a-ac9a-1b06e9434c35\") " Oct 30 12:18:53.454078 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:53.452714 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b3054913-cac1-435a-ac9a-1b06e9434c35-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3054913-cac1-435a-ac9a-1b06e9434c35-maas-api-xks-ca\") pod \"b3054913-cac1-435a-ac9a-1b06e9434c35\" (UID: \"b3054913-cac1-435a-ac9a-1b06e9434c35\") " Oct 30 12:18:53.454527 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:53.454505 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3054913-cac1-435a-ac9a-1b06e9434c35-kube-api-access-hdnb2\" (UniqueName: \"kubernetes.io/projected/b3054913-cac1-435a-ac9a-1b06e9434c35-kube-api-access-hdnb2\") pod \"b3054913-cac1-435a-ac9a-1b06e9434c35\" (UID: \"b3054913-cac1-435a-ac9a-1b06e9434c35\") " Oct 30 12:18:53.455413 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:53.454215 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3054913-cac1-435a-ac9a-1b06e9434c35-maas-api-openshift-ca" pod "b3054913-cac1-435a-ac9a-1b06e9434c35" (UID: "b3054913-cac1-435a-ac9a-1b06e9434c35"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:18:53.455413 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:53.455289 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3054913-cac1-435a-ac9a-1b06e9434c35-maas-api-openshift-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:18:53.456820 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:53.456782 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3054913-cac1-435a-ac9a-1b06e9434c35-maas-api-xks-ca" pod "b3054913-cac1-435a-ac9a-1b06e9434c35" (UID: "b3054913-cac1-435a-ac9a-1b06e9434c35"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:18:53.507099 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:53.507020 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3054913-cac1-435a-ac9a-1b06e9434c35-kube-api-access-hdnb2" pod "b3054913-cac1-435a-ac9a-1b06e9434c35" (UID: "b3054913-cac1-435a-ac9a-1b06e9434c35"). InnerVolumeSpecName "kube-api-access-hdnb2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:18:53.556944 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:53.556871 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3054913-cac1-435a-ac9a-1b06e9434c35-maas-api-xks-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:18:53.556944 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:53.556911 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdnb2\" (UniqueName: \"kubernetes.io/projected/b3054913-cac1-435a-ac9a-1b06e9434c35-kube-api-access-hdnb2\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:18:53.686949 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:53.686046 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9b22a7d-dtk4r"] Oct 30 12:18:53.708223 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:18:53.708179 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d5b74c2_a2b1_4e10_a9e2_7690fd383724.slice/crio-e414b9824d3fa828c37223f23b3dd3effa4d8a647863f1491192e0b518abce41 WatchSource:0}: Error finding container e414b9824d3fa828c37223f23b3dd3effa4d8a647863f1491192e0b518abce41: Status 404 returned error can't find the container with id e414b9824d3fa828c37223f23b3dd3effa4d8a647863f1491192e0b518abce41 Oct 30 12:18:53.729667 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:53.729640 4908 patch_prober.go:29] interesting pod/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rx4tf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 30 12:18:53.729768 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:53.729688 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rx4tf" podUID="76c80954-caf1-42b7-a812-fa52488f2e1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 30 12:18:54.109029 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:54.108996 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de045378-pcjbr" Oct 30 12:18:54.109310 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:54.109041 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de045378-pcjbr" event={"ID":"b3054913-cac1-435a-ac9a-1b06e9434c35","Type":"ContainerDied","Data":"d6e7591542c43d578ef6470405ddf70a5cfb216266847ede67dfcd65850ff2c8"} Oct 30 12:18:54.109397 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:54.109377 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d6e7591542c43d578ef6470405ddf70a5cfb216266847ede67dfcd65850ff2c8" Oct 30 12:18:54.115073 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:54.114421 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9b22a7d-dtk4r" event={"ID":"5d5b74c2-a2b1-4e10-a9e2-7690fd383724","Type":"ContainerStarted","Data":"e414b9824d3fa828c37223f23b3dd3effa4d8a647863f1491192e0b518abce41"} Oct 30 12:18:54.170894 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:54.168691 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9b22a7d-dtk4r" podStartSLOduration=2.168669258 podStartE2EDuration="2.168669258s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:18:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:18:54.167226135 +0000 UTC m=+1490.681654881" watchObservedRunningTime="2026-10-30 12:18:54.168669258 +0000 UTC m=+1490.683098005" Oct 30 12:18:54.729108 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:54.729066 4908 patch_prober.go:29] interesting pod/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rx4tf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 30 12:18:54.729293 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:54.729131 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rx4tf" podUID="76c80954-caf1-42b7-a812-fa52488f2e1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 30 12:18:55.124985 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:55.124883 4908 generic.go:356] "Generic (PLEG): container finished" podID="5d5b74c2-a2b1-4e10-a9e2-7690fd383724" containerID="916e69059bde9f089ac49330da767098b619fef80f78c4c2a2026664b18ac9d6" exitCode=0 Oct 30 12:18:55.125170 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:55.124983 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9b22a7d-dtk4r" event={"ID":"5d5b74c2-a2b1-4e10-a9e2-7690fd383724","Type":"ContainerDied","Data":"916e69059bde9f089ac49330da767098b619fef80f78c4c2a2026664b18ac9d6"} Oct 30 12:18:55.729357 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:55.729319 4908 patch_prober.go:29] interesting pod/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rx4tf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 30 12:18:55.729817 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:55.729376 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rx4tf" podUID="76c80954-caf1-42b7-a812-fa52488f2e1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 30 12:18:56.323581 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:56.323553 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9b22a7d-dtk4r" Oct 30 12:18:56.413006 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:56.412889 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d5b74c2-a2b1-4e10-a9e2-7690fd383724-kube-api-access-qzn9g\" (UniqueName: \"kubernetes.io/projected/5d5b74c2-a2b1-4e10-a9e2-7690fd383724-kube-api-access-qzn9g\") pod \"5d5b74c2-a2b1-4e10-a9e2-7690fd383724\" (UID: \"5d5b74c2-a2b1-4e10-a9e2-7690fd383724\") " Oct 30 12:18:56.413006 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:56.412998 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d5b74c2-a2b1-4e10-a9e2-7690fd383724-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d5b74c2-a2b1-4e10-a9e2-7690fd383724-maas-api-openshift-ca\") pod \"5d5b74c2-a2b1-4e10-a9e2-7690fd383724\" (UID: \"5d5b74c2-a2b1-4e10-a9e2-7690fd383724\") " Oct 30 12:18:56.413239 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:56.413046 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5d5b74c2-a2b1-4e10-a9e2-7690fd383724-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d5b74c2-a2b1-4e10-a9e2-7690fd383724-maas-api-xks-ca\") pod \"5d5b74c2-a2b1-4e10-a9e2-7690fd383724\" (UID: \"5d5b74c2-a2b1-4e10-a9e2-7690fd383724\") " Oct 30 12:18:56.413747 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:56.413715 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d5b74c2-a2b1-4e10-a9e2-7690fd383724-maas-api-openshift-ca" pod "5d5b74c2-a2b1-4e10-a9e2-7690fd383724" (UID: "5d5b74c2-a2b1-4e10-a9e2-7690fd383724"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:18:56.415687 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:56.415653 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d5b74c2-a2b1-4e10-a9e2-7690fd383724-maas-api-xks-ca" pod "5d5b74c2-a2b1-4e10-a9e2-7690fd383724" (UID: "5d5b74c2-a2b1-4e10-a9e2-7690fd383724"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:18:56.416271 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:56.416244 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d5b74c2-a2b1-4e10-a9e2-7690fd383724-kube-api-access-qzn9g" pod "5d5b74c2-a2b1-4e10-a9e2-7690fd383724" (UID: "5d5b74c2-a2b1-4e10-a9e2-7690fd383724"). InnerVolumeSpecName "kube-api-access-qzn9g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:18:56.514911 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:56.514865 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qzn9g\" (UniqueName: \"kubernetes.io/projected/5d5b74c2-a2b1-4e10-a9e2-7690fd383724-kube-api-access-qzn9g\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:18:56.514911 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:56.514907 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d5b74c2-a2b1-4e10-a9e2-7690fd383724-maas-api-openshift-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:18:56.514911 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:56.514921 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d5b74c2-a2b1-4e10-a9e2-7690fd383724-maas-api-xks-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:18:56.729123 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:56.729070 4908 patch_prober.go:29] interesting pod/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rx4tf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 30 12:18:56.729316 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:56.729146 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rx4tf" podUID="76c80954-caf1-42b7-a812-fa52488f2e1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 30 12:18:57.145261 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:57.145170 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9b22a7d-dtk4r" Oct 30 12:18:57.145661 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:57.145167 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9b22a7d-dtk4r" event={"ID":"5d5b74c2-a2b1-4e10-a9e2-7690fd383724","Type":"ContainerDied","Data":"e414b9824d3fa828c37223f23b3dd3effa4d8a647863f1491192e0b518abce41"} Oct 30 12:18:57.145661 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:57.145319 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e414b9824d3fa828c37223f23b3dd3effa4d8a647863f1491192e0b518abce41" Oct 30 12:18:57.729334 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:57.729294 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-6a7148/distinct-2-w3-6a7148-kserve-67bd5fb976-9nfff"] Oct 30 12:18:57.729897 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:57.729878 4908 cpu_manager.go:414] "containerMap: removing container" podUID="b3054913-cac1-435a-ac9a-1b06e9434c35" containerName="revoke-keys" Oct 30 12:18:57.730013 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:57.729897 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3054913-cac1-435a-ac9a-1b06e9434c35" containerName="revoke-keys" Oct 30 12:18:57.730013 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:57.729934 4908 cpu_manager.go:414] "containerMap: removing container" podUID="5d5b74c2-a2b1-4e10-a9e2-7690fd383724" containerName="revoke-keys" Oct 30 12:18:57.730013 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:57.729942 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d5b74c2-a2b1-4e10-a9e2-7690fd383724" containerName="revoke-keys" Oct 30 12:18:57.730169 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:57.730105 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d5b74c2-a2b1-4e10-a9e2-7690fd383724" containerName="revoke-keys" Oct 30 12:18:57.730169 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:57.730128 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3054913-cac1-435a-ac9a-1b06e9434c35" containerName="revoke-keys" Oct 30 12:18:57.734121 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:57.734089 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-6a7148/distinct-2-w3-6a7148-kserve-67bd5fb976-9nfff" Oct 30 12:18:57.734782 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:57.734750 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rx4tf" Oct 30 12:18:57.735103 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:57.735079 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rx4tf" Oct 30 12:18:57.736012 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:57.735989 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rx4tf" Oct 30 12:18:57.751613 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:57.751585 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-6a7148/distinct-2-w3-6a7148-kserve-67bd5fb976-9nfff"] Oct 30 12:18:57.757223 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:57.757197 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-6a7148\"/\"distinct-2-w3-6a7148-kserve-self-signed-certs\"" Oct 30 12:18:57.837397 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:57.837345 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbwbp\" (UniqueName: \"kubernetes.io/projected/6046cc02-29fe-4f59-b522-8f57f4557277-kube-api-access-mbwbp\") pod \"distinct-2-w3-6a7148-kserve-67bd5fb976-9nfff\" (UID: \"6046cc02-29fe-4f59-b522-8f57f4557277\") " pod="e2e-models-e2e-worker-w3-6a7148/distinct-2-w3-6a7148-kserve-67bd5fb976-9nfff" Oct 30 12:18:57.837397 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:57.837401 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6046cc02-29fe-4f59-b522-8f57f4557277-dshm\") pod \"distinct-2-w3-6a7148-kserve-67bd5fb976-9nfff\" (UID: \"6046cc02-29fe-4f59-b522-8f57f4557277\") " pod="e2e-models-e2e-worker-w3-6a7148/distinct-2-w3-6a7148-kserve-67bd5fb976-9nfff" Oct 30 12:18:57.837649 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:57.837432 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6046cc02-29fe-4f59-b522-8f57f4557277-tmp-dir\") pod \"distinct-2-w3-6a7148-kserve-67bd5fb976-9nfff\" (UID: \"6046cc02-29fe-4f59-b522-8f57f4557277\") " pod="e2e-models-e2e-worker-w3-6a7148/distinct-2-w3-6a7148-kserve-67bd5fb976-9nfff" Oct 30 12:18:57.837649 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:57.837459 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6046cc02-29fe-4f59-b522-8f57f4557277-tls-certs\") pod \"distinct-2-w3-6a7148-kserve-67bd5fb976-9nfff\" (UID: \"6046cc02-29fe-4f59-b522-8f57f4557277\") " pod="e2e-models-e2e-worker-w3-6a7148/distinct-2-w3-6a7148-kserve-67bd5fb976-9nfff" Oct 30 12:18:57.837649 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:57.837494 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6046cc02-29fe-4f59-b522-8f57f4557277-home\") pod \"distinct-2-w3-6a7148-kserve-67bd5fb976-9nfff\" (UID: \"6046cc02-29fe-4f59-b522-8f57f4557277\") " pod="e2e-models-e2e-worker-w3-6a7148/distinct-2-w3-6a7148-kserve-67bd5fb976-9nfff" Oct 30 12:18:57.838281 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:57.838230 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6046cc02-29fe-4f59-b522-8f57f4557277-model-cache\") pod \"distinct-2-w3-6a7148-kserve-67bd5fb976-9nfff\" (UID: \"6046cc02-29fe-4f59-b522-8f57f4557277\") " pod="e2e-models-e2e-worker-w3-6a7148/distinct-2-w3-6a7148-kserve-67bd5fb976-9nfff" Oct 30 12:18:57.939682 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:57.939642 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbwbp\" (UniqueName: \"kubernetes.io/projected/6046cc02-29fe-4f59-b522-8f57f4557277-kube-api-access-mbwbp\") pod \"distinct-2-w3-6a7148-kserve-67bd5fb976-9nfff\" (UID: \"6046cc02-29fe-4f59-b522-8f57f4557277\") " pod="e2e-models-e2e-worker-w3-6a7148/distinct-2-w3-6a7148-kserve-67bd5fb976-9nfff" Oct 30 12:18:57.939894 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:57.939698 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6046cc02-29fe-4f59-b522-8f57f4557277-dshm\") pod \"distinct-2-w3-6a7148-kserve-67bd5fb976-9nfff\" (UID: \"6046cc02-29fe-4f59-b522-8f57f4557277\") " pod="e2e-models-e2e-worker-w3-6a7148/distinct-2-w3-6a7148-kserve-67bd5fb976-9nfff" Oct 30 12:18:57.939894 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:57.939727 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6046cc02-29fe-4f59-b522-8f57f4557277-tmp-dir\") pod \"distinct-2-w3-6a7148-kserve-67bd5fb976-9nfff\" (UID: \"6046cc02-29fe-4f59-b522-8f57f4557277\") " pod="e2e-models-e2e-worker-w3-6a7148/distinct-2-w3-6a7148-kserve-67bd5fb976-9nfff" Oct 30 12:18:57.939894 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:57.939754 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6046cc02-29fe-4f59-b522-8f57f4557277-tls-certs\") pod \"distinct-2-w3-6a7148-kserve-67bd5fb976-9nfff\" (UID: \"6046cc02-29fe-4f59-b522-8f57f4557277\") " pod="e2e-models-e2e-worker-w3-6a7148/distinct-2-w3-6a7148-kserve-67bd5fb976-9nfff" Oct 30 12:18:57.939894 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:57.939787 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6046cc02-29fe-4f59-b522-8f57f4557277-home\") pod \"distinct-2-w3-6a7148-kserve-67bd5fb976-9nfff\" (UID: \"6046cc02-29fe-4f59-b522-8f57f4557277\") " pod="e2e-models-e2e-worker-w3-6a7148/distinct-2-w3-6a7148-kserve-67bd5fb976-9nfff" Oct 30 12:18:57.940136 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:57.939918 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6046cc02-29fe-4f59-b522-8f57f4557277-model-cache\") pod \"distinct-2-w3-6a7148-kserve-67bd5fb976-9nfff\" (UID: \"6046cc02-29fe-4f59-b522-8f57f4557277\") " pod="e2e-models-e2e-worker-w3-6a7148/distinct-2-w3-6a7148-kserve-67bd5fb976-9nfff" Oct 30 12:18:57.940320 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:57.940295 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6046cc02-29fe-4f59-b522-8f57f4557277-tmp-dir\") pod \"distinct-2-w3-6a7148-kserve-67bd5fb976-9nfff\" (UID: \"6046cc02-29fe-4f59-b522-8f57f4557277\") " pod="e2e-models-e2e-worker-w3-6a7148/distinct-2-w3-6a7148-kserve-67bd5fb976-9nfff" Oct 30 12:18:57.940598 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:57.940571 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6046cc02-29fe-4f59-b522-8f57f4557277-model-cache\") pod \"distinct-2-w3-6a7148-kserve-67bd5fb976-9nfff\" (UID: \"6046cc02-29fe-4f59-b522-8f57f4557277\") " pod="e2e-models-e2e-worker-w3-6a7148/distinct-2-w3-6a7148-kserve-67bd5fb976-9nfff" Oct 30 12:18:57.940731 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:57.940613 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6046cc02-29fe-4f59-b522-8f57f4557277-home\") pod \"distinct-2-w3-6a7148-kserve-67bd5fb976-9nfff\" (UID: \"6046cc02-29fe-4f59-b522-8f57f4557277\") " pod="e2e-models-e2e-worker-w3-6a7148/distinct-2-w3-6a7148-kserve-67bd5fb976-9nfff" Oct 30 12:18:57.942624 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:57.942599 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6046cc02-29fe-4f59-b522-8f57f4557277-dshm\") pod \"distinct-2-w3-6a7148-kserve-67bd5fb976-9nfff\" (UID: \"6046cc02-29fe-4f59-b522-8f57f4557277\") " pod="e2e-models-e2e-worker-w3-6a7148/distinct-2-w3-6a7148-kserve-67bd5fb976-9nfff" Oct 30 12:18:57.942941 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:57.942922 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6046cc02-29fe-4f59-b522-8f57f4557277-tls-certs\") pod \"distinct-2-w3-6a7148-kserve-67bd5fb976-9nfff\" (UID: \"6046cc02-29fe-4f59-b522-8f57f4557277\") " pod="e2e-models-e2e-worker-w3-6a7148/distinct-2-w3-6a7148-kserve-67bd5fb976-9nfff" Oct 30 12:18:57.994741 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:57.994620 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbwbp\" (UniqueName: \"kubernetes.io/projected/6046cc02-29fe-4f59-b522-8f57f4557277-kube-api-access-mbwbp\") pod \"distinct-2-w3-6a7148-kserve-67bd5fb976-9nfff\" (UID: \"6046cc02-29fe-4f59-b522-8f57f4557277\") " pod="e2e-models-e2e-worker-w3-6a7148/distinct-2-w3-6a7148-kserve-67bd5fb976-9nfff" Oct 30 12:18:58.050659 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:58.050110 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-6a7148/distinct-2-w3-6a7148-kserve-67bd5fb976-9nfff" Oct 30 12:18:58.392235 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:58.392202 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-6a7148/distinct-2-w3-6a7148-kserve-67bd5fb976-9nfff"] Oct 30 12:18:58.394658 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:18:58.394633 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6046cc02_29fe_4f59_b522_8f57f4557277.slice/crio-53a13d2aa52225f36269c9fde43848eea68c017bbefdb9a46b030177c8e73a7f WatchSource:0}: Error finding container 53a13d2aa52225f36269c9fde43848eea68c017bbefdb9a46b030177c8e73a7f: Status 404 returned error can't find the container with id 53a13d2aa52225f36269c9fde43848eea68c017bbefdb9a46b030177c8e73a7f Oct 30 12:18:59.169603 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:59.169551 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-6a7148/distinct-2-w3-6a7148-kserve-67bd5fb976-9nfff" event={"ID":"6046cc02-29fe-4f59-b522-8f57f4557277","Type":"ContainerStarted","Data":"c0e007f741bf504d35e94c7a7d5ff0af3d7c5505c8fd05bad82dca1037226ad6"} Oct 30 12:18:59.169926 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:59.169619 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-6a7148/distinct-2-w3-6a7148-kserve-67bd5fb976-9nfff" event={"ID":"6046cc02-29fe-4f59-b522-8f57f4557277","Type":"ContainerStarted","Data":"53a13d2aa52225f36269c9fde43848eea68c017bbefdb9a46b030177c8e73a7f"} Oct 30 12:18:59.218243 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:59.218186 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-6a7148/distinct-2-w3-6a7148-kserve-67bd5fb976-9nfff" podStartSLOduration=2.218170427 podStartE2EDuration="2.218170427s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:18:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:18:59.214370733 +0000 UTC m=+1495.728799479" watchObservedRunningTime="2026-10-30 12:18:59.218170427 +0000 UTC m=+1495.732599173" Oct 30 12:18:59.366255 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:59.366210 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21b8dc27-bptpx"] Oct 30 12:18:59.371895 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:59.371864 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21b8dc27-bptpx" Oct 30 12:18:59.386523 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:59.386462 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 12:18:59.387281 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:59.387211 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s75hq\"" Oct 30 12:18:59.387652 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:59.387630 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 12:18:59.387738 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:59.387700 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 12:18:59.404368 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:59.404328 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21b8dc27-bptpx"] Oct 30 12:18:59.458422 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:59.458327 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3e63d4d-0257-41a0-9b25-24a4c7514b95-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-21b8dc27-bptpx\" (UID: \"d3e63d4d-0257-41a0-9b25-24a4c7514b95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21b8dc27-bptpx" Oct 30 12:18:59.458585 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:59.458492 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3e63d4d-0257-41a0-9b25-24a4c7514b95-maas-api-xks-ca\") pod \"maas-api-revoke-sub-21b8dc27-bptpx\" (UID: \"d3e63d4d-0257-41a0-9b25-24a4c7514b95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21b8dc27-bptpx" Oct 30 12:18:59.458585 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:59.458552 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nphm5\" (UniqueName: \"kubernetes.io/projected/d3e63d4d-0257-41a0-9b25-24a4c7514b95-kube-api-access-nphm5\") pod \"maas-api-revoke-sub-21b8dc27-bptpx\" (UID: \"d3e63d4d-0257-41a0-9b25-24a4c7514b95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21b8dc27-bptpx" Oct 30 12:18:59.559971 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:59.559888 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nphm5\" (UniqueName: \"kubernetes.io/projected/d3e63d4d-0257-41a0-9b25-24a4c7514b95-kube-api-access-nphm5\") pod \"maas-api-revoke-sub-21b8dc27-bptpx\" (UID: \"d3e63d4d-0257-41a0-9b25-24a4c7514b95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21b8dc27-bptpx" Oct 30 12:18:59.560395 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:59.560167 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3e63d4d-0257-41a0-9b25-24a4c7514b95-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-21b8dc27-bptpx\" (UID: \"d3e63d4d-0257-41a0-9b25-24a4c7514b95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21b8dc27-bptpx" Oct 30 12:18:59.560395 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:59.560378 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3e63d4d-0257-41a0-9b25-24a4c7514b95-maas-api-xks-ca\") pod \"maas-api-revoke-sub-21b8dc27-bptpx\" (UID: \"d3e63d4d-0257-41a0-9b25-24a4c7514b95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21b8dc27-bptpx" Oct 30 12:18:59.561200 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:59.561155 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3e63d4d-0257-41a0-9b25-24a4c7514b95-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-21b8dc27-bptpx\" (UID: \"d3e63d4d-0257-41a0-9b25-24a4c7514b95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21b8dc27-bptpx" Oct 30 12:18:59.563198 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:59.563170 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3e63d4d-0257-41a0-9b25-24a4c7514b95-maas-api-xks-ca\") pod \"maas-api-revoke-sub-21b8dc27-bptpx\" (UID: \"d3e63d4d-0257-41a0-9b25-24a4c7514b95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21b8dc27-bptpx" Oct 30 12:18:59.583016 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:59.582982 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nphm5\" (UniqueName: \"kubernetes.io/projected/d3e63d4d-0257-41a0-9b25-24a4c7514b95-kube-api-access-nphm5\") pod \"maas-api-revoke-sub-21b8dc27-bptpx\" (UID: \"d3e63d4d-0257-41a0-9b25-24a4c7514b95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21b8dc27-bptpx" Oct 30 12:18:59.702260 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:18:59.702228 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21b8dc27-bptpx" Oct 30 12:19:00.007979 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:00.007945 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21b8dc27-bptpx"] Oct 30 12:19:00.008500 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:19:00.008473 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3e63d4d_0257_41a0_9b25_24a4c7514b95.slice/crio-38a69cb27068ac64b838ba37362f4e701b43deb3a8a55a6a6102e7a1d91b8233 WatchSource:0}: Error finding container 38a69cb27068ac64b838ba37362f4e701b43deb3a8a55a6a6102e7a1d91b8233: Status 404 returned error can't find the container with id 38a69cb27068ac64b838ba37362f4e701b43deb3a8a55a6a6102e7a1d91b8233 Oct 30 12:19:00.050651 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:00.050622 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-6a7148/distinct-2-w3-6a7148-kserve-67bd5fb976-9nfff" Oct 30 12:19:00.181548 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:00.181512 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21b8dc27-bptpx" event={"ID":"d3e63d4d-0257-41a0-9b25-24a4c7514b95","Type":"ContainerStarted","Data":"38a69cb27068ac64b838ba37362f4e701b43deb3a8a55a6a6102e7a1d91b8233"} Oct 30 12:19:01.193842 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:01.193731 4908 generic.go:356] "Generic (PLEG): container finished" podID="d3e63d4d-0257-41a0-9b25-24a4c7514b95" containerID="0d3ae493403b988ef3f8fdcaa54318423bb1557314f08a321c1f5963ffdc0816" exitCode=0 Oct 30 12:19:01.193842 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:01.193811 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21b8dc27-bptpx" event={"ID":"d3e63d4d-0257-41a0-9b25-24a4c7514b95","Type":"ContainerDied","Data":"0d3ae493403b988ef3f8fdcaa54318423bb1557314f08a321c1f5963ffdc0816"} Oct 30 12:19:03.383854 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:03.383743 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21b8dc27-bptpx" Oct 30 12:19:03.407852 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:03.406473 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d3e63d4d-0257-41a0-9b25-24a4c7514b95-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3e63d4d-0257-41a0-9b25-24a4c7514b95-maas-api-xks-ca\") pod \"d3e63d4d-0257-41a0-9b25-24a4c7514b95\" (UID: \"d3e63d4d-0257-41a0-9b25-24a4c7514b95\") " Oct 30 12:19:03.407852 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:03.406651 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3e63d4d-0257-41a0-9b25-24a4c7514b95-kube-api-access-nphm5\" (UniqueName: \"kubernetes.io/projected/d3e63d4d-0257-41a0-9b25-24a4c7514b95-kube-api-access-nphm5\") pod \"d3e63d4d-0257-41a0-9b25-24a4c7514b95\" (UID: \"d3e63d4d-0257-41a0-9b25-24a4c7514b95\") " Oct 30 12:19:03.407852 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:03.406686 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3e63d4d-0257-41a0-9b25-24a4c7514b95-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3e63d4d-0257-41a0-9b25-24a4c7514b95-maas-api-openshift-ca\") pod \"d3e63d4d-0257-41a0-9b25-24a4c7514b95\" (UID: \"d3e63d4d-0257-41a0-9b25-24a4c7514b95\") " Oct 30 12:19:03.408756 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:03.408720 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3e63d4d-0257-41a0-9b25-24a4c7514b95-maas-api-openshift-ca" pod "d3e63d4d-0257-41a0-9b25-24a4c7514b95" (UID: "d3e63d4d-0257-41a0-9b25-24a4c7514b95"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:19:03.437509 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:03.437468 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3e63d4d-0257-41a0-9b25-24a4c7514b95-kube-api-access-nphm5" pod "d3e63d4d-0257-41a0-9b25-24a4c7514b95" (UID: "d3e63d4d-0257-41a0-9b25-24a4c7514b95"). InnerVolumeSpecName "kube-api-access-nphm5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:19:03.455426 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:03.455384 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3e63d4d-0257-41a0-9b25-24a4c7514b95-maas-api-xks-ca" pod "d3e63d4d-0257-41a0-9b25-24a4c7514b95" (UID: "d3e63d4d-0257-41a0-9b25-24a4c7514b95"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:19:03.508775 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:03.508732 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nphm5\" (UniqueName: \"kubernetes.io/projected/d3e63d4d-0257-41a0-9b25-24a4c7514b95-kube-api-access-nphm5\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:19:03.509025 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:03.509004 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3e63d4d-0257-41a0-9b25-24a4c7514b95-maas-api-openshift-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:19:03.509127 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:03.509114 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3e63d4d-0257-41a0-9b25-24a4c7514b95-maas-api-xks-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:19:04.161902 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:04.158709 4908 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-plczt_a70ee11c-c20e-4a4f-8476-ac5eb02816d0/console-operator/3.log" Oct 30 12:19:04.161902 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:04.158792 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-hbpxv"] Oct 30 12:19:04.161902 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:04.159641 4908 cpu_manager.go:414] "containerMap: removing container" podUID="d3e63d4d-0257-41a0-9b25-24a4c7514b95" containerName="revoke-keys" Oct 30 12:19:04.161902 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:04.159664 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3e63d4d-0257-41a0-9b25-24a4c7514b95" containerName="revoke-keys" Oct 30 12:19:04.161902 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:04.159852 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3e63d4d-0257-41a0-9b25-24a4c7514b95" containerName="revoke-keys" Oct 30 12:19:04.225853 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:04.216103 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-hbpxv" Oct 30 12:19:04.282753 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:04.277893 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-hbpxv"] Oct 30 12:19:04.335407 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:04.301813 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e36676bf-3529-43a7-b65f-ddb63f3b9278-istiod-ca-cert\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-hbpxv\" (UID: \"e36676bf-3529-43a7-b65f-ddb63f3b9278\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-hbpxv" Oct 30 12:19:04.335823 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:04.335793 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e36676bf-3529-43a7-b65f-ddb63f3b9278-istio-token\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-hbpxv\" (UID: \"e36676bf-3529-43a7-b65f-ddb63f3b9278\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-hbpxv" Oct 30 12:19:04.336142 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:04.336122 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e36676bf-3529-43a7-b65f-ddb63f3b9278-istio-envoy\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-hbpxv\" (UID: \"e36676bf-3529-43a7-b65f-ddb63f3b9278\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-hbpxv" Oct 30 12:19:04.336471 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:04.336429 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e36676bf-3529-43a7-b65f-ddb63f3b9278-workload-socket\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-hbpxv\" (UID: \"e36676bf-3529-43a7-b65f-ddb63f3b9278\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-hbpxv" Oct 30 12:19:04.336569 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:04.336551 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e36676bf-3529-43a7-b65f-ddb63f3b9278-istio-data\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-hbpxv\" (UID: \"e36676bf-3529-43a7-b65f-ddb63f3b9278\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-hbpxv" Oct 30 12:19:04.336641 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:04.336610 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e36676bf-3529-43a7-b65f-ddb63f3b9278-workload-certs\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-hbpxv\" (UID: \"e36676bf-3529-43a7-b65f-ddb63f3b9278\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-hbpxv" Oct 30 12:19:04.336693 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:04.336638 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7d4j\" (UniqueName: \"kubernetes.io/projected/e36676bf-3529-43a7-b65f-ddb63f3b9278-kube-api-access-m7d4j\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-hbpxv\" (UID: \"e36676bf-3529-43a7-b65f-ddb63f3b9278\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-hbpxv" Oct 30 12:19:04.336693 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:04.336673 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e36676bf-3529-43a7-b65f-ddb63f3b9278-credential-socket\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-hbpxv\" (UID: \"e36676bf-3529-43a7-b65f-ddb63f3b9278\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-hbpxv" Oct 30 12:19:04.336807 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:04.336792 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e36676bf-3529-43a7-b65f-ddb63f3b9278-istio-podinfo\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-hbpxv\" (UID: \"e36676bf-3529-43a7-b65f-ddb63f3b9278\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-hbpxv" Oct 30 12:19:04.442908 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:04.439721 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e36676bf-3529-43a7-b65f-ddb63f3b9278-istio-data\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-hbpxv\" (UID: \"e36676bf-3529-43a7-b65f-ddb63f3b9278\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-hbpxv" Oct 30 12:19:04.442908 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:04.439791 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e36676bf-3529-43a7-b65f-ddb63f3b9278-workload-certs\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-hbpxv\" (UID: \"e36676bf-3529-43a7-b65f-ddb63f3b9278\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-hbpxv" Oct 30 12:19:04.442908 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:04.439817 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7d4j\" (UniqueName: \"kubernetes.io/projected/e36676bf-3529-43a7-b65f-ddb63f3b9278-kube-api-access-m7d4j\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-hbpxv\" (UID: \"e36676bf-3529-43a7-b65f-ddb63f3b9278\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-hbpxv" Oct 30 12:19:04.442908 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:04.439868 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e36676bf-3529-43a7-b65f-ddb63f3b9278-credential-socket\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-hbpxv\" (UID: \"e36676bf-3529-43a7-b65f-ddb63f3b9278\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-hbpxv" Oct 30 12:19:04.442908 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:04.439960 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e36676bf-3529-43a7-b65f-ddb63f3b9278-istio-podinfo\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-hbpxv\" (UID: \"e36676bf-3529-43a7-b65f-ddb63f3b9278\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-hbpxv" Oct 30 12:19:04.442908 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:04.440044 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e36676bf-3529-43a7-b65f-ddb63f3b9278-istiod-ca-cert\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-hbpxv\" (UID: \"e36676bf-3529-43a7-b65f-ddb63f3b9278\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-hbpxv" Oct 30 12:19:04.442908 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:04.440071 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e36676bf-3529-43a7-b65f-ddb63f3b9278-istio-token\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-hbpxv\" (UID: \"e36676bf-3529-43a7-b65f-ddb63f3b9278\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-hbpxv" Oct 30 12:19:04.442908 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:04.440125 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e36676bf-3529-43a7-b65f-ddb63f3b9278-istio-envoy\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-hbpxv\" (UID: \"e36676bf-3529-43a7-b65f-ddb63f3b9278\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-hbpxv" Oct 30 12:19:04.442908 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:04.440165 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e36676bf-3529-43a7-b65f-ddb63f3b9278-workload-socket\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-hbpxv\" (UID: \"e36676bf-3529-43a7-b65f-ddb63f3b9278\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-hbpxv" Oct 30 12:19:04.442908 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:04.440680 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e36676bf-3529-43a7-b65f-ddb63f3b9278-workload-socket\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-hbpxv\" (UID: \"e36676bf-3529-43a7-b65f-ddb63f3b9278\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-hbpxv" Oct 30 12:19:04.452447 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:04.452398 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e36676bf-3529-43a7-b65f-ddb63f3b9278-workload-certs\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-hbpxv\" (UID: \"e36676bf-3529-43a7-b65f-ddb63f3b9278\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-hbpxv" Oct 30 12:19:04.453442 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:04.453419 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e36676bf-3529-43a7-b65f-ddb63f3b9278-credential-socket\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-hbpxv\" (UID: \"e36676bf-3529-43a7-b65f-ddb63f3b9278\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-hbpxv" Oct 30 12:19:04.481882 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:04.474085 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e36676bf-3529-43a7-b65f-ddb63f3b9278-istiod-ca-cert\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-hbpxv\" (UID: \"e36676bf-3529-43a7-b65f-ddb63f3b9278\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-hbpxv" Oct 30 12:19:04.489839 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:04.489768 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21b8dc27-bptpx" event={"ID":"d3e63d4d-0257-41a0-9b25-24a4c7514b95","Type":"ContainerDied","Data":"38a69cb27068ac64b838ba37362f4e701b43deb3a8a55a6a6102e7a1d91b8233"} Oct 30 12:19:04.490067 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:04.490045 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="38a69cb27068ac64b838ba37362f4e701b43deb3a8a55a6a6102e7a1d91b8233" Oct 30 12:19:04.490554 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:04.490534 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21b8dc27-bptpx" Oct 30 12:19:04.492364 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:04.492346 4908 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b8bxz_644e19b3-10ad-4145-9ed3-595d19ed9b07/ovn-acl-logging/0.log" Oct 30 12:19:04.505874 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:04.505427 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e36676bf-3529-43a7-b65f-ddb63f3b9278-istio-podinfo\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-hbpxv\" (UID: \"e36676bf-3529-43a7-b65f-ddb63f3b9278\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-hbpxv" Oct 30 12:19:04.511336 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:04.511253 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7d4j\" (UniqueName: \"kubernetes.io/projected/e36676bf-3529-43a7-b65f-ddb63f3b9278-kube-api-access-m7d4j\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-hbpxv\" (UID: \"e36676bf-3529-43a7-b65f-ddb63f3b9278\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-hbpxv" Oct 30 12:19:04.541851 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:04.526890 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e36676bf-3529-43a7-b65f-ddb63f3b9278-istio-data\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-hbpxv\" (UID: \"e36676bf-3529-43a7-b65f-ddb63f3b9278\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-hbpxv" Oct 30 12:19:04.604943 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:04.601791 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e36676bf-3529-43a7-b65f-ddb63f3b9278-istio-token\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-hbpxv\" (UID: \"e36676bf-3529-43a7-b65f-ddb63f3b9278\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-hbpxv" Oct 30 12:19:04.604943 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:04.602567 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e36676bf-3529-43a7-b65f-ddb63f3b9278-istio-envoy\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-hbpxv\" (UID: \"e36676bf-3529-43a7-b65f-ddb63f3b9278\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-hbpxv" Oct 30 12:19:04.628621 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:04.627667 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-hbpxv" Oct 30 12:19:04.692852 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:04.679433 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-799d024f-6gftm"] Oct 30 12:19:04.774936 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:04.728644 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-799d024f-6gftm"] Oct 30 12:19:04.774936 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:04.728792 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-799d024f-6gftm" Oct 30 12:19:04.803205 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:04.795960 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 12:19:04.803205 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:04.796438 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s75hq\"" Oct 30 12:19:04.803205 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:04.797098 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 12:19:04.803205 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:04.797137 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 12:19:05.139017 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:05.112966 4908 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47fd99/distinct-2-w2-47fd99-kserve-77cf84b469-zxs8s" podUID="baf44d3d-42e6-43f7-a0ad-f0f9932320c1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:19:05.165370 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:05.165334 4908 patch_prober.go:29] interesting pod/e2e-worker-w0-f920f7-openshift-default-7c9767c97-tsxml container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 12:19:05.165648 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:05.165619 4908 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-tsxml" podUID="c8dc0dee-2192-42cf-a386-1a02e5ceb4f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 12:19:05.171134 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:05.171109 4908 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-plczt_a70ee11c-c20e-4a4f-8476-ac5eb02816d0/console-operator/3.log" Oct 30 12:19:05.315119 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:05.315081 4908 patch_prober.go:29] interesting pod/e2e-worker-w0-f920f7-openshift-default-7c9767c97-ljzls container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 12:19:05.315472 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:05.315439 4908 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-ljzls" podUID="6f0ed460-7572-4518-af84-589e8ac4ad29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 12:19:05.316725 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:05.316609 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6276351f-7c08-4c6d-b841-1d0e259c3fb8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-799d024f-6gftm\" (UID: \"6276351f-7c08-4c6d-b841-1d0e259c3fb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-799d024f-6gftm" Oct 30 12:19:05.317581 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:05.317351 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48hm7\" (UniqueName: \"kubernetes.io/projected/6276351f-7c08-4c6d-b841-1d0e259c3fb8-kube-api-access-48hm7\") pod \"maas-api-revoke-sub-799d024f-6gftm\" (UID: \"6276351f-7c08-4c6d-b841-1d0e259c3fb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-799d024f-6gftm" Oct 30 12:19:05.344104 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:05.344050 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6276351f-7c08-4c6d-b841-1d0e259c3fb8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-799d024f-6gftm\" (UID: \"6276351f-7c08-4c6d-b841-1d0e259c3fb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-799d024f-6gftm" Oct 30 12:19:05.385040 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:05.385008 4908 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b8bxz_644e19b3-10ad-4145-9ed3-595d19ed9b07/ovn-acl-logging/0.log" Oct 30 12:19:05.549486 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:05.532580 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6276351f-7c08-4c6d-b841-1d0e259c3fb8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-799d024f-6gftm\" (UID: \"6276351f-7c08-4c6d-b841-1d0e259c3fb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-799d024f-6gftm" Oct 30 12:19:05.549486 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:05.532639 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48hm7\" (UniqueName: \"kubernetes.io/projected/6276351f-7c08-4c6d-b841-1d0e259c3fb8-kube-api-access-48hm7\") pod \"maas-api-revoke-sub-799d024f-6gftm\" (UID: \"6276351f-7c08-4c6d-b841-1d0e259c3fb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-799d024f-6gftm" Oct 30 12:19:05.549486 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:05.532770 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6276351f-7c08-4c6d-b841-1d0e259c3fb8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-799d024f-6gftm\" (UID: \"6276351f-7c08-4c6d-b841-1d0e259c3fb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-799d024f-6gftm" Oct 30 12:19:05.549486 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:05.533966 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6276351f-7c08-4c6d-b841-1d0e259c3fb8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-799d024f-6gftm\" (UID: \"6276351f-7c08-4c6d-b841-1d0e259c3fb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-799d024f-6gftm" Oct 30 12:19:05.693853 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:05.673691 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48hm7\" (UniqueName: \"kubernetes.io/projected/6276351f-7c08-4c6d-b841-1d0e259c3fb8-kube-api-access-48hm7\") pod \"maas-api-revoke-sub-799d024f-6gftm\" (UID: \"6276351f-7c08-4c6d-b841-1d0e259c3fb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-799d024f-6gftm" Oct 30 12:19:05.748863 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:05.737923 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6276351f-7c08-4c6d-b841-1d0e259c3fb8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-799d024f-6gftm\" (UID: \"6276351f-7c08-4c6d-b841-1d0e259c3fb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-799d024f-6gftm" Oct 30 12:19:05.847088 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:05.846990 4908 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47fd99/premium-w2-47fd99-kserve-5d85f9bb95-hqkmf" podUID="fd2cbeda-dbc4-41f6-a730-e1c6f9343c5e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 12:19:05.848738 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:05.848707 4908 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6a7148/premium-w3-6a7148-kserve-67b976c888-tgd6c" podUID="13de4441-e7f6-4cbd-97d9-0f079d11cb8e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:19:06.033888 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:06.033607 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-799d024f-6gftm" Oct 30 12:19:06.199723 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:06.198959 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-hbpxv"] Oct 30 12:19:06.212665 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:06.212627 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:19:06.212790 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:06.212717 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:19:06.212887 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:06.212807 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:19:06.490814 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:06.490776 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-799d024f-6gftm"] Oct 30 12:19:06.497290 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:19:06.496271 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6276351f_7c08_4c6d_b841_1d0e259c3fb8.slice/crio-9fdf04b35794d56f43c6bc4721849835d5d176112201b388d2593cc1f96a485e WatchSource:0}: Error finding container 9fdf04b35794d56f43c6bc4721849835d5d176112201b388d2593cc1f96a485e: Status 404 returned error can't find the container with id 9fdf04b35794d56f43c6bc4721849835d5d176112201b388d2593cc1f96a485e Oct 30 12:19:06.568988 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:06.568950 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-sqv2j"] Oct 30 12:19:06.575717 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:06.575678 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-sqv2j" Oct 30 12:19:06.614993 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:06.614955 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-sqv2j"] Oct 30 12:19:06.620985 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:06.619504 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-gk4rg"] Oct 30 12:19:06.624034 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:06.623998 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-gk4rg" Oct 30 12:19:06.630655 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:06.630617 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-gk4rg"] Oct 30 12:19:06.669719 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:06.669680 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f110318-3125-4a76-ba6e-b9b0e6861051-workload-socket\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-sqv2j\" (UID: \"8f110318-3125-4a76-ba6e-b9b0e6861051\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-sqv2j" Oct 30 12:19:06.669932 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:06.669728 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ac0ce00-fe15-408d-90eb-9633ea3e67d0-workload-certs\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-gk4rg\" (UID: \"9ac0ce00-fe15-408d-90eb-9633ea3e67d0\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-gk4rg" Oct 30 12:19:06.669932 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:06.669753 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ac0ce00-fe15-408d-90eb-9633ea3e67d0-istiod-ca-cert\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-gk4rg\" (UID: \"9ac0ce00-fe15-408d-90eb-9633ea3e67d0\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-gk4rg" Oct 30 12:19:06.669932 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:06.669784 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f110318-3125-4a76-ba6e-b9b0e6861051-istiod-ca-cert\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-sqv2j\" (UID: \"8f110318-3125-4a76-ba6e-b9b0e6861051\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-sqv2j" Oct 30 12:19:06.669932 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:06.669882 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f110318-3125-4a76-ba6e-b9b0e6861051-istio-envoy\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-sqv2j\" (UID: \"8f110318-3125-4a76-ba6e-b9b0e6861051\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-sqv2j" Oct 30 12:19:06.670148 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:06.669946 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzcf5\" (UniqueName: \"kubernetes.io/projected/9ac0ce00-fe15-408d-90eb-9633ea3e67d0-kube-api-access-fzcf5\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-gk4rg\" (UID: \"9ac0ce00-fe15-408d-90eb-9633ea3e67d0\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-gk4rg" Oct 30 12:19:06.670148 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:06.669995 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ac0ce00-fe15-408d-90eb-9633ea3e67d0-istio-podinfo\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-gk4rg\" (UID: \"9ac0ce00-fe15-408d-90eb-9633ea3e67d0\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-gk4rg" Oct 30 12:19:06.670148 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:06.670035 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qhjn\" (UniqueName: \"kubernetes.io/projected/8f110318-3125-4a76-ba6e-b9b0e6861051-kube-api-access-7qhjn\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-sqv2j\" (UID: \"8f110318-3125-4a76-ba6e-b9b0e6861051\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-sqv2j" Oct 30 12:19:06.670148 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:06.670091 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ac0ce00-fe15-408d-90eb-9633ea3e67d0-credential-socket\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-gk4rg\" (UID: \"9ac0ce00-fe15-408d-90eb-9633ea3e67d0\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-gk4rg" Oct 30 12:19:06.670943 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:06.670498 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f110318-3125-4a76-ba6e-b9b0e6861051-credential-socket\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-sqv2j\" (UID: \"8f110318-3125-4a76-ba6e-b9b0e6861051\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-sqv2j" Oct 30 12:19:06.670943 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:06.670557 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ac0ce00-fe15-408d-90eb-9633ea3e67d0-istio-data\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-gk4rg\" (UID: \"9ac0ce00-fe15-408d-90eb-9633ea3e67d0\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-gk4rg" Oct 30 12:19:06.670943 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:06.670683 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ac0ce00-fe15-408d-90eb-9633ea3e67d0-istio-envoy\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-gk4rg\" (UID: \"9ac0ce00-fe15-408d-90eb-9633ea3e67d0\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-gk4rg" Oct 30 12:19:06.670943 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:06.670852 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f110318-3125-4a76-ba6e-b9b0e6861051-workload-certs\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-sqv2j\" (UID: \"8f110318-3125-4a76-ba6e-b9b0e6861051\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-sqv2j" Oct 30 12:19:06.670943 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:06.670903 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f110318-3125-4a76-ba6e-b9b0e6861051-istio-podinfo\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-sqv2j\" (UID: \"8f110318-3125-4a76-ba6e-b9b0e6861051\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-sqv2j" Oct 30 12:19:06.670943 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:06.670942 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f110318-3125-4a76-ba6e-b9b0e6861051-istio-data\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-sqv2j\" (UID: \"8f110318-3125-4a76-ba6e-b9b0e6861051\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-sqv2j" Oct 30 12:19:06.671392 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:06.670996 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f110318-3125-4a76-ba6e-b9b0e6861051-istio-token\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-sqv2j\" (UID: \"8f110318-3125-4a76-ba6e-b9b0e6861051\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-sqv2j" Oct 30 12:19:06.671392 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:06.671057 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ac0ce00-fe15-408d-90eb-9633ea3e67d0-workload-socket\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-gk4rg\" (UID: \"9ac0ce00-fe15-408d-90eb-9633ea3e67d0\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-gk4rg" Oct 30 12:19:06.671392 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:06.671098 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ac0ce00-fe15-408d-90eb-9633ea3e67d0-istio-token\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-gk4rg\" (UID: \"9ac0ce00-fe15-408d-90eb-9633ea3e67d0\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-gk4rg" Oct 30 12:19:06.772152 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:06.772118 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f110318-3125-4a76-ba6e-b9b0e6861051-workload-socket\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-sqv2j\" (UID: \"8f110318-3125-4a76-ba6e-b9b0e6861051\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-sqv2j" Oct 30 12:19:06.772334 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:06.772159 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ac0ce00-fe15-408d-90eb-9633ea3e67d0-workload-certs\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-gk4rg\" (UID: \"9ac0ce00-fe15-408d-90eb-9633ea3e67d0\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-gk4rg" Oct 30 12:19:06.772334 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:06.772183 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ac0ce00-fe15-408d-90eb-9633ea3e67d0-istiod-ca-cert\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-gk4rg\" (UID: \"9ac0ce00-fe15-408d-90eb-9633ea3e67d0\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-gk4rg" Oct 30 12:19:06.772334 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:06.772226 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f110318-3125-4a76-ba6e-b9b0e6861051-istiod-ca-cert\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-sqv2j\" (UID: \"8f110318-3125-4a76-ba6e-b9b0e6861051\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-sqv2j" Oct 30 12:19:06.772334 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:06.772274 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f110318-3125-4a76-ba6e-b9b0e6861051-istio-envoy\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-sqv2j\" (UID: \"8f110318-3125-4a76-ba6e-b9b0e6861051\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-sqv2j" Oct 30 12:19:06.772334 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:06.772325 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzcf5\" (UniqueName: \"kubernetes.io/projected/9ac0ce00-fe15-408d-90eb-9633ea3e67d0-kube-api-access-fzcf5\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-gk4rg\" (UID: \"9ac0ce00-fe15-408d-90eb-9633ea3e67d0\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-gk4rg" Oct 30 12:19:06.772592 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:06.772368 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ac0ce00-fe15-408d-90eb-9633ea3e67d0-istio-podinfo\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-gk4rg\" (UID: \"9ac0ce00-fe15-408d-90eb-9633ea3e67d0\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-gk4rg" Oct 30 12:19:06.772592 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:06.772403 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qhjn\" (UniqueName: \"kubernetes.io/projected/8f110318-3125-4a76-ba6e-b9b0e6861051-kube-api-access-7qhjn\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-sqv2j\" (UID: \"8f110318-3125-4a76-ba6e-b9b0e6861051\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-sqv2j" Oct 30 12:19:06.772592 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:06.772440 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ac0ce00-fe15-408d-90eb-9633ea3e67d0-credential-socket\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-gk4rg\" (UID: \"9ac0ce00-fe15-408d-90eb-9633ea3e67d0\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-gk4rg" Oct 30 12:19:06.772592 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:06.772472 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f110318-3125-4a76-ba6e-b9b0e6861051-credential-socket\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-sqv2j\" (UID: \"8f110318-3125-4a76-ba6e-b9b0e6861051\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-sqv2j" Oct 30 12:19:06.772592 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:06.772499 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ac0ce00-fe15-408d-90eb-9633ea3e67d0-istio-data\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-gk4rg\" (UID: \"9ac0ce00-fe15-408d-90eb-9633ea3e67d0\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-gk4rg" Oct 30 12:19:06.772592 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:06.772532 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ac0ce00-fe15-408d-90eb-9633ea3e67d0-istio-envoy\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-gk4rg\" (UID: \"9ac0ce00-fe15-408d-90eb-9633ea3e67d0\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-gk4rg" Oct 30 12:19:06.772592 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:06.772583 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f110318-3125-4a76-ba6e-b9b0e6861051-workload-certs\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-sqv2j\" (UID: \"8f110318-3125-4a76-ba6e-b9b0e6861051\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-sqv2j" Oct 30 12:19:06.772944 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:06.772604 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f110318-3125-4a76-ba6e-b9b0e6861051-workload-socket\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-sqv2j\" (UID: \"8f110318-3125-4a76-ba6e-b9b0e6861051\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-sqv2j" Oct 30 12:19:06.772944 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:06.772627 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f110318-3125-4a76-ba6e-b9b0e6861051-istio-podinfo\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-sqv2j\" (UID: \"8f110318-3125-4a76-ba6e-b9b0e6861051\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-sqv2j" Oct 30 12:19:06.772944 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:06.772663 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f110318-3125-4a76-ba6e-b9b0e6861051-istio-data\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-sqv2j\" (UID: \"8f110318-3125-4a76-ba6e-b9b0e6861051\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-sqv2j" Oct 30 12:19:06.772944 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:06.772718 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f110318-3125-4a76-ba6e-b9b0e6861051-istio-token\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-sqv2j\" (UID: \"8f110318-3125-4a76-ba6e-b9b0e6861051\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-sqv2j" Oct 30 12:19:06.772944 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:06.772768 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ac0ce00-fe15-408d-90eb-9633ea3e67d0-workload-socket\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-gk4rg\" (UID: \"9ac0ce00-fe15-408d-90eb-9633ea3e67d0\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-gk4rg" Oct 30 12:19:06.772944 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:06.772809 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ac0ce00-fe15-408d-90eb-9633ea3e67d0-istio-token\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-gk4rg\" (UID: \"9ac0ce00-fe15-408d-90eb-9633ea3e67d0\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-gk4rg" Oct 30 12:19:06.773223 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:06.773081 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ac0ce00-fe15-408d-90eb-9633ea3e67d0-istiod-ca-cert\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-gk4rg\" (UID: \"9ac0ce00-fe15-408d-90eb-9633ea3e67d0\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-gk4rg" Oct 30 12:19:06.773392 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:06.773367 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f110318-3125-4a76-ba6e-b9b0e6861051-workload-certs\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-sqv2j\" (UID: \"8f110318-3125-4a76-ba6e-b9b0e6861051\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-sqv2j" Oct 30 12:19:06.773474 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:06.773450 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ac0ce00-fe15-408d-90eb-9633ea3e67d0-credential-socket\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-gk4rg\" (UID: \"9ac0ce00-fe15-408d-90eb-9633ea3e67d0\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-gk4rg" Oct 30 12:19:06.773627 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:06.773608 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f110318-3125-4a76-ba6e-b9b0e6861051-credential-socket\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-sqv2j\" (UID: \"8f110318-3125-4a76-ba6e-b9b0e6861051\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-sqv2j" Oct 30 12:19:06.773741 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:06.773713 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f110318-3125-4a76-ba6e-b9b0e6861051-istiod-ca-cert\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-sqv2j\" (UID: \"8f110318-3125-4a76-ba6e-b9b0e6861051\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-sqv2j" Oct 30 12:19:06.773879 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:06.773851 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ac0ce00-fe15-408d-90eb-9633ea3e67d0-istio-data\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-gk4rg\" (UID: \"9ac0ce00-fe15-408d-90eb-9633ea3e67d0\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-gk4rg" Oct 30 12:19:06.774503 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:06.774421 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f110318-3125-4a76-ba6e-b9b0e6861051-istio-data\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-sqv2j\" (UID: \"8f110318-3125-4a76-ba6e-b9b0e6861051\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-sqv2j" Oct 30 12:19:06.774656 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:06.774634 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ac0ce00-fe15-408d-90eb-9633ea3e67d0-workload-socket\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-gk4rg\" (UID: \"9ac0ce00-fe15-408d-90eb-9633ea3e67d0\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-gk4rg" Oct 30 12:19:06.774963 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:06.774940 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ac0ce00-fe15-408d-90eb-9633ea3e67d0-workload-certs\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-gk4rg\" (UID: \"9ac0ce00-fe15-408d-90eb-9633ea3e67d0\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-gk4rg" Oct 30 12:19:06.776517 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:06.776497 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f110318-3125-4a76-ba6e-b9b0e6861051-istio-podinfo\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-sqv2j\" (UID: \"8f110318-3125-4a76-ba6e-b9b0e6861051\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-sqv2j" Oct 30 12:19:06.776695 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:06.776676 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f110318-3125-4a76-ba6e-b9b0e6861051-istio-envoy\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-sqv2j\" (UID: \"8f110318-3125-4a76-ba6e-b9b0e6861051\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-sqv2j" Oct 30 12:19:06.776903 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:06.776879 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ac0ce00-fe15-408d-90eb-9633ea3e67d0-istio-podinfo\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-gk4rg\" (UID: \"9ac0ce00-fe15-408d-90eb-9633ea3e67d0\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-gk4rg" Oct 30 12:19:06.777040 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:06.777022 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ac0ce00-fe15-408d-90eb-9633ea3e67d0-istio-envoy\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-gk4rg\" (UID: \"9ac0ce00-fe15-408d-90eb-9633ea3e67d0\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-gk4rg" Oct 30 12:19:06.795501 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:06.795347 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ac0ce00-fe15-408d-90eb-9633ea3e67d0-istio-token\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-gk4rg\" (UID: \"9ac0ce00-fe15-408d-90eb-9633ea3e67d0\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-gk4rg" Oct 30 12:19:06.795501 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:06.795471 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qhjn\" (UniqueName: \"kubernetes.io/projected/8f110318-3125-4a76-ba6e-b9b0e6861051-kube-api-access-7qhjn\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-sqv2j\" (UID: \"8f110318-3125-4a76-ba6e-b9b0e6861051\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-sqv2j" Oct 30 12:19:06.795810 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:06.795786 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f110318-3125-4a76-ba6e-b9b0e6861051-istio-token\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-sqv2j\" (UID: \"8f110318-3125-4a76-ba6e-b9b0e6861051\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-sqv2j" Oct 30 12:19:06.796367 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:06.796339 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzcf5\" (UniqueName: \"kubernetes.io/projected/9ac0ce00-fe15-408d-90eb-9633ea3e67d0-kube-api-access-fzcf5\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-gk4rg\" (UID: \"9ac0ce00-fe15-408d-90eb-9633ea3e67d0\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-gk4rg" Oct 30 12:19:06.903882 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:06.903816 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-sqv2j" Oct 30 12:19:06.949368 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:06.949332 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-gk4rg" Oct 30 12:19:07.058366 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:07.057354 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-799d024f-6gftm" event={"ID":"6276351f-7c08-4c6d-b841-1d0e259c3fb8","Type":"ContainerStarted","Data":"4434116c2ca20b1f86d50f26d6f26123623105046510f0c5c2eae5b4f45417ec"} Oct 30 12:19:07.058366 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:07.057421 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-799d024f-6gftm" event={"ID":"6276351f-7c08-4c6d-b841-1d0e259c3fb8","Type":"ContainerStarted","Data":"9fdf04b35794d56f43c6bc4721849835d5d176112201b388d2593cc1f96a485e"} Oct 30 12:19:07.064149 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:07.060759 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-hbpxv" event={"ID":"e36676bf-3529-43a7-b65f-ddb63f3b9278","Type":"ContainerStarted","Data":"e27cf3a01e8bd3563f2b72a43bfb22f9db8711899c13825047573c48bcdbf8d9"} Oct 30 12:19:07.064149 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:07.060841 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-hbpxv" event={"ID":"e36676bf-3529-43a7-b65f-ddb63f3b9278","Type":"ContainerStarted","Data":"4981c804ef11b5a353f31b3736310360e91fa62518e7e76937b1a76a65b9c035"} Oct 30 12:19:07.123688 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:07.123626 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-799d024f-6gftm" podStartSLOduration=3.123606847 podStartE2EDuration="3.123606847s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:19:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:19:07.119394338 +0000 UTC m=+1503.633823085" watchObservedRunningTime="2026-10-30 12:19:07.123606847 +0000 UTC m=+1503.638035594" Oct 30 12:19:07.197763 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:07.197700 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-hbpxv" podStartSLOduration=3.197678768 podStartE2EDuration="3.197678768s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:19:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:19:07.184213047 +0000 UTC m=+1503.698641796" watchObservedRunningTime="2026-10-30 12:19:07.197678768 +0000 UTC m=+1503.712107514" Oct 30 12:19:07.240050 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:07.240023 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-sqv2j"] Oct 30 12:19:07.241689 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:19:07.241661 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f110318_3125_4a76_ba6e_b9b0e6861051.slice/crio-706623eddc8ecad79b4a616106169db48c0729848a9b86584db6e8398dbe766e WatchSource:0}: Error finding container 706623eddc8ecad79b4a616106169db48c0729848a9b86584db6e8398dbe766e: Status 404 returned error can't find the container with id 706623eddc8ecad79b4a616106169db48c0729848a9b86584db6e8398dbe766e Oct 30 12:19:07.244335 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:07.244303 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:19:07.244393 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:07.244369 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:19:07.244440 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:07.244395 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:19:07.309282 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:07.308523 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:19:07.309282 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:07.308642 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:19:07.309282 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:07.308685 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:19:07.309515 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:07.309357 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-gk4rg"] Oct 30 12:19:07.628723 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:07.628693 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-hbpxv" Oct 30 12:19:07.629960 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:07.629667 4908 patch_prober.go:29] interesting pod/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-hbpxv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 30 12:19:07.629960 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:07.629719 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-hbpxv" podUID="e36676bf-3529-43a7-b65f-ddb63f3b9278" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 30 12:19:07.940280 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:19:07.940201 4908 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3e63d4d_0257_41a0_9b25_24a4c7514b95.slice\": RecentStats: unable to find data in memory cache]" Oct 30 12:19:08.051188 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:08.051132 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-6a7148/distinct-2-w3-6a7148-kserve-67bd5fb976-9nfff" Oct 30 12:19:08.100363 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:08.100320 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-gk4rg" event={"ID":"9ac0ce00-fe15-408d-90eb-9633ea3e67d0","Type":"ContainerStarted","Data":"763181d0f06ab314d19a69c97efef3ce3e5f17327808faa7187c9d8d99668eb6"} Oct 30 12:19:08.100672 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:08.100656 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-gk4rg" event={"ID":"9ac0ce00-fe15-408d-90eb-9633ea3e67d0","Type":"ContainerStarted","Data":"f22db1aea7ee9ed274a2deae674830cc32fadea3fc21d20484d4ef0bd6bbbc4d"} Oct 30 12:19:08.102792 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:08.102756 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-6a7148/distinct-2-w3-6a7148-kserve-67bd5fb976-9nfff" Oct 30 12:19:08.102927 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:08.102807 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-sqv2j" event={"ID":"8f110318-3125-4a76-ba6e-b9b0e6861051","Type":"ContainerStarted","Data":"c090aea8285182d5fe43510bf0bb33c033249b4dc80999f1896128a59d9c7615"} Oct 30 12:19:08.102927 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:08.102867 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-sqv2j" event={"ID":"8f110318-3125-4a76-ba6e-b9b0e6861051","Type":"ContainerStarted","Data":"706623eddc8ecad79b4a616106169db48c0729848a9b86584db6e8398dbe766e"} Oct 30 12:19:08.105146 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:08.105121 4908 generic.go:356] "Generic (PLEG): container finished" podID="6276351f-7c08-4c6d-b841-1d0e259c3fb8" containerID="4434116c2ca20b1f86d50f26d6f26123623105046510f0c5c2eae5b4f45417ec" exitCode=0 Oct 30 12:19:08.105331 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:08.105185 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-799d024f-6gftm" event={"ID":"6276351f-7c08-4c6d-b841-1d0e259c3fb8","Type":"ContainerDied","Data":"4434116c2ca20b1f86d50f26d6f26123623105046510f0c5c2eae5b4f45417ec"} Oct 30 12:19:08.119671 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:08.119640 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-6a7148/distinct-2-w3-6a7148-kserve-67bd5fb976-9nfff" Oct 30 12:19:08.147191 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:08.147137 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-gk4rg" podStartSLOduration=2.147122923 podStartE2EDuration="2.147122923s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:19:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:19:08.142611476 +0000 UTC m=+1504.657040222" watchObservedRunningTime="2026-10-30 12:19:08.147122923 +0000 UTC m=+1504.661551669" Oct 30 12:19:08.197226 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:08.197097 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-sqv2j" podStartSLOduration=2.197079961 podStartE2EDuration="2.197079961s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:19:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:19:08.186399567 +0000 UTC m=+1504.700828314" watchObservedRunningTime="2026-10-30 12:19:08.197079961 +0000 UTC m=+1504.711508707" Oct 30 12:19:08.628865 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:08.628247 4908 patch_prober.go:29] interesting pod/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-hbpxv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 30 12:19:08.628865 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:08.628311 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-hbpxv" podUID="e36676bf-3529-43a7-b65f-ddb63f3b9278" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 30 12:19:08.905281 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:08.905183 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-sqv2j" Oct 30 12:19:08.924572 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:08.924513 4908 patch_prober.go:29] interesting pod/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-sqv2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Oct 30 12:19:08.924984 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:08.924934 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-sqv2j" podUID="8f110318-3125-4a76-ba6e-b9b0e6861051" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Oct 30 12:19:08.952237 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:08.952200 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-gk4rg" Oct 30 12:19:08.954965 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:08.954938 4908 patch_prober.go:29] interesting pod/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-gk4rg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Oct 30 12:19:08.955169 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:08.955149 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-gk4rg" podUID="9ac0ce00-fe15-408d-90eb-9633ea3e67d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Oct 30 12:19:09.385858 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:09.373642 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e57de56-tns89"] Oct 30 12:19:09.656913 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:09.635116 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e57de56-tns89"] Oct 30 12:19:09.656913 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:09.655549 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e57de56-tns89" Oct 30 12:19:09.730928 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:09.703980 4908 patch_prober.go:29] interesting pod/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-hbpxv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 30 12:19:09.730928 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:09.704050 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-hbpxv" podUID="e36676bf-3529-43a7-b65f-ddb63f3b9278" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 30 12:19:09.905227 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:09.904703 4908 patch_prober.go:29] interesting pod/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-sqv2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Oct 30 12:19:09.905227 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:09.904767 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-sqv2j" podUID="8f110318-3125-4a76-ba6e-b9b0e6861051" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Oct 30 12:19:09.951193 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:19:09.951098 4908 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3e63d4d_0257_41a0_9b25_24a4c7514b95.slice\": RecentStats: unable to find data in memory cache]" Oct 30 12:19:09.993855 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:09.953560 4908 patch_prober.go:29] interesting pod/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-gk4rg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Oct 30 12:19:09.993855 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:09.953626 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-gk4rg" podUID="9ac0ce00-fe15-408d-90eb-9633ea3e67d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Oct 30 12:19:10.057870 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:10.053636 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21d19651-5c90-47ba-acf5-5004aa8e11f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e57de56-tns89\" (UID: \"21d19651-5c90-47ba-acf5-5004aa8e11f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e57de56-tns89" Oct 30 12:19:10.057870 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:10.053746 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trht6\" (UniqueName: \"kubernetes.io/projected/21d19651-5c90-47ba-acf5-5004aa8e11f5-kube-api-access-trht6\") pod \"maas-api-revoke-sub-3e57de56-tns89\" (UID: \"21d19651-5c90-47ba-acf5-5004aa8e11f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e57de56-tns89" Oct 30 12:19:10.057870 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:10.053966 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21d19651-5c90-47ba-acf5-5004aa8e11f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e57de56-tns89\" (UID: \"21d19651-5c90-47ba-acf5-5004aa8e11f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e57de56-tns89" Oct 30 12:19:10.178853 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:10.169007 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21d19651-5c90-47ba-acf5-5004aa8e11f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e57de56-tns89\" (UID: \"21d19651-5c90-47ba-acf5-5004aa8e11f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e57de56-tns89" Oct 30 12:19:10.178853 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:10.169133 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21d19651-5c90-47ba-acf5-5004aa8e11f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e57de56-tns89\" (UID: \"21d19651-5c90-47ba-acf5-5004aa8e11f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e57de56-tns89" Oct 30 12:19:10.178853 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:10.169194 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trht6\" (UniqueName: \"kubernetes.io/projected/21d19651-5c90-47ba-acf5-5004aa8e11f5-kube-api-access-trht6\") pod \"maas-api-revoke-sub-3e57de56-tns89\" (UID: \"21d19651-5c90-47ba-acf5-5004aa8e11f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e57de56-tns89" Oct 30 12:19:10.178853 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:10.172621 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21d19651-5c90-47ba-acf5-5004aa8e11f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e57de56-tns89\" (UID: \"21d19651-5c90-47ba-acf5-5004aa8e11f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e57de56-tns89" Oct 30 12:19:10.205918 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:10.184676 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21d19651-5c90-47ba-acf5-5004aa8e11f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e57de56-tns89\" (UID: \"21d19651-5c90-47ba-acf5-5004aa8e11f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e57de56-tns89" Oct 30 12:19:10.234861 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:10.217590 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-trht6\" (UniqueName: \"kubernetes.io/projected/21d19651-5c90-47ba-acf5-5004aa8e11f5-kube-api-access-trht6\") pod \"maas-api-revoke-sub-3e57de56-tns89\" (UID: \"21d19651-5c90-47ba-acf5-5004aa8e11f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e57de56-tns89" Oct 30 12:19:10.235113 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:10.234057 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-799d024f-6gftm" Oct 30 12:19:10.255853 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:10.250581 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-799d024f-6gftm" event={"ID":"6276351f-7c08-4c6d-b841-1d0e259c3fb8","Type":"ContainerDied","Data":"9fdf04b35794d56f43c6bc4721849835d5d176112201b388d2593cc1f96a485e"} Oct 30 12:19:10.255853 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:10.250644 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9fdf04b35794d56f43c6bc4721849835d5d176112201b388d2593cc1f96a485e" Oct 30 12:19:10.255853 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:10.252721 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e57de56-tns89" Oct 30 12:19:10.405855 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:10.401287 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6276351f-7c08-4c6d-b841-1d0e259c3fb8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6276351f-7c08-4c6d-b841-1d0e259c3fb8-maas-api-xks-ca\") pod \"6276351f-7c08-4c6d-b841-1d0e259c3fb8\" (UID: \"6276351f-7c08-4c6d-b841-1d0e259c3fb8\") " Oct 30 12:19:10.450886 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:10.416402 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6276351f-7c08-4c6d-b841-1d0e259c3fb8-kube-api-access-48hm7\" (UniqueName: \"kubernetes.io/projected/6276351f-7c08-4c6d-b841-1d0e259c3fb8-kube-api-access-48hm7\") pod \"6276351f-7c08-4c6d-b841-1d0e259c3fb8\" (UID: \"6276351f-7c08-4c6d-b841-1d0e259c3fb8\") " Oct 30 12:19:10.454131 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:10.454080 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6276351f-7c08-4c6d-b841-1d0e259c3fb8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6276351f-7c08-4c6d-b841-1d0e259c3fb8-maas-api-openshift-ca\") pod \"6276351f-7c08-4c6d-b841-1d0e259c3fb8\" (UID: \"6276351f-7c08-4c6d-b841-1d0e259c3fb8\") " Oct 30 12:19:10.454897 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:10.432653 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6276351f-7c08-4c6d-b841-1d0e259c3fb8-maas-api-xks-ca" pod "6276351f-7c08-4c6d-b841-1d0e259c3fb8" (UID: "6276351f-7c08-4c6d-b841-1d0e259c3fb8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:19:10.455456 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:10.455395 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6276351f-7c08-4c6d-b841-1d0e259c3fb8-maas-api-xks-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:19:10.456846 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:10.456753 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6276351f-7c08-4c6d-b841-1d0e259c3fb8-maas-api-openshift-ca" pod "6276351f-7c08-4c6d-b841-1d0e259c3fb8" (UID: "6276351f-7c08-4c6d-b841-1d0e259c3fb8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:19:10.458231 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:10.458211 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6276351f-7c08-4c6d-b841-1d0e259c3fb8-kube-api-access-48hm7" pod "6276351f-7c08-4c6d-b841-1d0e259c3fb8" (UID: "6276351f-7c08-4c6d-b841-1d0e259c3fb8"). InnerVolumeSpecName "kube-api-access-48hm7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:19:10.557157 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:10.557118 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-48hm7\" (UniqueName: \"kubernetes.io/projected/6276351f-7c08-4c6d-b841-1d0e259c3fb8-kube-api-access-48hm7\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:19:10.557395 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:10.557378 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6276351f-7c08-4c6d-b841-1d0e259c3fb8-maas-api-openshift-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:19:10.631862 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:10.631805 4908 patch_prober.go:29] interesting pod/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-hbpxv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 30 12:19:10.632123 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:10.632091 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-hbpxv" podUID="e36676bf-3529-43a7-b65f-ddb63f3b9278" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 30 12:19:10.939050 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:10.939012 4908 patch_prober.go:29] interesting pod/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-sqv2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Oct 30 12:19:10.939788 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:10.939758 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-sqv2j" podUID="8f110318-3125-4a76-ba6e-b9b0e6861051" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Oct 30 12:19:10.950331 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:10.950304 4908 patch_prober.go:29] interesting pod/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-gk4rg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Oct 30 12:19:10.950500 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:10.950481 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-gk4rg" podUID="9ac0ce00-fe15-408d-90eb-9633ea3e67d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Oct 30 12:19:11.145898 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:11.144329 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e57de56-tns89"] Oct 30 12:19:11.458598 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:11.458558 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-799d024f-6gftm" Oct 30 12:19:11.496916 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:11.474446 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e57de56-tns89" event={"ID":"21d19651-5c90-47ba-acf5-5004aa8e11f5","Type":"ContainerStarted","Data":"8fe22ae33ffceb5dfe3b9e7bc02aa5aee0b613115ea8a1764943f8d9a22521cd"} Oct 30 12:19:11.648719 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:11.647577 4908 patch_prober.go:29] interesting pod/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-hbpxv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 30 12:19:11.648719 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:11.647640 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-hbpxv" podUID="e36676bf-3529-43a7-b65f-ddb63f3b9278" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 30 12:19:11.910854 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:11.908261 4908 patch_prober.go:29] interesting pod/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-sqv2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Oct 30 12:19:11.910854 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:11.908331 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-sqv2j" podUID="8f110318-3125-4a76-ba6e-b9b0e6861051" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Oct 30 12:19:11.951520 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:11.951334 4908 patch_prober.go:29] interesting pod/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-gk4rg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Oct 30 12:19:11.951520 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:11.951402 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-gk4rg" podUID="9ac0ce00-fe15-408d-90eb-9633ea3e67d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Oct 30 12:19:12.546855 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:12.545135 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e57de56-tns89" event={"ID":"21d19651-5c90-47ba-acf5-5004aa8e11f5","Type":"ContainerStarted","Data":"015d2e03956594024194f11affa0ded6923d588575dc88e6a9740203b15b23d0"} Oct 30 12:19:12.610904 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:12.604012 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e57de56-tns89" podStartSLOduration=3.603991413 podStartE2EDuration="3.603991413s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:19:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:19:12.601569774 +0000 UTC m=+1509.115998521" watchObservedRunningTime="2026-10-30 12:19:12.603991413 +0000 UTC m=+1509.118420161" Oct 30 12:19:12.650685 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:12.634227 4908 patch_prober.go:29] interesting pod/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-hbpxv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 30 12:19:12.651030 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:12.650988 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-hbpxv" podUID="e36676bf-3529-43a7-b65f-ddb63f3b9278" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 30 12:19:12.905327 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:12.905228 4908 patch_prober.go:29] interesting pod/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-sqv2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Oct 30 12:19:12.905611 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:12.905570 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-sqv2j" podUID="8f110318-3125-4a76-ba6e-b9b0e6861051" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Oct 30 12:19:12.949890 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:12.949856 4908 patch_prober.go:29] interesting pod/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-gk4rg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Oct 30 12:19:12.950075 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:12.949913 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-gk4rg" podUID="9ac0ce00-fe15-408d-90eb-9633ea3e67d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Oct 30 12:19:13.595949 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:13.595375 4908 generic.go:356] "Generic (PLEG): container finished" podID="21d19651-5c90-47ba-acf5-5004aa8e11f5" containerID="015d2e03956594024194f11affa0ded6923d588575dc88e6a9740203b15b23d0" exitCode=0 Oct 30 12:19:13.595949 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:13.595512 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e57de56-tns89" event={"ID":"21d19651-5c90-47ba-acf5-5004aa8e11f5","Type":"ContainerDied","Data":"015d2e03956594024194f11affa0ded6923d588575dc88e6a9740203b15b23d0"} Oct 30 12:19:13.631388 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:13.631350 4908 patch_prober.go:29] interesting pod/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-hbpxv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 30 12:19:13.631655 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:13.631622 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-hbpxv" podUID="e36676bf-3529-43a7-b65f-ddb63f3b9278" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 30 12:19:13.970958 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:13.957335 4908 patch_prober.go:29] interesting pod/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-sqv2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Oct 30 12:19:13.970958 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:13.957400 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-sqv2j" podUID="8f110318-3125-4a76-ba6e-b9b0e6861051" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Oct 30 12:19:14.006027 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:14.005957 4908 patch_prober.go:29] interesting pod/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-gk4rg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Oct 30 12:19:14.006027 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:14.006031 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-gk4rg" podUID="9ac0ce00-fe15-408d-90eb-9633ea3e67d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Oct 30 12:19:14.726180 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:14.724136 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-hbpxv" Oct 30 12:19:14.726180 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:14.725786 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-hbpxv" Oct 30 12:19:14.733370 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:14.732992 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-hbpxv" Oct 30 12:19:14.819232 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:14.819202 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e57de56-tns89" Oct 30 12:19:14.904567 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:14.904452 4908 patch_prober.go:29] interesting pod/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-sqv2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Oct 30 12:19:14.904567 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:14.904510 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-sqv2j" podUID="8f110318-3125-4a76-ba6e-b9b0e6861051" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Oct 30 12:19:14.938603 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:14.938535 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21d19651-5c90-47ba-acf5-5004aa8e11f5-kube-api-access-trht6\" (UniqueName: \"kubernetes.io/projected/21d19651-5c90-47ba-acf5-5004aa8e11f5-kube-api-access-trht6\") pod \"21d19651-5c90-47ba-acf5-5004aa8e11f5\" (UID: \"21d19651-5c90-47ba-acf5-5004aa8e11f5\") " Oct 30 12:19:14.938958 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:14.938932 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21d19651-5c90-47ba-acf5-5004aa8e11f5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21d19651-5c90-47ba-acf5-5004aa8e11f5-maas-api-xks-ca\") pod \"21d19651-5c90-47ba-acf5-5004aa8e11f5\" (UID: \"21d19651-5c90-47ba-acf5-5004aa8e11f5\") " Oct 30 12:19:14.939417 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:14.939394 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21d19651-5c90-47ba-acf5-5004aa8e11f5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21d19651-5c90-47ba-acf5-5004aa8e11f5-maas-api-openshift-ca\") pod \"21d19651-5c90-47ba-acf5-5004aa8e11f5\" (UID: \"21d19651-5c90-47ba-acf5-5004aa8e11f5\") " Oct 30 12:19:14.944417 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:14.942361 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21d19651-5c90-47ba-acf5-5004aa8e11f5-maas-api-openshift-ca" pod "21d19651-5c90-47ba-acf5-5004aa8e11f5" (UID: "21d19651-5c90-47ba-acf5-5004aa8e11f5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:19:14.944417 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:14.943920 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21d19651-5c90-47ba-acf5-5004aa8e11f5-maas-api-openshift-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:19:14.945996 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:14.945969 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21d19651-5c90-47ba-acf5-5004aa8e11f5-kube-api-access-trht6" pod "21d19651-5c90-47ba-acf5-5004aa8e11f5" (UID: "21d19651-5c90-47ba-acf5-5004aa8e11f5"). InnerVolumeSpecName "kube-api-access-trht6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:19:14.946730 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:14.946705 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21d19651-5c90-47ba-acf5-5004aa8e11f5-maas-api-xks-ca" pod "21d19651-5c90-47ba-acf5-5004aa8e11f5" (UID: "21d19651-5c90-47ba-acf5-5004aa8e11f5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:19:14.950060 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:14.950032 4908 patch_prober.go:29] interesting pod/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-gk4rg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Oct 30 12:19:14.950175 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:14.950091 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-gk4rg" podUID="9ac0ce00-fe15-408d-90eb-9633ea3e67d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Oct 30 12:19:15.045602 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:15.045562 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-trht6\" (UniqueName: \"kubernetes.io/projected/21d19651-5c90-47ba-acf5-5004aa8e11f5-kube-api-access-trht6\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:19:15.045602 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:15.045607 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21d19651-5c90-47ba-acf5-5004aa8e11f5-maas-api-xks-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:19:15.706163 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:15.705940 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e57de56-tns89" Oct 30 12:19:15.707847 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:15.705999 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e57de56-tns89" event={"ID":"21d19651-5c90-47ba-acf5-5004aa8e11f5","Type":"ContainerDied","Data":"8fe22ae33ffceb5dfe3b9e7bc02aa5aee0b613115ea8a1764943f8d9a22521cd"} Oct 30 12:19:15.708164 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:15.708095 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8fe22ae33ffceb5dfe3b9e7bc02aa5aee0b613115ea8a1764943f8d9a22521cd" Oct 30 12:19:15.955323 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:15.955288 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-gk4rg" Oct 30 12:19:15.955975 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:15.955800 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-gk4rg" Oct 30 12:19:15.958084 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:15.958019 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-gk4rg" Oct 30 12:19:15.958526 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:15.958503 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-sqv2j" Oct 30 12:19:15.958908 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:15.958882 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-sqv2j" Oct 30 12:19:15.960435 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:15.960415 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-sqv2j" Oct 30 12:19:19.688624 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:19:19.688572 4908 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vn6vv" containerName="extractor" Oct 30 12:19:20.090031 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:19:20.089991 4908 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3e63d4d_0257_41a0_9b25_24a4c7514b95.slice\": RecentStats: unable to find data in memory cache]" Oct 30 12:19:21.899845 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:21.899789 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9wwfc"] Oct 30 12:19:22.863854 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:19:22.854734 4908 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3e63d4d_0257_41a0_9b25_24a4c7514b95.slice\": RecentStats: unable to find data in memory cache]" Oct 30 12:19:23.482856 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:23.479886 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eacb4712-5xgh4"] Oct 30 12:19:23.482856 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:23.480645 4908 cpu_manager.go:414] "containerMap: removing container" podUID="21d19651-5c90-47ba-acf5-5004aa8e11f5" containerName="revoke-keys" Oct 30 12:19:23.482856 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:23.480664 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21d19651-5c90-47ba-acf5-5004aa8e11f5" containerName="revoke-keys" Oct 30 12:19:23.482856 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:23.480682 4908 cpu_manager.go:414] "containerMap: removing container" podUID="6276351f-7c08-4c6d-b841-1d0e259c3fb8" containerName="revoke-keys" Oct 30 12:19:23.482856 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:23.480689 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6276351f-7c08-4c6d-b841-1d0e259c3fb8" containerName="revoke-keys" Oct 30 12:19:23.482856 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:23.480876 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="21d19651-5c90-47ba-acf5-5004aa8e11f5" containerName="revoke-keys" Oct 30 12:19:23.482856 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:23.480889 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="6276351f-7c08-4c6d-b841-1d0e259c3fb8" containerName="revoke-keys" Oct 30 12:19:23.875926 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:23.867538 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82946bf7-7840-4bc5-b599-04b3939f1f53-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eacb4712-5xgh4\" (UID: \"82946bf7-7840-4bc5-b599-04b3939f1f53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eacb4712-5xgh4" Oct 30 12:19:23.955074 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:23.955035 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82946bf7-7840-4bc5-b599-04b3939f1f53-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eacb4712-5xgh4\" (UID: \"82946bf7-7840-4bc5-b599-04b3939f1f53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eacb4712-5xgh4" Oct 30 12:19:23.955386 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:23.955368 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgddk\" (UniqueName: \"kubernetes.io/projected/82946bf7-7840-4bc5-b599-04b3939f1f53-kube-api-access-qgddk\") pod \"maas-api-revoke-sub-eacb4712-5xgh4\" (UID: \"82946bf7-7840-4bc5-b599-04b3939f1f53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eacb4712-5xgh4" Oct 30 12:19:23.997411 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:23.997365 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eacb4712-5xgh4" Oct 30 12:19:24.050215 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:24.049851 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 12:19:24.050215 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:24.050159 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 12:19:24.068629 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:24.068599 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s75hq\"" Oct 30 12:19:24.086855 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:24.082273 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eacb4712-5xgh4"] Oct 30 12:19:24.086855 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:24.083068 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 12:19:24.143566 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:24.110861 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82946bf7-7840-4bc5-b599-04b3939f1f53-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eacb4712-5xgh4\" (UID: \"82946bf7-7840-4bc5-b599-04b3939f1f53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eacb4712-5xgh4" Oct 30 12:19:24.143566 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:24.110915 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82946bf7-7840-4bc5-b599-04b3939f1f53-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eacb4712-5xgh4\" (UID: \"82946bf7-7840-4bc5-b599-04b3939f1f53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eacb4712-5xgh4" Oct 30 12:19:24.143566 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:24.110975 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgddk\" (UniqueName: \"kubernetes.io/projected/82946bf7-7840-4bc5-b599-04b3939f1f53-kube-api-access-qgddk\") pod \"maas-api-revoke-sub-eacb4712-5xgh4\" (UID: \"82946bf7-7840-4bc5-b599-04b3939f1f53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eacb4712-5xgh4" Oct 30 12:19:24.143566 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:24.112640 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82946bf7-7840-4bc5-b599-04b3939f1f53-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eacb4712-5xgh4\" (UID: \"82946bf7-7840-4bc5-b599-04b3939f1f53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eacb4712-5xgh4" Oct 30 12:19:24.203228 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:24.203193 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82946bf7-7840-4bc5-b599-04b3939f1f53-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eacb4712-5xgh4\" (UID: \"82946bf7-7840-4bc5-b599-04b3939f1f53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eacb4712-5xgh4" Oct 30 12:19:24.214854 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:24.204058 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgddk\" (UniqueName: \"kubernetes.io/projected/82946bf7-7840-4bc5-b599-04b3939f1f53-kube-api-access-qgddk\") pod \"maas-api-revoke-sub-eacb4712-5xgh4\" (UID: \"82946bf7-7840-4bc5-b599-04b3939f1f53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eacb4712-5xgh4" Oct 30 12:19:24.482931 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:24.470201 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eacb4712-5xgh4" Oct 30 12:19:24.987031 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:24.986980 4908 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47fd99/premium-w2-47fd99-kserve-5d85f9bb95-hqkmf" podUID="fd2cbeda-dbc4-41f6-a730-e1c6f9343c5e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:19:24.987472 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:24.987437 4908 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6a7148/premium-w3-6a7148-kserve-67b976c888-tgd6c" podUID="13de4441-e7f6-4cbd-97d9-0f079d11cb8e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:19:25.087021 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:25.086952 4908 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47fd99/distinct-2-w2-47fd99-kserve-77cf84b469-zxs8s" podUID="baf44d3d-42e6-43f7-a0ad-f0f9932320c1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:19:25.109982 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:25.109929 4908 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6a7148/distinct-2-w3-6a7148-kserve-67bd5fb976-9nfff" podUID="6046cc02-29fe-4f59-b522-8f57f4557277" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:19:25.348010 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:25.347935 4908 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-512ec5/simulator-w4-512ec5-kserve-7df7674f8-8rqwb" podUID="8a5dfcc9-dec3-4bc6-b0a8-c928350b3596" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:19:25.956387 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:25.885026 4908 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f920f7/simulator-w0-f920f7-kserve-599b7ff45f-g79d7" podUID="a85e562f-d655-4c22-b4f4-a299a4e494a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:19:26.004222 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:26.004160 4908 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6a7148/premium-w3-6a7148-kserve-67b976c888-tgd6c" podUID="13de4441-e7f6-4cbd-97d9-0f079d11cb8e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:19:26.005022 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:26.004970 4908 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47fd99/premium-w2-47fd99-kserve-5d85f9bb95-hqkmf" podUID="fd2cbeda-dbc4-41f6-a730-e1c6f9343c5e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:19:26.037857 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:26.012362 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-6a7148/premium-w3-6a7148-kserve-67b976c888-tgd6c" Oct 30 12:19:26.038984 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:26.038951 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-47fd99/premium-w2-47fd99-kserve-5d85f9bb95-hqkmf" Oct 30 12:19:26.143855 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:26.112510 4908 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6a7148/distinct-2-w3-6a7148-kserve-67bd5fb976-9nfff" podUID="6046cc02-29fe-4f59-b522-8f57f4557277" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:19:26.144583 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:26.112952 4908 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47fd99/distinct-2-w2-47fd99-kserve-77cf84b469-zxs8s" podUID="baf44d3d-42e6-43f7-a0ad-f0f9932320c1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:19:26.145127 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:26.145102 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-6a7148/distinct-2-w3-6a7148-kserve-67bd5fb976-9nfff" Oct 30 12:19:26.148853 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:26.145305 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-47fd99/distinct-2-w2-47fd99-kserve-77cf84b469-zxs8s" Oct 30 12:19:26.180856 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:26.123075 4908 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-47fd99/distinct-2-w2-47fd99-kserve-77cf84b469-zxs8s" podUID="baf44d3d-42e6-43f7-a0ad-f0f9932320c1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:19:26.276927 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:26.262078 4908 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-6a7148/premium-w3-6a7148-kserve-67b976c888-tgd6c" podUID="13de4441-e7f6-4cbd-97d9-0f079d11cb8e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:19:26.363370 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:26.363322 4908 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-512ec5/simulator-w4-512ec5-kserve-7df7674f8-8rqwb" podUID="8a5dfcc9-dec3-4bc6-b0a8-c928350b3596" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:19:26.412861 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:26.388801 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-512ec5/simulator-w4-512ec5-kserve-7df7674f8-8rqwb" Oct 30 12:19:27.003914 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:26.998977 4908 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f920f7/simulator-w0-f920f7-kserve-599b7ff45f-g79d7" podUID="a85e562f-d655-4c22-b4f4-a299a4e494a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:19:27.003914 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:27.001727 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f920f7/simulator-w0-f920f7-kserve-599b7ff45f-g79d7" Oct 30 12:19:27.027853 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:27.020653 4908 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47fd99/premium-w2-47fd99-kserve-5d85f9bb95-hqkmf" podUID="fd2cbeda-dbc4-41f6-a730-e1c6f9343c5e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Oct 30 12:19:27.134853 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:27.028171 4908 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6a7148/premium-w3-6a7148-kserve-67b976c888-tgd6c" podUID="13de4441-e7f6-4cbd-97d9-0f079d11cb8e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 30 12:19:27.148995 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:27.148940 4908 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6a7148/distinct-2-w3-6a7148-kserve-67bd5fb976-9nfff" podUID="6046cc02-29fe-4f59-b522-8f57f4557277" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": context deadline exceeded" Oct 30 12:19:27.157250 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:27.157191 4908 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47fd99/distinct-2-w2-47fd99-kserve-77cf84b469-zxs8s" podUID="baf44d3d-42e6-43f7-a0ad-f0f9932320c1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 12:19:27.377909 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:27.374238 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eacb4712-5xgh4"] Oct 30 12:19:27.404033 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:27.403983 4908 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-512ec5/simulator-w4-512ec5-kserve-7df7674f8-8rqwb" podUID="8a5dfcc9-dec3-4bc6-b0a8-c928350b3596" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:19:27.564316 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:27.542574 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eacb4712-5xgh4" event={"ID":"82946bf7-7840-4bc5-b599-04b3939f1f53","Type":"ContainerStarted","Data":"19f98c2ccc859afbbd5dab235454a7ebc7f9f50d92cfc50e08f7726811ee74cc"} Oct 30 12:19:28.021014 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:28.019719 4908 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f920f7/simulator-w0-f920f7-kserve-599b7ff45f-g79d7" podUID="a85e562f-d655-4c22-b4f4-a299a4e494a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Oct 30 12:19:28.118379 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:28.115443 4908 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47fd99/premium-w2-47fd99-kserve-5d85f9bb95-hqkmf" podUID="fd2cbeda-dbc4-41f6-a730-e1c6f9343c5e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Oct 30 12:19:28.131622 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:28.129921 4908 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6a7148/premium-w3-6a7148-kserve-67b976c888-tgd6c" podUID="13de4441-e7f6-4cbd-97d9-0f079d11cb8e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:19:28.158573 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:28.158528 4908 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6a7148/distinct-2-w3-6a7148-kserve-67bd5fb976-9nfff" podUID="6046cc02-29fe-4f59-b522-8f57f4557277" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:19:28.176054 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:28.159610 4908 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47fd99/distinct-2-w2-47fd99-kserve-77cf84b469-zxs8s" podUID="baf44d3d-42e6-43f7-a0ad-f0f9932320c1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:19:28.436856 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:28.436081 4908 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-512ec5/simulator-w4-512ec5-kserve-7df7674f8-8rqwb" podUID="8a5dfcc9-dec3-4bc6-b0a8-c928350b3596" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:19:28.489861 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:28.487163 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-6a7148/distinct-2-w3-6a7148-kserve-67bd5fb976-9nfff" Oct 30 12:19:28.796155 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:28.796013 4908 patch_prober.go:29] interesting pod/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rx4tf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 12:19:28.796920 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:28.796453 4908 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rx4tf" podUID="76c80954-caf1-42b7-a812-fa52488f2e1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 12:19:28.858444 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:28.853780 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eacb4712-5xgh4" event={"ID":"82946bf7-7840-4bc5-b599-04b3939f1f53","Type":"ContainerStarted","Data":"4481120c8d3e36ddd0dd41a1a29ebaec0b22f0fd5d8ffa3a0214c1b26f040fe7"} Oct 30 12:19:29.044851 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:29.023935 4908 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f920f7/simulator-w0-f920f7-kserve-599b7ff45f-g79d7" podUID="a85e562f-d655-4c22-b4f4-a299a4e494a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:19:29.044851 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:29.040926 4908 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-47fd99/premium-w2-47fd99-kserve-5d85f9bb95-hqkmf" podUID="fd2cbeda-dbc4-41f6-a730-e1c6f9343c5e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:19:29.207237 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:29.207111 4908 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6a7148/premium-w3-6a7148-kserve-67b976c888-tgd6c" podUID="13de4441-e7f6-4cbd-97d9-0f079d11cb8e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:19:29.224275 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:29.224230 4908 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47fd99/distinct-2-w2-47fd99-kserve-77cf84b469-zxs8s" podUID="baf44d3d-42e6-43f7-a0ad-f0f9932320c1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 30 12:19:29.247044 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:29.236932 4908 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47fd99/premium-w2-47fd99-kserve-5d85f9bb95-hqkmf" podUID="fd2cbeda-dbc4-41f6-a730-e1c6f9343c5e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Oct 30 12:19:29.433874 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:29.430921 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f920f7/simulator-w0-f920f7-kserve-599b7ff45f-g79d7" Oct 30 12:19:29.433874 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:29.432117 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-47fd99/premium-w2-47fd99-kserve-5d85f9bb95-hqkmf" Oct 30 12:19:29.433874 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:29.433569 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-512ec5/simulator-w4-512ec5-kserve-7df7674f8-8rqwb" Oct 30 12:19:29.463112 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:29.461654 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-47fd99/distinct-2-w2-47fd99-kserve-77cf84b469-zxs8s" Oct 30 12:19:29.473946 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:29.472713 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-6a7148/premium-w3-6a7148-kserve-67b976c888-tgd6c" Oct 30 12:19:29.494992 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:29.494911 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eacb4712-5xgh4" podStartSLOduration=6.494873803 podStartE2EDuration="6.494873803s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:19:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:19:28.943804659 +0000 UTC m=+1525.458233406" watchObservedRunningTime="2026-10-30 12:19:29.494873803 +0000 UTC m=+1526.009302552" Oct 30 12:19:29.868382 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:29.868340 4908 generic.go:356] "Generic (PLEG): container finished" podID="82946bf7-7840-4bc5-b599-04b3939f1f53" containerID="4481120c8d3e36ddd0dd41a1a29ebaec0b22f0fd5d8ffa3a0214c1b26f040fe7" exitCode=0 Oct 30 12:19:29.868688 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:29.868662 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eacb4712-5xgh4" event={"ID":"82946bf7-7840-4bc5-b599-04b3939f1f53","Type":"ContainerDied","Data":"4481120c8d3e36ddd0dd41a1a29ebaec0b22f0fd5d8ffa3a0214c1b26f040fe7"} Oct 30 12:19:30.069644 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:30.069614 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f9ae990-bmc72"] Oct 30 12:19:30.074511 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:30.074484 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f9ae990-bmc72" Oct 30 12:19:30.115849 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:30.102134 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae8121a7-b304-4a56-8ecf-8d3bfdeee659-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9f9ae990-bmc72\" (UID: \"ae8121a7-b304-4a56-8ecf-8d3bfdeee659\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f9ae990-bmc72" Oct 30 12:19:30.115849 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:30.102236 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae8121a7-b304-4a56-8ecf-8d3bfdeee659-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9f9ae990-bmc72\" (UID: \"ae8121a7-b304-4a56-8ecf-8d3bfdeee659\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f9ae990-bmc72" Oct 30 12:19:30.115849 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:30.102319 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnxhb\" (UniqueName: \"kubernetes.io/projected/ae8121a7-b304-4a56-8ecf-8d3bfdeee659-kube-api-access-fnxhb\") pod \"maas-api-revoke-sub-9f9ae990-bmc72\" (UID: \"ae8121a7-b304-4a56-8ecf-8d3bfdeee659\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f9ae990-bmc72" Oct 30 12:19:30.115849 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:30.113863 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f9ae990-bmc72"] Oct 30 12:19:30.204784 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:30.203192 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae8121a7-b304-4a56-8ecf-8d3bfdeee659-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9f9ae990-bmc72\" (UID: \"ae8121a7-b304-4a56-8ecf-8d3bfdeee659\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f9ae990-bmc72" Oct 30 12:19:30.204784 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:30.203306 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnxhb\" (UniqueName: \"kubernetes.io/projected/ae8121a7-b304-4a56-8ecf-8d3bfdeee659-kube-api-access-fnxhb\") pod \"maas-api-revoke-sub-9f9ae990-bmc72\" (UID: \"ae8121a7-b304-4a56-8ecf-8d3bfdeee659\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f9ae990-bmc72" Oct 30 12:19:30.204784 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:30.203407 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae8121a7-b304-4a56-8ecf-8d3bfdeee659-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9f9ae990-bmc72\" (UID: \"ae8121a7-b304-4a56-8ecf-8d3bfdeee659\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f9ae990-bmc72" Oct 30 12:19:30.204784 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:30.204315 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae8121a7-b304-4a56-8ecf-8d3bfdeee659-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9f9ae990-bmc72\" (UID: \"ae8121a7-b304-4a56-8ecf-8d3bfdeee659\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f9ae990-bmc72" Oct 30 12:19:30.214019 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:30.213978 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae8121a7-b304-4a56-8ecf-8d3bfdeee659-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9f9ae990-bmc72\" (UID: \"ae8121a7-b304-4a56-8ecf-8d3bfdeee659\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f9ae990-bmc72" Oct 30 12:19:30.224166 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:19:30.224130 4908 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3e63d4d_0257_41a0_9b25_24a4c7514b95.slice\": RecentStats: unable to find data in memory cache]" Oct 30 12:19:30.229608 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:30.229548 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnxhb\" (UniqueName: \"kubernetes.io/projected/ae8121a7-b304-4a56-8ecf-8d3bfdeee659-kube-api-access-fnxhb\") pod \"maas-api-revoke-sub-9f9ae990-bmc72\" (UID: \"ae8121a7-b304-4a56-8ecf-8d3bfdeee659\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f9ae990-bmc72" Oct 30 12:19:30.414822 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:30.414773 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f9ae990-bmc72" Oct 30 12:19:30.742412 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:30.742385 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f9ae990-bmc72"] Oct 30 12:19:30.744729 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:19:30.744698 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae8121a7_b304_4a56_8ecf_8d3bfdeee659.slice/crio-4c2062fc7d4cf69e140de4df229fde9639467f58bce0ef85a37399d486ecc584 WatchSource:0}: Error finding container 4c2062fc7d4cf69e140de4df229fde9639467f58bce0ef85a37399d486ecc584: Status 404 returned error can't find the container with id 4c2062fc7d4cf69e140de4df229fde9639467f58bce0ef85a37399d486ecc584 Oct 30 12:19:30.880124 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:30.880081 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f9ae990-bmc72" event={"ID":"ae8121a7-b304-4a56-8ecf-8d3bfdeee659","Type":"ContainerStarted","Data":"4c2062fc7d4cf69e140de4df229fde9639467f58bce0ef85a37399d486ecc584"} Oct 30 12:19:31.032245 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:31.032217 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eacb4712-5xgh4" Oct 30 12:19:31.114559 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:31.114528 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82946bf7-7840-4bc5-b599-04b3939f1f53-kube-api-access-qgddk\" (UniqueName: \"kubernetes.io/projected/82946bf7-7840-4bc5-b599-04b3939f1f53-kube-api-access-qgddk\") pod \"82946bf7-7840-4bc5-b599-04b3939f1f53\" (UID: \"82946bf7-7840-4bc5-b599-04b3939f1f53\") " Oct 30 12:19:31.115061 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:31.114585 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/82946bf7-7840-4bc5-b599-04b3939f1f53-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82946bf7-7840-4bc5-b599-04b3939f1f53-maas-api-xks-ca\") pod \"82946bf7-7840-4bc5-b599-04b3939f1f53\" (UID: \"82946bf7-7840-4bc5-b599-04b3939f1f53\") " Oct 30 12:19:31.115061 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:31.114619 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82946bf7-7840-4bc5-b599-04b3939f1f53-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82946bf7-7840-4bc5-b599-04b3939f1f53-maas-api-openshift-ca\") pod \"82946bf7-7840-4bc5-b599-04b3939f1f53\" (UID: \"82946bf7-7840-4bc5-b599-04b3939f1f53\") " Oct 30 12:19:31.115061 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:31.115051 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82946bf7-7840-4bc5-b599-04b3939f1f53-maas-api-openshift-ca" pod "82946bf7-7840-4bc5-b599-04b3939f1f53" (UID: "82946bf7-7840-4bc5-b599-04b3939f1f53"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:19:31.116914 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:31.116885 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82946bf7-7840-4bc5-b599-04b3939f1f53-maas-api-xks-ca" pod "82946bf7-7840-4bc5-b599-04b3939f1f53" (UID: "82946bf7-7840-4bc5-b599-04b3939f1f53"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:19:31.117042 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:31.116915 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82946bf7-7840-4bc5-b599-04b3939f1f53-kube-api-access-qgddk" pod "82946bf7-7840-4bc5-b599-04b3939f1f53" (UID: "82946bf7-7840-4bc5-b599-04b3939f1f53"). InnerVolumeSpecName "kube-api-access-qgddk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:19:31.216614 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:31.216575 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qgddk\" (UniqueName: \"kubernetes.io/projected/82946bf7-7840-4bc5-b599-04b3939f1f53-kube-api-access-qgddk\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:19:31.216614 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:31.216608 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82946bf7-7840-4bc5-b599-04b3939f1f53-maas-api-xks-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:19:31.216614 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:31.216619 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82946bf7-7840-4bc5-b599-04b3939f1f53-maas-api-openshift-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:19:31.890388 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:31.890292 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eacb4712-5xgh4" Oct 30 12:19:31.890388 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:31.890329 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eacb4712-5xgh4" event={"ID":"82946bf7-7840-4bc5-b599-04b3939f1f53","Type":"ContainerDied","Data":"19f98c2ccc859afbbd5dab235454a7ebc7f9f50d92cfc50e08f7726811ee74cc"} Oct 30 12:19:31.890624 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:31.890393 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="19f98c2ccc859afbbd5dab235454a7ebc7f9f50d92cfc50e08f7726811ee74cc" Oct 30 12:19:31.892644 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:31.892612 4908 generic.go:356] "Generic (PLEG): container finished" podID="ae8121a7-b304-4a56-8ecf-8d3bfdeee659" containerID="05de84f1b1c4cd5cb4f3fba550466a3e586f3ac60fa195501ee7b613756b9503" exitCode=0 Oct 30 12:19:31.892860 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:31.892813 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f9ae990-bmc72" event={"ID":"ae8121a7-b304-4a56-8ecf-8d3bfdeee659","Type":"ContainerDied","Data":"05de84f1b1c4cd5cb4f3fba550466a3e586f3ac60fa195501ee7b613756b9503"} Oct 30 12:19:33.063246 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:33.063203 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f9ae990-bmc72" Oct 30 12:19:33.138011 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:33.137975 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae8121a7-b304-4a56-8ecf-8d3bfdeee659-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae8121a7-b304-4a56-8ecf-8d3bfdeee659-maas-api-xks-ca\") pod \"ae8121a7-b304-4a56-8ecf-8d3bfdeee659\" (UID: \"ae8121a7-b304-4a56-8ecf-8d3bfdeee659\") " Oct 30 12:19:33.138243 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:33.138210 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae8121a7-b304-4a56-8ecf-8d3bfdeee659-kube-api-access-fnxhb\" (UniqueName: \"kubernetes.io/projected/ae8121a7-b304-4a56-8ecf-8d3bfdeee659-kube-api-access-fnxhb\") pod \"ae8121a7-b304-4a56-8ecf-8d3bfdeee659\" (UID: \"ae8121a7-b304-4a56-8ecf-8d3bfdeee659\") " Oct 30 12:19:33.138334 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:33.138248 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae8121a7-b304-4a56-8ecf-8d3bfdeee659-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae8121a7-b304-4a56-8ecf-8d3bfdeee659-maas-api-openshift-ca\") pod \"ae8121a7-b304-4a56-8ecf-8d3bfdeee659\" (UID: \"ae8121a7-b304-4a56-8ecf-8d3bfdeee659\") " Oct 30 12:19:33.138664 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:33.138634 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae8121a7-b304-4a56-8ecf-8d3bfdeee659-maas-api-openshift-ca" pod "ae8121a7-b304-4a56-8ecf-8d3bfdeee659" (UID: "ae8121a7-b304-4a56-8ecf-8d3bfdeee659"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:19:33.140711 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:33.140689 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae8121a7-b304-4a56-8ecf-8d3bfdeee659-maas-api-xks-ca" pod "ae8121a7-b304-4a56-8ecf-8d3bfdeee659" (UID: "ae8121a7-b304-4a56-8ecf-8d3bfdeee659"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:19:33.140972 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:33.140941 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae8121a7-b304-4a56-8ecf-8d3bfdeee659-kube-api-access-fnxhb" pod "ae8121a7-b304-4a56-8ecf-8d3bfdeee659" (UID: "ae8121a7-b304-4a56-8ecf-8d3bfdeee659"). InnerVolumeSpecName "kube-api-access-fnxhb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:19:33.240332 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:33.240296 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fnxhb\" (UniqueName: \"kubernetes.io/projected/ae8121a7-b304-4a56-8ecf-8d3bfdeee659-kube-api-access-fnxhb\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:19:33.240332 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:33.240329 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae8121a7-b304-4a56-8ecf-8d3bfdeee659-maas-api-openshift-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:19:33.240555 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:33.240341 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae8121a7-b304-4a56-8ecf-8d3bfdeee659-maas-api-xks-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:19:33.917460 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:33.917291 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f9ae990-bmc72" Oct 30 12:19:33.917460 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:33.917291 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f9ae990-bmc72" event={"ID":"ae8121a7-b304-4a56-8ecf-8d3bfdeee659","Type":"ContainerDied","Data":"4c2062fc7d4cf69e140de4df229fde9639467f58bce0ef85a37399d486ecc584"} Oct 30 12:19:33.917918 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:33.917506 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4c2062fc7d4cf69e140de4df229fde9639467f58bce0ef85a37399d486ecc584" Oct 30 12:19:37.824155 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:19:37.824115 4908 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3e63d4d_0257_41a0_9b25_24a4c7514b95.slice\": RecentStats: unable to find data in memory cache]" Oct 30 12:19:40.245225 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:19:40.245186 4908 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3e63d4d_0257_41a0_9b25_24a4c7514b95.slice\": RecentStats: unable to find data in memory cache]" Oct 30 12:19:41.639389 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:41.639348 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-pk78c"] Oct 30 12:19:41.639981 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:41.639965 4908 cpu_manager.go:414] "containerMap: removing container" podUID="ae8121a7-b304-4a56-8ecf-8d3bfdeee659" containerName="revoke-keys" Oct 30 12:19:41.640035 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:41.639982 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae8121a7-b304-4a56-8ecf-8d3bfdeee659" containerName="revoke-keys" Oct 30 12:19:41.640035 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:41.640021 4908 cpu_manager.go:414] "containerMap: removing container" podUID="82946bf7-7840-4bc5-b599-04b3939f1f53" containerName="revoke-keys" Oct 30 12:19:41.640035 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:41.640026 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82946bf7-7840-4bc5-b599-04b3939f1f53" containerName="revoke-keys" Oct 30 12:19:41.640124 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:41.640117 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="82946bf7-7840-4bc5-b599-04b3939f1f53" containerName="revoke-keys" Oct 30 12:19:41.640154 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:41.640126 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae8121a7-b304-4a56-8ecf-8d3bfdeee659" containerName="revoke-keys" Oct 30 12:19:41.645063 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:41.645043 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-pk78c" Oct 30 12:19:41.676548 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:41.676507 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-pk78c"] Oct 30 12:19:41.749625 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:41.749583 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/714b47ac-9d7f-422f-a676-08be901b4216-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-pk78c\" (UID: \"714b47ac-9d7f-422f-a676-08be901b4216\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-pk78c" Oct 30 12:19:41.749625 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:41.749627 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/714b47ac-9d7f-422f-a676-08be901b4216-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-pk78c\" (UID: \"714b47ac-9d7f-422f-a676-08be901b4216\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-pk78c" Oct 30 12:19:41.749881 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:41.749726 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/714b47ac-9d7f-422f-a676-08be901b4216-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-pk78c\" (UID: \"714b47ac-9d7f-422f-a676-08be901b4216\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-pk78c" Oct 30 12:19:41.749881 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:41.749791 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/714b47ac-9d7f-422f-a676-08be901b4216-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-pk78c\" (UID: \"714b47ac-9d7f-422f-a676-08be901b4216\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-pk78c" Oct 30 12:19:41.749881 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:41.749819 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/714b47ac-9d7f-422f-a676-08be901b4216-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-pk78c\" (UID: \"714b47ac-9d7f-422f-a676-08be901b4216\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-pk78c" Oct 30 12:19:41.749980 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:41.749895 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvcrl\" (UniqueName: \"kubernetes.io/projected/714b47ac-9d7f-422f-a676-08be901b4216-kube-api-access-cvcrl\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-pk78c\" (UID: \"714b47ac-9d7f-422f-a676-08be901b4216\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-pk78c" Oct 30 12:19:41.749980 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:41.749914 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/714b47ac-9d7f-422f-a676-08be901b4216-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-pk78c\" (UID: \"714b47ac-9d7f-422f-a676-08be901b4216\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-pk78c" Oct 30 12:19:41.750074 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:41.750061 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/714b47ac-9d7f-422f-a676-08be901b4216-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-pk78c\" (UID: \"714b47ac-9d7f-422f-a676-08be901b4216\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-pk78c" Oct 30 12:19:41.750110 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:41.750097 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/714b47ac-9d7f-422f-a676-08be901b4216-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-pk78c\" (UID: \"714b47ac-9d7f-422f-a676-08be901b4216\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-pk78c" Oct 30 12:19:41.851741 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:41.851702 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/714b47ac-9d7f-422f-a676-08be901b4216-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-pk78c\" (UID: \"714b47ac-9d7f-422f-a676-08be901b4216\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-pk78c" Oct 30 12:19:41.851938 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:41.851753 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/714b47ac-9d7f-422f-a676-08be901b4216-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-pk78c\" (UID: \"714b47ac-9d7f-422f-a676-08be901b4216\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-pk78c" Oct 30 12:19:41.851938 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:41.851802 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/714b47ac-9d7f-422f-a676-08be901b4216-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-pk78c\" (UID: \"714b47ac-9d7f-422f-a676-08be901b4216\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-pk78c" Oct 30 12:19:41.851938 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:41.851820 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/714b47ac-9d7f-422f-a676-08be901b4216-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-pk78c\" (UID: \"714b47ac-9d7f-422f-a676-08be901b4216\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-pk78c" Oct 30 12:19:41.852062 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:41.851939 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/714b47ac-9d7f-422f-a676-08be901b4216-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-pk78c\" (UID: \"714b47ac-9d7f-422f-a676-08be901b4216\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-pk78c" Oct 30 12:19:41.852062 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:41.851995 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/714b47ac-9d7f-422f-a676-08be901b4216-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-pk78c\" (UID: \"714b47ac-9d7f-422f-a676-08be901b4216\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-pk78c" Oct 30 12:19:41.852062 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:41.852020 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/714b47ac-9d7f-422f-a676-08be901b4216-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-pk78c\" (UID: \"714b47ac-9d7f-422f-a676-08be901b4216\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-pk78c" Oct 30 12:19:41.852217 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:41.852072 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvcrl\" (UniqueName: \"kubernetes.io/projected/714b47ac-9d7f-422f-a676-08be901b4216-kube-api-access-cvcrl\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-pk78c\" (UID: \"714b47ac-9d7f-422f-a676-08be901b4216\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-pk78c" Oct 30 12:19:41.852217 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:41.852105 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/714b47ac-9d7f-422f-a676-08be901b4216-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-pk78c\" (UID: \"714b47ac-9d7f-422f-a676-08be901b4216\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-pk78c" Oct 30 12:19:41.852217 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:41.852186 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/714b47ac-9d7f-422f-a676-08be901b4216-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-pk78c\" (UID: \"714b47ac-9d7f-422f-a676-08be901b4216\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-pk78c" Oct 30 12:19:41.852377 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:41.852266 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/714b47ac-9d7f-422f-a676-08be901b4216-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-pk78c\" (UID: \"714b47ac-9d7f-422f-a676-08be901b4216\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-pk78c" Oct 30 12:19:41.852377 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:41.852299 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/714b47ac-9d7f-422f-a676-08be901b4216-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-pk78c\" (UID: \"714b47ac-9d7f-422f-a676-08be901b4216\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-pk78c" Oct 30 12:19:41.852483 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:41.852410 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/714b47ac-9d7f-422f-a676-08be901b4216-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-pk78c\" (UID: \"714b47ac-9d7f-422f-a676-08be901b4216\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-pk78c" Oct 30 12:19:41.852760 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:41.852739 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/714b47ac-9d7f-422f-a676-08be901b4216-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-pk78c\" (UID: \"714b47ac-9d7f-422f-a676-08be901b4216\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-pk78c" Oct 30 12:19:41.854275 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:41.854250 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/714b47ac-9d7f-422f-a676-08be901b4216-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-pk78c\" (UID: \"714b47ac-9d7f-422f-a676-08be901b4216\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-pk78c" Oct 30 12:19:41.854590 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:41.854570 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/714b47ac-9d7f-422f-a676-08be901b4216-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-pk78c\" (UID: \"714b47ac-9d7f-422f-a676-08be901b4216\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-pk78c" Oct 30 12:19:41.875302 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:41.875258 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/714b47ac-9d7f-422f-a676-08be901b4216-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-pk78c\" (UID: \"714b47ac-9d7f-422f-a676-08be901b4216\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-pk78c" Oct 30 12:19:41.875465 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:41.875374 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvcrl\" (UniqueName: \"kubernetes.io/projected/714b47ac-9d7f-422f-a676-08be901b4216-kube-api-access-cvcrl\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-pk78c\" (UID: \"714b47ac-9d7f-422f-a676-08be901b4216\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-pk78c" Oct 30 12:19:41.978799 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:41.978768 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-pk78c" Oct 30 12:19:42.353539 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:42.352005 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-pk78c"] Oct 30 12:19:42.356509 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:19:42.356465 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod714b47ac_9d7f_422f_a676_08be901b4216.slice/crio-ef82112fbeea6cff10d2534850fc1aaec7bdc607e736f8617e939305579e4cb6 WatchSource:0}: Error finding container ef82112fbeea6cff10d2534850fc1aaec7bdc607e736f8617e939305579e4cb6: Status 404 returned error can't find the container with id ef82112fbeea6cff10d2534850fc1aaec7bdc607e736f8617e939305579e4cb6 Oct 30 12:19:42.361925 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:42.360917 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:19:42.361925 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:42.361025 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:19:42.361925 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:42.361054 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:19:42.999261 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:42.999215 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-pk78c" event={"ID":"714b47ac-9d7f-422f-a676-08be901b4216","Type":"ContainerStarted","Data":"34fd5eed27439bb58111269fac7ed1ab03f468eb330ed99a8593e6b33321ca95"} Oct 30 12:19:42.999933 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:42.999896 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-pk78c" event={"ID":"714b47ac-9d7f-422f-a676-08be901b4216","Type":"ContainerStarted","Data":"ef82112fbeea6cff10d2534850fc1aaec7bdc607e736f8617e939305579e4cb6"} Oct 30 12:19:43.050127 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:43.050066 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-pk78c" podStartSLOduration=2.050050857 podStartE2EDuration="2.050050857s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:19:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:19:43.046137426 +0000 UTC m=+1539.560566172" watchObservedRunningTime="2026-10-30 12:19:43.050050857 +0000 UTC m=+1539.564479602" Oct 30 12:19:43.981995 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:43.981956 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-pk78c" Oct 30 12:19:43.982456 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:43.982420 4908 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-pk78c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Oct 30 12:19:43.982666 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:43.982640 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-pk78c" podUID="714b47ac-9d7f-422f-a676-08be901b4216" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Oct 30 12:19:44.985856 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:44.985743 4908 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-pk78c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Oct 30 12:19:44.985856 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:44.985814 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-pk78c" podUID="714b47ac-9d7f-422f-a676-08be901b4216" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Oct 30 12:19:45.982368 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:45.982330 4908 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-pk78c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Oct 30 12:19:45.982698 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:45.982655 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-pk78c" podUID="714b47ac-9d7f-422f-a676-08be901b4216" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Oct 30 12:19:46.979297 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:46.979255 4908 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-pk78c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Oct 30 12:19:46.979991 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:46.979954 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-pk78c" podUID="714b47ac-9d7f-422f-a676-08be901b4216" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Oct 30 12:19:48.008858 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:47.982292 4908 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-pk78c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Oct 30 12:19:48.009564 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:48.009517 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-pk78c" podUID="714b47ac-9d7f-422f-a676-08be901b4216" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Oct 30 12:19:48.155033 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:48.154982 4908 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6a7148/premium-w3-6a7148-kserve-67b976c888-tgd6c" podUID="13de4441-e7f6-4cbd-97d9-0f079d11cb8e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:19:48.207277 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:48.201925 4908 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47fd99/distinct-2-w2-47fd99-kserve-77cf84b469-zxs8s" podUID="baf44d3d-42e6-43f7-a0ad-f0f9932320c1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 30 12:19:48.215855 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:48.203276 4908 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47fd99/premium-w2-47fd99-kserve-5d85f9bb95-hqkmf" podUID="fd2cbeda-dbc4-41f6-a730-e1c6f9343c5e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:19:48.260864 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:48.259525 4908 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6a7148/distinct-2-w3-6a7148-kserve-67bd5fb976-9nfff" podUID="6046cc02-29fe-4f59-b522-8f57f4557277" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 12:19:48.440594 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:48.440538 4908 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-512ec5/simulator-w4-512ec5-kserve-7df7674f8-8rqwb" podUID="8a5dfcc9-dec3-4bc6-b0a8-c928350b3596" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Oct 30 12:19:48.982785 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:48.982297 4908 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-pk78c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Oct 30 12:19:48.982785 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:48.982369 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-pk78c" podUID="714b47ac-9d7f-422f-a676-08be901b4216" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Oct 30 12:19:49.035985 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:49.035934 4908 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f920f7/simulator-w0-f920f7-kserve-599b7ff45f-g79d7" podUID="a85e562f-d655-4c22-b4f4-a299a4e494a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:19:49.049860 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:49.042174 4908 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-47fd99/premium-w2-47fd99-kserve-5d85f9bb95-hqkmf" podUID="fd2cbeda-dbc4-41f6-a730-e1c6f9343c5e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": context deadline exceeded" Oct 30 12:19:49.294903 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:49.268052 4908 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6a7148/premium-w3-6a7148-kserve-67b976c888-tgd6c" podUID="13de4441-e7f6-4cbd-97d9-0f079d11cb8e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 30 12:19:49.313895 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:49.284605 4908 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47fd99/distinct-2-w2-47fd99-kserve-77cf84b469-zxs8s" podUID="baf44d3d-42e6-43f7-a0ad-f0f9932320c1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 30 12:19:49.346992 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:49.315373 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-6a7148/premium-w3-6a7148-kserve-67b976c888-tgd6c" Oct 30 12:19:49.346992 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:49.315445 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-47fd99/distinct-2-w2-47fd99-kserve-77cf84b469-zxs8s" Oct 30 12:19:49.346992 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:49.284646 4908 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47fd99/premium-w2-47fd99-kserve-5d85f9bb95-hqkmf" podUID="fd2cbeda-dbc4-41f6-a730-e1c6f9343c5e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Oct 30 12:19:49.346992 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:49.322104 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-47fd99/premium-w2-47fd99-kserve-5d85f9bb95-hqkmf" Oct 30 12:19:49.358852 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:49.348599 4908 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6a7148/distinct-2-w3-6a7148-kserve-67bd5fb976-9nfff" podUID="6046cc02-29fe-4f59-b522-8f57f4557277" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": context deadline exceeded" Oct 30 12:19:49.358852 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:49.348938 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-6a7148/distinct-2-w3-6a7148-kserve-67bd5fb976-9nfff" Oct 30 12:19:49.533421 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:49.480170 4908 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-512ec5/simulator-w4-512ec5-kserve-7df7674f8-8rqwb" podUID="8a5dfcc9-dec3-4bc6-b0a8-c928350b3596" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:19:49.533421 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:49.503185 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-512ec5/simulator-w4-512ec5-kserve-7df7674f8-8rqwb" Oct 30 12:19:49.547501 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:49.547001 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-47fd99/distinct-2-w2-47fd99-kserve-77cf84b469-zxs8s" Oct 30 12:19:49.802927 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:49.801914 4908 patch_prober.go:29] interesting pod/e2e-worker-w0-f920f7-openshift-default-7c9767c97-ljzls container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 12:19:49.802927 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:49.801999 4908 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-ljzls" podUID="6f0ed460-7572-4518-af84-589e8ac4ad29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 12:19:49.958857 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:49.958189 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rtxmt"] Oct 30 12:19:50.024864 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:19:50.017014 4908 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vn6vv" containerName="extractor" Oct 30 12:19:50.047887 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:50.024432 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rtxmt" Oct 30 12:19:50.047887 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:50.033994 4908 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-pk78c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Oct 30 12:19:50.047887 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:50.034638 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-pk78c" podUID="714b47ac-9d7f-422f-a676-08be901b4216" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Oct 30 12:19:50.051921 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:50.048617 4908 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f920f7/simulator-w0-f920f7-kserve-599b7ff45f-g79d7" podUID="a85e562f-d655-4c22-b4f4-a299a4e494a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:19:50.064932 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:50.055860 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f920f7/simulator-w0-f920f7-kserve-599b7ff45f-g79d7" Oct 30 12:19:50.117749 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:50.111273 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rtxmt"] Oct 30 12:19:50.182725 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:50.179402 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f36337d3-669a-4d7d-a157-b8a5f78aff17-istio-token\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rtxmt\" (UID: \"f36337d3-669a-4d7d-a157-b8a5f78aff17\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rtxmt" Oct 30 12:19:50.186659 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:50.186613 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f36337d3-669a-4d7d-a157-b8a5f78aff17-workload-certs\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rtxmt\" (UID: \"f36337d3-669a-4d7d-a157-b8a5f78aff17\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rtxmt" Oct 30 12:19:50.186960 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:50.186924 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f36337d3-669a-4d7d-a157-b8a5f78aff17-istiod-ca-cert\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rtxmt\" (UID: \"f36337d3-669a-4d7d-a157-b8a5f78aff17\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rtxmt" Oct 30 12:19:50.200853 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:50.190177 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqrr6\" (UniqueName: \"kubernetes.io/projected/f36337d3-669a-4d7d-a157-b8a5f78aff17-kube-api-access-jqrr6\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rtxmt\" (UID: \"f36337d3-669a-4d7d-a157-b8a5f78aff17\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rtxmt" Oct 30 12:19:50.200853 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:50.190480 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f36337d3-669a-4d7d-a157-b8a5f78aff17-istio-envoy\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rtxmt\" (UID: \"f36337d3-669a-4d7d-a157-b8a5f78aff17\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rtxmt" Oct 30 12:19:50.200853 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:50.190534 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f36337d3-669a-4d7d-a157-b8a5f78aff17-credential-socket\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rtxmt\" (UID: \"f36337d3-669a-4d7d-a157-b8a5f78aff17\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rtxmt" Oct 30 12:19:50.200853 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:50.190751 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f36337d3-669a-4d7d-a157-b8a5f78aff17-workload-socket\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rtxmt\" (UID: \"f36337d3-669a-4d7d-a157-b8a5f78aff17\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rtxmt" Oct 30 12:19:50.200853 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:50.190898 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f36337d3-669a-4d7d-a157-b8a5f78aff17-istio-podinfo\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rtxmt\" (UID: \"f36337d3-669a-4d7d-a157-b8a5f78aff17\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rtxmt" Oct 30 12:19:50.200853 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:50.191042 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f36337d3-669a-4d7d-a157-b8a5f78aff17-istio-data\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rtxmt\" (UID: \"f36337d3-669a-4d7d-a157-b8a5f78aff17\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rtxmt" Oct 30 12:19:50.353863 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:50.353721 4908 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6a7148/distinct-2-w3-6a7148-kserve-67bd5fb976-9nfff" podUID="6046cc02-29fe-4f59-b522-8f57f4557277" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:19:50.468856 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:50.443769 4908 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6a7148/premium-w3-6a7148-kserve-67b976c888-tgd6c" podUID="13de4441-e7f6-4cbd-97d9-0f079d11cb8e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 30 12:19:50.590973 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:50.499169 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f36337d3-669a-4d7d-a157-b8a5f78aff17-istio-data\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rtxmt\" (UID: \"f36337d3-669a-4d7d-a157-b8a5f78aff17\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rtxmt" Oct 30 12:19:50.609463 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:50.499738 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f36337d3-669a-4d7d-a157-b8a5f78aff17-istio-data\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rtxmt\" (UID: \"f36337d3-669a-4d7d-a157-b8a5f78aff17\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rtxmt" Oct 30 12:19:50.609688 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:50.579499 4908 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47fd99/premium-w2-47fd99-kserve-5d85f9bb95-hqkmf" podUID="fd2cbeda-dbc4-41f6-a730-e1c6f9343c5e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Oct 30 12:19:50.610776 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:50.582930 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-512ec5/simulator-w4-512ec5-kserve-7df7674f8-8rqwb" Oct 30 12:19:50.611764 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:50.611141 4908 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47fd99/distinct-2-w2-47fd99-kserve-77cf84b469-zxs8s" podUID="baf44d3d-42e6-43f7-a0ad-f0f9932320c1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 12:19:50.614739 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:50.612305 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f36337d3-669a-4d7d-a157-b8a5f78aff17-istio-token\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rtxmt\" (UID: \"f36337d3-669a-4d7d-a157-b8a5f78aff17\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rtxmt" Oct 30 12:19:50.735431 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:50.735392 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f36337d3-669a-4d7d-a157-b8a5f78aff17-workload-certs\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rtxmt\" (UID: \"f36337d3-669a-4d7d-a157-b8a5f78aff17\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rtxmt" Oct 30 12:19:50.735698 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:50.735673 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f36337d3-669a-4d7d-a157-b8a5f78aff17-istiod-ca-cert\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rtxmt\" (UID: \"f36337d3-669a-4d7d-a157-b8a5f78aff17\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rtxmt" Oct 30 12:19:50.750228 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:50.750182 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqrr6\" (UniqueName: \"kubernetes.io/projected/f36337d3-669a-4d7d-a157-b8a5f78aff17-kube-api-access-jqrr6\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rtxmt\" (UID: \"f36337d3-669a-4d7d-a157-b8a5f78aff17\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rtxmt" Oct 30 12:19:50.750522 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:50.750506 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f36337d3-669a-4d7d-a157-b8a5f78aff17-istio-envoy\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rtxmt\" (UID: \"f36337d3-669a-4d7d-a157-b8a5f78aff17\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rtxmt" Oct 30 12:19:50.966025 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:50.965934 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f36337d3-669a-4d7d-a157-b8a5f78aff17-workload-certs\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rtxmt\" (UID: \"f36337d3-669a-4d7d-a157-b8a5f78aff17\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rtxmt" Oct 30 12:19:50.967208 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:50.967184 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f36337d3-669a-4d7d-a157-b8a5f78aff17-istiod-ca-cert\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rtxmt\" (UID: \"f36337d3-669a-4d7d-a157-b8a5f78aff17\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rtxmt" Oct 30 12:19:51.015296 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:50.750678 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f36337d3-669a-4d7d-a157-b8a5f78aff17-credential-socket\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rtxmt\" (UID: \"f36337d3-669a-4d7d-a157-b8a5f78aff17\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rtxmt" Oct 30 12:19:51.020863 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:51.015588 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f36337d3-669a-4d7d-a157-b8a5f78aff17-workload-socket\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rtxmt\" (UID: \"f36337d3-669a-4d7d-a157-b8a5f78aff17\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rtxmt" Oct 30 12:19:51.021254 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:51.021224 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f36337d3-669a-4d7d-a157-b8a5f78aff17-istio-podinfo\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rtxmt\" (UID: \"f36337d3-669a-4d7d-a157-b8a5f78aff17\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rtxmt" Oct 30 12:19:51.043395 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:51.043355 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f36337d3-669a-4d7d-a157-b8a5f78aff17-credential-socket\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rtxmt\" (UID: \"f36337d3-669a-4d7d-a157-b8a5f78aff17\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rtxmt" Oct 30 12:19:51.053858 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:51.052541 4908 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f920f7/simulator-w0-f920f7-kserve-599b7ff45f-g79d7" podUID="a85e562f-d655-4c22-b4f4-a299a4e494a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 12:19:51.075851 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:51.069773 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f36337d3-669a-4d7d-a157-b8a5f78aff17-istio-token\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rtxmt\" (UID: \"f36337d3-669a-4d7d-a157-b8a5f78aff17\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rtxmt" Oct 30 12:19:51.075851 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:51.072396 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f36337d3-669a-4d7d-a157-b8a5f78aff17-workload-socket\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rtxmt\" (UID: \"f36337d3-669a-4d7d-a157-b8a5f78aff17\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rtxmt" Oct 30 12:19:51.089997 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:51.078363 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqrr6\" (UniqueName: \"kubernetes.io/projected/f36337d3-669a-4d7d-a157-b8a5f78aff17-kube-api-access-jqrr6\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rtxmt\" (UID: \"f36337d3-669a-4d7d-a157-b8a5f78aff17\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rtxmt" Oct 30 12:19:51.117854 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:51.099330 4908 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-pk78c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Oct 30 12:19:51.117854 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:51.099411 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-pk78c" podUID="714b47ac-9d7f-422f-a676-08be901b4216" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Oct 30 12:19:51.150854 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:51.137984 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-6a7148/premium-w3-6a7148-kserve-67b976c888-tgd6c" Oct 30 12:19:51.199855 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:51.184789 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f36337d3-669a-4d7d-a157-b8a5f78aff17-istio-podinfo\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rtxmt\" (UID: \"f36337d3-669a-4d7d-a157-b8a5f78aff17\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rtxmt" Oct 30 12:19:51.315159 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:51.315123 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f36337d3-669a-4d7d-a157-b8a5f78aff17-istio-envoy\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rtxmt\" (UID: \"f36337d3-669a-4d7d-a157-b8a5f78aff17\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rtxmt" Oct 30 12:19:51.336342 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:51.335351 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rtxmt" Oct 30 12:19:51.567913 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:51.566842 4908 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6a7148/distinct-2-w3-6a7148-kserve-67bd5fb976-9nfff" podUID="6046cc02-29fe-4f59-b522-8f57f4557277" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": context deadline exceeded" Oct 30 12:19:51.589067 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:51.589023 4908 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-512ec5/simulator-w4-512ec5-kserve-7df7674f8-8rqwb" podUID="8a5dfcc9-dec3-4bc6-b0a8-c928350b3596" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:19:51.635236 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:51.633932 4908 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47fd99/premium-w2-47fd99-kserve-5d85f9bb95-hqkmf" podUID="fd2cbeda-dbc4-41f6-a730-e1c6f9343c5e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:19:52.061853 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:52.044218 4908 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-pk78c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Oct 30 12:19:52.061853 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:52.044281 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-pk78c" podUID="714b47ac-9d7f-422f-a676-08be901b4216" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Oct 30 12:19:52.073200 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:52.071917 4908 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f920f7/simulator-w0-f920f7-kserve-599b7ff45f-g79d7" podUID="a85e562f-d655-4c22-b4f4-a299a4e494a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:19:52.222172 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:52.222055 4908 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-f920f7/simulator-w0-f920f7-kserve-599b7ff45f-g79d7" podUID="a85e562f-d655-4c22-b4f4-a299a4e494a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:19:52.542854 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:52.526995 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f920f7/simulator-w0-f920f7-kserve-599b7ff45f-g79d7" Oct 30 12:19:52.589490 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:52.589390 4908 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6a7148/distinct-2-w3-6a7148-kserve-67bd5fb976-9nfff" podUID="6046cc02-29fe-4f59-b522-8f57f4557277" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:19:52.616044 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:52.601991 4908 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6a7148/premium-w3-6a7148-kserve-67b976c888-tgd6c" podUID="13de4441-e7f6-4cbd-97d9-0f079d11cb8e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 30 12:19:52.616563 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:52.603941 4908 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-512ec5/simulator-w4-512ec5-kserve-7df7674f8-8rqwb" podUID="8a5dfcc9-dec3-4bc6-b0a8-c928350b3596" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Oct 30 12:19:52.617259 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:52.604517 4908 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47fd99/distinct-2-w2-47fd99-kserve-77cf84b469-zxs8s" podUID="baf44d3d-42e6-43f7-a0ad-f0f9932320c1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 30 12:19:52.617696 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:52.617674 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-512ec5/simulator-w4-512ec5-kserve-7df7674f8-8rqwb" Oct 30 12:19:52.642617 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:52.642572 4908 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47fd99/premium-w2-47fd99-kserve-5d85f9bb95-hqkmf" podUID="fd2cbeda-dbc4-41f6-a730-e1c6f9343c5e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:19:52.829908 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:19:52.683802 4908 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3e63d4d_0257_41a0_9b25_24a4c7514b95.slice\": RecentStats: unable to find data in memory cache]" Oct 30 12:19:52.959036 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:19:52.958989 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf36337d3_669a_4d7d_a157_b8a5f78aff17.slice/crio-343d896d6ab1251e837ee528f06dc59c02bf13b4e38fb93d5bbf20715391b254 WatchSource:0}: Error finding container 343d896d6ab1251e837ee528f06dc59c02bf13b4e38fb93d5bbf20715391b254: Status 404 returned error can't find the container with id 343d896d6ab1251e837ee528f06dc59c02bf13b4e38fb93d5bbf20715391b254 Oct 30 12:19:52.986854 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:19:52.965398 4908 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3e63d4d_0257_41a0_9b25_24a4c7514b95.slice\": RecentStats: unable to find data in memory cache]" Oct 30 12:19:52.986854 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:52.979243 4908 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-pk78c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Oct 30 12:19:52.986854 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:52.979305 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-pk78c" podUID="714b47ac-9d7f-422f-a676-08be901b4216" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Oct 30 12:19:53.029971 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:53.029928 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:19:53.030238 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:53.030213 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:19:53.030509 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:53.030468 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:19:53.148994 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:53.148855 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-512ec5/simulator-w4-512ec5-kserve-7df7674f8-8rqwb" Oct 30 12:19:53.222865 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:53.222784 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rtxmt" event={"ID":"f36337d3-669a-4d7d-a157-b8a5f78aff17","Type":"ContainerStarted","Data":"343d896d6ab1251e837ee528f06dc59c02bf13b4e38fb93d5bbf20715391b254"} Oct 30 12:19:53.539082 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:53.539033 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vmxrp"] Oct 30 12:19:53.802857 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:53.797317 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fhq7\" (UniqueName: \"kubernetes.io/projected/8c130b2b-0577-4020-82e0-2c6c1e48f653-kube-api-access-5fhq7\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vmxrp\" (UID: \"8c130b2b-0577-4020-82e0-2c6c1e48f653\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vmxrp" Oct 30 12:19:53.802857 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:53.797414 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/8c130b2b-0577-4020-82e0-2c6c1e48f653-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vmxrp\" (UID: \"8c130b2b-0577-4020-82e0-2c6c1e48f653\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vmxrp" Oct 30 12:19:53.802857 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:53.797455 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/8c130b2b-0577-4020-82e0-2c6c1e48f653-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vmxrp\" (UID: \"8c130b2b-0577-4020-82e0-2c6c1e48f653\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vmxrp" Oct 30 12:19:53.802857 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:53.797535 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8c130b2b-0577-4020-82e0-2c6c1e48f653-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vmxrp\" (UID: \"8c130b2b-0577-4020-82e0-2c6c1e48f653\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vmxrp" Oct 30 12:19:53.802857 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:53.797644 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c130b2b-0577-4020-82e0-2c6c1e48f653-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vmxrp\" (UID: \"8c130b2b-0577-4020-82e0-2c6c1e48f653\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vmxrp" Oct 30 12:19:53.802857 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:53.797680 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/8c130b2b-0577-4020-82e0-2c6c1e48f653-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vmxrp\" (UID: \"8c130b2b-0577-4020-82e0-2c6c1e48f653\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vmxrp" Oct 30 12:19:53.802857 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:53.797734 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c130b2b-0577-4020-82e0-2c6c1e48f653-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vmxrp\" (UID: \"8c130b2b-0577-4020-82e0-2c6c1e48f653\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vmxrp" Oct 30 12:19:53.877347 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:53.877310 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vmxrp" Oct 30 12:19:53.877778 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:53.877762 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-6a7148/distinct-2-w3-6a7148-kserve-67bd5fb976-9nfff" Oct 30 12:19:53.877965 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:53.877952 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vmxrp"] Oct 30 12:19:53.912159 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:53.909325 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 30 12:19:53.912372 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:53.912162 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/8c130b2b-0577-4020-82e0-2c6c1e48f653-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vmxrp\" (UID: \"8c130b2b-0577-4020-82e0-2c6c1e48f653\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vmxrp" Oct 30 12:19:53.912372 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:53.912237 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/8c130b2b-0577-4020-82e0-2c6c1e48f653-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vmxrp\" (UID: \"8c130b2b-0577-4020-82e0-2c6c1e48f653\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vmxrp" Oct 30 12:19:53.912706 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:53.912679 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8c130b2b-0577-4020-82e0-2c6c1e48f653-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vmxrp\" (UID: \"8c130b2b-0577-4020-82e0-2c6c1e48f653\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vmxrp" Oct 30 12:19:53.928322 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:53.912929 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c130b2b-0577-4020-82e0-2c6c1e48f653-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vmxrp\" (UID: \"8c130b2b-0577-4020-82e0-2c6c1e48f653\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vmxrp" Oct 30 12:19:53.944551 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:53.944515 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/8c130b2b-0577-4020-82e0-2c6c1e48f653-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vmxrp\" (UID: \"8c130b2b-0577-4020-82e0-2c6c1e48f653\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vmxrp" Oct 30 12:19:53.944820 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:53.944803 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c130b2b-0577-4020-82e0-2c6c1e48f653-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vmxrp\" (UID: \"8c130b2b-0577-4020-82e0-2c6c1e48f653\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vmxrp" Oct 30 12:19:53.945148 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:53.945131 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fhq7\" (UniqueName: \"kubernetes.io/projected/8c130b2b-0577-4020-82e0-2c6c1e48f653-kube-api-access-5fhq7\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vmxrp\" (UID: \"8c130b2b-0577-4020-82e0-2c6c1e48f653\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vmxrp" Oct 30 12:19:53.971540 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:53.971499 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/8c130b2b-0577-4020-82e0-2c6c1e48f653-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vmxrp\" (UID: \"8c130b2b-0577-4020-82e0-2c6c1e48f653\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vmxrp" Oct 30 12:19:53.972350 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:53.972328 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 30 12:19:53.978356 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:53.978333 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 30 12:19:53.991413 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:53.991345 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/8c130b2b-0577-4020-82e0-2c6c1e48f653-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vmxrp\" (UID: \"8c130b2b-0577-4020-82e0-2c6c1e48f653\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vmxrp" Oct 30 12:19:53.992762 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:53.992685 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fhq7\" (UniqueName: \"kubernetes.io/projected/8c130b2b-0577-4020-82e0-2c6c1e48f653-kube-api-access-5fhq7\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vmxrp\" (UID: \"8c130b2b-0577-4020-82e0-2c6c1e48f653\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vmxrp" Oct 30 12:19:53.995720 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:53.995164 4908 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-pk78c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Oct 30 12:19:53.995720 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:53.995216 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-pk78c" podUID="714b47ac-9d7f-422f-a676-08be901b4216" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Oct 30 12:19:53.997760 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:53.997602 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-47fd99/premium-w2-47fd99-kserve-5d85f9bb95-hqkmf" Oct 30 12:19:53.997760 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:53.997680 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rtxmt"] Oct 30 12:19:54.002176 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:54.002121 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c130b2b-0577-4020-82e0-2c6c1e48f653-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vmxrp\" (UID: \"8c130b2b-0577-4020-82e0-2c6c1e48f653\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vmxrp" Oct 30 12:19:54.004489 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:54.004432 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c130b2b-0577-4020-82e0-2c6c1e48f653-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vmxrp\" (UID: \"8c130b2b-0577-4020-82e0-2c6c1e48f653\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vmxrp" Oct 30 12:19:54.030886 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:54.030375 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-qvjrm\"" Oct 30 12:19:54.031948 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:54.031918 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8c130b2b-0577-4020-82e0-2c6c1e48f653-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vmxrp\" (UID: \"8c130b2b-0577-4020-82e0-2c6c1e48f653\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vmxrp" Oct 30 12:19:54.042860 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:54.036286 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/8c130b2b-0577-4020-82e0-2c6c1e48f653-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vmxrp\" (UID: \"8c130b2b-0577-4020-82e0-2c6c1e48f653\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vmxrp" Oct 30 12:19:54.074440 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:54.074184 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vmxrp" Oct 30 12:19:54.247528 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:54.247475 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rtxmt" event={"ID":"f36337d3-669a-4d7d-a157-b8a5f78aff17","Type":"ContainerStarted","Data":"e0a5a8db655d7522436768c1698da13bc7a08d3987e5c844ad7f864ae168b490"} Oct 30 12:19:54.326499 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:54.326378 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rtxmt" podStartSLOduration=5.326354531 podStartE2EDuration="5.326354531s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:19:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:19:54.320359784 +0000 UTC m=+1550.834788532" watchObservedRunningTime="2026-10-30 12:19:54.326354531 +0000 UTC m=+1550.840783279" Oct 30 12:19:54.344689 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:54.344632 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rtxmt" Oct 30 12:19:54.348633 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:54.347981 4908 patch_prober.go:29] interesting pod/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rtxmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Oct 30 12:19:54.348633 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:54.348048 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rtxmt" podUID="f36337d3-669a-4d7d-a157-b8a5f78aff17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Oct 30 12:19:54.472451 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:54.472405 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vmxrp"] Oct 30 12:19:54.478702 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:54.478680 4908 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 12:19:55.295872 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:55.295029 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vmxrp" event={"ID":"8c130b2b-0577-4020-82e0-2c6c1e48f653","Type":"ContainerStarted","Data":"74fc9abd9f6bc55646766edf9516d687990e30c006894d3d0d896b5bb57152fc"} Oct 30 12:19:55.344649 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:55.344607 4908 patch_prober.go:29] interesting pod/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rtxmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Oct 30 12:19:55.344895 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:55.344679 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rtxmt" podUID="f36337d3-669a-4d7d-a157-b8a5f78aff17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Oct 30 12:19:56.028987 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:56.028944 4908 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-pk78c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 12:19:56.029215 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:56.029001 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-pk78c" podUID="714b47ac-9d7f-422f-a676-08be901b4216" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 12:19:56.283247 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:56.283154 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-pk78c" Oct 30 12:19:56.297850 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:56.285021 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-pk78c" Oct 30 12:19:56.305254 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:56.305157 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-pk78c" Oct 30 12:19:56.344953 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:56.344914 4908 patch_prober.go:29] interesting pod/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rtxmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Oct 30 12:19:56.345160 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:56.344974 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rtxmt" podUID="f36337d3-669a-4d7d-a157-b8a5f78aff17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Oct 30 12:19:57.344971 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:57.344933 4908 patch_prober.go:29] interesting pod/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rtxmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Oct 30 12:19:57.345504 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:57.344997 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rtxmt" podUID="f36337d3-669a-4d7d-a157-b8a5f78aff17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Oct 30 12:19:58.344916 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:58.344689 4908 patch_prober.go:29] interesting pod/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rtxmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Oct 30 12:19:58.344916 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:58.344772 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rtxmt" podUID="f36337d3-669a-4d7d-a157-b8a5f78aff17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Oct 30 12:19:58.518090 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:19:58.518048 4908 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3e63d4d_0257_41a0_9b25_24a4c7514b95.slice\": RecentStats: unable to find data in memory cache]" Oct 30 12:19:58.518480 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:19:58.518337 4908 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3e63d4d_0257_41a0_9b25_24a4c7514b95.slice\": RecentStats: unable to find data in memory cache]" Oct 30 12:19:59.345042 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:59.345005 4908 patch_prober.go:29] interesting pod/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rtxmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Oct 30 12:19:59.345231 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:19:59.345060 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rtxmt" podUID="f36337d3-669a-4d7d-a157-b8a5f78aff17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Oct 30 12:20:00.054001 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:00.053965 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:20:00.054092 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:00.054044 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:20:00.345209 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:00.345122 4908 patch_prober.go:29] interesting pod/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rtxmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Oct 30 12:20:00.345209 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:00.345194 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rtxmt" podUID="f36337d3-669a-4d7d-a157-b8a5f78aff17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Oct 30 12:20:00.365840 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:00.365781 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vmxrp" event={"ID":"8c130b2b-0577-4020-82e0-2c6c1e48f653","Type":"ContainerStarted","Data":"1f4b13342a0b1375685138fdfca93c9cd3227a3e44e33b996e934543d07b8a91"} Oct 30 12:20:00.366030 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:00.365885 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vmxrp" Oct 30 12:20:00.432626 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:00.432569 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vmxrp" podStartSLOduration=3.858715733 podStartE2EDuration="9.432554112s" totalImagesPullingTime="5.573838379s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:19:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:20:00.416505958 +0000 UTC m=+1556.930934704" watchObservedRunningTime="2026-10-30 12:20:00.432554112 +0000 UTC m=+1556.946982858" Oct 30 12:20:01.378687 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:01.378657 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vmxrp" Oct 30 12:20:01.661429 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:01.661257 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rtxmt" Oct 30 12:20:01.662101 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:01.662074 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rtxmt" Oct 30 12:20:01.663204 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:01.663186 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rtxmt" Oct 30 12:20:03.939263 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:03.939231 4908 scope.go:122] "RemoveContainer" containerID="76b21b36f8f7a30e6f4f9d0c0616e068c21fb97c98ed0fd4c372a071dc8172e9" Oct 30 12:20:03.955119 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:03.955093 4908 scope.go:122] "RemoveContainer" containerID="e03de40fb42cb31c9361facc54b227acdd9afbb115b21e5bca29dbb7af57fe0d" Oct 30 12:20:03.965663 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:03.965639 4908 scope.go:122] "RemoveContainer" containerID="d49837e4250ee2677404f86d1053b22de1e8df5ea0d5a7b693608d931120b6bc" Oct 30 12:20:03.976030 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:03.976008 4908 scope.go:122] "RemoveContainer" containerID="7d3ba06bae620988613653f8cd2c0a3682cccec5434f6a132a1e7f7cdb944915" Oct 30 12:20:03.986579 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:03.986546 4908 scope.go:122] "RemoveContainer" containerID="ff1502cb89ada31dd2b2dfcbcf8e41d629eaddc0c51786985575fa91f36d9f4f" Oct 30 12:20:07.265681 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:07.265636 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9wwfc"] Oct 30 12:20:09.762126 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:09.762088 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e447a5b1-crx2n"] Oct 30 12:20:09.768276 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:09.768250 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e447a5b1-crx2n" Oct 30 12:20:09.782071 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:09.782040 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 12:20:09.782341 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:09.782325 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 12:20:09.782610 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:09.782594 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 12:20:09.782770 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:09.782387 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s75hq\"" Oct 30 12:20:09.793672 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:09.793646 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e447a5b1-crx2n"] Oct 30 12:20:09.855646 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:09.855590 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71f7eeee-de32-4636-a9d1-314b6046c3a3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e447a5b1-crx2n\" (UID: \"71f7eeee-de32-4636-a9d1-314b6046c3a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e447a5b1-crx2n" Oct 30 12:20:09.855848 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:09.855760 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71f7eeee-de32-4636-a9d1-314b6046c3a3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e447a5b1-crx2n\" (UID: \"71f7eeee-de32-4636-a9d1-314b6046c3a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e447a5b1-crx2n" Oct 30 12:20:09.855848 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:09.855820 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnwx5\" (UniqueName: \"kubernetes.io/projected/71f7eeee-de32-4636-a9d1-314b6046c3a3-kube-api-access-qnwx5\") pod \"maas-api-revoke-sub-e447a5b1-crx2n\" (UID: \"71f7eeee-de32-4636-a9d1-314b6046c3a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e447a5b1-crx2n" Oct 30 12:20:09.956973 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:09.956939 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71f7eeee-de32-4636-a9d1-314b6046c3a3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e447a5b1-crx2n\" (UID: \"71f7eeee-de32-4636-a9d1-314b6046c3a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e447a5b1-crx2n" Oct 30 12:20:09.957138 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:09.956988 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnwx5\" (UniqueName: \"kubernetes.io/projected/71f7eeee-de32-4636-a9d1-314b6046c3a3-kube-api-access-qnwx5\") pod \"maas-api-revoke-sub-e447a5b1-crx2n\" (UID: \"71f7eeee-de32-4636-a9d1-314b6046c3a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e447a5b1-crx2n" Oct 30 12:20:09.957222 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:09.957166 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71f7eeee-de32-4636-a9d1-314b6046c3a3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e447a5b1-crx2n\" (UID: \"71f7eeee-de32-4636-a9d1-314b6046c3a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e447a5b1-crx2n" Oct 30 12:20:09.957787 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:09.957765 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71f7eeee-de32-4636-a9d1-314b6046c3a3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e447a5b1-crx2n\" (UID: \"71f7eeee-de32-4636-a9d1-314b6046c3a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e447a5b1-crx2n" Oct 30 12:20:09.959575 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:09.959549 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71f7eeee-de32-4636-a9d1-314b6046c3a3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e447a5b1-crx2n\" (UID: \"71f7eeee-de32-4636-a9d1-314b6046c3a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e447a5b1-crx2n" Oct 30 12:20:09.980882 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:09.980847 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnwx5\" (UniqueName: \"kubernetes.io/projected/71f7eeee-de32-4636-a9d1-314b6046c3a3-kube-api-access-qnwx5\") pod \"maas-api-revoke-sub-e447a5b1-crx2n\" (UID: \"71f7eeee-de32-4636-a9d1-314b6046c3a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e447a5b1-crx2n" Oct 30 12:20:10.085300 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:10.085210 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e447a5b1-crx2n" Oct 30 12:20:10.396976 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:10.396940 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e447a5b1-crx2n"] Oct 30 12:20:10.465026 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:10.464981 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e447a5b1-crx2n" event={"ID":"71f7eeee-de32-4636-a9d1-314b6046c3a3","Type":"ContainerStarted","Data":"849f038a169941832edd28dcb1e53281ead302d76553ea2658a45434ef160f31"} Oct 30 12:20:11.476922 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:11.476885 4908 generic.go:356] "Generic (PLEG): container finished" podID="71f7eeee-de32-4636-a9d1-314b6046c3a3" containerID="17f8e796b274bbbe570e8163d5d003f6b963b3c0f8b47123d7a208011d25332f" exitCode=0 Oct 30 12:20:11.477353 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:11.476977 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e447a5b1-crx2n" event={"ID":"71f7eeee-de32-4636-a9d1-314b6046c3a3","Type":"ContainerDied","Data":"17f8e796b274bbbe570e8163d5d003f6b963b3c0f8b47123d7a208011d25332f"} Oct 30 12:20:12.695157 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:12.695130 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e447a5b1-crx2n" Oct 30 12:20:12.799512 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:12.799404 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71f7eeee-de32-4636-a9d1-314b6046c3a3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71f7eeee-de32-4636-a9d1-314b6046c3a3-maas-api-openshift-ca\") pod \"71f7eeee-de32-4636-a9d1-314b6046c3a3\" (UID: \"71f7eeee-de32-4636-a9d1-314b6046c3a3\") " Oct 30 12:20:12.799696 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:12.799571 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/71f7eeee-de32-4636-a9d1-314b6046c3a3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71f7eeee-de32-4636-a9d1-314b6046c3a3-maas-api-xks-ca\") pod \"71f7eeee-de32-4636-a9d1-314b6046c3a3\" (UID: \"71f7eeee-de32-4636-a9d1-314b6046c3a3\") " Oct 30 12:20:12.799696 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:12.799614 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71f7eeee-de32-4636-a9d1-314b6046c3a3-kube-api-access-qnwx5\" (UniqueName: \"kubernetes.io/projected/71f7eeee-de32-4636-a9d1-314b6046c3a3-kube-api-access-qnwx5\") pod \"71f7eeee-de32-4636-a9d1-314b6046c3a3\" (UID: \"71f7eeee-de32-4636-a9d1-314b6046c3a3\") " Oct 30 12:20:12.799816 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:12.799774 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71f7eeee-de32-4636-a9d1-314b6046c3a3-maas-api-openshift-ca" pod "71f7eeee-de32-4636-a9d1-314b6046c3a3" (UID: "71f7eeee-de32-4636-a9d1-314b6046c3a3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:20:12.800269 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:12.800246 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71f7eeee-de32-4636-a9d1-314b6046c3a3-maas-api-openshift-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:20:12.801841 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:12.801799 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71f7eeee-de32-4636-a9d1-314b6046c3a3-maas-api-xks-ca" pod "71f7eeee-de32-4636-a9d1-314b6046c3a3" (UID: "71f7eeee-de32-4636-a9d1-314b6046c3a3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:20:12.801969 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:12.801951 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71f7eeee-de32-4636-a9d1-314b6046c3a3-kube-api-access-qnwx5" pod "71f7eeee-de32-4636-a9d1-314b6046c3a3" (UID: "71f7eeee-de32-4636-a9d1-314b6046c3a3"). InnerVolumeSpecName "kube-api-access-qnwx5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:20:12.901455 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:12.901410 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71f7eeee-de32-4636-a9d1-314b6046c3a3-maas-api-xks-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:20:12.901455 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:12.901452 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qnwx5\" (UniqueName: \"kubernetes.io/projected/71f7eeee-de32-4636-a9d1-314b6046c3a3-kube-api-access-qnwx5\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:20:13.497026 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:13.496977 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e447a5b1-crx2n" event={"ID":"71f7eeee-de32-4636-a9d1-314b6046c3a3","Type":"ContainerDied","Data":"849f038a169941832edd28dcb1e53281ead302d76553ea2658a45434ef160f31"} Oct 30 12:20:13.497209 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:13.497043 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="849f038a169941832edd28dcb1e53281ead302d76553ea2658a45434ef160f31" Oct 30 12:20:13.497209 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:13.496995 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e447a5b1-crx2n" Oct 30 12:20:19.691017 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:20:19.690930 4908 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vn6vv" containerName="extractor" Oct 30 12:20:20.104965 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:20.104928 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a04a4d4c-4jxqh"] Oct 30 12:20:20.105549 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:20.105529 4908 cpu_manager.go:414] "containerMap: removing container" podUID="71f7eeee-de32-4636-a9d1-314b6046c3a3" containerName="revoke-keys" Oct 30 12:20:20.105549 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:20.105549 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71f7eeee-de32-4636-a9d1-314b6046c3a3" containerName="revoke-keys" Oct 30 12:20:20.105678 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:20.105659 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="71f7eeee-de32-4636-a9d1-314b6046c3a3" containerName="revoke-keys" Oct 30 12:20:20.112035 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:20.112013 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a04a4d4c-4jxqh" Oct 30 12:20:20.128070 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:20.128044 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a04a4d4c-4jxqh"] Oct 30 12:20:20.133735 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:20.133465 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 12:20:20.133735 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:20.133620 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s75hq\"" Oct 30 12:20:20.133735 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:20.133700 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 12:20:20.136393 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:20.136372 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 12:20:20.195742 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:20.195715 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/479b056c-a79f-489b-8e05-10bddcb95072-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a04a4d4c-4jxqh\" (UID: \"479b056c-a79f-489b-8e05-10bddcb95072\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a04a4d4c-4jxqh" Oct 30 12:20:20.195921 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:20.195821 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/479b056c-a79f-489b-8e05-10bddcb95072-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a04a4d4c-4jxqh\" (UID: \"479b056c-a79f-489b-8e05-10bddcb95072\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a04a4d4c-4jxqh" Oct 30 12:20:20.196007 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:20.195992 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gp6z\" (UniqueName: \"kubernetes.io/projected/479b056c-a79f-489b-8e05-10bddcb95072-kube-api-access-2gp6z\") pod \"maas-api-revoke-sub-a04a4d4c-4jxqh\" (UID: \"479b056c-a79f-489b-8e05-10bddcb95072\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a04a4d4c-4jxqh" Oct 30 12:20:20.296661 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:20.296630 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gp6z\" (UniqueName: \"kubernetes.io/projected/479b056c-a79f-489b-8e05-10bddcb95072-kube-api-access-2gp6z\") pod \"maas-api-revoke-sub-a04a4d4c-4jxqh\" (UID: \"479b056c-a79f-489b-8e05-10bddcb95072\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a04a4d4c-4jxqh" Oct 30 12:20:20.296813 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:20.296700 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/479b056c-a79f-489b-8e05-10bddcb95072-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a04a4d4c-4jxqh\" (UID: \"479b056c-a79f-489b-8e05-10bddcb95072\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a04a4d4c-4jxqh" Oct 30 12:20:20.296813 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:20.296744 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/479b056c-a79f-489b-8e05-10bddcb95072-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a04a4d4c-4jxqh\" (UID: \"479b056c-a79f-489b-8e05-10bddcb95072\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a04a4d4c-4jxqh" Oct 30 12:20:20.297681 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:20.297652 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/479b056c-a79f-489b-8e05-10bddcb95072-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a04a4d4c-4jxqh\" (UID: \"479b056c-a79f-489b-8e05-10bddcb95072\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a04a4d4c-4jxqh" Oct 30 12:20:20.299157 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:20.299126 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/479b056c-a79f-489b-8e05-10bddcb95072-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a04a4d4c-4jxqh\" (UID: \"479b056c-a79f-489b-8e05-10bddcb95072\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a04a4d4c-4jxqh" Oct 30 12:20:20.319343 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:20.319321 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gp6z\" (UniqueName: \"kubernetes.io/projected/479b056c-a79f-489b-8e05-10bddcb95072-kube-api-access-2gp6z\") pod \"maas-api-revoke-sub-a04a4d4c-4jxqh\" (UID: \"479b056c-a79f-489b-8e05-10bddcb95072\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a04a4d4c-4jxqh" Oct 30 12:20:20.426495 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:20.426426 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a04a4d4c-4jxqh" Oct 30 12:20:20.743296 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:20.743261 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a04a4d4c-4jxqh"] Oct 30 12:20:20.744639 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:20:20.744614 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod479b056c_a79f_489b_8e05_10bddcb95072.slice/crio-afbd12afa8fd62dd773edddb52c00a793b740e5bc36ca0b5c8e7fdfc1acb6dad WatchSource:0}: Error finding container afbd12afa8fd62dd773edddb52c00a793b740e5bc36ca0b5c8e7fdfc1acb6dad: Status 404 returned error can't find the container with id afbd12afa8fd62dd773edddb52c00a793b740e5bc36ca0b5c8e7fdfc1acb6dad Oct 30 12:20:21.568736 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:21.568699 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a04a4d4c-4jxqh" event={"ID":"479b056c-a79f-489b-8e05-10bddcb95072","Type":"ContainerStarted","Data":"61c5899fd11c9c752ab32ac7a53fbdc834f5954fd4bbedd6e606f0fdbad41b56"} Oct 30 12:20:21.568950 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:21.568755 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a04a4d4c-4jxqh" event={"ID":"479b056c-a79f-489b-8e05-10bddcb95072","Type":"ContainerStarted","Data":"afbd12afa8fd62dd773edddb52c00a793b740e5bc36ca0b5c8e7fdfc1acb6dad"} Oct 30 12:20:21.612003 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:21.611933 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a04a4d4c-4jxqh" podStartSLOduration=1.611916889 podStartE2EDuration="1.611916889s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:20:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:20:21.606883253 +0000 UTC m=+1578.121312004" watchObservedRunningTime="2026-10-30 12:20:21.611916889 +0000 UTC m=+1578.126345635" Oct 30 12:20:22.582384 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:22.582345 4908 generic.go:356] "Generic (PLEG): container finished" podID="479b056c-a79f-489b-8e05-10bddcb95072" containerID="61c5899fd11c9c752ab32ac7a53fbdc834f5954fd4bbedd6e606f0fdbad41b56" exitCode=0 Oct 30 12:20:22.582931 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:22.582582 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a04a4d4c-4jxqh" event={"ID":"479b056c-a79f-489b-8e05-10bddcb95072","Type":"ContainerDied","Data":"61c5899fd11c9c752ab32ac7a53fbdc834f5954fd4bbedd6e606f0fdbad41b56"} Oct 30 12:20:23.849681 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:23.849614 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a04a4d4c-4jxqh" Oct 30 12:20:23.954329 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:23.954291 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/479b056c-a79f-489b-8e05-10bddcb95072-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/479b056c-a79f-489b-8e05-10bddcb95072-maas-api-openshift-ca\") pod \"479b056c-a79f-489b-8e05-10bddcb95072\" (UID: \"479b056c-a79f-489b-8e05-10bddcb95072\") " Oct 30 12:20:23.954529 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:23.954448 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/479b056c-a79f-489b-8e05-10bddcb95072-kube-api-access-2gp6z\" (UniqueName: \"kubernetes.io/projected/479b056c-a79f-489b-8e05-10bddcb95072-kube-api-access-2gp6z\") pod \"479b056c-a79f-489b-8e05-10bddcb95072\" (UID: \"479b056c-a79f-489b-8e05-10bddcb95072\") " Oct 30 12:20:23.954600 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:23.954580 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/479b056c-a79f-489b-8e05-10bddcb95072-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/479b056c-a79f-489b-8e05-10bddcb95072-maas-api-xks-ca\") pod \"479b056c-a79f-489b-8e05-10bddcb95072\" (UID: \"479b056c-a79f-489b-8e05-10bddcb95072\") " Oct 30 12:20:23.954676 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:23.954647 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/479b056c-a79f-489b-8e05-10bddcb95072-maas-api-openshift-ca" pod "479b056c-a79f-489b-8e05-10bddcb95072" (UID: "479b056c-a79f-489b-8e05-10bddcb95072"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:20:23.955584 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:23.955556 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/479b056c-a79f-489b-8e05-10bddcb95072-maas-api-openshift-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:20:23.956612 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:23.956586 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/479b056c-a79f-489b-8e05-10bddcb95072-maas-api-xks-ca" pod "479b056c-a79f-489b-8e05-10bddcb95072" (UID: "479b056c-a79f-489b-8e05-10bddcb95072"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:20:23.956786 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:23.956768 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/479b056c-a79f-489b-8e05-10bddcb95072-kube-api-access-2gp6z" pod "479b056c-a79f-489b-8e05-10bddcb95072" (UID: "479b056c-a79f-489b-8e05-10bddcb95072"). InnerVolumeSpecName "kube-api-access-2gp6z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:20:24.056652 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:24.056620 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2gp6z\" (UniqueName: \"kubernetes.io/projected/479b056c-a79f-489b-8e05-10bddcb95072-kube-api-access-2gp6z\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:20:24.056652 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:24.056650 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/479b056c-a79f-489b-8e05-10bddcb95072-maas-api-xks-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:20:24.599969 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:24.599930 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a04a4d4c-4jxqh" event={"ID":"479b056c-a79f-489b-8e05-10bddcb95072","Type":"ContainerDied","Data":"afbd12afa8fd62dd773edddb52c00a793b740e5bc36ca0b5c8e7fdfc1acb6dad"} Oct 30 12:20:24.600128 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:24.599986 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="afbd12afa8fd62dd773edddb52c00a793b740e5bc36ca0b5c8e7fdfc1acb6dad" Oct 30 12:20:24.600128 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:24.599942 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a04a4d4c-4jxqh" Oct 30 12:20:45.886776 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:45.886732 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9wwfc"] Oct 30 12:20:47.898514 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:47.898482 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-tjnq5"] Oct 30 12:20:47.899057 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:47.899041 4908 cpu_manager.go:414] "containerMap: removing container" podUID="479b056c-a79f-489b-8e05-10bddcb95072" containerName="revoke-keys" Oct 30 12:20:47.899106 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:47.899057 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="479b056c-a79f-489b-8e05-10bddcb95072" containerName="revoke-keys" Oct 30 12:20:47.899171 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:47.899162 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="479b056c-a79f-489b-8e05-10bddcb95072" containerName="revoke-keys" Oct 30 12:20:47.903193 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:47.903172 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-tjnq5" Oct 30 12:20:47.927303 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:47.927277 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-tjnq5"] Oct 30 12:20:47.935554 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:47.935530 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-rv92b"] Oct 30 12:20:47.941140 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:47.941117 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-rv92b" Oct 30 12:20:47.959393 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:47.959337 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-rv92b"] Oct 30 12:20:47.985411 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:47.985384 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3e7bd6e-7283-451a-8df8-cc9634908630-istio-podinfo\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-tjnq5\" (UID: \"c3e7bd6e-7283-451a-8df8-cc9634908630\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-tjnq5" Oct 30 12:20:47.985533 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:47.985417 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stglb\" (UniqueName: \"kubernetes.io/projected/c3e7bd6e-7283-451a-8df8-cc9634908630-kube-api-access-stglb\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-tjnq5\" (UID: \"c3e7bd6e-7283-451a-8df8-cc9634908630\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-tjnq5" Oct 30 12:20:47.985533 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:47.985447 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3e7bd6e-7283-451a-8df8-cc9634908630-credential-socket\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-tjnq5\" (UID: \"c3e7bd6e-7283-451a-8df8-cc9634908630\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-tjnq5" Oct 30 12:20:47.985671 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:47.985556 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3e7bd6e-7283-451a-8df8-cc9634908630-workload-socket\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-tjnq5\" (UID: \"c3e7bd6e-7283-451a-8df8-cc9634908630\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-tjnq5" Oct 30 12:20:47.985671 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:47.985619 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3e7bd6e-7283-451a-8df8-cc9634908630-istiod-ca-cert\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-tjnq5\" (UID: \"c3e7bd6e-7283-451a-8df8-cc9634908630\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-tjnq5" Oct 30 12:20:47.985671 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:47.985659 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3e7bd6e-7283-451a-8df8-cc9634908630-istio-envoy\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-tjnq5\" (UID: \"c3e7bd6e-7283-451a-8df8-cc9634908630\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-tjnq5" Oct 30 12:20:47.985853 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:47.985678 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3e7bd6e-7283-451a-8df8-cc9634908630-istio-data\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-tjnq5\" (UID: \"c3e7bd6e-7283-451a-8df8-cc9634908630\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-tjnq5" Oct 30 12:20:47.985918 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:47.985862 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3e7bd6e-7283-451a-8df8-cc9634908630-istio-token\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-tjnq5\" (UID: \"c3e7bd6e-7283-451a-8df8-cc9634908630\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-tjnq5" Oct 30 12:20:47.985974 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:47.985915 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3e7bd6e-7283-451a-8df8-cc9634908630-workload-certs\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-tjnq5\" (UID: \"c3e7bd6e-7283-451a-8df8-cc9634908630\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-tjnq5" Oct 30 12:20:48.087252 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:48.087225 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3e7bd6e-7283-451a-8df8-cc9634908630-istiod-ca-cert\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-tjnq5\" (UID: \"c3e7bd6e-7283-451a-8df8-cc9634908630\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-tjnq5" Oct 30 12:20:48.087420 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:48.087284 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3e7bd6e-7283-451a-8df8-cc9634908630-istio-envoy\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-tjnq5\" (UID: \"c3e7bd6e-7283-451a-8df8-cc9634908630\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-tjnq5" Oct 30 12:20:48.087420 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:48.087306 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3e7bd6e-7283-451a-8df8-cc9634908630-istio-data\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-tjnq5\" (UID: \"c3e7bd6e-7283-451a-8df8-cc9634908630\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-tjnq5" Oct 30 12:20:48.087420 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:48.087330 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4630b841-bf2e-4019-b450-12e85497afc1-workload-certs\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-rv92b\" (UID: \"4630b841-bf2e-4019-b450-12e85497afc1\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-rv92b" Oct 30 12:20:48.087420 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:48.087368 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4630b841-bf2e-4019-b450-12e85497afc1-istiod-ca-cert\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-rv92b\" (UID: \"4630b841-bf2e-4019-b450-12e85497afc1\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-rv92b" Oct 30 12:20:48.087603 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:48.087439 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4630b841-bf2e-4019-b450-12e85497afc1-istio-podinfo\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-rv92b\" (UID: \"4630b841-bf2e-4019-b450-12e85497afc1\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-rv92b" Oct 30 12:20:48.087603 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:48.087466 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7md78\" (UniqueName: \"kubernetes.io/projected/4630b841-bf2e-4019-b450-12e85497afc1-kube-api-access-7md78\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-rv92b\" (UID: \"4630b841-bf2e-4019-b450-12e85497afc1\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-rv92b" Oct 30 12:20:48.087603 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:48.087567 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4630b841-bf2e-4019-b450-12e85497afc1-istio-data\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-rv92b\" (UID: \"4630b841-bf2e-4019-b450-12e85497afc1\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-rv92b" Oct 30 12:20:48.087716 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:48.087615 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3e7bd6e-7283-451a-8df8-cc9634908630-istio-token\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-tjnq5\" (UID: \"c3e7bd6e-7283-451a-8df8-cc9634908630\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-tjnq5" Oct 30 12:20:48.087716 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:48.087688 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3e7bd6e-7283-451a-8df8-cc9634908630-istio-data\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-tjnq5\" (UID: \"c3e7bd6e-7283-451a-8df8-cc9634908630\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-tjnq5" Oct 30 12:20:48.087820 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:48.087732 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4630b841-bf2e-4019-b450-12e85497afc1-istio-token\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-rv92b\" (UID: \"4630b841-bf2e-4019-b450-12e85497afc1\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-rv92b" Oct 30 12:20:48.087820 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:48.087773 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3e7bd6e-7283-451a-8df8-cc9634908630-workload-certs\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-tjnq5\" (UID: \"c3e7bd6e-7283-451a-8df8-cc9634908630\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-tjnq5" Oct 30 12:20:48.087975 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:48.087956 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3e7bd6e-7283-451a-8df8-cc9634908630-istiod-ca-cert\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-tjnq5\" (UID: \"c3e7bd6e-7283-451a-8df8-cc9634908630\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-tjnq5" Oct 30 12:20:48.087975 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:48.087957 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4630b841-bf2e-4019-b450-12e85497afc1-workload-socket\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-rv92b\" (UID: \"4630b841-bf2e-4019-b450-12e85497afc1\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-rv92b" Oct 30 12:20:48.088063 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:48.088004 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4630b841-bf2e-4019-b450-12e85497afc1-istio-envoy\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-rv92b\" (UID: \"4630b841-bf2e-4019-b450-12e85497afc1\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-rv92b" Oct 30 12:20:48.088063 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:48.088058 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3e7bd6e-7283-451a-8df8-cc9634908630-istio-podinfo\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-tjnq5\" (UID: \"c3e7bd6e-7283-451a-8df8-cc9634908630\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-tjnq5" Oct 30 12:20:48.088139 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:48.088075 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stglb\" (UniqueName: \"kubernetes.io/projected/c3e7bd6e-7283-451a-8df8-cc9634908630-kube-api-access-stglb\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-tjnq5\" (UID: \"c3e7bd6e-7283-451a-8df8-cc9634908630\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-tjnq5" Oct 30 12:20:48.088139 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:48.088077 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3e7bd6e-7283-451a-8df8-cc9634908630-workload-certs\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-tjnq5\" (UID: \"c3e7bd6e-7283-451a-8df8-cc9634908630\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-tjnq5" Oct 30 12:20:48.088139 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:48.088112 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3e7bd6e-7283-451a-8df8-cc9634908630-credential-socket\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-tjnq5\" (UID: \"c3e7bd6e-7283-451a-8df8-cc9634908630\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-tjnq5" Oct 30 12:20:48.088139 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:48.088134 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4630b841-bf2e-4019-b450-12e85497afc1-credential-socket\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-rv92b\" (UID: \"4630b841-bf2e-4019-b450-12e85497afc1\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-rv92b" Oct 30 12:20:48.088332 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:48.088170 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3e7bd6e-7283-451a-8df8-cc9634908630-workload-socket\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-tjnq5\" (UID: \"c3e7bd6e-7283-451a-8df8-cc9634908630\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-tjnq5" Oct 30 12:20:48.088477 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:48.088458 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3e7bd6e-7283-451a-8df8-cc9634908630-credential-socket\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-tjnq5\" (UID: \"c3e7bd6e-7283-451a-8df8-cc9634908630\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-tjnq5" Oct 30 12:20:48.088477 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:48.088469 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3e7bd6e-7283-451a-8df8-cc9634908630-workload-socket\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-tjnq5\" (UID: \"c3e7bd6e-7283-451a-8df8-cc9634908630\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-tjnq5" Oct 30 12:20:48.089735 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:48.089713 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3e7bd6e-7283-451a-8df8-cc9634908630-istio-envoy\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-tjnq5\" (UID: \"c3e7bd6e-7283-451a-8df8-cc9634908630\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-tjnq5" Oct 30 12:20:48.090199 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:48.090180 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3e7bd6e-7283-451a-8df8-cc9634908630-istio-podinfo\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-tjnq5\" (UID: \"c3e7bd6e-7283-451a-8df8-cc9634908630\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-tjnq5" Oct 30 12:20:48.109035 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:48.109008 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3e7bd6e-7283-451a-8df8-cc9634908630-istio-token\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-tjnq5\" (UID: \"c3e7bd6e-7283-451a-8df8-cc9634908630\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-tjnq5" Oct 30 12:20:48.109414 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:48.109396 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-stglb\" (UniqueName: \"kubernetes.io/projected/c3e7bd6e-7283-451a-8df8-cc9634908630-kube-api-access-stglb\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-tjnq5\" (UID: \"c3e7bd6e-7283-451a-8df8-cc9634908630\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-tjnq5" Oct 30 12:20:48.189739 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:48.189689 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4630b841-bf2e-4019-b450-12e85497afc1-workload-socket\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-rv92b\" (UID: \"4630b841-bf2e-4019-b450-12e85497afc1\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-rv92b" Oct 30 12:20:48.189739 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:48.189723 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4630b841-bf2e-4019-b450-12e85497afc1-istio-envoy\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-rv92b\" (UID: \"4630b841-bf2e-4019-b450-12e85497afc1\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-rv92b" Oct 30 12:20:48.189903 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:48.189874 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4630b841-bf2e-4019-b450-12e85497afc1-credential-socket\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-rv92b\" (UID: \"4630b841-bf2e-4019-b450-12e85497afc1\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-rv92b" Oct 30 12:20:48.189955 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:48.189940 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4630b841-bf2e-4019-b450-12e85497afc1-workload-certs\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-rv92b\" (UID: \"4630b841-bf2e-4019-b450-12e85497afc1\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-rv92b" Oct 30 12:20:48.190000 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:48.189985 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4630b841-bf2e-4019-b450-12e85497afc1-istiod-ca-cert\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-rv92b\" (UID: \"4630b841-bf2e-4019-b450-12e85497afc1\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-rv92b" Oct 30 12:20:48.190050 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:48.190021 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4630b841-bf2e-4019-b450-12e85497afc1-istio-podinfo\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-rv92b\" (UID: \"4630b841-bf2e-4019-b450-12e85497afc1\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-rv92b" Oct 30 12:20:48.190050 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:48.190042 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7md78\" (UniqueName: \"kubernetes.io/projected/4630b841-bf2e-4019-b450-12e85497afc1-kube-api-access-7md78\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-rv92b\" (UID: \"4630b841-bf2e-4019-b450-12e85497afc1\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-rv92b" Oct 30 12:20:48.190154 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:48.190079 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4630b841-bf2e-4019-b450-12e85497afc1-istio-data\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-rv92b\" (UID: \"4630b841-bf2e-4019-b450-12e85497afc1\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-rv92b" Oct 30 12:20:48.190154 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:48.190119 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4630b841-bf2e-4019-b450-12e85497afc1-istio-token\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-rv92b\" (UID: \"4630b841-bf2e-4019-b450-12e85497afc1\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-rv92b" Oct 30 12:20:48.190154 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:48.190138 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4630b841-bf2e-4019-b450-12e85497afc1-workload-socket\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-rv92b\" (UID: \"4630b841-bf2e-4019-b450-12e85497afc1\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-rv92b" Oct 30 12:20:48.190319 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:48.190239 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4630b841-bf2e-4019-b450-12e85497afc1-credential-socket\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-rv92b\" (UID: \"4630b841-bf2e-4019-b450-12e85497afc1\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-rv92b" Oct 30 12:20:48.190319 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:48.190288 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4630b841-bf2e-4019-b450-12e85497afc1-workload-certs\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-rv92b\" (UID: \"4630b841-bf2e-4019-b450-12e85497afc1\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-rv92b" Oct 30 12:20:48.190535 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:48.190511 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4630b841-bf2e-4019-b450-12e85497afc1-istio-data\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-rv92b\" (UID: \"4630b841-bf2e-4019-b450-12e85497afc1\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-rv92b" Oct 30 12:20:48.190737 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:48.190717 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4630b841-bf2e-4019-b450-12e85497afc1-istiod-ca-cert\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-rv92b\" (UID: \"4630b841-bf2e-4019-b450-12e85497afc1\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-rv92b" Oct 30 12:20:48.191722 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:48.191703 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4630b841-bf2e-4019-b450-12e85497afc1-istio-envoy\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-rv92b\" (UID: \"4630b841-bf2e-4019-b450-12e85497afc1\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-rv92b" Oct 30 12:20:48.192274 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:48.192254 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4630b841-bf2e-4019-b450-12e85497afc1-istio-podinfo\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-rv92b\" (UID: \"4630b841-bf2e-4019-b450-12e85497afc1\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-rv92b" Oct 30 12:20:48.215008 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:48.214986 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4630b841-bf2e-4019-b450-12e85497afc1-istio-token\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-rv92b\" (UID: \"4630b841-bf2e-4019-b450-12e85497afc1\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-rv92b" Oct 30 12:20:48.215091 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:48.215033 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7md78\" (UniqueName: \"kubernetes.io/projected/4630b841-bf2e-4019-b450-12e85497afc1-kube-api-access-7md78\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-rv92b\" (UID: \"4630b841-bf2e-4019-b450-12e85497afc1\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-rv92b" Oct 30 12:20:48.217892 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:48.217873 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-tjnq5" Oct 30 12:20:48.261569 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:48.261539 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-rv92b" Oct 30 12:20:48.545964 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:48.545935 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-tjnq5"] Oct 30 12:20:48.547293 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:20:48.547265 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3e7bd6e_7283_451a_8df8_cc9634908630.slice/crio-29e660461b82d900aa1754b7f3b05f576a4a76c8ee3fd53706d79bd0ba3305f5 WatchSource:0}: Error finding container 29e660461b82d900aa1754b7f3b05f576a4a76c8ee3fd53706d79bd0ba3305f5: Status 404 returned error can't find the container with id 29e660461b82d900aa1754b7f3b05f576a4a76c8ee3fd53706d79bd0ba3305f5 Oct 30 12:20:48.550266 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:48.550230 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:20:48.550365 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:48.550295 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:20:48.550365 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:48.550323 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:20:48.632430 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:20:48.632399 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4630b841_bf2e_4019_b450_12e85497afc1.slice/crio-86b211fca371add4df61097c94ef65e489f7e8720ba3918129f1c2d26f02ff9c WatchSource:0}: Error finding container 86b211fca371add4df61097c94ef65e489f7e8720ba3918129f1c2d26f02ff9c: Status 404 returned error can't find the container with id 86b211fca371add4df61097c94ef65e489f7e8720ba3918129f1c2d26f02ff9c Oct 30 12:20:48.635233 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:48.635204 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:20:48.635320 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:48.635268 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:20:48.635320 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:48.635314 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:20:48.637907 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:48.637879 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-rv92b"] Oct 30 12:20:48.854389 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:48.854009 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-rv92b" event={"ID":"4630b841-bf2e-4019-b450-12e85497afc1","Type":"ContainerStarted","Data":"8244474829cc517bf079988c2803e3cb5b3b5c8d9e715f7210f6e7a4a1620646"} Oct 30 12:20:48.854663 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:48.854626 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-rv92b" event={"ID":"4630b841-bf2e-4019-b450-12e85497afc1","Type":"ContainerStarted","Data":"86b211fca371add4df61097c94ef65e489f7e8720ba3918129f1c2d26f02ff9c"} Oct 30 12:20:48.856059 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:48.856036 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-tjnq5" event={"ID":"c3e7bd6e-7283-451a-8df8-cc9634908630","Type":"ContainerStarted","Data":"6c617bec82991ea0e7d318e4505a289d67bffd53172b1f9794c5a407e8884554"} Oct 30 12:20:48.856150 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:48.856078 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-tjnq5" event={"ID":"c3e7bd6e-7283-451a-8df8-cc9634908630","Type":"ContainerStarted","Data":"29e660461b82d900aa1754b7f3b05f576a4a76c8ee3fd53706d79bd0ba3305f5"} Oct 30 12:20:48.947441 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:48.947396 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-rv92b" podStartSLOduration=1.9473826779999999 podStartE2EDuration="1.947382678s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:20:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:20:48.902488434 +0000 UTC m=+1605.416917180" watchObservedRunningTime="2026-10-30 12:20:48.947382678 +0000 UTC m=+1605.461811425" Oct 30 12:20:48.951140 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:48.951100 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-tjnq5" podStartSLOduration=1.9510877180000001 podStartE2EDuration="1.951087718s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:20:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:20:48.94670186 +0000 UTC m=+1605.461130606" watchObservedRunningTime="2026-10-30 12:20:48.951087718 +0000 UTC m=+1605.465516463" Oct 30 12:20:49.220247 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:49.218885 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-tjnq5" Oct 30 12:20:49.229908 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:49.226430 4908 patch_prober.go:29] interesting pod/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-tjnq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Oct 30 12:20:49.229908 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:49.226492 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-tjnq5" podUID="c3e7bd6e-7283-451a-8df8-cc9634908630" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Oct 30 12:20:49.283851 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:49.278462 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-rv92b" Oct 30 12:20:49.283851 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:49.278658 4908 patch_prober.go:29] interesting pod/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-rv92b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Oct 30 12:20:49.283851 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:49.278703 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-rv92b" podUID="4630b841-bf2e-4019-b450-12e85497afc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Oct 30 12:20:49.735984 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:20:49.735949 4908 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vn6vv" containerName="extractor" Oct 30 12:20:50.049312 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:50.049231 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2a841b0-wnzdf"] Oct 30 12:20:50.054509 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:50.054482 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2a841b0-wnzdf"] Oct 30 12:20:50.218732 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:50.218697 4908 patch_prober.go:29] interesting pod/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-tjnq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Oct 30 12:20:50.218932 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:50.218758 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-tjnq5" podUID="c3e7bd6e-7283-451a-8df8-cc9634908630" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Oct 30 12:20:50.266853 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:50.263124 4908 patch_prober.go:29] interesting pod/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-rv92b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Oct 30 12:20:50.266853 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:50.263179 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-rv92b" podUID="4630b841-bf2e-4019-b450-12e85497afc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Oct 30 12:20:50.330773 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:50.330695 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef80232e-6rnpp"] Oct 30 12:20:50.335140 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:50.335122 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef80232e-6rnpp" Oct 30 12:20:50.348993 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:50.348972 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 12:20:50.349127 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:50.349115 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 12:20:50.349192 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:50.349172 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s75hq\"" Oct 30 12:20:50.349297 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:50.349215 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 12:20:50.352841 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:50.352806 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef80232e-6rnpp"] Oct 30 12:20:50.441841 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:50.441788 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53e329b8-a4e4-4ced-8c81-d141b31076c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ef80232e-6rnpp\" (UID: \"53e329b8-a4e4-4ced-8c81-d141b31076c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef80232e-6rnpp" Oct 30 12:20:50.442089 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:50.441952 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53e329b8-a4e4-4ced-8c81-d141b31076c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ef80232e-6rnpp\" (UID: \"53e329b8-a4e4-4ced-8c81-d141b31076c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef80232e-6rnpp" Oct 30 12:20:50.442089 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:50.442024 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpl8n\" (UniqueName: \"kubernetes.io/projected/53e329b8-a4e4-4ced-8c81-d141b31076c3-kube-api-access-cpl8n\") pod \"maas-api-revoke-sub-ef80232e-6rnpp\" (UID: \"53e329b8-a4e4-4ced-8c81-d141b31076c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef80232e-6rnpp" Oct 30 12:20:50.543422 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:50.543391 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53e329b8-a4e4-4ced-8c81-d141b31076c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ef80232e-6rnpp\" (UID: \"53e329b8-a4e4-4ced-8c81-d141b31076c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef80232e-6rnpp" Oct 30 12:20:50.543696 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:50.543458 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpl8n\" (UniqueName: \"kubernetes.io/projected/53e329b8-a4e4-4ced-8c81-d141b31076c3-kube-api-access-cpl8n\") pod \"maas-api-revoke-sub-ef80232e-6rnpp\" (UID: \"53e329b8-a4e4-4ced-8c81-d141b31076c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef80232e-6rnpp" Oct 30 12:20:50.543696 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:50.543564 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53e329b8-a4e4-4ced-8c81-d141b31076c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ef80232e-6rnpp\" (UID: \"53e329b8-a4e4-4ced-8c81-d141b31076c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef80232e-6rnpp" Oct 30 12:20:50.544109 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:50.544084 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53e329b8-a4e4-4ced-8c81-d141b31076c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ef80232e-6rnpp\" (UID: \"53e329b8-a4e4-4ced-8c81-d141b31076c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef80232e-6rnpp" Oct 30 12:20:50.545917 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:50.545894 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53e329b8-a4e4-4ced-8c81-d141b31076c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ef80232e-6rnpp\" (UID: \"53e329b8-a4e4-4ced-8c81-d141b31076c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef80232e-6rnpp" Oct 30 12:20:50.597597 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:50.597521 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpl8n\" (UniqueName: \"kubernetes.io/projected/53e329b8-a4e4-4ced-8c81-d141b31076c3-kube-api-access-cpl8n\") pod \"maas-api-revoke-sub-ef80232e-6rnpp\" (UID: \"53e329b8-a4e4-4ced-8c81-d141b31076c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef80232e-6rnpp" Oct 30 12:20:50.649204 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:50.649173 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef80232e-6rnpp" Oct 30 12:20:51.022781 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:51.022682 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef80232e-6rnpp"] Oct 30 12:20:51.024484 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:20:51.024451 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53e329b8_a4e4_4ced_8c81_d141b31076c3.slice/crio-122164d26851a8e3711b1d0b5d010e0ad3e0d5b00f791eb95a446c62c00f68d7 WatchSource:0}: Error finding container 122164d26851a8e3711b1d0b5d010e0ad3e0d5b00f791eb95a446c62c00f68d7: Status 404 returned error can't find the container with id 122164d26851a8e3711b1d0b5d010e0ad3e0d5b00f791eb95a446c62c00f68d7 Oct 30 12:20:51.218389 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:51.218339 4908 patch_prober.go:29] interesting pod/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-tjnq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Oct 30 12:20:51.218817 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:51.218419 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-tjnq5" podUID="c3e7bd6e-7283-451a-8df8-cc9634908630" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Oct 30 12:20:51.278892 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:51.264248 4908 patch_prober.go:29] interesting pod/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-rv92b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Oct 30 12:20:51.278892 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:51.264317 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-rv92b" podUID="4630b841-bf2e-4019-b450-12e85497afc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Oct 30 12:20:51.630903 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:51.628471 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="025f63d3-09a6-401f-8403-98241d3a4e24" path="/var/lib/kubelet/pods/025f63d3-09a6-401f-8403-98241d3a4e24/volumes" Oct 30 12:20:51.945165 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:51.945078 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef80232e-6rnpp" event={"ID":"53e329b8-a4e4-4ced-8c81-d141b31076c3","Type":"ContainerStarted","Data":"05d9d1e688bd95421cb442d5eae1a7345069394957ae6bc622bbcf8c78f53921"} Oct 30 12:20:51.945165 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:51.945132 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef80232e-6rnpp" event={"ID":"53e329b8-a4e4-4ced-8c81-d141b31076c3","Type":"ContainerStarted","Data":"122164d26851a8e3711b1d0b5d010e0ad3e0d5b00f791eb95a446c62c00f68d7"} Oct 30 12:20:51.999677 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:51.999617 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef80232e-6rnpp" podStartSLOduration=1.9995959239999999 podStartE2EDuration="1.999595924s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:20:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:20:51.990189423 +0000 UTC m=+1608.504618171" watchObservedRunningTime="2026-10-30 12:20:51.999595924 +0000 UTC m=+1608.514024689" Oct 30 12:20:52.219877 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:52.219192 4908 patch_prober.go:29] interesting pod/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-tjnq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Oct 30 12:20:52.219877 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:52.219251 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-tjnq5" podUID="c3e7bd6e-7283-451a-8df8-cc9634908630" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Oct 30 12:20:52.266188 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:52.266157 4908 patch_prober.go:29] interesting pod/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-rv92b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Oct 30 12:20:52.269849 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:52.266372 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-rv92b" podUID="4630b841-bf2e-4019-b450-12e85497afc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Oct 30 12:20:52.957705 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:52.957673 4908 generic.go:356] "Generic (PLEG): container finished" podID="53e329b8-a4e4-4ced-8c81-d141b31076c3" containerID="05d9d1e688bd95421cb442d5eae1a7345069394957ae6bc622bbcf8c78f53921" exitCode=0 Oct 30 12:20:52.958010 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:52.957979 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef80232e-6rnpp" event={"ID":"53e329b8-a4e4-4ced-8c81-d141b31076c3","Type":"ContainerDied","Data":"05d9d1e688bd95421cb442d5eae1a7345069394957ae6bc622bbcf8c78f53921"} Oct 30 12:20:53.221312 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:53.221193 4908 patch_prober.go:29] interesting pod/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-tjnq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Oct 30 12:20:53.221312 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:53.221252 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-tjnq5" podUID="c3e7bd6e-7283-451a-8df8-cc9634908630" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Oct 30 12:20:53.263191 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:53.263152 4908 patch_prober.go:29] interesting pod/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-rv92b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Oct 30 12:20:53.263467 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:53.263440 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-rv92b" podUID="4630b841-bf2e-4019-b450-12e85497afc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Oct 30 12:20:54.246148 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:54.245660 4908 patch_prober.go:29] interesting pod/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-tjnq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Oct 30 12:20:54.246148 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:54.245729 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-tjnq5" podUID="c3e7bd6e-7283-451a-8df8-cc9634908630" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Oct 30 12:20:54.318909 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:54.304571 4908 patch_prober.go:29] interesting pod/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-rv92b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Oct 30 12:20:54.319228 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:54.319190 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-rv92b" podUID="4630b841-bf2e-4019-b450-12e85497afc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Oct 30 12:20:54.814036 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:54.813999 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef80232e-6rnpp" Oct 30 12:20:54.907244 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:54.904692 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53e329b8-a4e4-4ced-8c81-d141b31076c3-kube-api-access-cpl8n\" (UniqueName: \"kubernetes.io/projected/53e329b8-a4e4-4ced-8c81-d141b31076c3-kube-api-access-cpl8n\") pod \"53e329b8-a4e4-4ced-8c81-d141b31076c3\" (UID: \"53e329b8-a4e4-4ced-8c81-d141b31076c3\") " Oct 30 12:20:54.909224 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:54.909187 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/53e329b8-a4e4-4ced-8c81-d141b31076c3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53e329b8-a4e4-4ced-8c81-d141b31076c3-maas-api-xks-ca\") pod \"53e329b8-a4e4-4ced-8c81-d141b31076c3\" (UID: \"53e329b8-a4e4-4ced-8c81-d141b31076c3\") " Oct 30 12:20:54.909943 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:54.909918 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53e329b8-a4e4-4ced-8c81-d141b31076c3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53e329b8-a4e4-4ced-8c81-d141b31076c3-maas-api-openshift-ca\") pod \"53e329b8-a4e4-4ced-8c81-d141b31076c3\" (UID: \"53e329b8-a4e4-4ced-8c81-d141b31076c3\") " Oct 30 12:20:54.911044 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:54.910399 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53e329b8-a4e4-4ced-8c81-d141b31076c3-maas-api-openshift-ca" pod "53e329b8-a4e4-4ced-8c81-d141b31076c3" (UID: "53e329b8-a4e4-4ced-8c81-d141b31076c3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:20:54.914655 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:54.914633 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53e329b8-a4e4-4ced-8c81-d141b31076c3-kube-api-access-cpl8n" pod "53e329b8-a4e4-4ced-8c81-d141b31076c3" (UID: "53e329b8-a4e4-4ced-8c81-d141b31076c3"). InnerVolumeSpecName "kube-api-access-cpl8n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:20:54.916739 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:54.916704 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53e329b8-a4e4-4ced-8c81-d141b31076c3-maas-api-xks-ca" pod "53e329b8-a4e4-4ced-8c81-d141b31076c3" (UID: "53e329b8-a4e4-4ced-8c81-d141b31076c3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:20:55.006165 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:55.006129 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef80232e-6rnpp" Oct 30 12:20:55.006338 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:55.006129 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef80232e-6rnpp" event={"ID":"53e329b8-a4e4-4ced-8c81-d141b31076c3","Type":"ContainerDied","Data":"122164d26851a8e3711b1d0b5d010e0ad3e0d5b00f791eb95a446c62c00f68d7"} Oct 30 12:20:55.006338 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:55.006269 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="122164d26851a8e3711b1d0b5d010e0ad3e0d5b00f791eb95a446c62c00f68d7" Oct 30 12:20:55.012356 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:55.012328 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cpl8n\" (UniqueName: \"kubernetes.io/projected/53e329b8-a4e4-4ced-8c81-d141b31076c3-kube-api-access-cpl8n\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:20:55.012356 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:55.012355 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53e329b8-a4e4-4ced-8c81-d141b31076c3-maas-api-xks-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:20:55.012526 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:55.012366 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53e329b8-a4e4-4ced-8c81-d141b31076c3-maas-api-openshift-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:20:55.218763 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:55.218648 4908 patch_prober.go:29] interesting pod/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-tjnq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Oct 30 12:20:55.218763 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:55.218735 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-tjnq5" podUID="c3e7bd6e-7283-451a-8df8-cc9634908630" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Oct 30 12:20:55.262892 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:55.262806 4908 patch_prober.go:29] interesting pod/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-rv92b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Oct 30 12:20:55.263381 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:55.262926 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-rv92b" podUID="4630b841-bf2e-4019-b450-12e85497afc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Oct 30 12:20:56.359561 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:56.359530 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-rv92b" Oct 30 12:20:56.361671 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:56.361603 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-rv92b" Oct 30 12:20:56.363128 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:56.363081 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-rv92b" Oct 30 12:20:56.444299 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:56.444271 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-tjnq5" Oct 30 12:20:56.444782 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:56.444755 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-tjnq5" Oct 30 12:20:56.446732 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:20:56.446706 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-tjnq5" Oct 30 12:21:05.030715 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:05.030674 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7024660-sqz4g"] Oct 30 12:21:05.041479 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:05.041452 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7024660-sqz4g"] Oct 30 12:21:05.609611 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:05.609573 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c36c58da-716d-47d8-ac5d-8230d92f60f3" path="/var/lib/kubelet/pods/c36c58da-716d-47d8-ac5d-8230d92f60f3/volumes" Oct 30 12:21:07.032797 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:07.032742 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e534bd0-rhl8b"] Oct 30 12:21:07.044256 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:07.044227 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e534bd0-rhl8b"] Oct 30 12:21:07.607081 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:07.607051 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5db1278-dd68-45f3-a5ed-384b3d9a721d" path="/var/lib/kubelet/pods/f5db1278-dd68-45f3-a5ed-384b3d9a721d/volumes" Oct 30 12:21:11.426626 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:11.426592 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5649cd-x2x4m"] Oct 30 12:21:11.427405 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:11.427381 4908 cpu_manager.go:414] "containerMap: removing container" podUID="53e329b8-a4e4-4ced-8c81-d141b31076c3" containerName="revoke-keys" Oct 30 12:21:11.427510 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:11.427405 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53e329b8-a4e4-4ced-8c81-d141b31076c3" containerName="revoke-keys" Oct 30 12:21:11.427578 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:11.427569 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="53e329b8-a4e4-4ced-8c81-d141b31076c3" containerName="revoke-keys" Oct 30 12:21:11.431448 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:11.431424 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5649cd-x2x4m" Oct 30 12:21:11.443142 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:11.443117 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qm6lb\" (UniqueName: \"kubernetes.io/projected/0e9e432a-f985-46d8-8545-4164ac8fc051-kube-api-access-qm6lb\") pod \"maas-api-revoke-keys-e2e-api-a-5649cd-x2x4m\" (UID: \"0e9e432a-f985-46d8-8545-4164ac8fc051\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5649cd-x2x4m" Oct 30 12:21:11.443228 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:11.443153 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e9e432a-f985-46d8-8545-4164ac8fc051-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-5649cd-x2x4m\" (UID: \"0e9e432a-f985-46d8-8545-4164ac8fc051\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5649cd-x2x4m" Oct 30 12:21:11.443450 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:11.443429 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e9e432a-f985-46d8-8545-4164ac8fc051-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-5649cd-x2x4m\" (UID: \"0e9e432a-f985-46d8-8545-4164ac8fc051\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5649cd-x2x4m" Oct 30 12:21:11.450063 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:11.450042 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5649cd-x2x4m"] Oct 30 12:21:11.455107 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:11.455071 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 12:21:11.455217 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:11.455123 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 12:21:11.455217 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:11.455136 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 12:21:11.456046 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:11.456022 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s75hq\"" Oct 30 12:21:11.544364 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:11.544330 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qm6lb\" (UniqueName: \"kubernetes.io/projected/0e9e432a-f985-46d8-8545-4164ac8fc051-kube-api-access-qm6lb\") pod \"maas-api-revoke-keys-e2e-api-a-5649cd-x2x4m\" (UID: \"0e9e432a-f985-46d8-8545-4164ac8fc051\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5649cd-x2x4m" Oct 30 12:21:11.544525 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:11.544370 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e9e432a-f985-46d8-8545-4164ac8fc051-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-5649cd-x2x4m\" (UID: \"0e9e432a-f985-46d8-8545-4164ac8fc051\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5649cd-x2x4m" Oct 30 12:21:11.544525 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:11.544466 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e9e432a-f985-46d8-8545-4164ac8fc051-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-5649cd-x2x4m\" (UID: \"0e9e432a-f985-46d8-8545-4164ac8fc051\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5649cd-x2x4m" Oct 30 12:21:11.545095 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:11.545071 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e9e432a-f985-46d8-8545-4164ac8fc051-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-5649cd-x2x4m\" (UID: \"0e9e432a-f985-46d8-8545-4164ac8fc051\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5649cd-x2x4m" Oct 30 12:21:11.546800 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:11.546776 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e9e432a-f985-46d8-8545-4164ac8fc051-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-5649cd-x2x4m\" (UID: \"0e9e432a-f985-46d8-8545-4164ac8fc051\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5649cd-x2x4m" Oct 30 12:21:11.570063 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:11.570036 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qm6lb\" (UniqueName: \"kubernetes.io/projected/0e9e432a-f985-46d8-8545-4164ac8fc051-kube-api-access-qm6lb\") pod \"maas-api-revoke-keys-e2e-api-a-5649cd-x2x4m\" (UID: \"0e9e432a-f985-46d8-8545-4164ac8fc051\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5649cd-x2x4m" Oct 30 12:21:11.745416 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:11.745390 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5649cd-x2x4m" Oct 30 12:21:12.053725 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:12.053701 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5649cd-x2x4m"] Oct 30 12:21:12.055196 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:21:12.055167 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e9e432a_f985_46d8_8545_4164ac8fc051.slice/crio-1fc9c799cc78e7477c24361bef9b3575f698ea24a79c8c2343ffc7815a1a0a49 WatchSource:0}: Error finding container 1fc9c799cc78e7477c24361bef9b3575f698ea24a79c8c2343ffc7815a1a0a49: Status 404 returned error can't find the container with id 1fc9c799cc78e7477c24361bef9b3575f698ea24a79c8c2343ffc7815a1a0a49 Oct 30 12:21:12.167781 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:12.167750 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5649cd-x2x4m" event={"ID":"0e9e432a-f985-46d8-8545-4164ac8fc051","Type":"ContainerStarted","Data":"1fc9c799cc78e7477c24361bef9b3575f698ea24a79c8c2343ffc7815a1a0a49"} Oct 30 12:21:13.177050 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:13.177013 4908 generic.go:356] "Generic (PLEG): container finished" podID="0e9e432a-f985-46d8-8545-4164ac8fc051" containerID="4bdc6be23f7f4ec21790d10426ebf1f6bf812654532836d9d133ccab32866834" exitCode=0 Oct 30 12:21:13.177465 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:13.177093 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5649cd-x2x4m" event={"ID":"0e9e432a-f985-46d8-8545-4164ac8fc051","Type":"ContainerDied","Data":"4bdc6be23f7f4ec21790d10426ebf1f6bf812654532836d9d133ccab32866834"} Oct 30 12:21:14.363879 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:14.363854 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5649cd-x2x4m" Oct 30 12:21:14.487641 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:14.487609 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e9e432a-f985-46d8-8545-4164ac8fc051-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e9e432a-f985-46d8-8545-4164ac8fc051-maas-api-xks-ca\") pod \"0e9e432a-f985-46d8-8545-4164ac8fc051\" (UID: \"0e9e432a-f985-46d8-8545-4164ac8fc051\") " Oct 30 12:21:14.487819 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:14.487679 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e9e432a-f985-46d8-8545-4164ac8fc051-kube-api-access-qm6lb\" (UniqueName: \"kubernetes.io/projected/0e9e432a-f985-46d8-8545-4164ac8fc051-kube-api-access-qm6lb\") pod \"0e9e432a-f985-46d8-8545-4164ac8fc051\" (UID: \"0e9e432a-f985-46d8-8545-4164ac8fc051\") " Oct 30 12:21:14.487908 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:14.487874 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e9e432a-f985-46d8-8545-4164ac8fc051-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e9e432a-f985-46d8-8545-4164ac8fc051-maas-api-openshift-ca\") pod \"0e9e432a-f985-46d8-8545-4164ac8fc051\" (UID: \"0e9e432a-f985-46d8-8545-4164ac8fc051\") " Oct 30 12:21:14.488556 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:14.488520 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e9e432a-f985-46d8-8545-4164ac8fc051-maas-api-openshift-ca" pod "0e9e432a-f985-46d8-8545-4164ac8fc051" (UID: "0e9e432a-f985-46d8-8545-4164ac8fc051"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:21:14.490159 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:14.490122 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e9e432a-f985-46d8-8545-4164ac8fc051-maas-api-xks-ca" pod "0e9e432a-f985-46d8-8545-4164ac8fc051" (UID: "0e9e432a-f985-46d8-8545-4164ac8fc051"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:21:14.490293 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:14.490270 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e9e432a-f985-46d8-8545-4164ac8fc051-kube-api-access-qm6lb" pod "0e9e432a-f985-46d8-8545-4164ac8fc051" (UID: "0e9e432a-f985-46d8-8545-4164ac8fc051"). InnerVolumeSpecName "kube-api-access-qm6lb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:21:14.589733 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:14.589703 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e9e432a-f985-46d8-8545-4164ac8fc051-maas-api-xks-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:21:14.589733 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:14.589728 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qm6lb\" (UniqueName: \"kubernetes.io/projected/0e9e432a-f985-46d8-8545-4164ac8fc051-kube-api-access-qm6lb\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:21:14.589913 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:14.589740 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e9e432a-f985-46d8-8545-4164ac8fc051-maas-api-openshift-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:21:15.198093 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:15.198063 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5649cd-x2x4m" Oct 30 12:21:15.198093 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:15.198071 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5649cd-x2x4m" event={"ID":"0e9e432a-f985-46d8-8545-4164ac8fc051","Type":"ContainerDied","Data":"1fc9c799cc78e7477c24361bef9b3575f698ea24a79c8c2343ffc7815a1a0a49"} Oct 30 12:21:15.198318 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:15.198131 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1fc9c799cc78e7477c24361bef9b3575f698ea24a79c8c2343ffc7815a1a0a49" Oct 30 12:21:18.713584 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:18.713550 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-ebc6bb-openshift-default-84d66c85b7-ssbvj"] Oct 30 12:21:18.714400 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:18.714378 4908 cpu_manager.go:414] "containerMap: removing container" podUID="0e9e432a-f985-46d8-8545-4164ac8fc051" containerName="revoke-keys" Oct 30 12:21:18.714400 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:18.714400 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e9e432a-f985-46d8-8545-4164ac8fc051" containerName="revoke-keys" Oct 30 12:21:18.714624 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:18.714610 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e9e432a-f985-46d8-8545-4164ac8fc051" containerName="revoke-keys" Oct 30 12:21:18.718931 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:18.718911 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-ebc6bb-openshift-default-84d66c85b7-ssbvj" Oct 30 12:21:18.745254 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:18.745230 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-ebc6bb-openshift-default-dockercfg-4bxft\"" Oct 30 12:21:18.751880 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:18.751852 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-ebc6bb-openshift-default-84d66c85b7-ssbvj"] Oct 30 12:21:18.844851 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:18.844343 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd7c01ae-47be-43dd-93d3-c4702d08ef42-istio-data\") pod \"e2e-api-b-ebc6bb-openshift-default-84d66c85b7-ssbvj\" (UID: \"cd7c01ae-47be-43dd-93d3-c4702d08ef42\") " pod="openshift-ingress/e2e-api-b-ebc6bb-openshift-default-84d66c85b7-ssbvj" Oct 30 12:21:18.844851 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:18.844446 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd7c01ae-47be-43dd-93d3-c4702d08ef42-credential-socket\") pod \"e2e-api-b-ebc6bb-openshift-default-84d66c85b7-ssbvj\" (UID: \"cd7c01ae-47be-43dd-93d3-c4702d08ef42\") " pod="openshift-ingress/e2e-api-b-ebc6bb-openshift-default-84d66c85b7-ssbvj" Oct 30 12:21:18.844851 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:18.844481 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd7c01ae-47be-43dd-93d3-c4702d08ef42-workload-certs\") pod \"e2e-api-b-ebc6bb-openshift-default-84d66c85b7-ssbvj\" (UID: \"cd7c01ae-47be-43dd-93d3-c4702d08ef42\") " pod="openshift-ingress/e2e-api-b-ebc6bb-openshift-default-84d66c85b7-ssbvj" Oct 30 12:21:18.844851 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:18.844511 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd7c01ae-47be-43dd-93d3-c4702d08ef42-istio-token\") pod \"e2e-api-b-ebc6bb-openshift-default-84d66c85b7-ssbvj\" (UID: \"cd7c01ae-47be-43dd-93d3-c4702d08ef42\") " pod="openshift-ingress/e2e-api-b-ebc6bb-openshift-default-84d66c85b7-ssbvj" Oct 30 12:21:18.844851 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:18.844584 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmz8s\" (UniqueName: \"kubernetes.io/projected/cd7c01ae-47be-43dd-93d3-c4702d08ef42-kube-api-access-rmz8s\") pod \"e2e-api-b-ebc6bb-openshift-default-84d66c85b7-ssbvj\" (UID: \"cd7c01ae-47be-43dd-93d3-c4702d08ef42\") " pod="openshift-ingress/e2e-api-b-ebc6bb-openshift-default-84d66c85b7-ssbvj" Oct 30 12:21:18.844851 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:18.844615 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd7c01ae-47be-43dd-93d3-c4702d08ef42-workload-socket\") pod \"e2e-api-b-ebc6bb-openshift-default-84d66c85b7-ssbvj\" (UID: \"cd7c01ae-47be-43dd-93d3-c4702d08ef42\") " pod="openshift-ingress/e2e-api-b-ebc6bb-openshift-default-84d66c85b7-ssbvj" Oct 30 12:21:18.844851 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:18.844656 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd7c01ae-47be-43dd-93d3-c4702d08ef42-istio-envoy\") pod \"e2e-api-b-ebc6bb-openshift-default-84d66c85b7-ssbvj\" (UID: \"cd7c01ae-47be-43dd-93d3-c4702d08ef42\") " pod="openshift-ingress/e2e-api-b-ebc6bb-openshift-default-84d66c85b7-ssbvj" Oct 30 12:21:18.844851 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:18.844734 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd7c01ae-47be-43dd-93d3-c4702d08ef42-istio-podinfo\") pod \"e2e-api-b-ebc6bb-openshift-default-84d66c85b7-ssbvj\" (UID: \"cd7c01ae-47be-43dd-93d3-c4702d08ef42\") " pod="openshift-ingress/e2e-api-b-ebc6bb-openshift-default-84d66c85b7-ssbvj" Oct 30 12:21:18.844851 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:18.844796 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd7c01ae-47be-43dd-93d3-c4702d08ef42-istiod-ca-cert\") pod \"e2e-api-b-ebc6bb-openshift-default-84d66c85b7-ssbvj\" (UID: \"cd7c01ae-47be-43dd-93d3-c4702d08ef42\") " pod="openshift-ingress/e2e-api-b-ebc6bb-openshift-default-84d66c85b7-ssbvj" Oct 30 12:21:18.949769 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:18.949728 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd7c01ae-47be-43dd-93d3-c4702d08ef42-istio-podinfo\") pod \"e2e-api-b-ebc6bb-openshift-default-84d66c85b7-ssbvj\" (UID: \"cd7c01ae-47be-43dd-93d3-c4702d08ef42\") " pod="openshift-ingress/e2e-api-b-ebc6bb-openshift-default-84d66c85b7-ssbvj" Oct 30 12:21:18.949953 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:18.949821 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd7c01ae-47be-43dd-93d3-c4702d08ef42-istiod-ca-cert\") pod \"e2e-api-b-ebc6bb-openshift-default-84d66c85b7-ssbvj\" (UID: \"cd7c01ae-47be-43dd-93d3-c4702d08ef42\") " pod="openshift-ingress/e2e-api-b-ebc6bb-openshift-default-84d66c85b7-ssbvj" Oct 30 12:21:18.949953 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:18.949932 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd7c01ae-47be-43dd-93d3-c4702d08ef42-istio-data\") pod \"e2e-api-b-ebc6bb-openshift-default-84d66c85b7-ssbvj\" (UID: \"cd7c01ae-47be-43dd-93d3-c4702d08ef42\") " pod="openshift-ingress/e2e-api-b-ebc6bb-openshift-default-84d66c85b7-ssbvj" Oct 30 12:21:18.950058 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:18.950010 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd7c01ae-47be-43dd-93d3-c4702d08ef42-credential-socket\") pod \"e2e-api-b-ebc6bb-openshift-default-84d66c85b7-ssbvj\" (UID: \"cd7c01ae-47be-43dd-93d3-c4702d08ef42\") " pod="openshift-ingress/e2e-api-b-ebc6bb-openshift-default-84d66c85b7-ssbvj" Oct 30 12:21:18.950058 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:18.950037 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd7c01ae-47be-43dd-93d3-c4702d08ef42-workload-certs\") pod \"e2e-api-b-ebc6bb-openshift-default-84d66c85b7-ssbvj\" (UID: \"cd7c01ae-47be-43dd-93d3-c4702d08ef42\") " pod="openshift-ingress/e2e-api-b-ebc6bb-openshift-default-84d66c85b7-ssbvj" Oct 30 12:21:18.950126 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:18.950069 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd7c01ae-47be-43dd-93d3-c4702d08ef42-istio-token\") pod \"e2e-api-b-ebc6bb-openshift-default-84d66c85b7-ssbvj\" (UID: \"cd7c01ae-47be-43dd-93d3-c4702d08ef42\") " pod="openshift-ingress/e2e-api-b-ebc6bb-openshift-default-84d66c85b7-ssbvj" Oct 30 12:21:18.950163 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:18.950149 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmz8s\" (UniqueName: \"kubernetes.io/projected/cd7c01ae-47be-43dd-93d3-c4702d08ef42-kube-api-access-rmz8s\") pod \"e2e-api-b-ebc6bb-openshift-default-84d66c85b7-ssbvj\" (UID: \"cd7c01ae-47be-43dd-93d3-c4702d08ef42\") " pod="openshift-ingress/e2e-api-b-ebc6bb-openshift-default-84d66c85b7-ssbvj" Oct 30 12:21:18.950207 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:18.950188 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd7c01ae-47be-43dd-93d3-c4702d08ef42-workload-socket\") pod \"e2e-api-b-ebc6bb-openshift-default-84d66c85b7-ssbvj\" (UID: \"cd7c01ae-47be-43dd-93d3-c4702d08ef42\") " pod="openshift-ingress/e2e-api-b-ebc6bb-openshift-default-84d66c85b7-ssbvj" Oct 30 12:21:18.950258 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:18.950245 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd7c01ae-47be-43dd-93d3-c4702d08ef42-istio-envoy\") pod \"e2e-api-b-ebc6bb-openshift-default-84d66c85b7-ssbvj\" (UID: \"cd7c01ae-47be-43dd-93d3-c4702d08ef42\") " pod="openshift-ingress/e2e-api-b-ebc6bb-openshift-default-84d66c85b7-ssbvj" Oct 30 12:21:18.951136 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:18.951109 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd7c01ae-47be-43dd-93d3-c4702d08ef42-credential-socket\") pod \"e2e-api-b-ebc6bb-openshift-default-84d66c85b7-ssbvj\" (UID: \"cd7c01ae-47be-43dd-93d3-c4702d08ef42\") " pod="openshift-ingress/e2e-api-b-ebc6bb-openshift-default-84d66c85b7-ssbvj" Oct 30 12:21:18.952578 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:18.952549 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd7c01ae-47be-43dd-93d3-c4702d08ef42-istiod-ca-cert\") pod \"e2e-api-b-ebc6bb-openshift-default-84d66c85b7-ssbvj\" (UID: \"cd7c01ae-47be-43dd-93d3-c4702d08ef42\") " pod="openshift-ingress/e2e-api-b-ebc6bb-openshift-default-84d66c85b7-ssbvj" Oct 30 12:21:18.952884 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:18.952847 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd7c01ae-47be-43dd-93d3-c4702d08ef42-istio-data\") pod \"e2e-api-b-ebc6bb-openshift-default-84d66c85b7-ssbvj\" (UID: \"cd7c01ae-47be-43dd-93d3-c4702d08ef42\") " pod="openshift-ingress/e2e-api-b-ebc6bb-openshift-default-84d66c85b7-ssbvj" Oct 30 12:21:18.953438 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:18.953409 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd7c01ae-47be-43dd-93d3-c4702d08ef42-workload-certs\") pod \"e2e-api-b-ebc6bb-openshift-default-84d66c85b7-ssbvj\" (UID: \"cd7c01ae-47be-43dd-93d3-c4702d08ef42\") " pod="openshift-ingress/e2e-api-b-ebc6bb-openshift-default-84d66c85b7-ssbvj" Oct 30 12:21:18.953952 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:18.953917 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd7c01ae-47be-43dd-93d3-c4702d08ef42-workload-socket\") pod \"e2e-api-b-ebc6bb-openshift-default-84d66c85b7-ssbvj\" (UID: \"cd7c01ae-47be-43dd-93d3-c4702d08ef42\") " pod="openshift-ingress/e2e-api-b-ebc6bb-openshift-default-84d66c85b7-ssbvj" Oct 30 12:21:18.955439 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:18.955402 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd7c01ae-47be-43dd-93d3-c4702d08ef42-istio-envoy\") pod \"e2e-api-b-ebc6bb-openshift-default-84d66c85b7-ssbvj\" (UID: \"cd7c01ae-47be-43dd-93d3-c4702d08ef42\") " pod="openshift-ingress/e2e-api-b-ebc6bb-openshift-default-84d66c85b7-ssbvj" Oct 30 12:21:18.963380 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:18.963354 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd7c01ae-47be-43dd-93d3-c4702d08ef42-istio-podinfo\") pod \"e2e-api-b-ebc6bb-openshift-default-84d66c85b7-ssbvj\" (UID: \"cd7c01ae-47be-43dd-93d3-c4702d08ef42\") " pod="openshift-ingress/e2e-api-b-ebc6bb-openshift-default-84d66c85b7-ssbvj" Oct 30 12:21:18.982324 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:18.982298 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmz8s\" (UniqueName: \"kubernetes.io/projected/cd7c01ae-47be-43dd-93d3-c4702d08ef42-kube-api-access-rmz8s\") pod \"e2e-api-b-ebc6bb-openshift-default-84d66c85b7-ssbvj\" (UID: \"cd7c01ae-47be-43dd-93d3-c4702d08ef42\") " pod="openshift-ingress/e2e-api-b-ebc6bb-openshift-default-84d66c85b7-ssbvj" Oct 30 12:21:18.982964 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:18.982940 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd7c01ae-47be-43dd-93d3-c4702d08ef42-istio-token\") pod \"e2e-api-b-ebc6bb-openshift-default-84d66c85b7-ssbvj\" (UID: \"cd7c01ae-47be-43dd-93d3-c4702d08ef42\") " pod="openshift-ingress/e2e-api-b-ebc6bb-openshift-default-84d66c85b7-ssbvj" Oct 30 12:21:19.032771 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:19.032733 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b703a455-llp76"] Oct 30 12:21:19.038375 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:19.038340 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d554a235-x5ccx"] Oct 30 12:21:19.039903 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:19.039863 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-ebc6bb-openshift-default-84d66c85b7-ssbvj" Oct 30 12:21:19.043190 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:19.043162 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b703a455-llp76"] Oct 30 12:21:19.047625 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:19.047606 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d554a235-x5ccx"] Oct 30 12:21:19.384459 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:19.384428 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-ebc6bb-openshift-default-84d66c85b7-ssbvj"] Oct 30 12:21:19.384598 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:21:19.384550 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd7c01ae_47be_43dd_93d3_c4702d08ef42.slice/crio-4f20024605c3e48a934abd83019fa003c078d470786520cfbcc64b5adbcd8481 WatchSource:0}: Error finding container 4f20024605c3e48a934abd83019fa003c078d470786520cfbcc64b5adbcd8481: Status 404 returned error can't find the container with id 4f20024605c3e48a934abd83019fa003c078d470786520cfbcc64b5adbcd8481 Oct 30 12:21:19.387536 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:19.387498 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:21:19.387644 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:19.387582 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:21:19.387644 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:19.387628 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:21:19.607496 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:19.607465 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76c0e83d-ebe5-4e89-be6b-043568973731" path="/var/lib/kubelet/pods/76c0e83d-ebe5-4e89-be6b-043568973731/volumes" Oct 30 12:21:19.608015 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:19.607999 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dfeb8f6c-afd6-4dd2-99af-a4829db10a20" path="/var/lib/kubelet/pods/dfeb8f6c-afd6-4dd2-99af-a4829db10a20/volumes" Oct 30 12:21:19.688483 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:21:19.688390 4908 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vn6vv" containerName="extractor" Oct 30 12:21:20.247425 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:20.247389 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ebc6bb-openshift-default-84d66c85b7-ssbvj" event={"ID":"cd7c01ae-47be-43dd-93d3-c4702d08ef42","Type":"ContainerStarted","Data":"ff15f396b5ac70fd1d2a8dea2c2f98be7f0a9d96dd278f371bc1db4f595ccbbd"} Oct 30 12:21:20.247820 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:20.247445 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ebc6bb-openshift-default-84d66c85b7-ssbvj" event={"ID":"cd7c01ae-47be-43dd-93d3-c4702d08ef42","Type":"ContainerStarted","Data":"4f20024605c3e48a934abd83019fa003c078d470786520cfbcc64b5adbcd8481"} Oct 30 12:21:20.305353 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:20.305308 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-ebc6bb-openshift-default-84d66c85b7-ssbvj" podStartSLOduration=2.305295574 podStartE2EDuration="2.305295574s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:21:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:21:20.300029439 +0000 UTC m=+1636.814458178" watchObservedRunningTime="2026-10-30 12:21:20.305295574 +0000 UTC m=+1636.819724320" Oct 30 12:21:20.413961 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:20.413933 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-11581-8a9ac2"] Oct 30 12:21:20.420255 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:20.420231 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11581-8a9ac2" Oct 30 12:21:20.436073 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:20.434570 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-ttfhc\"" Oct 30 12:21:20.436936 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:20.436908 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11581-8a9ac2"] Oct 30 12:21:20.580025 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:20.579931 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8m7k\" (UniqueName: \"kubernetes.io/projected/b76fdf52-5a71-4785-825f-a7ca897f81e6-kube-api-access-t8m7k\") pod \"test-curl-11581-8a9ac2\" (UID: \"b76fdf52-5a71-4785-825f-a7ca897f81e6\") " pod="opendatahub/test-curl-11581-8a9ac2" Oct 30 12:21:20.682088 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:20.682047 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8m7k\" (UniqueName: \"kubernetes.io/projected/b76fdf52-5a71-4785-825f-a7ca897f81e6-kube-api-access-t8m7k\") pod \"test-curl-11581-8a9ac2\" (UID: \"b76fdf52-5a71-4785-825f-a7ca897f81e6\") " pod="opendatahub/test-curl-11581-8a9ac2" Oct 30 12:21:20.704131 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:20.704099 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8m7k\" (UniqueName: \"kubernetes.io/projected/b76fdf52-5a71-4785-825f-a7ca897f81e6-kube-api-access-t8m7k\") pod \"test-curl-11581-8a9ac2\" (UID: \"b76fdf52-5a71-4785-825f-a7ca897f81e6\") " pod="opendatahub/test-curl-11581-8a9ac2" Oct 30 12:21:20.738819 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:20.738790 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11581-8a9ac2" Oct 30 12:21:21.040081 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:21.040044 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11581-8a9ac2"] Oct 30 12:21:21.040190 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:21.040104 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-ebc6bb-openshift-default-84d66c85b7-ssbvj" Oct 30 12:21:21.041614 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:21:21.041582 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb76fdf52_5a71_4785_825f_a7ca897f81e6.slice/crio-0ec784c6d34e677d1453a9086ff53803941d51cf544c45efcbe059bc9694cd9d WatchSource:0}: Error finding container 0ec784c6d34e677d1453a9086ff53803941d51cf544c45efcbe059bc9694cd9d: Status 404 returned error can't find the container with id 0ec784c6d34e677d1453a9086ff53803941d51cf544c45efcbe059bc9694cd9d Oct 30 12:21:21.046273 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:21.046251 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-ebc6bb-openshift-default-84d66c85b7-ssbvj" Oct 30 12:21:21.257446 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:21.257401 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11581-8a9ac2" event={"ID":"b76fdf52-5a71-4785-825f-a7ca897f81e6","Type":"ContainerStarted","Data":"0ec784c6d34e677d1453a9086ff53803941d51cf544c45efcbe059bc9694cd9d"} Oct 30 12:21:21.257992 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:21.257788 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-ebc6bb-openshift-default-84d66c85b7-ssbvj" Oct 30 12:21:21.258816 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:21.258798 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-ebc6bb-openshift-default-84d66c85b7-ssbvj" Oct 30 12:21:23.049945 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:23.049906 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71c0f7bc-nd54p"] Oct 30 12:21:23.060887 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:23.060857 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71c0f7bc-nd54p"] Oct 30 12:21:23.619756 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:23.619720 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af769e53-bc80-4f62-be1b-86c7eac896f3" path="/var/lib/kubelet/pods/af769e53-bc80-4f62-be1b-86c7eac896f3/volumes" Oct 30 12:21:24.301022 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:24.300986 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11581-8a9ac2" event={"ID":"b76fdf52-5a71-4785-825f-a7ca897f81e6","Type":"ContainerStarted","Data":"718b39a8e39e85c4e5ae2f9872c18267c03d55e4865cf68681bf5275d4e6631b"} Oct 30 12:21:24.344539 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:24.344487 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-11581-8a9ac2" podStartSLOduration=1.61343824 podStartE2EDuration="4.344472433s" totalImagesPullingTime="2.731034193s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:21:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:21:24.340244808 +0000 UTC m=+1640.854673554" watchObservedRunningTime="2026-10-30 12:21:24.344472433 +0000 UTC m=+1640.858901178" Oct 30 12:21:25.372891 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:25.372853 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-11581-8a9ac2"] Oct 30 12:21:25.377960 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:25.377932 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-11581-8a9ac2"] Oct 30 12:21:26.082366 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:26.082332 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-060f41f2-bx7wc"] Oct 30 12:21:26.082926 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:26.082910 4908 cpu_manager.go:414] "containerMap: removing container" podUID="b76fdf52-5a71-4785-825f-a7ca897f81e6" containerName="test-curl-11581-8a9ac2" Oct 30 12:21:26.082926 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:26.082925 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b76fdf52-5a71-4785-825f-a7ca897f81e6" containerName="test-curl-11581-8a9ac2" Oct 30 12:21:26.083063 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:26.083033 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="b76fdf52-5a71-4785-825f-a7ca897f81e6" containerName="test-curl-11581-8a9ac2" Oct 30 12:21:26.086744 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:26.086723 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-060f41f2-bx7wc" Oct 30 12:21:26.105778 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:26.105747 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-060f41f2-bx7wc"] Oct 30 12:21:26.111230 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:26.111204 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 12:21:26.111363 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:26.111261 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s75hq\"" Oct 30 12:21:26.111363 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:26.111311 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 12:21:26.111363 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:26.111327 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 12:21:26.277443 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:26.277409 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6756f1a3-1c7f-4ab9-9ef9-8b67ec55eaf3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-060f41f2-bx7wc\" (UID: \"6756f1a3-1c7f-4ab9-9ef9-8b67ec55eaf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-060f41f2-bx7wc" Oct 30 12:21:26.277620 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:26.277561 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6756f1a3-1c7f-4ab9-9ef9-8b67ec55eaf3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-060f41f2-bx7wc\" (UID: \"6756f1a3-1c7f-4ab9-9ef9-8b67ec55eaf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-060f41f2-bx7wc" Oct 30 12:21:26.277693 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:26.277676 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkd7t\" (UniqueName: \"kubernetes.io/projected/6756f1a3-1c7f-4ab9-9ef9-8b67ec55eaf3-kube-api-access-xkd7t\") pod \"maas-api-revoke-keys-e2e-mt-060f41f2-bx7wc\" (UID: \"6756f1a3-1c7f-4ab9-9ef9-8b67ec55eaf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-060f41f2-bx7wc" Oct 30 12:21:26.321166 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:26.321123 4908 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-11581-8a9ac2" podUID="b76fdf52-5a71-4785-825f-a7ca897f81e6" containerName="test-curl-11581-8a9ac2" containerID="cri-o://718b39a8e39e85c4e5ae2f9872c18267c03d55e4865cf68681bf5275d4e6631b" gracePeriod=2 Oct 30 12:21:26.380943 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:26.380544 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6756f1a3-1c7f-4ab9-9ef9-8b67ec55eaf3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-060f41f2-bx7wc\" (UID: \"6756f1a3-1c7f-4ab9-9ef9-8b67ec55eaf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-060f41f2-bx7wc" Oct 30 12:21:26.380943 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:26.380678 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkd7t\" (UniqueName: \"kubernetes.io/projected/6756f1a3-1c7f-4ab9-9ef9-8b67ec55eaf3-kube-api-access-xkd7t\") pod \"maas-api-revoke-keys-e2e-mt-060f41f2-bx7wc\" (UID: \"6756f1a3-1c7f-4ab9-9ef9-8b67ec55eaf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-060f41f2-bx7wc" Oct 30 12:21:26.380943 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:26.380847 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6756f1a3-1c7f-4ab9-9ef9-8b67ec55eaf3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-060f41f2-bx7wc\" (UID: \"6756f1a3-1c7f-4ab9-9ef9-8b67ec55eaf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-060f41f2-bx7wc" Oct 30 12:21:26.381979 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:26.381706 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6756f1a3-1c7f-4ab9-9ef9-8b67ec55eaf3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-060f41f2-bx7wc\" (UID: \"6756f1a3-1c7f-4ab9-9ef9-8b67ec55eaf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-060f41f2-bx7wc" Oct 30 12:21:26.385588 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:26.385559 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6756f1a3-1c7f-4ab9-9ef9-8b67ec55eaf3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-060f41f2-bx7wc\" (UID: \"6756f1a3-1c7f-4ab9-9ef9-8b67ec55eaf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-060f41f2-bx7wc" Oct 30 12:21:26.408283 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:26.408245 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkd7t\" (UniqueName: \"kubernetes.io/projected/6756f1a3-1c7f-4ab9-9ef9-8b67ec55eaf3-kube-api-access-xkd7t\") pod \"maas-api-revoke-keys-e2e-mt-060f41f2-bx7wc\" (UID: \"6756f1a3-1c7f-4ab9-9ef9-8b67ec55eaf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-060f41f2-bx7wc" Oct 30 12:21:26.701512 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:26.701413 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-060f41f2-bx7wc" Oct 30 12:21:27.006156 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:27.006120 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-060f41f2-bx7wc"] Oct 30 12:21:27.009555 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:21:27.009518 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6756f1a3_1c7f_4ab9_9ef9_8b67ec55eaf3.slice/crio-401bb3390f8fe218441045c3091ea71e63cab6e556918d1cc0daa614e4b3c141 WatchSource:0}: Error finding container 401bb3390f8fe218441045c3091ea71e63cab6e556918d1cc0daa614e4b3c141: Status 404 returned error can't find the container with id 401bb3390f8fe218441045c3091ea71e63cab6e556918d1cc0daa614e4b3c141 Oct 30 12:21:27.332222 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:27.332193 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-060f41f2-bx7wc" event={"ID":"6756f1a3-1c7f-4ab9-9ef9-8b67ec55eaf3","Type":"ContainerStarted","Data":"ffbb95821eba745fa706df06b2824b5d0eabdb354866bcdf034ce8404d648c7c"} Oct 30 12:21:27.332343 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:27.332246 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-060f41f2-bx7wc" event={"ID":"6756f1a3-1c7f-4ab9-9ef9-8b67ec55eaf3","Type":"ContainerStarted","Data":"401bb3390f8fe218441045c3091ea71e63cab6e556918d1cc0daa614e4b3c141"} Oct 30 12:21:27.371361 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:27.371321 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-060f41f2-bx7wc" podStartSLOduration=1.371306881 podStartE2EDuration="1.371306881s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:21:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:21:27.370069054 +0000 UTC m=+1643.884497799" watchObservedRunningTime="2026-10-30 12:21:27.371306881 +0000 UTC m=+1643.885735676" Oct 30 12:21:28.344884 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:28.344855 4908 generic.go:356] "Generic (PLEG): container finished" podID="6756f1a3-1c7f-4ab9-9ef9-8b67ec55eaf3" containerID="ffbb95821eba745fa706df06b2824b5d0eabdb354866bcdf034ce8404d648c7c" exitCode=0 Oct 30 12:21:28.345215 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:28.344928 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-060f41f2-bx7wc" event={"ID":"6756f1a3-1c7f-4ab9-9ef9-8b67ec55eaf3","Type":"ContainerDied","Data":"ffbb95821eba745fa706df06b2824b5d0eabdb354866bcdf034ce8404d648c7c"} Oct 30 12:21:28.551179 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:28.551155 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11581-8a9ac2" Oct 30 12:21:28.607335 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:28.607258 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b76fdf52-5a71-4785-825f-a7ca897f81e6-kube-api-access-t8m7k\" (UniqueName: \"kubernetes.io/projected/b76fdf52-5a71-4785-825f-a7ca897f81e6-kube-api-access-t8m7k\") pod \"b76fdf52-5a71-4785-825f-a7ca897f81e6\" (UID: \"b76fdf52-5a71-4785-825f-a7ca897f81e6\") " Oct 30 12:21:28.609383 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:28.609355 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b76fdf52-5a71-4785-825f-a7ca897f81e6-kube-api-access-t8m7k" pod "b76fdf52-5a71-4785-825f-a7ca897f81e6" (UID: "b76fdf52-5a71-4785-825f-a7ca897f81e6"). InnerVolumeSpecName "kube-api-access-t8m7k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:21:28.714948 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:28.714918 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t8m7k\" (UniqueName: \"kubernetes.io/projected/b76fdf52-5a71-4785-825f-a7ca897f81e6-kube-api-access-t8m7k\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:21:29.354222 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:29.354182 4908 generic.go:356] "Generic (PLEG): container finished" podID="b76fdf52-5a71-4785-825f-a7ca897f81e6" containerID="718b39a8e39e85c4e5ae2f9872c18267c03d55e4865cf68681bf5275d4e6631b" exitCode=137 Oct 30 12:21:29.354643 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:29.354246 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11581-8a9ac2" Oct 30 12:21:29.354643 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:29.354283 4908 scope.go:122] "RemoveContainer" containerID="718b39a8e39e85c4e5ae2f9872c18267c03d55e4865cf68681bf5275d4e6631b" Oct 30 12:21:29.375944 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:21:29.375906 4908 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11581-8a9ac2\" is forbidden: User \"system:node:ip-10-0-141-251.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-251.ec2.internal' and this object" podUID="b76fdf52-5a71-4785-825f-a7ca897f81e6" pod="opendatahub/test-curl-11581-8a9ac2" Oct 30 12:21:29.376800 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:29.376778 4908 scope.go:122] "RemoveContainer" containerID="718b39a8e39e85c4e5ae2f9872c18267c03d55e4865cf68681bf5275d4e6631b" Oct 30 12:21:29.377169 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:21:29.377134 4908 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"718b39a8e39e85c4e5ae2f9872c18267c03d55e4865cf68681bf5275d4e6631b\": container with ID starting with 718b39a8e39e85c4e5ae2f9872c18267c03d55e4865cf68681bf5275d4e6631b not found: ID does not exist" containerID="718b39a8e39e85c4e5ae2f9872c18267c03d55e4865cf68681bf5275d4e6631b" Oct 30 12:21:29.377244 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:29.377170 4908 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"718b39a8e39e85c4e5ae2f9872c18267c03d55e4865cf68681bf5275d4e6631b"} err="failed to get container status \"718b39a8e39e85c4e5ae2f9872c18267c03d55e4865cf68681bf5275d4e6631b\": rpc error: code = NotFound desc = could not find container \"718b39a8e39e85c4e5ae2f9872c18267c03d55e4865cf68681bf5275d4e6631b\": container with ID starting with 718b39a8e39e85c4e5ae2f9872c18267c03d55e4865cf68681bf5275d4e6631b not found: ID does not exist" Oct 30 12:21:29.540682 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:29.540657 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-060f41f2-bx7wc" Oct 30 12:21:29.600346 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:21:29.593081 4908 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11581-8a9ac2\" is forbidden: User \"system:node:ip-10-0-141-251.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-251.ec2.internal' and this object" podUID="b76fdf52-5a71-4785-825f-a7ca897f81e6" pod="opendatahub/test-curl-11581-8a9ac2" Oct 30 12:21:29.625505 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:29.625400 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6756f1a3-1c7f-4ab9-9ef9-8b67ec55eaf3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6756f1a3-1c7f-4ab9-9ef9-8b67ec55eaf3-maas-api-openshift-ca\") pod \"6756f1a3-1c7f-4ab9-9ef9-8b67ec55eaf3\" (UID: \"6756f1a3-1c7f-4ab9-9ef9-8b67ec55eaf3\") " Oct 30 12:21:29.625708 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:29.625536 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6756f1a3-1c7f-4ab9-9ef9-8b67ec55eaf3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6756f1a3-1c7f-4ab9-9ef9-8b67ec55eaf3-maas-api-xks-ca\") pod \"6756f1a3-1c7f-4ab9-9ef9-8b67ec55eaf3\" (UID: \"6756f1a3-1c7f-4ab9-9ef9-8b67ec55eaf3\") " Oct 30 12:21:29.625708 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:29.625619 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6756f1a3-1c7f-4ab9-9ef9-8b67ec55eaf3-kube-api-access-xkd7t\" (UniqueName: \"kubernetes.io/projected/6756f1a3-1c7f-4ab9-9ef9-8b67ec55eaf3-kube-api-access-xkd7t\") pod \"6756f1a3-1c7f-4ab9-9ef9-8b67ec55eaf3\" (UID: \"6756f1a3-1c7f-4ab9-9ef9-8b67ec55eaf3\") " Oct 30 12:21:29.626767 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:29.626731 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6756f1a3-1c7f-4ab9-9ef9-8b67ec55eaf3-maas-api-openshift-ca" pod "6756f1a3-1c7f-4ab9-9ef9-8b67ec55eaf3" (UID: "6756f1a3-1c7f-4ab9-9ef9-8b67ec55eaf3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:21:29.634894 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:29.630652 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b76fdf52-5a71-4785-825f-a7ca897f81e6" path="/var/lib/kubelet/pods/b76fdf52-5a71-4785-825f-a7ca897f81e6/volumes" Oct 30 12:21:29.647091 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:29.646470 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6756f1a3-1c7f-4ab9-9ef9-8b67ec55eaf3-maas-api-xks-ca" pod "6756f1a3-1c7f-4ab9-9ef9-8b67ec55eaf3" (UID: "6756f1a3-1c7f-4ab9-9ef9-8b67ec55eaf3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:21:29.653422 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:29.653388 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6756f1a3-1c7f-4ab9-9ef9-8b67ec55eaf3-kube-api-access-xkd7t" pod "6756f1a3-1c7f-4ab9-9ef9-8b67ec55eaf3" (UID: "6756f1a3-1c7f-4ab9-9ef9-8b67ec55eaf3"). InnerVolumeSpecName "kube-api-access-xkd7t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:21:29.727604 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:29.727572 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6756f1a3-1c7f-4ab9-9ef9-8b67ec55eaf3-maas-api-xks-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:21:29.727604 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:29.727601 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkd7t\" (UniqueName: \"kubernetes.io/projected/6756f1a3-1c7f-4ab9-9ef9-8b67ec55eaf3-kube-api-access-xkd7t\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:21:29.727604 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:29.727610 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6756f1a3-1c7f-4ab9-9ef9-8b67ec55eaf3-maas-api-openshift-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:21:30.363514 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:30.363480 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-060f41f2-bx7wc" Oct 30 12:21:30.363514 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:30.363500 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-060f41f2-bx7wc" event={"ID":"6756f1a3-1c7f-4ab9-9ef9-8b67ec55eaf3","Type":"ContainerDied","Data":"401bb3390f8fe218441045c3091ea71e63cab6e556918d1cc0daa614e4b3c141"} Oct 30 12:21:30.364087 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:30.363558 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="401bb3390f8fe218441045c3091ea71e63cab6e556918d1cc0daa614e4b3c141" Oct 30 12:21:34.872680 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:34.872644 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9wwfc"] Oct 30 12:21:37.053853 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:37.052883 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d8be2a15-88g2p"] Oct 30 12:21:37.084861 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:37.080865 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d8be2a15-88g2p"] Oct 30 12:21:37.660852 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:37.655623 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1608674-e935-4a04-87fa-d20ce21e7218" path="/var/lib/kubelet/pods/e1608674-e935-4a04-87fa-d20ce21e7218/volumes" Oct 30 12:21:38.546863 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:38.542904 4908 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f920f7/simulator-w0-f920f7-kserve-599b7ff45f-g79d7" podUID="a85e562f-d655-4c22-b4f4-a299a4e494a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:21:38.576132 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:38.576088 4908 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6a7148/distinct-2-w3-6a7148-kserve-67bd5fb976-9nfff" podUID="6046cc02-29fe-4f59-b522-8f57f4557277" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:21:38.650851 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:38.636173 4908 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6a7148/premium-w3-6a7148-kserve-67b976c888-tgd6c" podUID="13de4441-e7f6-4cbd-97d9-0f079d11cb8e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:21:39.076956 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:39.076906 4908 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-6a7148/distinct-2-w3-6a7148-kserve-67bd5fb976-9nfff" podUID="6046cc02-29fe-4f59-b522-8f57f4557277" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:21:44.631678 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:44.631636 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5649cd-x2x4m"] Oct 30 12:21:44.642206 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:44.642172 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5649cd-x2x4m"] Oct 30 12:21:45.607894 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:45.607860 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e9e432a-f985-46d8-8545-4164ac8fc051" path="/var/lib/kubelet/pods/0e9e432a-f985-46d8-8545-4164ac8fc051/volumes" Oct 30 12:21:47.033004 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:47.032974 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-747839d7-thvml"] Oct 30 12:21:47.043657 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:47.043629 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-747839d7-thvml"] Oct 30 12:21:47.607263 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:47.607224 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea24c359-cda6-4d11-a757-df67563772f2" path="/var/lib/kubelet/pods/ea24c359-cda6-4d11-a757-df67563772f2/volumes" Oct 30 12:21:48.339259 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:48.339226 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb601c3c-2w42x"] Oct 30 12:21:48.339798 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:48.339785 4908 cpu_manager.go:414] "containerMap: removing container" podUID="6756f1a3-1c7f-4ab9-9ef9-8b67ec55eaf3" containerName="revoke-keys" Oct 30 12:21:48.339886 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:48.339799 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6756f1a3-1c7f-4ab9-9ef9-8b67ec55eaf3" containerName="revoke-keys" Oct 30 12:21:48.339930 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:48.339924 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="6756f1a3-1c7f-4ab9-9ef9-8b67ec55eaf3" containerName="revoke-keys" Oct 30 12:21:48.345129 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:48.345110 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb601c3c-2w42x" Oct 30 12:21:48.358824 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:48.358798 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 12:21:48.363365 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:48.363342 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb601c3c-2w42x"] Oct 30 12:21:48.368355 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:48.368336 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 12:21:48.368493 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:48.368336 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s75hq\"" Oct 30 12:21:48.368493 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:48.368436 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 12:21:48.456577 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:48.456540 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e1fbc3b-d000-48eb-b78e-b5b74d952dad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb601c3c-2w42x\" (UID: \"5e1fbc3b-d000-48eb-b78e-b5b74d952dad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb601c3c-2w42x" Oct 30 12:21:48.456859 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:48.456821 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfgqf\" (UniqueName: \"kubernetes.io/projected/5e1fbc3b-d000-48eb-b78e-b5b74d952dad-kube-api-access-dfgqf\") pod \"maas-api-revoke-sub-fb601c3c-2w42x\" (UID: \"5e1fbc3b-d000-48eb-b78e-b5b74d952dad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb601c3c-2w42x" Oct 30 12:21:48.456993 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:48.456974 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e1fbc3b-d000-48eb-b78e-b5b74d952dad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb601c3c-2w42x\" (UID: \"5e1fbc3b-d000-48eb-b78e-b5b74d952dad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb601c3c-2w42x" Oct 30 12:21:48.558401 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:48.558363 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e1fbc3b-d000-48eb-b78e-b5b74d952dad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb601c3c-2w42x\" (UID: \"5e1fbc3b-d000-48eb-b78e-b5b74d952dad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb601c3c-2w42x" Oct 30 12:21:48.558561 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:48.558409 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfgqf\" (UniqueName: \"kubernetes.io/projected/5e1fbc3b-d000-48eb-b78e-b5b74d952dad-kube-api-access-dfgqf\") pod \"maas-api-revoke-sub-fb601c3c-2w42x\" (UID: \"5e1fbc3b-d000-48eb-b78e-b5b74d952dad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb601c3c-2w42x" Oct 30 12:21:48.558561 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:48.558438 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e1fbc3b-d000-48eb-b78e-b5b74d952dad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb601c3c-2w42x\" (UID: \"5e1fbc3b-d000-48eb-b78e-b5b74d952dad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb601c3c-2w42x" Oct 30 12:21:48.559181 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:48.559154 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e1fbc3b-d000-48eb-b78e-b5b74d952dad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb601c3c-2w42x\" (UID: \"5e1fbc3b-d000-48eb-b78e-b5b74d952dad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb601c3c-2w42x" Oct 30 12:21:48.560880 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:48.560856 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e1fbc3b-d000-48eb-b78e-b5b74d952dad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb601c3c-2w42x\" (UID: \"5e1fbc3b-d000-48eb-b78e-b5b74d952dad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb601c3c-2w42x" Oct 30 12:21:48.581116 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:48.581087 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfgqf\" (UniqueName: \"kubernetes.io/projected/5e1fbc3b-d000-48eb-b78e-b5b74d952dad-kube-api-access-dfgqf\") pod \"maas-api-revoke-sub-fb601c3c-2w42x\" (UID: \"5e1fbc3b-d000-48eb-b78e-b5b74d952dad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb601c3c-2w42x" Oct 30 12:21:48.659261 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:48.659205 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb601c3c-2w42x" Oct 30 12:21:48.966582 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:48.966557 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb601c3c-2w42x"] Oct 30 12:21:48.968332 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:21:48.968306 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e1fbc3b_d000_48eb_b78e_b5b74d952dad.slice/crio-b7dcfdfa14fd932cfaa8c5935440bba9c00f0349dcf581088618174b70c62fdd WatchSource:0}: Error finding container b7dcfdfa14fd932cfaa8c5935440bba9c00f0349dcf581088618174b70c62fdd: Status 404 returned error can't find the container with id b7dcfdfa14fd932cfaa8c5935440bba9c00f0349dcf581088618174b70c62fdd Oct 30 12:21:49.497140 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:49.497109 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a476f81-4hcqx"] Oct 30 12:21:49.501375 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:49.501357 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a476f81-4hcqx" Oct 30 12:21:49.520864 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:49.520612 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a476f81-4hcqx"] Oct 30 12:21:49.571377 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:49.571342 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2q7w\" (UniqueName: \"kubernetes.io/projected/dd0f25d5-fa88-41a5-88c7-c787799b40b3-kube-api-access-f2q7w\") pod \"maas-api-revoke-sub-7a476f81-4hcqx\" (UID: \"dd0f25d5-fa88-41a5-88c7-c787799b40b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a476f81-4hcqx" Oct 30 12:21:49.571520 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:49.571498 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd0f25d5-fa88-41a5-88c7-c787799b40b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7a476f81-4hcqx\" (UID: \"dd0f25d5-fa88-41a5-88c7-c787799b40b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a476f81-4hcqx" Oct 30 12:21:49.571614 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:49.571597 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd0f25d5-fa88-41a5-88c7-c787799b40b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7a476f81-4hcqx\" (UID: \"dd0f25d5-fa88-41a5-88c7-c787799b40b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a476f81-4hcqx" Oct 30 12:21:49.672552 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:49.672522 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2q7w\" (UniqueName: \"kubernetes.io/projected/dd0f25d5-fa88-41a5-88c7-c787799b40b3-kube-api-access-f2q7w\") pod \"maas-api-revoke-sub-7a476f81-4hcqx\" (UID: \"dd0f25d5-fa88-41a5-88c7-c787799b40b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a476f81-4hcqx" Oct 30 12:21:49.672740 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:49.672661 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd0f25d5-fa88-41a5-88c7-c787799b40b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7a476f81-4hcqx\" (UID: \"dd0f25d5-fa88-41a5-88c7-c787799b40b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a476f81-4hcqx" Oct 30 12:21:49.672808 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:49.672757 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd0f25d5-fa88-41a5-88c7-c787799b40b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7a476f81-4hcqx\" (UID: \"dd0f25d5-fa88-41a5-88c7-c787799b40b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a476f81-4hcqx" Oct 30 12:21:49.673499 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:49.673474 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd0f25d5-fa88-41a5-88c7-c787799b40b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7a476f81-4hcqx\" (UID: \"dd0f25d5-fa88-41a5-88c7-c787799b40b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a476f81-4hcqx" Oct 30 12:21:49.675248 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:49.675224 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd0f25d5-fa88-41a5-88c7-c787799b40b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7a476f81-4hcqx\" (UID: \"dd0f25d5-fa88-41a5-88c7-c787799b40b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a476f81-4hcqx" Oct 30 12:21:49.685112 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:49.685085 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb601c3c-2w42x" event={"ID":"5e1fbc3b-d000-48eb-b78e-b5b74d952dad","Type":"ContainerStarted","Data":"a023ce7cf4600d83691a751af80028cad5e0e527798d47da82d1649261808a87"} Oct 30 12:21:49.685217 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:49.685133 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb601c3c-2w42x" event={"ID":"5e1fbc3b-d000-48eb-b78e-b5b74d952dad","Type":"ContainerStarted","Data":"b7dcfdfa14fd932cfaa8c5935440bba9c00f0349dcf581088618174b70c62fdd"} Oct 30 12:21:49.685715 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:21:49.685690 4908 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vn6vv" containerName="extractor" Oct 30 12:21:49.694704 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:49.694681 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2q7w\" (UniqueName: \"kubernetes.io/projected/dd0f25d5-fa88-41a5-88c7-c787799b40b3-kube-api-access-f2q7w\") pod \"maas-api-revoke-sub-7a476f81-4hcqx\" (UID: \"dd0f25d5-fa88-41a5-88c7-c787799b40b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a476f81-4hcqx" Oct 30 12:21:49.727013 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:49.726968 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb601c3c-2w42x" podStartSLOduration=1.726957133 podStartE2EDuration="1.726957133s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:21:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:21:49.726212963 +0000 UTC m=+1666.240641709" watchObservedRunningTime="2026-10-30 12:21:49.726957133 +0000 UTC m=+1666.241385879" Oct 30 12:21:49.816133 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:49.816110 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a476f81-4hcqx" Oct 30 12:21:50.123325 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:50.123299 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a476f81-4hcqx"] Oct 30 12:21:50.124233 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:21:50.124210 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd0f25d5_fa88_41a5_88c7_c787799b40b3.slice/crio-1a2958e7c77bdf5a0fd8a7eaff9c66cba1e34fed76967206805363fce2f298b1 WatchSource:0}: Error finding container 1a2958e7c77bdf5a0fd8a7eaff9c66cba1e34fed76967206805363fce2f298b1: Status 404 returned error can't find the container with id 1a2958e7c77bdf5a0fd8a7eaff9c66cba1e34fed76967206805363fce2f298b1 Oct 30 12:21:50.698098 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:50.698009 4908 generic.go:356] "Generic (PLEG): container finished" podID="5e1fbc3b-d000-48eb-b78e-b5b74d952dad" containerID="a023ce7cf4600d83691a751af80028cad5e0e527798d47da82d1649261808a87" exitCode=0 Oct 30 12:21:50.698545 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:50.698092 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb601c3c-2w42x" event={"ID":"5e1fbc3b-d000-48eb-b78e-b5b74d952dad","Type":"ContainerDied","Data":"a023ce7cf4600d83691a751af80028cad5e0e527798d47da82d1649261808a87"} Oct 30 12:21:50.700025 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:50.699995 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a476f81-4hcqx" event={"ID":"dd0f25d5-fa88-41a5-88c7-c787799b40b3","Type":"ContainerStarted","Data":"0dfb5696ff79c2d508333f8b27b3103dca71d2128003a0ec721c09a90047bd13"} Oct 30 12:21:50.700179 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:50.700046 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a476f81-4hcqx" event={"ID":"dd0f25d5-fa88-41a5-88c7-c787799b40b3","Type":"ContainerStarted","Data":"1a2958e7c77bdf5a0fd8a7eaff9c66cba1e34fed76967206805363fce2f298b1"} Oct 30 12:21:50.774929 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:50.774881 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a476f81-4hcqx" podStartSLOduration=1.774822406 podStartE2EDuration="1.774822406s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:21:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:21:50.771658194 +0000 UTC m=+1667.286086945" watchObservedRunningTime="2026-10-30 12:21:50.774822406 +0000 UTC m=+1667.289251173" Oct 30 12:21:51.712879 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:51.712808 4908 generic.go:356] "Generic (PLEG): container finished" podID="dd0f25d5-fa88-41a5-88c7-c787799b40b3" containerID="0dfb5696ff79c2d508333f8b27b3103dca71d2128003a0ec721c09a90047bd13" exitCode=0 Oct 30 12:21:51.713292 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:51.712927 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a476f81-4hcqx" event={"ID":"dd0f25d5-fa88-41a5-88c7-c787799b40b3","Type":"ContainerDied","Data":"0dfb5696ff79c2d508333f8b27b3103dca71d2128003a0ec721c09a90047bd13"} Oct 30 12:21:51.944202 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:51.944174 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb601c3c-2w42x" Oct 30 12:21:52.020593 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:52.018895 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e1fbc3b-d000-48eb-b78e-b5b74d952dad-kube-api-access-dfgqf\" (UniqueName: \"kubernetes.io/projected/5e1fbc3b-d000-48eb-b78e-b5b74d952dad-kube-api-access-dfgqf\") pod \"5e1fbc3b-d000-48eb-b78e-b5b74d952dad\" (UID: \"5e1fbc3b-d000-48eb-b78e-b5b74d952dad\") " Oct 30 12:21:52.020593 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:52.018974 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e1fbc3b-d000-48eb-b78e-b5b74d952dad-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e1fbc3b-d000-48eb-b78e-b5b74d952dad-maas-api-openshift-ca\") pod \"5e1fbc3b-d000-48eb-b78e-b5b74d952dad\" (UID: \"5e1fbc3b-d000-48eb-b78e-b5b74d952dad\") " Oct 30 12:21:52.020593 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:52.019036 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e1fbc3b-d000-48eb-b78e-b5b74d952dad-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e1fbc3b-d000-48eb-b78e-b5b74d952dad-maas-api-xks-ca\") pod \"5e1fbc3b-d000-48eb-b78e-b5b74d952dad\" (UID: \"5e1fbc3b-d000-48eb-b78e-b5b74d952dad\") " Oct 30 12:21:52.033848 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:52.026427 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e1fbc3b-d000-48eb-b78e-b5b74d952dad-maas-api-openshift-ca" pod "5e1fbc3b-d000-48eb-b78e-b5b74d952dad" (UID: "5e1fbc3b-d000-48eb-b78e-b5b74d952dad"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:21:52.050705 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:52.050648 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e1fbc3b-d000-48eb-b78e-b5b74d952dad-maas-api-xks-ca" pod "5e1fbc3b-d000-48eb-b78e-b5b74d952dad" (UID: "5e1fbc3b-d000-48eb-b78e-b5b74d952dad"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:21:52.138391 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:52.138353 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e1fbc3b-d000-48eb-b78e-b5b74d952dad-kube-api-access-dfgqf" pod "5e1fbc3b-d000-48eb-b78e-b5b74d952dad" (UID: "5e1fbc3b-d000-48eb-b78e-b5b74d952dad"). InnerVolumeSpecName "kube-api-access-dfgqf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:21:52.205847 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:52.197082 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e1fbc3b-d000-48eb-b78e-b5b74d952dad-maas-api-openshift-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:21:52.205847 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:52.197132 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e1fbc3b-d000-48eb-b78e-b5b74d952dad-maas-api-xks-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:21:52.298492 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:52.298410 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dfgqf\" (UniqueName: \"kubernetes.io/projected/5e1fbc3b-d000-48eb-b78e-b5b74d952dad-kube-api-access-dfgqf\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:21:52.522758 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:52.522722 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7566f5-whqb6"] Oct 30 12:21:52.523348 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:52.523332 4908 cpu_manager.go:414] "containerMap: removing container" podUID="5e1fbc3b-d000-48eb-b78e-b5b74d952dad" containerName="revoke-keys" Oct 30 12:21:52.523397 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:52.523349 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e1fbc3b-d000-48eb-b78e-b5b74d952dad" containerName="revoke-keys" Oct 30 12:21:52.523471 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:52.523462 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e1fbc3b-d000-48eb-b78e-b5b74d952dad" containerName="revoke-keys" Oct 30 12:21:52.527154 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:52.527130 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7566f5-whqb6" Oct 30 12:21:52.546352 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:52.546324 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7566f5-whqb6"] Oct 30 12:21:52.602135 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:52.602068 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/135b84cd-3715-4846-90d7-14c2e5c6985c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-7566f5-whqb6\" (UID: \"135b84cd-3715-4846-90d7-14c2e5c6985c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7566f5-whqb6" Oct 30 12:21:52.602252 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:52.602227 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/135b84cd-3715-4846-90d7-14c2e5c6985c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-7566f5-whqb6\" (UID: \"135b84cd-3715-4846-90d7-14c2e5c6985c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7566f5-whqb6" Oct 30 12:21:52.602294 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:52.602272 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b96pn\" (UniqueName: \"kubernetes.io/projected/135b84cd-3715-4846-90d7-14c2e5c6985c-kube-api-access-b96pn\") pod \"maas-api-revoke-keys-e2e-api-b-7566f5-whqb6\" (UID: \"135b84cd-3715-4846-90d7-14c2e5c6985c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7566f5-whqb6" Oct 30 12:21:52.703745 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:52.703709 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/135b84cd-3715-4846-90d7-14c2e5c6985c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-7566f5-whqb6\" (UID: \"135b84cd-3715-4846-90d7-14c2e5c6985c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7566f5-whqb6" Oct 30 12:21:52.703920 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:52.703767 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b96pn\" (UniqueName: \"kubernetes.io/projected/135b84cd-3715-4846-90d7-14c2e5c6985c-kube-api-access-b96pn\") pod \"maas-api-revoke-keys-e2e-api-b-7566f5-whqb6\" (UID: \"135b84cd-3715-4846-90d7-14c2e5c6985c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7566f5-whqb6" Oct 30 12:21:52.704053 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:52.704030 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/135b84cd-3715-4846-90d7-14c2e5c6985c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-7566f5-whqb6\" (UID: \"135b84cd-3715-4846-90d7-14c2e5c6985c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7566f5-whqb6" Oct 30 12:21:52.704402 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:52.704375 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/135b84cd-3715-4846-90d7-14c2e5c6985c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-7566f5-whqb6\" (UID: \"135b84cd-3715-4846-90d7-14c2e5c6985c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7566f5-whqb6" Oct 30 12:21:52.706420 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:52.706397 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/135b84cd-3715-4846-90d7-14c2e5c6985c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-7566f5-whqb6\" (UID: \"135b84cd-3715-4846-90d7-14c2e5c6985c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7566f5-whqb6" Oct 30 12:21:52.725359 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:52.725341 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb601c3c-2w42x" Oct 30 12:21:52.725798 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:52.725379 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb601c3c-2w42x" event={"ID":"5e1fbc3b-d000-48eb-b78e-b5b74d952dad","Type":"ContainerDied","Data":"b7dcfdfa14fd932cfaa8c5935440bba9c00f0349dcf581088618174b70c62fdd"} Oct 30 12:21:52.725798 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:52.725440 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b7dcfdfa14fd932cfaa8c5935440bba9c00f0349dcf581088618174b70c62fdd" Oct 30 12:21:52.725798 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:52.725759 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b96pn\" (UniqueName: \"kubernetes.io/projected/135b84cd-3715-4846-90d7-14c2e5c6985c-kube-api-access-b96pn\") pod \"maas-api-revoke-keys-e2e-api-b-7566f5-whqb6\" (UID: \"135b84cd-3715-4846-90d7-14c2e5c6985c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7566f5-whqb6" Oct 30 12:21:52.841969 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:52.841933 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7566f5-whqb6" Oct 30 12:21:52.879209 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:52.879186 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a476f81-4hcqx" Oct 30 12:21:53.007379 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:53.007348 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dd0f25d5-fa88-41a5-88c7-c787799b40b3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd0f25d5-fa88-41a5-88c7-c787799b40b3-maas-api-xks-ca\") pod \"dd0f25d5-fa88-41a5-88c7-c787799b40b3\" (UID: \"dd0f25d5-fa88-41a5-88c7-c787799b40b3\") " Oct 30 12:21:53.007553 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:53.007402 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd0f25d5-fa88-41a5-88c7-c787799b40b3-kube-api-access-f2q7w\" (UniqueName: \"kubernetes.io/projected/dd0f25d5-fa88-41a5-88c7-c787799b40b3-kube-api-access-f2q7w\") pod \"dd0f25d5-fa88-41a5-88c7-c787799b40b3\" (UID: \"dd0f25d5-fa88-41a5-88c7-c787799b40b3\") " Oct 30 12:21:53.007638 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:53.007599 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd0f25d5-fa88-41a5-88c7-c787799b40b3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd0f25d5-fa88-41a5-88c7-c787799b40b3-maas-api-openshift-ca\") pod \"dd0f25d5-fa88-41a5-88c7-c787799b40b3\" (UID: \"dd0f25d5-fa88-41a5-88c7-c787799b40b3\") " Oct 30 12:21:53.008005 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:53.007948 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd0f25d5-fa88-41a5-88c7-c787799b40b3-maas-api-openshift-ca" pod "dd0f25d5-fa88-41a5-88c7-c787799b40b3" (UID: "dd0f25d5-fa88-41a5-88c7-c787799b40b3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:21:53.008213 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:53.008191 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd0f25d5-fa88-41a5-88c7-c787799b40b3-maas-api-openshift-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:21:53.009601 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:53.009579 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd0f25d5-fa88-41a5-88c7-c787799b40b3-maas-api-xks-ca" pod "dd0f25d5-fa88-41a5-88c7-c787799b40b3" (UID: "dd0f25d5-fa88-41a5-88c7-c787799b40b3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:21:53.009960 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:53.009944 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd0f25d5-fa88-41a5-88c7-c787799b40b3-kube-api-access-f2q7w" pod "dd0f25d5-fa88-41a5-88c7-c787799b40b3" (UID: "dd0f25d5-fa88-41a5-88c7-c787799b40b3"). InnerVolumeSpecName "kube-api-access-f2q7w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:21:53.109983 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:53.109922 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd0f25d5-fa88-41a5-88c7-c787799b40b3-maas-api-xks-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:21:53.109983 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:53.109947 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2q7w\" (UniqueName: \"kubernetes.io/projected/dd0f25d5-fa88-41a5-88c7-c787799b40b3-kube-api-access-f2q7w\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:21:53.154393 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:53.154371 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7566f5-whqb6"] Oct 30 12:21:53.156699 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:21:53.156671 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod135b84cd_3715_4846_90d7_14c2e5c6985c.slice/crio-b5056d67ebf3c4e6ddf76298f5a4193391697243c0a00301399ea147d0a50139 WatchSource:0}: Error finding container b5056d67ebf3c4e6ddf76298f5a4193391697243c0a00301399ea147d0a50139: Status 404 returned error can't find the container with id b5056d67ebf3c4e6ddf76298f5a4193391697243c0a00301399ea147d0a50139 Oct 30 12:21:53.735005 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:53.734971 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7566f5-whqb6" event={"ID":"135b84cd-3715-4846-90d7-14c2e5c6985c","Type":"ContainerStarted","Data":"fb52762311b20e114a5fa60f4e035c67ab461ec406f7fdb9f7b426eb2a76dbf1"} Oct 30 12:21:53.735479 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:53.735034 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7566f5-whqb6" event={"ID":"135b84cd-3715-4846-90d7-14c2e5c6985c","Type":"ContainerStarted","Data":"b5056d67ebf3c4e6ddf76298f5a4193391697243c0a00301399ea147d0a50139"} Oct 30 12:21:53.736449 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:53.736427 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a476f81-4hcqx" Oct 30 12:21:53.736515 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:53.736463 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a476f81-4hcqx" event={"ID":"dd0f25d5-fa88-41a5-88c7-c787799b40b3","Type":"ContainerDied","Data":"1a2958e7c77bdf5a0fd8a7eaff9c66cba1e34fed76967206805363fce2f298b1"} Oct 30 12:21:53.736515 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:53.736507 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1a2958e7c77bdf5a0fd8a7eaff9c66cba1e34fed76967206805363fce2f298b1" Oct 30 12:21:53.775267 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:53.775217 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7566f5-whqb6" podStartSLOduration=1.775200618 podStartE2EDuration="1.775200618s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:21:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:21:53.771918037 +0000 UTC m=+1670.286346803" watchObservedRunningTime="2026-10-30 12:21:53.775200618 +0000 UTC m=+1670.289629368" Oct 30 12:21:54.745837 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:54.745791 4908 generic.go:356] "Generic (PLEG): container finished" podID="135b84cd-3715-4846-90d7-14c2e5c6985c" containerID="fb52762311b20e114a5fa60f4e035c67ab461ec406f7fdb9f7b426eb2a76dbf1" exitCode=0 Oct 30 12:21:54.746245 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:54.745876 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7566f5-whqb6" event={"ID":"135b84cd-3715-4846-90d7-14c2e5c6985c","Type":"ContainerDied","Data":"fb52762311b20e114a5fa60f4e035c67ab461ec406f7fdb9f7b426eb2a76dbf1"} Oct 30 12:21:55.940636 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:55.940612 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7566f5-whqb6" Oct 30 12:21:56.046320 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:56.046243 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/135b84cd-3715-4846-90d7-14c2e5c6985c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/135b84cd-3715-4846-90d7-14c2e5c6985c-maas-api-openshift-ca\") pod \"135b84cd-3715-4846-90d7-14c2e5c6985c\" (UID: \"135b84cd-3715-4846-90d7-14c2e5c6985c\") " Oct 30 12:21:56.046459 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:56.046418 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/135b84cd-3715-4846-90d7-14c2e5c6985c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/135b84cd-3715-4846-90d7-14c2e5c6985c-maas-api-xks-ca\") pod \"135b84cd-3715-4846-90d7-14c2e5c6985c\" (UID: \"135b84cd-3715-4846-90d7-14c2e5c6985c\") " Oct 30 12:21:56.046526 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:56.046470 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/135b84cd-3715-4846-90d7-14c2e5c6985c-kube-api-access-b96pn\" (UniqueName: \"kubernetes.io/projected/135b84cd-3715-4846-90d7-14c2e5c6985c-kube-api-access-b96pn\") pod \"135b84cd-3715-4846-90d7-14c2e5c6985c\" (UID: \"135b84cd-3715-4846-90d7-14c2e5c6985c\") " Oct 30 12:21:56.046603 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:56.046577 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/135b84cd-3715-4846-90d7-14c2e5c6985c-maas-api-openshift-ca" pod "135b84cd-3715-4846-90d7-14c2e5c6985c" (UID: "135b84cd-3715-4846-90d7-14c2e5c6985c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:21:56.046907 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:56.046886 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/135b84cd-3715-4846-90d7-14c2e5c6985c-maas-api-openshift-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:21:56.048535 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:56.048509 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/135b84cd-3715-4846-90d7-14c2e5c6985c-maas-api-xks-ca" pod "135b84cd-3715-4846-90d7-14c2e5c6985c" (UID: "135b84cd-3715-4846-90d7-14c2e5c6985c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:21:56.048693 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:56.048673 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/135b84cd-3715-4846-90d7-14c2e5c6985c-kube-api-access-b96pn" pod "135b84cd-3715-4846-90d7-14c2e5c6985c" (UID: "135b84cd-3715-4846-90d7-14c2e5c6985c"). InnerVolumeSpecName "kube-api-access-b96pn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:21:56.147718 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:56.147688 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b96pn\" (UniqueName: \"kubernetes.io/projected/135b84cd-3715-4846-90d7-14c2e5c6985c-kube-api-access-b96pn\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:21:56.147718 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:56.147716 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/135b84cd-3715-4846-90d7-14c2e5c6985c-maas-api-xks-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:21:56.763782 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:56.763745 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7566f5-whqb6" Oct 30 12:21:56.764010 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:56.763774 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7566f5-whqb6" event={"ID":"135b84cd-3715-4846-90d7-14c2e5c6985c","Type":"ContainerDied","Data":"b5056d67ebf3c4e6ddf76298f5a4193391697243c0a00301399ea147d0a50139"} Oct 30 12:21:56.764010 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:21:56.763841 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b5056d67ebf3c4e6ddf76298f5a4193391697243c0a00301399ea147d0a50139" Oct 30 12:22:04.055482 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:04.055370 4908 scope.go:122] "RemoveContainer" containerID="9a1867aeeb0939f788a96922b46eda07aecfdff6db2e9fff725dab1a86849d68" Oct 30 12:22:04.078939 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:04.078304 4908 scope.go:122] "RemoveContainer" containerID="ad7d08afbd2bc0372f06b82b997ae0659db7c10388b55545f92d849665bfb202" Oct 30 12:22:04.089587 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:04.089566 4908 scope.go:122] "RemoveContainer" containerID="9a2dc1d3afb0fc36dd8fa5a5f60dfa3397a176f33b44c018966c9f61cb28d8ff" Oct 30 12:22:05.201850 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:05.201318 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-82299a-4kjdd"] Oct 30 12:22:05.228852 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:05.202507 4908 cpu_manager.go:414] "containerMap: removing container" podUID="dd0f25d5-fa88-41a5-88c7-c787799b40b3" containerName="revoke-keys" Oct 30 12:22:05.228852 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:05.202531 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd0f25d5-fa88-41a5-88c7-c787799b40b3" containerName="revoke-keys" Oct 30 12:22:05.228852 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:05.202564 4908 cpu_manager.go:414] "containerMap: removing container" podUID="135b84cd-3715-4846-90d7-14c2e5c6985c" containerName="revoke-keys" Oct 30 12:22:05.228852 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:05.202572 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="135b84cd-3715-4846-90d7-14c2e5c6985c" containerName="revoke-keys" Oct 30 12:22:05.228852 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:05.215145 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="135b84cd-3715-4846-90d7-14c2e5c6985c" containerName="revoke-keys" Oct 30 12:22:05.228852 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:05.215194 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd0f25d5-fa88-41a5-88c7-c787799b40b3" containerName="revoke-keys" Oct 30 12:22:05.274850 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:05.271126 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-82299a-4kjdd"] Oct 30 12:22:05.274850 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:05.271245 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-82299a-4kjdd" Oct 30 12:22:05.314848 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:05.299346 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 12:22:05.314848 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:05.299408 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 12:22:05.314848 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:05.311048 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s75hq\"" Oct 30 12:22:05.314848 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:05.311186 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 12:22:05.340857 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:05.339920 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/175fd62e-7735-4976-b8e7-e3c087f14441-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-82299a-4kjdd\" (UID: \"175fd62e-7735-4976-b8e7-e3c087f14441\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-82299a-4kjdd" Oct 30 12:22:05.340857 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:05.339975 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/175fd62e-7735-4976-b8e7-e3c087f14441-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-82299a-4kjdd\" (UID: \"175fd62e-7735-4976-b8e7-e3c087f14441\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-82299a-4kjdd" Oct 30 12:22:05.340857 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:05.340053 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kg297\" (UniqueName: \"kubernetes.io/projected/175fd62e-7735-4976-b8e7-e3c087f14441-kube-api-access-kg297\") pod \"maas-api-revoke-keys-e2e-api-a-82299a-4kjdd\" (UID: \"175fd62e-7735-4976-b8e7-e3c087f14441\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-82299a-4kjdd" Oct 30 12:22:05.443162 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:05.441512 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kg297\" (UniqueName: \"kubernetes.io/projected/175fd62e-7735-4976-b8e7-e3c087f14441-kube-api-access-kg297\") pod \"maas-api-revoke-keys-e2e-api-a-82299a-4kjdd\" (UID: \"175fd62e-7735-4976-b8e7-e3c087f14441\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-82299a-4kjdd" Oct 30 12:22:05.443162 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:05.441782 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/175fd62e-7735-4976-b8e7-e3c087f14441-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-82299a-4kjdd\" (UID: \"175fd62e-7735-4976-b8e7-e3c087f14441\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-82299a-4kjdd" Oct 30 12:22:05.443162 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:05.441844 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/175fd62e-7735-4976-b8e7-e3c087f14441-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-82299a-4kjdd\" (UID: \"175fd62e-7735-4976-b8e7-e3c087f14441\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-82299a-4kjdd" Oct 30 12:22:05.443162 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:05.442729 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/175fd62e-7735-4976-b8e7-e3c087f14441-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-82299a-4kjdd\" (UID: \"175fd62e-7735-4976-b8e7-e3c087f14441\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-82299a-4kjdd" Oct 30 12:22:05.444976 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:05.444949 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/175fd62e-7735-4976-b8e7-e3c087f14441-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-82299a-4kjdd\" (UID: \"175fd62e-7735-4976-b8e7-e3c087f14441\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-82299a-4kjdd" Oct 30 12:22:05.463897 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:05.463784 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kg297\" (UniqueName: \"kubernetes.io/projected/175fd62e-7735-4976-b8e7-e3c087f14441-kube-api-access-kg297\") pod \"maas-api-revoke-keys-e2e-api-a-82299a-4kjdd\" (UID: \"175fd62e-7735-4976-b8e7-e3c087f14441\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-82299a-4kjdd" Oct 30 12:22:05.673055 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:05.673027 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-82299a-4kjdd" Oct 30 12:22:05.992764 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:05.992740 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-82299a-4kjdd"] Oct 30 12:22:05.994676 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:22:05.994639 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod175fd62e_7735_4976_b8e7_e3c087f14441.slice/crio-eff27537b7ad8dbc090bc1ace53450f50b4f28e3912d61dadb4f7aabd913d0d2 WatchSource:0}: Error finding container eff27537b7ad8dbc090bc1ace53450f50b4f28e3912d61dadb4f7aabd913d0d2: Status 404 returned error can't find the container with id eff27537b7ad8dbc090bc1ace53450f50b4f28e3912d61dadb4f7aabd913d0d2 Oct 30 12:22:06.957217 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:06.954103 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-82299a-4kjdd" event={"ID":"175fd62e-7735-4976-b8e7-e3c087f14441","Type":"ContainerStarted","Data":"c7bb35db053aab00a2954af1791efee329d653d43d7980f7ac1c8720b1e373f3"} Oct 30 12:22:06.957217 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:06.954174 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-82299a-4kjdd" event={"ID":"175fd62e-7735-4976-b8e7-e3c087f14441","Type":"ContainerStarted","Data":"eff27537b7ad8dbc090bc1ace53450f50b4f28e3912d61dadb4f7aabd913d0d2"} Oct 30 12:22:07.007847 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:07.006925 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-82299a-4kjdd" podStartSLOduration=2.006906927 podStartE2EDuration="2.006906927s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:22:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:22:06.994862779 +0000 UTC m=+1683.509291526" watchObservedRunningTime="2026-10-30 12:22:07.006906927 +0000 UTC m=+1683.521335694" Oct 30 12:22:07.300041 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:07.299988 4908 patch_prober.go:29] interesting pod/e2e-api-b-ebc6bb-openshift-default-84d66c85b7-ssbvj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 12:22:07.300041 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:07.300070 4908 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-ebc6bb-openshift-default-84d66c85b7-ssbvj" podUID="cd7c01ae-47be-43dd-93d3-c4702d08ef42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 12:22:07.774618 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:07.774589 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-060f41f2-bx7wc"] Oct 30 12:22:07.783646 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:07.782809 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-060f41f2-bx7wc"] Oct 30 12:22:07.966487 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:07.966389 4908 generic.go:356] "Generic (PLEG): container finished" podID="cd7c01ae-47be-43dd-93d3-c4702d08ef42" containerID="ff15f396b5ac70fd1d2a8dea2c2f98be7f0a9d96dd278f371bc1db4f595ccbbd" exitCode=0 Oct 30 12:22:07.966487 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:07.966467 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ebc6bb-openshift-default-84d66c85b7-ssbvj" event={"ID":"cd7c01ae-47be-43dd-93d3-c4702d08ef42","Type":"ContainerDied","Data":"ff15f396b5ac70fd1d2a8dea2c2f98be7f0a9d96dd278f371bc1db4f595ccbbd"} Oct 30 12:22:07.967271 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:07.967249 4908 scope.go:122] "RemoveContainer" containerID="ff15f396b5ac70fd1d2a8dea2c2f98be7f0a9d96dd278f371bc1db4f595ccbbd" Oct 30 12:22:07.969836 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:07.969489 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:22:07.969836 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:07.969614 4908 generic.go:356] "Generic (PLEG): container finished" podID="175fd62e-7735-4976-b8e7-e3c087f14441" containerID="c7bb35db053aab00a2954af1791efee329d653d43d7980f7ac1c8720b1e373f3" exitCode=0 Oct 30 12:22:07.969836 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:07.969635 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:22:07.969836 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:07.969664 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-82299a-4kjdd" event={"ID":"175fd62e-7735-4976-b8e7-e3c087f14441","Type":"ContainerDied","Data":"c7bb35db053aab00a2954af1791efee329d653d43d7980f7ac1c8720b1e373f3"} Oct 30 12:22:07.969836 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:07.969693 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:22:09.003503 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:09.003461 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ebc6bb-openshift-default-84d66c85b7-ssbvj" event={"ID":"cd7c01ae-47be-43dd-93d3-c4702d08ef42","Type":"ContainerStarted","Data":"fd5af407cd770b40e16d7568d6c178d5626a7b9ee02663acd3acd3121470de38"} Oct 30 12:22:09.040319 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:09.040285 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-ebc6bb-openshift-default-84d66c85b7-ssbvj" Oct 30 12:22:09.040479 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:09.040460 4908 patch_prober.go:29] interesting pod/e2e-api-b-ebc6bb-openshift-default-84d66c85b7-ssbvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Oct 30 12:22:09.040544 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:09.040521 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ebc6bb-openshift-default-84d66c85b7-ssbvj" podUID="cd7c01ae-47be-43dd-93d3-c4702d08ef42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Oct 30 12:22:09.239203 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:09.239178 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-82299a-4kjdd" Oct 30 12:22:09.334883 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:09.333670 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/175fd62e-7735-4976-b8e7-e3c087f14441-kube-api-access-kg297\" (UniqueName: \"kubernetes.io/projected/175fd62e-7735-4976-b8e7-e3c087f14441-kube-api-access-kg297\") pod \"175fd62e-7735-4976-b8e7-e3c087f14441\" (UID: \"175fd62e-7735-4976-b8e7-e3c087f14441\") " Oct 30 12:22:09.334883 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:09.333745 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/175fd62e-7735-4976-b8e7-e3c087f14441-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/175fd62e-7735-4976-b8e7-e3c087f14441-maas-api-xks-ca\") pod \"175fd62e-7735-4976-b8e7-e3c087f14441\" (UID: \"175fd62e-7735-4976-b8e7-e3c087f14441\") " Oct 30 12:22:09.334883 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:09.334061 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/175fd62e-7735-4976-b8e7-e3c087f14441-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/175fd62e-7735-4976-b8e7-e3c087f14441-maas-api-openshift-ca\") pod \"175fd62e-7735-4976-b8e7-e3c087f14441\" (UID: \"175fd62e-7735-4976-b8e7-e3c087f14441\") " Oct 30 12:22:09.335486 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:09.335455 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/175fd62e-7735-4976-b8e7-e3c087f14441-maas-api-openshift-ca" pod "175fd62e-7735-4976-b8e7-e3c087f14441" (UID: "175fd62e-7735-4976-b8e7-e3c087f14441"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:22:09.337488 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:09.337450 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/175fd62e-7735-4976-b8e7-e3c087f14441-maas-api-xks-ca" pod "175fd62e-7735-4976-b8e7-e3c087f14441" (UID: "175fd62e-7735-4976-b8e7-e3c087f14441"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:22:09.337587 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:09.337496 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/175fd62e-7735-4976-b8e7-e3c087f14441-kube-api-access-kg297" pod "175fd62e-7735-4976-b8e7-e3c087f14441" (UID: "175fd62e-7735-4976-b8e7-e3c087f14441"). InnerVolumeSpecName "kube-api-access-kg297". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:22:09.435978 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:09.435942 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/175fd62e-7735-4976-b8e7-e3c087f14441-maas-api-openshift-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:22:09.435978 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:09.435973 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kg297\" (UniqueName: \"kubernetes.io/projected/175fd62e-7735-4976-b8e7-e3c087f14441-kube-api-access-kg297\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:22:09.435978 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:09.435984 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/175fd62e-7735-4976-b8e7-e3c087f14441-maas-api-xks-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:22:09.609149 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:09.609063 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6756f1a3-1c7f-4ab9-9ef9-8b67ec55eaf3" path="/var/lib/kubelet/pods/6756f1a3-1c7f-4ab9-9ef9-8b67ec55eaf3/volumes" Oct 30 12:22:10.013836 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:10.013792 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-82299a-4kjdd" Oct 30 12:22:10.014365 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:10.013859 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-82299a-4kjdd" event={"ID":"175fd62e-7735-4976-b8e7-e3c087f14441","Type":"ContainerDied","Data":"eff27537b7ad8dbc090bc1ace53450f50b4f28e3912d61dadb4f7aabd913d0d2"} Oct 30 12:22:10.014365 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:10.013917 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eff27537b7ad8dbc090bc1ace53450f50b4f28e3912d61dadb4f7aabd913d0d2" Oct 30 12:22:10.040460 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:10.040435 4908 patch_prober.go:29] interesting pod/e2e-api-b-ebc6bb-openshift-default-84d66c85b7-ssbvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Oct 30 12:22:10.040591 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:10.040488 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ebc6bb-openshift-default-84d66c85b7-ssbvj" podUID="cd7c01ae-47be-43dd-93d3-c4702d08ef42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Oct 30 12:22:11.040997 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:11.040967 4908 patch_prober.go:29] interesting pod/e2e-api-b-ebc6bb-openshift-default-84d66c85b7-ssbvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Oct 30 12:22:11.041388 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:11.041018 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ebc6bb-openshift-default-84d66c85b7-ssbvj" podUID="cd7c01ae-47be-43dd-93d3-c4702d08ef42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Oct 30 12:22:12.040220 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:12.040187 4908 patch_prober.go:29] interesting pod/e2e-api-b-ebc6bb-openshift-default-84d66c85b7-ssbvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Oct 30 12:22:12.040424 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:12.040253 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ebc6bb-openshift-default-84d66c85b7-ssbvj" podUID="cd7c01ae-47be-43dd-93d3-c4702d08ef42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Oct 30 12:22:13.042505 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:13.042454 4908 patch_prober.go:29] interesting pod/e2e-api-b-ebc6bb-openshift-default-84d66c85b7-ssbvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Oct 30 12:22:13.042945 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:13.042511 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ebc6bb-openshift-default-84d66c85b7-ssbvj" podUID="cd7c01ae-47be-43dd-93d3-c4702d08ef42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Oct 30 12:22:13.691443 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:13.691401 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9wwfc"] Oct 30 12:22:14.041106 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:14.041066 4908 patch_prober.go:29] interesting pod/e2e-api-b-ebc6bb-openshift-default-84d66c85b7-ssbvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Oct 30 12:22:14.041322 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:14.041143 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ebc6bb-openshift-default-84d66c85b7-ssbvj" podUID="cd7c01ae-47be-43dd-93d3-c4702d08ef42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Oct 30 12:22:15.072195 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:15.072168 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-ebc6bb-openshift-default-84d66c85b7-ssbvj" Oct 30 12:22:15.074512 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:15.074491 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-ebc6bb-openshift-default-84d66c85b7-ssbvj" Oct 30 12:22:15.076718 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:15.076701 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-ebc6bb-openshift-default-84d66c85b7-ssbvj" Oct 30 12:22:19.685760 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:22:19.685721 4908 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vn6vv" containerName="extractor" Oct 30 12:22:33.957710 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:33.957673 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7566f5-whqb6"] Oct 30 12:22:33.971120 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:33.971084 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7566f5-whqb6"] Oct 30 12:22:35.607267 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:35.607228 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="135b84cd-3715-4846-90d7-14c2e5c6985c" path="/var/lib/kubelet/pods/135b84cd-3715-4846-90d7-14c2e5c6985c/volumes" Oct 30 12:22:47.486921 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:47.486883 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-11581-ed6f81"] Oct 30 12:22:47.491865 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:47.491537 4908 cpu_manager.go:414] "containerMap: removing container" podUID="175fd62e-7735-4976-b8e7-e3c087f14441" containerName="revoke-keys" Oct 30 12:22:47.491865 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:47.491572 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="175fd62e-7735-4976-b8e7-e3c087f14441" containerName="revoke-keys" Oct 30 12:22:47.492886 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:47.492275 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="175fd62e-7735-4976-b8e7-e3c087f14441" containerName="revoke-keys" Oct 30 12:22:47.507069 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:47.507038 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-82299a-4kjdd"] Oct 30 12:22:47.507207 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:47.507145 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11581-ed6f81" Oct 30 12:22:47.513265 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:47.513244 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-82299a-4kjdd"] Oct 30 12:22:47.518080 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:47.518057 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11581-ed6f81"] Oct 30 12:22:47.521480 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:47.521460 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-ttfhc\"" Oct 30 12:22:47.616157 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:47.616127 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="175fd62e-7735-4976-b8e7-e3c087f14441" path="/var/lib/kubelet/pods/175fd62e-7735-4976-b8e7-e3c087f14441/volumes" Oct 30 12:22:47.629603 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:47.629567 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wncm\" (UniqueName: \"kubernetes.io/projected/094a54c2-63b2-4ed3-82e7-bc2d7c42d3bb-kube-api-access-9wncm\") pod \"test-curl-11581-ed6f81\" (UID: \"094a54c2-63b2-4ed3-82e7-bc2d7c42d3bb\") " pod="opendatahub/test-curl-11581-ed6f81" Oct 30 12:22:47.730904 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:47.730867 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wncm\" (UniqueName: \"kubernetes.io/projected/094a54c2-63b2-4ed3-82e7-bc2d7c42d3bb-kube-api-access-9wncm\") pod \"test-curl-11581-ed6f81\" (UID: \"094a54c2-63b2-4ed3-82e7-bc2d7c42d3bb\") " pod="opendatahub/test-curl-11581-ed6f81" Oct 30 12:22:47.753265 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:47.753197 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wncm\" (UniqueName: \"kubernetes.io/projected/094a54c2-63b2-4ed3-82e7-bc2d7c42d3bb-kube-api-access-9wncm\") pod \"test-curl-11581-ed6f81\" (UID: \"094a54c2-63b2-4ed3-82e7-bc2d7c42d3bb\") " pod="opendatahub/test-curl-11581-ed6f81" Oct 30 12:22:47.826082 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:47.826051 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11581-ed6f81" Oct 30 12:22:48.031277 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:48.031182 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a164f939-sbn28"] Oct 30 12:22:48.035464 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:48.035434 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a164f939-sbn28"] Oct 30 12:22:48.143937 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:48.143910 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11581-ed6f81"] Oct 30 12:22:48.146096 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:22:48.146039 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod094a54c2_63b2_4ed3_82e7_bc2d7c42d3bb.slice/crio-f0e97ddeb8389fef9f0482b5ff5ae6b9c0c2702d0a54e253663b283454b6de45 WatchSource:0}: Error finding container f0e97ddeb8389fef9f0482b5ff5ae6b9c0c2702d0a54e253663b283454b6de45: Status 404 returned error can't find the container with id f0e97ddeb8389fef9f0482b5ff5ae6b9c0c2702d0a54e253663b283454b6de45 Oct 30 12:22:48.780648 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:48.780612 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11581-ed6f81" event={"ID":"094a54c2-63b2-4ed3-82e7-bc2d7c42d3bb","Type":"ContainerStarted","Data":"bd6bebbf84a1c1fd673fa6fb878e2e403ef1a2b5b378ce78d4dcc4d8856929f9"} Oct 30 12:22:48.781074 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:48.780663 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11581-ed6f81" event={"ID":"094a54c2-63b2-4ed3-82e7-bc2d7c42d3bb","Type":"ContainerStarted","Data":"f0e97ddeb8389fef9f0482b5ff5ae6b9c0c2702d0a54e253663b283454b6de45"} Oct 30 12:22:48.819213 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:48.819157 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-11581-ed6f81" podStartSLOduration=1.81913886 podStartE2EDuration="1.81913886s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:22:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:22:48.815862833 +0000 UTC m=+1725.330291572" watchObservedRunningTime="2026-10-30 12:22:48.81913886 +0000 UTC m=+1725.333567605" Oct 30 12:22:49.609612 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:49.609575 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87a04e72-f817-438e-890c-9e6843ff8c18" path="/var/lib/kubelet/pods/87a04e72-f817-438e-890c-9e6843ff8c18/volumes" Oct 30 12:22:49.685892 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:22:49.685852 4908 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vn6vv" containerName="extractor" Oct 30 12:22:49.774354 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:49.774319 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-11581-ed6f81"] Oct 30 12:22:49.781090 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:49.781061 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-11581-ed6f81"] Oct 30 12:22:50.080770 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:50.080739 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-060f41f2-tpnss"] Oct 30 12:22:50.081318 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:50.081304 4908 cpu_manager.go:414] "containerMap: removing container" podUID="094a54c2-63b2-4ed3-82e7-bc2d7c42d3bb" containerName="test-curl-11581-ed6f81" Oct 30 12:22:50.081372 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:50.081319 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="094a54c2-63b2-4ed3-82e7-bc2d7c42d3bb" containerName="test-curl-11581-ed6f81" Oct 30 12:22:50.081435 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:50.081426 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="094a54c2-63b2-4ed3-82e7-bc2d7c42d3bb" containerName="test-curl-11581-ed6f81" Oct 30 12:22:50.085261 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:50.085241 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-060f41f2-tpnss" Oct 30 12:22:50.098778 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:50.098754 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 12:22:50.103978 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:50.103956 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-060f41f2-tpnss"] Oct 30 12:22:50.109646 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:50.109624 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 12:22:50.109794 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:50.109772 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s75hq\"" Oct 30 12:22:50.109903 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:50.109658 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 12:22:50.160880 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:50.160856 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe5514a3-8747-416c-aff9-91464fecac73-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-060f41f2-tpnss\" (UID: \"fe5514a3-8747-416c-aff9-91464fecac73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-060f41f2-tpnss" Oct 30 12:22:50.160992 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:50.160934 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pq5hb\" (UniqueName: \"kubernetes.io/projected/fe5514a3-8747-416c-aff9-91464fecac73-kube-api-access-pq5hb\") pod \"maas-api-revoke-keys-e2e-mt-060f41f2-tpnss\" (UID: \"fe5514a3-8747-416c-aff9-91464fecac73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-060f41f2-tpnss" Oct 30 12:22:50.161115 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:50.161096 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe5514a3-8747-416c-aff9-91464fecac73-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-060f41f2-tpnss\" (UID: \"fe5514a3-8747-416c-aff9-91464fecac73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-060f41f2-tpnss" Oct 30 12:22:50.262177 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:50.262145 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe5514a3-8747-416c-aff9-91464fecac73-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-060f41f2-tpnss\" (UID: \"fe5514a3-8747-416c-aff9-91464fecac73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-060f41f2-tpnss" Oct 30 12:22:50.262336 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:50.262188 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pq5hb\" (UniqueName: \"kubernetes.io/projected/fe5514a3-8747-416c-aff9-91464fecac73-kube-api-access-pq5hb\") pod \"maas-api-revoke-keys-e2e-mt-060f41f2-tpnss\" (UID: \"fe5514a3-8747-416c-aff9-91464fecac73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-060f41f2-tpnss" Oct 30 12:22:50.262336 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:50.262281 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe5514a3-8747-416c-aff9-91464fecac73-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-060f41f2-tpnss\" (UID: \"fe5514a3-8747-416c-aff9-91464fecac73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-060f41f2-tpnss" Oct 30 12:22:50.262922 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:50.262900 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe5514a3-8747-416c-aff9-91464fecac73-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-060f41f2-tpnss\" (UID: \"fe5514a3-8747-416c-aff9-91464fecac73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-060f41f2-tpnss" Oct 30 12:22:50.264570 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:50.264542 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe5514a3-8747-416c-aff9-91464fecac73-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-060f41f2-tpnss\" (UID: \"fe5514a3-8747-416c-aff9-91464fecac73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-060f41f2-tpnss" Oct 30 12:22:50.284440 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:50.284410 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pq5hb\" (UniqueName: \"kubernetes.io/projected/fe5514a3-8747-416c-aff9-91464fecac73-kube-api-access-pq5hb\") pod \"maas-api-revoke-keys-e2e-mt-060f41f2-tpnss\" (UID: \"fe5514a3-8747-416c-aff9-91464fecac73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-060f41f2-tpnss" Oct 30 12:22:50.398953 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:50.398875 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-060f41f2-tpnss" Oct 30 12:22:50.716425 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:50.716392 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-060f41f2-tpnss"] Oct 30 12:22:50.726136 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:22:50.726089 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe5514a3_8747_416c_aff9_91464fecac73.slice/crio-0c38beb5709935de99a180b03f62c6e9b125e6bd87dc4cfb44448ea971d72208 WatchSource:0}: Error finding container 0c38beb5709935de99a180b03f62c6e9b125e6bd87dc4cfb44448ea971d72208: Status 404 returned error can't find the container with id 0c38beb5709935de99a180b03f62c6e9b125e6bd87dc4cfb44448ea971d72208 Oct 30 12:22:50.804151 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:50.804120 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-060f41f2-tpnss" event={"ID":"fe5514a3-8747-416c-aff9-91464fecac73","Type":"ContainerStarted","Data":"0c38beb5709935de99a180b03f62c6e9b125e6bd87dc4cfb44448ea971d72208"} Oct 30 12:22:50.804482 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:50.804170 4908 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-11581-ed6f81" podUID="094a54c2-63b2-4ed3-82e7-bc2d7c42d3bb" containerName="test-curl-11581-ed6f81" containerID="cri-o://bd6bebbf84a1c1fd673fa6fb878e2e403ef1a2b5b378ce78d4dcc4d8856929f9" gracePeriod=2 Oct 30 12:22:50.816557 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:22:50.816533 4908 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11581-ed6f81\" is forbidden: User \"system:node:ip-10-0-141-251.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-251.ec2.internal' and this object" podUID="094a54c2-63b2-4ed3-82e7-bc2d7c42d3bb" pod="opendatahub/test-curl-11581-ed6f81" Oct 30 12:22:51.814136 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:51.814048 4908 generic.go:356] "Generic (PLEG): container finished" podID="fe5514a3-8747-416c-aff9-91464fecac73" containerID="d68e004e94124997d6f806a99f1b267fd18ca6574f077d1ed1b5b8996a873f16" exitCode=0 Oct 30 12:22:51.814507 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:51.814129 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-060f41f2-tpnss" event={"ID":"fe5514a3-8747-416c-aff9-91464fecac73","Type":"ContainerDied","Data":"d68e004e94124997d6f806a99f1b267fd18ca6574f077d1ed1b5b8996a873f16"} Oct 30 12:22:53.066437 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:53.066367 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-060f41f2-tpnss" Oct 30 12:22:53.099871 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:53.099816 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11581-ed6f81" Oct 30 12:22:53.204308 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:53.204273 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/094a54c2-63b2-4ed3-82e7-bc2d7c42d3bb-kube-api-access-9wncm\" (UniqueName: \"kubernetes.io/projected/094a54c2-63b2-4ed3-82e7-bc2d7c42d3bb-kube-api-access-9wncm\") pod \"094a54c2-63b2-4ed3-82e7-bc2d7c42d3bb\" (UID: \"094a54c2-63b2-4ed3-82e7-bc2d7c42d3bb\") " Oct 30 12:22:53.204475 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:53.204410 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe5514a3-8747-416c-aff9-91464fecac73-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe5514a3-8747-416c-aff9-91464fecac73-maas-api-openshift-ca\") pod \"fe5514a3-8747-416c-aff9-91464fecac73\" (UID: \"fe5514a3-8747-416c-aff9-91464fecac73\") " Oct 30 12:22:53.204582 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:53.204565 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe5514a3-8747-416c-aff9-91464fecac73-kube-api-access-pq5hb\" (UniqueName: \"kubernetes.io/projected/fe5514a3-8747-416c-aff9-91464fecac73-kube-api-access-pq5hb\") pod \"fe5514a3-8747-416c-aff9-91464fecac73\" (UID: \"fe5514a3-8747-416c-aff9-91464fecac73\") " Oct 30 12:22:53.204686 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:53.204669 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fe5514a3-8747-416c-aff9-91464fecac73-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe5514a3-8747-416c-aff9-91464fecac73-maas-api-xks-ca\") pod \"fe5514a3-8747-416c-aff9-91464fecac73\" (UID: \"fe5514a3-8747-416c-aff9-91464fecac73\") " Oct 30 12:22:53.204762 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:53.204736 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe5514a3-8747-416c-aff9-91464fecac73-maas-api-openshift-ca" pod "fe5514a3-8747-416c-aff9-91464fecac73" (UID: "fe5514a3-8747-416c-aff9-91464fecac73"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:22:53.205578 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:53.205542 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe5514a3-8747-416c-aff9-91464fecac73-maas-api-openshift-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:22:53.206735 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:53.206711 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe5514a3-8747-416c-aff9-91464fecac73-kube-api-access-pq5hb" pod "fe5514a3-8747-416c-aff9-91464fecac73" (UID: "fe5514a3-8747-416c-aff9-91464fecac73"). InnerVolumeSpecName "kube-api-access-pq5hb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:22:53.206968 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:53.206945 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe5514a3-8747-416c-aff9-91464fecac73-maas-api-xks-ca" pod "fe5514a3-8747-416c-aff9-91464fecac73" (UID: "fe5514a3-8747-416c-aff9-91464fecac73"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:22:53.207058 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:53.206968 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/094a54c2-63b2-4ed3-82e7-bc2d7c42d3bb-kube-api-access-9wncm" pod "094a54c2-63b2-4ed3-82e7-bc2d7c42d3bb" (UID: "094a54c2-63b2-4ed3-82e7-bc2d7c42d3bb"). InnerVolumeSpecName "kube-api-access-9wncm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:22:53.306998 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:53.306967 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pq5hb\" (UniqueName: \"kubernetes.io/projected/fe5514a3-8747-416c-aff9-91464fecac73-kube-api-access-pq5hb\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:22:53.306998 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:53.306999 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe5514a3-8747-416c-aff9-91464fecac73-maas-api-xks-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:22:53.307173 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:53.307015 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9wncm\" (UniqueName: \"kubernetes.io/projected/094a54c2-63b2-4ed3-82e7-bc2d7c42d3bb-kube-api-access-9wncm\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:22:53.613523 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:53.613492 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="094a54c2-63b2-4ed3-82e7-bc2d7c42d3bb" path="/var/lib/kubelet/pods/094a54c2-63b2-4ed3-82e7-bc2d7c42d3bb/volumes" Oct 30 12:22:53.832888 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:53.832848 4908 generic.go:356] "Generic (PLEG): container finished" podID="094a54c2-63b2-4ed3-82e7-bc2d7c42d3bb" containerID="bd6bebbf84a1c1fd673fa6fb878e2e403ef1a2b5b378ce78d4dcc4d8856929f9" exitCode=137 Oct 30 12:22:53.833005 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:53.832923 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11581-ed6f81" Oct 30 12:22:53.833005 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:53.832951 4908 scope.go:122] "RemoveContainer" containerID="bd6bebbf84a1c1fd673fa6fb878e2e403ef1a2b5b378ce78d4dcc4d8856929f9" Oct 30 12:22:53.834633 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:53.834608 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-060f41f2-tpnss" event={"ID":"fe5514a3-8747-416c-aff9-91464fecac73","Type":"ContainerDied","Data":"0c38beb5709935de99a180b03f62c6e9b125e6bd87dc4cfb44448ea971d72208"} Oct 30 12:22:53.834633 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:53.834626 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-060f41f2-tpnss" Oct 30 12:22:53.834838 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:53.834707 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0c38beb5709935de99a180b03f62c6e9b125e6bd87dc4cfb44448ea971d72208" Oct 30 12:22:53.843357 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:53.843341 4908 scope.go:122] "RemoveContainer" containerID="bd6bebbf84a1c1fd673fa6fb878e2e403ef1a2b5b378ce78d4dcc4d8856929f9" Oct 30 12:22:53.843624 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:22:53.843601 4908 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd6bebbf84a1c1fd673fa6fb878e2e403ef1a2b5b378ce78d4dcc4d8856929f9\": container with ID starting with bd6bebbf84a1c1fd673fa6fb878e2e403ef1a2b5b378ce78d4dcc4d8856929f9 not found: ID does not exist" containerID="bd6bebbf84a1c1fd673fa6fb878e2e403ef1a2b5b378ce78d4dcc4d8856929f9" Oct 30 12:22:53.843720 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:53.843630 4908 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd6bebbf84a1c1fd673fa6fb878e2e403ef1a2b5b378ce78d4dcc4d8856929f9"} err="failed to get container status \"bd6bebbf84a1c1fd673fa6fb878e2e403ef1a2b5b378ce78d4dcc4d8856929f9\": rpc error: code = NotFound desc = could not find container \"bd6bebbf84a1c1fd673fa6fb878e2e403ef1a2b5b378ce78d4dcc4d8856929f9\": container with ID starting with bd6bebbf84a1c1fd673fa6fb878e2e403ef1a2b5b378ce78d4dcc4d8856929f9 not found: ID does not exist" Oct 30 12:22:55.333443 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:55.333395 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ebc6bb-ngzj5"] Oct 30 12:22:55.334163 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:55.334142 4908 cpu_manager.go:414] "containerMap: removing container" podUID="fe5514a3-8747-416c-aff9-91464fecac73" containerName="revoke-keys" Oct 30 12:22:55.334267 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:55.334164 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe5514a3-8747-416c-aff9-91464fecac73" containerName="revoke-keys" Oct 30 12:22:55.334376 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:55.334362 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe5514a3-8747-416c-aff9-91464fecac73" containerName="revoke-keys" Oct 30 12:22:55.339449 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:55.339427 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ebc6bb-ngzj5" Oct 30 12:22:55.356285 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:55.356256 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ebc6bb-ngzj5"] Oct 30 12:22:55.361686 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:55.361664 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 12:22:55.361918 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:55.361896 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 12:22:55.428728 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:55.428701 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 12:22:55.428930 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:55.428915 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s75hq\"" Oct 30 12:22:55.433605 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:55.433583 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1afad88-b907-4719-ab94-8588ae5e24b2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-ebc6bb-ngzj5\" (UID: \"c1afad88-b907-4719-ab94-8588ae5e24b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ebc6bb-ngzj5" Oct 30 12:22:55.433670 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:55.433619 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57v78\" (UniqueName: \"kubernetes.io/projected/c1afad88-b907-4719-ab94-8588ae5e24b2-kube-api-access-57v78\") pod \"maas-api-revoke-keys-e2e-api-b-ebc6bb-ngzj5\" (UID: \"c1afad88-b907-4719-ab94-8588ae5e24b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ebc6bb-ngzj5" Oct 30 12:22:55.433722 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:55.433703 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1afad88-b907-4719-ab94-8588ae5e24b2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-ebc6bb-ngzj5\" (UID: \"c1afad88-b907-4719-ab94-8588ae5e24b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ebc6bb-ngzj5" Oct 30 12:22:55.535058 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:55.535027 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1afad88-b907-4719-ab94-8588ae5e24b2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-ebc6bb-ngzj5\" (UID: \"c1afad88-b907-4719-ab94-8588ae5e24b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ebc6bb-ngzj5" Oct 30 12:22:55.535173 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:55.535068 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57v78\" (UniqueName: \"kubernetes.io/projected/c1afad88-b907-4719-ab94-8588ae5e24b2-kube-api-access-57v78\") pod \"maas-api-revoke-keys-e2e-api-b-ebc6bb-ngzj5\" (UID: \"c1afad88-b907-4719-ab94-8588ae5e24b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ebc6bb-ngzj5" Oct 30 12:22:55.535173 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:55.535117 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1afad88-b907-4719-ab94-8588ae5e24b2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-ebc6bb-ngzj5\" (UID: \"c1afad88-b907-4719-ab94-8588ae5e24b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ebc6bb-ngzj5" Oct 30 12:22:55.535887 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:55.535864 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1afad88-b907-4719-ab94-8588ae5e24b2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-ebc6bb-ngzj5\" (UID: \"c1afad88-b907-4719-ab94-8588ae5e24b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ebc6bb-ngzj5" Oct 30 12:22:55.537444 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:55.537427 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1afad88-b907-4719-ab94-8588ae5e24b2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-ebc6bb-ngzj5\" (UID: \"c1afad88-b907-4719-ab94-8588ae5e24b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ebc6bb-ngzj5" Oct 30 12:22:55.556716 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:55.556693 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57v78\" (UniqueName: \"kubernetes.io/projected/c1afad88-b907-4719-ab94-8588ae5e24b2-kube-api-access-57v78\") pod \"maas-api-revoke-keys-e2e-api-b-ebc6bb-ngzj5\" (UID: \"c1afad88-b907-4719-ab94-8588ae5e24b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ebc6bb-ngzj5" Oct 30 12:22:55.652946 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:55.652882 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ebc6bb-ngzj5" Oct 30 12:22:55.957720 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:55.957699 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ebc6bb-ngzj5"] Oct 30 12:22:55.959092 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:22:55.959056 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1afad88_b907_4719_ab94_8588ae5e24b2.slice/crio-21b1cc759755ccd3d456e6063e3faa2d03469f7f7926f61b6d2cc380b026f9a9 WatchSource:0}: Error finding container 21b1cc759755ccd3d456e6063e3faa2d03469f7f7926f61b6d2cc380b026f9a9: Status 404 returned error can't find the container with id 21b1cc759755ccd3d456e6063e3faa2d03469f7f7926f61b6d2cc380b026f9a9 Oct 30 12:22:56.871130 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:56.871091 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ebc6bb-ngzj5" event={"ID":"c1afad88-b907-4719-ab94-8588ae5e24b2","Type":"ContainerStarted","Data":"76fdb648ad6498eaf1bab8ceb6f4a9bb82c58a32bd22c98840c00a43eba4de55"} Oct 30 12:22:56.871543 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:56.871207 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ebc6bb-ngzj5" event={"ID":"c1afad88-b907-4719-ab94-8588ae5e24b2","Type":"ContainerStarted","Data":"21b1cc759755ccd3d456e6063e3faa2d03469f7f7926f61b6d2cc380b026f9a9"} Oct 30 12:22:57.349471 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:57.349409 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9wwfc"] Oct 30 12:22:57.948523 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:57.938179 4908 generic.go:356] "Generic (PLEG): container finished" podID="c1afad88-b907-4719-ab94-8588ae5e24b2" containerID="76fdb648ad6498eaf1bab8ceb6f4a9bb82c58a32bd22c98840c00a43eba4de55" exitCode=0 Oct 30 12:22:57.948523 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:57.938417 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ebc6bb-ngzj5" event={"ID":"c1afad88-b907-4719-ab94-8588ae5e24b2","Type":"ContainerDied","Data":"76fdb648ad6498eaf1bab8ceb6f4a9bb82c58a32bd22c98840c00a43eba4de55"} Oct 30 12:22:58.132503 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:58.132475 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ebc6bb-ngzj5" Oct 30 12:22:58.272661 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:58.272630 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1afad88-b907-4719-ab94-8588ae5e24b2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1afad88-b907-4719-ab94-8588ae5e24b2-maas-api-openshift-ca\") pod \"c1afad88-b907-4719-ab94-8588ae5e24b2\" (UID: \"c1afad88-b907-4719-ab94-8588ae5e24b2\") " Oct 30 12:22:58.272800 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:58.272715 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1afad88-b907-4719-ab94-8588ae5e24b2-kube-api-access-57v78\" (UniqueName: \"kubernetes.io/projected/c1afad88-b907-4719-ab94-8588ae5e24b2-kube-api-access-57v78\") pod \"c1afad88-b907-4719-ab94-8588ae5e24b2\" (UID: \"c1afad88-b907-4719-ab94-8588ae5e24b2\") " Oct 30 12:22:58.272974 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:58.272954 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c1afad88-b907-4719-ab94-8588ae5e24b2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1afad88-b907-4719-ab94-8588ae5e24b2-maas-api-xks-ca\") pod \"c1afad88-b907-4719-ab94-8588ae5e24b2\" (UID: \"c1afad88-b907-4719-ab94-8588ae5e24b2\") " Oct 30 12:22:58.273050 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:58.273032 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1afad88-b907-4719-ab94-8588ae5e24b2-maas-api-openshift-ca" pod "c1afad88-b907-4719-ab94-8588ae5e24b2" (UID: "c1afad88-b907-4719-ab94-8588ae5e24b2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:22:58.273295 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:58.273275 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1afad88-b907-4719-ab94-8588ae5e24b2-maas-api-openshift-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:22:58.274784 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:58.274761 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1afad88-b907-4719-ab94-8588ae5e24b2-kube-api-access-57v78" pod "c1afad88-b907-4719-ab94-8588ae5e24b2" (UID: "c1afad88-b907-4719-ab94-8588ae5e24b2"). InnerVolumeSpecName "kube-api-access-57v78". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:22:58.274899 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:58.274796 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1afad88-b907-4719-ab94-8588ae5e24b2-maas-api-xks-ca" pod "c1afad88-b907-4719-ab94-8588ae5e24b2" (UID: "c1afad88-b907-4719-ab94-8588ae5e24b2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:22:58.374246 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:58.374204 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57v78\" (UniqueName: \"kubernetes.io/projected/c1afad88-b907-4719-ab94-8588ae5e24b2-kube-api-access-57v78\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:22:58.374246 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:58.374229 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1afad88-b907-4719-ab94-8588ae5e24b2-maas-api-xks-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:22:58.946937 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:58.946912 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ebc6bb-ngzj5" Oct 30 12:22:58.947088 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:58.946901 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ebc6bb-ngzj5" event={"ID":"c1afad88-b907-4719-ab94-8588ae5e24b2","Type":"ContainerDied","Data":"21b1cc759755ccd3d456e6063e3faa2d03469f7f7926f61b6d2cc380b026f9a9"} Oct 30 12:22:58.947088 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:22:58.947044 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="21b1cc759755ccd3d456e6063e3faa2d03469f7f7926f61b6d2cc380b026f9a9" Oct 30 12:23:00.029999 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:00.029968 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-504400d1-cktkq"] Oct 30 12:23:00.042317 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:00.042287 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-504400d1-cktkq"] Oct 30 12:23:01.625851 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:01.625354 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b2f3f3f-ca8b-4b15-a0e2-f3c1381664f9" path="/var/lib/kubelet/pods/6b2f3f3f-ca8b-4b15-a0e2-f3c1381664f9/volumes" Oct 30 12:23:03.301496 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:03.301432 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-060f41f2-openshift-default-5f86797c8f-7vcrc"] Oct 30 12:23:03.302888 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:03.302858 4908 cpu_manager.go:414] "containerMap: removing container" podUID="c1afad88-b907-4719-ab94-8588ae5e24b2" containerName="revoke-keys" Oct 30 12:23:03.302888 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:03.302886 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1afad88-b907-4719-ab94-8588ae5e24b2" containerName="revoke-keys" Oct 30 12:23:03.303092 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:03.303075 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1afad88-b907-4719-ab94-8588ae5e24b2" containerName="revoke-keys" Oct 30 12:23:03.308614 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:03.308588 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-060f41f2-openshift-default-5f86797c8f-7vcrc" Oct 30 12:23:03.321480 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:03.321459 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-060f41f2-openshift-default-5f86797c8f-7vcrc"] Oct 30 12:23:03.345216 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:03.345182 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-060f41f2-openshift-default-dockercfg-n2q4d\"" Oct 30 12:23:03.447351 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:03.447319 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca257393-2ed9-42fa-a81a-ce6001a4ac3b-istio-podinfo\") pod \"e2e-mt-060f41f2-openshift-default-5f86797c8f-7vcrc\" (UID: \"ca257393-2ed9-42fa-a81a-ce6001a4ac3b\") " pod="openshift-ingress/e2e-mt-060f41f2-openshift-default-5f86797c8f-7vcrc" Oct 30 12:23:03.447514 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:03.447429 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca257393-2ed9-42fa-a81a-ce6001a4ac3b-credential-socket\") pod \"e2e-mt-060f41f2-openshift-default-5f86797c8f-7vcrc\" (UID: \"ca257393-2ed9-42fa-a81a-ce6001a4ac3b\") " pod="openshift-ingress/e2e-mt-060f41f2-openshift-default-5f86797c8f-7vcrc" Oct 30 12:23:03.447514 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:03.447450 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca257393-2ed9-42fa-a81a-ce6001a4ac3b-workload-certs\") pod \"e2e-mt-060f41f2-openshift-default-5f86797c8f-7vcrc\" (UID: \"ca257393-2ed9-42fa-a81a-ce6001a4ac3b\") " pod="openshift-ingress/e2e-mt-060f41f2-openshift-default-5f86797c8f-7vcrc" Oct 30 12:23:03.447593 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:03.447518 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca257393-2ed9-42fa-a81a-ce6001a4ac3b-istio-data\") pod \"e2e-mt-060f41f2-openshift-default-5f86797c8f-7vcrc\" (UID: \"ca257393-2ed9-42fa-a81a-ce6001a4ac3b\") " pod="openshift-ingress/e2e-mt-060f41f2-openshift-default-5f86797c8f-7vcrc" Oct 30 12:23:03.447593 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:03.447543 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca257393-2ed9-42fa-a81a-ce6001a4ac3b-istiod-ca-cert\") pod \"e2e-mt-060f41f2-openshift-default-5f86797c8f-7vcrc\" (UID: \"ca257393-2ed9-42fa-a81a-ce6001a4ac3b\") " pod="openshift-ingress/e2e-mt-060f41f2-openshift-default-5f86797c8f-7vcrc" Oct 30 12:23:03.447667 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:03.447644 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca257393-2ed9-42fa-a81a-ce6001a4ac3b-workload-socket\") pod \"e2e-mt-060f41f2-openshift-default-5f86797c8f-7vcrc\" (UID: \"ca257393-2ed9-42fa-a81a-ce6001a4ac3b\") " pod="openshift-ingress/e2e-mt-060f41f2-openshift-default-5f86797c8f-7vcrc" Oct 30 12:23:03.447702 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:03.447680 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca257393-2ed9-42fa-a81a-ce6001a4ac3b-istio-envoy\") pod \"e2e-mt-060f41f2-openshift-default-5f86797c8f-7vcrc\" (UID: \"ca257393-2ed9-42fa-a81a-ce6001a4ac3b\") " pod="openshift-ingress/e2e-mt-060f41f2-openshift-default-5f86797c8f-7vcrc" Oct 30 12:23:03.447783 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:03.447771 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xphm\" (UniqueName: \"kubernetes.io/projected/ca257393-2ed9-42fa-a81a-ce6001a4ac3b-kube-api-access-5xphm\") pod \"e2e-mt-060f41f2-openshift-default-5f86797c8f-7vcrc\" (UID: \"ca257393-2ed9-42fa-a81a-ce6001a4ac3b\") " pod="openshift-ingress/e2e-mt-060f41f2-openshift-default-5f86797c8f-7vcrc" Oct 30 12:23:03.447839 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:03.447815 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca257393-2ed9-42fa-a81a-ce6001a4ac3b-istio-token\") pod \"e2e-mt-060f41f2-openshift-default-5f86797c8f-7vcrc\" (UID: \"ca257393-2ed9-42fa-a81a-ce6001a4ac3b\") " pod="openshift-ingress/e2e-mt-060f41f2-openshift-default-5f86797c8f-7vcrc" Oct 30 12:23:03.548978 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:03.548947 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca257393-2ed9-42fa-a81a-ce6001a4ac3b-istio-token\") pod \"e2e-mt-060f41f2-openshift-default-5f86797c8f-7vcrc\" (UID: \"ca257393-2ed9-42fa-a81a-ce6001a4ac3b\") " pod="openshift-ingress/e2e-mt-060f41f2-openshift-default-5f86797c8f-7vcrc" Oct 30 12:23:03.549156 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:03.549015 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca257393-2ed9-42fa-a81a-ce6001a4ac3b-istio-podinfo\") pod \"e2e-mt-060f41f2-openshift-default-5f86797c8f-7vcrc\" (UID: \"ca257393-2ed9-42fa-a81a-ce6001a4ac3b\") " pod="openshift-ingress/e2e-mt-060f41f2-openshift-default-5f86797c8f-7vcrc" Oct 30 12:23:03.549156 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:03.549076 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca257393-2ed9-42fa-a81a-ce6001a4ac3b-credential-socket\") pod \"e2e-mt-060f41f2-openshift-default-5f86797c8f-7vcrc\" (UID: \"ca257393-2ed9-42fa-a81a-ce6001a4ac3b\") " pod="openshift-ingress/e2e-mt-060f41f2-openshift-default-5f86797c8f-7vcrc" Oct 30 12:23:03.549156 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:03.549099 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca257393-2ed9-42fa-a81a-ce6001a4ac3b-workload-certs\") pod \"e2e-mt-060f41f2-openshift-default-5f86797c8f-7vcrc\" (UID: \"ca257393-2ed9-42fa-a81a-ce6001a4ac3b\") " pod="openshift-ingress/e2e-mt-060f41f2-openshift-default-5f86797c8f-7vcrc" Oct 30 12:23:03.549156 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:03.549143 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca257393-2ed9-42fa-a81a-ce6001a4ac3b-istio-data\") pod \"e2e-mt-060f41f2-openshift-default-5f86797c8f-7vcrc\" (UID: \"ca257393-2ed9-42fa-a81a-ce6001a4ac3b\") " pod="openshift-ingress/e2e-mt-060f41f2-openshift-default-5f86797c8f-7vcrc" Oct 30 12:23:03.549379 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:03.549168 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca257393-2ed9-42fa-a81a-ce6001a4ac3b-istiod-ca-cert\") pod \"e2e-mt-060f41f2-openshift-default-5f86797c8f-7vcrc\" (UID: \"ca257393-2ed9-42fa-a81a-ce6001a4ac3b\") " pod="openshift-ingress/e2e-mt-060f41f2-openshift-default-5f86797c8f-7vcrc" Oct 30 12:23:03.549379 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:03.549227 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca257393-2ed9-42fa-a81a-ce6001a4ac3b-workload-socket\") pod \"e2e-mt-060f41f2-openshift-default-5f86797c8f-7vcrc\" (UID: \"ca257393-2ed9-42fa-a81a-ce6001a4ac3b\") " pod="openshift-ingress/e2e-mt-060f41f2-openshift-default-5f86797c8f-7vcrc" Oct 30 12:23:03.549379 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:03.549258 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca257393-2ed9-42fa-a81a-ce6001a4ac3b-istio-envoy\") pod \"e2e-mt-060f41f2-openshift-default-5f86797c8f-7vcrc\" (UID: \"ca257393-2ed9-42fa-a81a-ce6001a4ac3b\") " pod="openshift-ingress/e2e-mt-060f41f2-openshift-default-5f86797c8f-7vcrc" Oct 30 12:23:03.549379 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:03.549336 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xphm\" (UniqueName: \"kubernetes.io/projected/ca257393-2ed9-42fa-a81a-ce6001a4ac3b-kube-api-access-5xphm\") pod \"e2e-mt-060f41f2-openshift-default-5f86797c8f-7vcrc\" (UID: \"ca257393-2ed9-42fa-a81a-ce6001a4ac3b\") " pod="openshift-ingress/e2e-mt-060f41f2-openshift-default-5f86797c8f-7vcrc" Oct 30 12:23:03.549573 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:03.549494 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca257393-2ed9-42fa-a81a-ce6001a4ac3b-credential-socket\") pod \"e2e-mt-060f41f2-openshift-default-5f86797c8f-7vcrc\" (UID: \"ca257393-2ed9-42fa-a81a-ce6001a4ac3b\") " pod="openshift-ingress/e2e-mt-060f41f2-openshift-default-5f86797c8f-7vcrc" Oct 30 12:23:03.549573 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:03.549530 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca257393-2ed9-42fa-a81a-ce6001a4ac3b-istio-data\") pod \"e2e-mt-060f41f2-openshift-default-5f86797c8f-7vcrc\" (UID: \"ca257393-2ed9-42fa-a81a-ce6001a4ac3b\") " pod="openshift-ingress/e2e-mt-060f41f2-openshift-default-5f86797c8f-7vcrc" Oct 30 12:23:03.549573 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:03.549554 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca257393-2ed9-42fa-a81a-ce6001a4ac3b-workload-socket\") pod \"e2e-mt-060f41f2-openshift-default-5f86797c8f-7vcrc\" (UID: \"ca257393-2ed9-42fa-a81a-ce6001a4ac3b\") " pod="openshift-ingress/e2e-mt-060f41f2-openshift-default-5f86797c8f-7vcrc" Oct 30 12:23:03.549884 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:03.549863 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca257393-2ed9-42fa-a81a-ce6001a4ac3b-workload-certs\") pod \"e2e-mt-060f41f2-openshift-default-5f86797c8f-7vcrc\" (UID: \"ca257393-2ed9-42fa-a81a-ce6001a4ac3b\") " pod="openshift-ingress/e2e-mt-060f41f2-openshift-default-5f86797c8f-7vcrc" Oct 30 12:23:03.550154 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:03.550134 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca257393-2ed9-42fa-a81a-ce6001a4ac3b-istiod-ca-cert\") pod \"e2e-mt-060f41f2-openshift-default-5f86797c8f-7vcrc\" (UID: \"ca257393-2ed9-42fa-a81a-ce6001a4ac3b\") " pod="openshift-ingress/e2e-mt-060f41f2-openshift-default-5f86797c8f-7vcrc" Oct 30 12:23:03.551650 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:03.551602 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca257393-2ed9-42fa-a81a-ce6001a4ac3b-istio-envoy\") pod \"e2e-mt-060f41f2-openshift-default-5f86797c8f-7vcrc\" (UID: \"ca257393-2ed9-42fa-a81a-ce6001a4ac3b\") " pod="openshift-ingress/e2e-mt-060f41f2-openshift-default-5f86797c8f-7vcrc" Oct 30 12:23:03.551947 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:03.551928 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca257393-2ed9-42fa-a81a-ce6001a4ac3b-istio-podinfo\") pod \"e2e-mt-060f41f2-openshift-default-5f86797c8f-7vcrc\" (UID: \"ca257393-2ed9-42fa-a81a-ce6001a4ac3b\") " pod="openshift-ingress/e2e-mt-060f41f2-openshift-default-5f86797c8f-7vcrc" Oct 30 12:23:03.571035 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:03.571015 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca257393-2ed9-42fa-a81a-ce6001a4ac3b-istio-token\") pod \"e2e-mt-060f41f2-openshift-default-5f86797c8f-7vcrc\" (UID: \"ca257393-2ed9-42fa-a81a-ce6001a4ac3b\") " pod="openshift-ingress/e2e-mt-060f41f2-openshift-default-5f86797c8f-7vcrc" Oct 30 12:23:03.571295 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:03.571274 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xphm\" (UniqueName: \"kubernetes.io/projected/ca257393-2ed9-42fa-a81a-ce6001a4ac3b-kube-api-access-5xphm\") pod \"e2e-mt-060f41f2-openshift-default-5f86797c8f-7vcrc\" (UID: \"ca257393-2ed9-42fa-a81a-ce6001a4ac3b\") " pod="openshift-ingress/e2e-mt-060f41f2-openshift-default-5f86797c8f-7vcrc" Oct 30 12:23:03.650396 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:03.650291 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-060f41f2-openshift-default-dockercfg-n2q4d\"" Oct 30 12:23:03.658289 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:03.658267 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-060f41f2-openshift-default-5f86797c8f-7vcrc" Oct 30 12:23:03.995065 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:23:03.995036 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca257393_2ed9_42fa_a81a_ce6001a4ac3b.slice/crio-9e03c1db8f721feb91500749ed6510ee44463fc39ee75db34a7b374c085b7a23 WatchSource:0}: Error finding container 9e03c1db8f721feb91500749ed6510ee44463fc39ee75db34a7b374c085b7a23: Status 404 returned error can't find the container with id 9e03c1db8f721feb91500749ed6510ee44463fc39ee75db34a7b374c085b7a23 Oct 30 12:23:03.996496 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:03.996471 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-060f41f2-openshift-default-5f86797c8f-7vcrc"] Oct 30 12:23:04.003323 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:04.001822 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:23:04.003323 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:04.001972 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:23:04.003323 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:04.002051 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:23:04.171343 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:04.171309 4908 scope.go:122] "RemoveContainer" containerID="6d0f0517fc6dc1f8f584c840277c316d6e140a762958499df768c036369af6fb" Oct 30 12:23:04.182312 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:04.182290 4908 scope.go:122] "RemoveContainer" containerID="d3c5aa0a4c4902f4adb29f5ca37dfcc09f89602f6d1fc0e340082df9d1560af6" Oct 30 12:23:04.194728 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:04.194703 4908 scope.go:122] "RemoveContainer" containerID="7756d5d15b4ff9f2a4fa1bcbe9ab639fd92451caaf336330fce2951e701ad21b" Oct 30 12:23:04.205817 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:04.205796 4908 scope.go:122] "RemoveContainer" containerID="8d46ceeea9ff9ac6015e4cef6b044464282b5e0c28e1fb0bc43a5c2ce4880e11" Oct 30 12:23:04.214577 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:04.214558 4908 scope.go:122] "RemoveContainer" containerID="10893ddb39916408c23cc3aa56971d59adb4d97407defbbd3cde1b9f21778c86" Oct 30 12:23:05.013267 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:05.013226 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-060f41f2-openshift-default-5f86797c8f-7vcrc" event={"ID":"ca257393-2ed9-42fa-a81a-ce6001a4ac3b","Type":"ContainerStarted","Data":"76ef52ef0419d17a6526ba1b2fdf3da59239cb587f03613f3a4a46e2115d3c1d"} Oct 30 12:23:05.013688 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:05.013292 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-060f41f2-openshift-default-5f86797c8f-7vcrc" event={"ID":"ca257393-2ed9-42fa-a81a-ce6001a4ac3b","Type":"ContainerStarted","Data":"9e03c1db8f721feb91500749ed6510ee44463fc39ee75db34a7b374c085b7a23"} Oct 30 12:23:05.658607 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:05.658569 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-060f41f2-openshift-default-5f86797c8f-7vcrc" Oct 30 12:23:05.681849 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:05.671916 4908 patch_prober.go:29] interesting pod/e2e-mt-060f41f2-openshift-default-5f86797c8f-7vcrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body= Oct 30 12:23:05.681849 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:05.671979 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-060f41f2-openshift-default-5f86797c8f-7vcrc" podUID="ca257393-2ed9-42fa-a81a-ce6001a4ac3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" Oct 30 12:23:05.993886 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:05.993773 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-060f41f2-openshift-default-5f86797c8f-7vcrc" podStartSLOduration=2.993741607 podStartE2EDuration="2.993741607s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:23:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:23:05.09732685 +0000 UTC m=+1741.611755594" watchObservedRunningTime="2026-10-30 12:23:05.993741607 +0000 UTC m=+1742.508170352" Oct 30 12:23:06.010583 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:06.009313 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93eb20b4-hmhp2"] Oct 30 12:23:06.042876 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:06.035922 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93eb20b4-hmhp2" Oct 30 12:23:06.048355 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:06.048330 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93eb20b4-hmhp2"] Oct 30 12:23:06.052012 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:06.051993 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 12:23:06.052218 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:06.052206 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 12:23:06.052415 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:06.052404 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 12:23:06.053050 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:06.052555 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s75hq\"" Oct 30 12:23:06.114532 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:06.114498 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96537d4e-4868-47ee-bb61-f7b2d2f3708c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-93eb20b4-hmhp2\" (UID: \"96537d4e-4868-47ee-bb61-f7b2d2f3708c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93eb20b4-hmhp2" Oct 30 12:23:06.114875 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:06.114822 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96537d4e-4868-47ee-bb61-f7b2d2f3708c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-93eb20b4-hmhp2\" (UID: \"96537d4e-4868-47ee-bb61-f7b2d2f3708c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93eb20b4-hmhp2" Oct 30 12:23:06.115322 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:06.115301 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkrlr\" (UniqueName: \"kubernetes.io/projected/96537d4e-4868-47ee-bb61-f7b2d2f3708c-kube-api-access-rkrlr\") pod \"maas-api-revoke-sub-93eb20b4-hmhp2\" (UID: \"96537d4e-4868-47ee-bb61-f7b2d2f3708c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93eb20b4-hmhp2" Oct 30 12:23:06.217271 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:06.217240 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkrlr\" (UniqueName: \"kubernetes.io/projected/96537d4e-4868-47ee-bb61-f7b2d2f3708c-kube-api-access-rkrlr\") pod \"maas-api-revoke-sub-93eb20b4-hmhp2\" (UID: \"96537d4e-4868-47ee-bb61-f7b2d2f3708c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93eb20b4-hmhp2" Oct 30 12:23:06.217448 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:06.217332 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96537d4e-4868-47ee-bb61-f7b2d2f3708c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-93eb20b4-hmhp2\" (UID: \"96537d4e-4868-47ee-bb61-f7b2d2f3708c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93eb20b4-hmhp2" Oct 30 12:23:06.217448 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:06.217383 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96537d4e-4868-47ee-bb61-f7b2d2f3708c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-93eb20b4-hmhp2\" (UID: \"96537d4e-4868-47ee-bb61-f7b2d2f3708c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93eb20b4-hmhp2" Oct 30 12:23:06.218086 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:06.218061 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96537d4e-4868-47ee-bb61-f7b2d2f3708c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-93eb20b4-hmhp2\" (UID: \"96537d4e-4868-47ee-bb61-f7b2d2f3708c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93eb20b4-hmhp2" Oct 30 12:23:06.219743 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:06.219718 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96537d4e-4868-47ee-bb61-f7b2d2f3708c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-93eb20b4-hmhp2\" (UID: \"96537d4e-4868-47ee-bb61-f7b2d2f3708c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93eb20b4-hmhp2" Oct 30 12:23:06.239571 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:06.239550 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkrlr\" (UniqueName: \"kubernetes.io/projected/96537d4e-4868-47ee-bb61-f7b2d2f3708c-kube-api-access-rkrlr\") pod \"maas-api-revoke-sub-93eb20b4-hmhp2\" (UID: \"96537d4e-4868-47ee-bb61-f7b2d2f3708c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93eb20b4-hmhp2" Oct 30 12:23:06.418589 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:06.412087 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93eb20b4-hmhp2" Oct 30 12:23:06.658803 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:06.658767 4908 patch_prober.go:29] interesting pod/e2e-mt-060f41f2-openshift-default-5f86797c8f-7vcrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body= Oct 30 12:23:06.659026 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:06.658862 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-060f41f2-openshift-default-5f86797c8f-7vcrc" podUID="ca257393-2ed9-42fa-a81a-ce6001a4ac3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" Oct 30 12:23:06.999594 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:23:06.999542 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96537d4e_4868_47ee_bb61_f7b2d2f3708c.slice/crio-bc80f79d4fd7f4de7aaa3e942294572267fe905174357b19a40a580cb10c24c2 WatchSource:0}: Error finding container bc80f79d4fd7f4de7aaa3e942294572267fe905174357b19a40a580cb10c24c2: Status 404 returned error can't find the container with id bc80f79d4fd7f4de7aaa3e942294572267fe905174357b19a40a580cb10c24c2 Oct 30 12:23:07.001580 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:07.001546 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93eb20b4-hmhp2"] Oct 30 12:23:07.098043 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:07.097992 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93eb20b4-hmhp2" event={"ID":"96537d4e-4868-47ee-bb61-f7b2d2f3708c","Type":"ContainerStarted","Data":"bc80f79d4fd7f4de7aaa3e942294572267fe905174357b19a40a580cb10c24c2"} Oct 30 12:23:07.658803 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:07.658726 4908 patch_prober.go:29] interesting pod/e2e-mt-060f41f2-openshift-default-5f86797c8f-7vcrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body= Oct 30 12:23:07.658803 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:07.658790 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-060f41f2-openshift-default-5f86797c8f-7vcrc" podUID="ca257393-2ed9-42fa-a81a-ce6001a4ac3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" Oct 30 12:23:08.110582 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:08.110548 4908 generic.go:356] "Generic (PLEG): container finished" podID="96537d4e-4868-47ee-bb61-f7b2d2f3708c" containerID="226c6a5e28afada6d8a15c17d8c23822c3ee8e3d2f738520caaeafc762a92c83" exitCode=0 Oct 30 12:23:08.111014 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:08.110582 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93eb20b4-hmhp2" event={"ID":"96537d4e-4868-47ee-bb61-f7b2d2f3708c","Type":"ContainerDied","Data":"226c6a5e28afada6d8a15c17d8c23822c3ee8e3d2f738520caaeafc762a92c83"} Oct 30 12:23:08.684874 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:08.678917 4908 patch_prober.go:29] interesting pod/e2e-mt-060f41f2-openshift-default-5f86797c8f-7vcrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body= Oct 30 12:23:08.684874 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:08.678975 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-060f41f2-openshift-default-5f86797c8f-7vcrc" podUID="ca257393-2ed9-42fa-a81a-ce6001a4ac3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" Oct 30 12:23:09.357734 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:09.357712 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93eb20b4-hmhp2" Oct 30 12:23:09.512141 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:09.512108 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/96537d4e-4868-47ee-bb61-f7b2d2f3708c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96537d4e-4868-47ee-bb61-f7b2d2f3708c-maas-api-xks-ca\") pod \"96537d4e-4868-47ee-bb61-f7b2d2f3708c\" (UID: \"96537d4e-4868-47ee-bb61-f7b2d2f3708c\") " Oct 30 12:23:09.512316 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:09.512201 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96537d4e-4868-47ee-bb61-f7b2d2f3708c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96537d4e-4868-47ee-bb61-f7b2d2f3708c-maas-api-openshift-ca\") pod \"96537d4e-4868-47ee-bb61-f7b2d2f3708c\" (UID: \"96537d4e-4868-47ee-bb61-f7b2d2f3708c\") " Oct 30 12:23:09.512399 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:09.512375 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96537d4e-4868-47ee-bb61-f7b2d2f3708c-kube-api-access-rkrlr\" (UniqueName: \"kubernetes.io/projected/96537d4e-4868-47ee-bb61-f7b2d2f3708c-kube-api-access-rkrlr\") pod \"96537d4e-4868-47ee-bb61-f7b2d2f3708c\" (UID: \"96537d4e-4868-47ee-bb61-f7b2d2f3708c\") " Oct 30 12:23:09.512549 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:09.512529 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96537d4e-4868-47ee-bb61-f7b2d2f3708c-maas-api-openshift-ca" pod "96537d4e-4868-47ee-bb61-f7b2d2f3708c" (UID: "96537d4e-4868-47ee-bb61-f7b2d2f3708c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:23:09.512930 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:09.512909 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96537d4e-4868-47ee-bb61-f7b2d2f3708c-maas-api-openshift-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:23:09.514395 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:09.514369 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96537d4e-4868-47ee-bb61-f7b2d2f3708c-maas-api-xks-ca" pod "96537d4e-4868-47ee-bb61-f7b2d2f3708c" (UID: "96537d4e-4868-47ee-bb61-f7b2d2f3708c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:23:09.515695 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:09.515645 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96537d4e-4868-47ee-bb61-f7b2d2f3708c-kube-api-access-rkrlr" pod "96537d4e-4868-47ee-bb61-f7b2d2f3708c" (UID: "96537d4e-4868-47ee-bb61-f7b2d2f3708c"). InnerVolumeSpecName "kube-api-access-rkrlr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:23:09.613747 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:09.613715 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rkrlr\" (UniqueName: \"kubernetes.io/projected/96537d4e-4868-47ee-bb61-f7b2d2f3708c-kube-api-access-rkrlr\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:23:09.613747 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:09.613744 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96537d4e-4868-47ee-bb61-f7b2d2f3708c-maas-api-xks-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:23:09.665851 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:09.664994 4908 patch_prober.go:29] interesting pod/e2e-mt-060f41f2-openshift-default-5f86797c8f-7vcrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body= Oct 30 12:23:09.665851 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:09.665047 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-060f41f2-openshift-default-5f86797c8f-7vcrc" podUID="ca257393-2ed9-42fa-a81a-ce6001a4ac3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" Oct 30 12:23:10.129804 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:10.129773 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93eb20b4-hmhp2" Oct 30 12:23:10.130026 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:10.129804 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93eb20b4-hmhp2" event={"ID":"96537d4e-4868-47ee-bb61-f7b2d2f3708c","Type":"ContainerDied","Data":"bc80f79d4fd7f4de7aaa3e942294572267fe905174357b19a40a580cb10c24c2"} Oct 30 12:23:10.130026 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:10.129870 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bc80f79d4fd7f4de7aaa3e942294572267fe905174357b19a40a580cb10c24c2" Oct 30 12:23:10.663261 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:10.663233 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-060f41f2-openshift-default-5f86797c8f-7vcrc" Oct 30 12:23:10.663671 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:10.663456 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-060f41f2-openshift-default-5f86797c8f-7vcrc" Oct 30 12:23:10.664087 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:10.664069 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-060f41f2-openshift-default-5f86797c8f-7vcrc" Oct 30 12:23:12.033169 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:12.033133 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-66838cc1-wwvtr"] Oct 30 12:23:12.042647 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:12.042619 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-66838cc1-wwvtr"] Oct 30 12:23:13.030893 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:13.030856 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d325dcc2-l845t"] Oct 30 12:23:13.042002 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:13.041975 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d325dcc2-l845t"] Oct 30 12:23:13.606507 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:13.606473 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4bfb80b4-66c6-42a0-90c2-9ddd522c2811" path="/var/lib/kubelet/pods/4bfb80b4-66c6-42a0-90c2-9ddd522c2811/volumes" Oct 30 12:23:13.606949 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:13.606934 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="590ad346-4bdd-4519-bfb8-adeefa86ba38" path="/var/lib/kubelet/pods/590ad346-4bdd-4519-bfb8-adeefa86ba38/volumes" Oct 30 12:23:15.496846 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:15.496776 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-060f41f2-tpnss"] Oct 30 12:23:15.513505 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:15.513472 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-060f41f2-tpnss"] Oct 30 12:23:15.609153 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:15.609123 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe5514a3-8747-416c-aff9-91464fecac73" path="/var/lib/kubelet/pods/fe5514a3-8747-416c-aff9-91464fecac73/volumes" Oct 30 12:23:19.687657 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:23:19.687620 4908 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vn6vv" containerName="extractor" Oct 30 12:23:20.266220 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:20.266188 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c35d6b-mw9n9"] Oct 30 12:23:20.266801 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:20.266786 4908 cpu_manager.go:414] "containerMap: removing container" podUID="96537d4e-4868-47ee-bb61-f7b2d2f3708c" containerName="revoke-keys" Oct 30 12:23:20.266859 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:20.266804 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96537d4e-4868-47ee-bb61-f7b2d2f3708c" containerName="revoke-keys" Oct 30 12:23:20.266952 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:20.266940 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="96537d4e-4868-47ee-bb61-f7b2d2f3708c" containerName="revoke-keys" Oct 30 12:23:20.272496 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:20.272478 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c35d6b-mw9n9" Oct 30 12:23:20.288989 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:20.288964 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c35d6b-mw9n9"] Oct 30 12:23:20.294394 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:20.294365 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s75hq\"" Oct 30 12:23:20.294394 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:20.294386 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 12:23:20.301869 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:20.301845 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 12:23:20.301973 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:20.301869 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 12:23:20.383824 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:20.383788 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnnzh\" (UniqueName: \"kubernetes.io/projected/d570f102-c33d-4383-b67e-faccb60fea14-kube-api-access-lnnzh\") pod \"maas-api-revoke-keys-e2e-api-a-c35d6b-mw9n9\" (UID: \"d570f102-c33d-4383-b67e-faccb60fea14\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c35d6b-mw9n9" Oct 30 12:23:20.384006 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:20.383988 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d570f102-c33d-4383-b67e-faccb60fea14-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c35d6b-mw9n9\" (UID: \"d570f102-c33d-4383-b67e-faccb60fea14\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c35d6b-mw9n9" Oct 30 12:23:20.384057 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:20.384042 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d570f102-c33d-4383-b67e-faccb60fea14-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c35d6b-mw9n9\" (UID: \"d570f102-c33d-4383-b67e-faccb60fea14\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c35d6b-mw9n9" Oct 30 12:23:20.485414 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:20.485383 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnnzh\" (UniqueName: \"kubernetes.io/projected/d570f102-c33d-4383-b67e-faccb60fea14-kube-api-access-lnnzh\") pod \"maas-api-revoke-keys-e2e-api-a-c35d6b-mw9n9\" (UID: \"d570f102-c33d-4383-b67e-faccb60fea14\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c35d6b-mw9n9" Oct 30 12:23:20.485565 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:20.485488 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d570f102-c33d-4383-b67e-faccb60fea14-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c35d6b-mw9n9\" (UID: \"d570f102-c33d-4383-b67e-faccb60fea14\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c35d6b-mw9n9" Oct 30 12:23:20.485565 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:20.485526 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d570f102-c33d-4383-b67e-faccb60fea14-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c35d6b-mw9n9\" (UID: \"d570f102-c33d-4383-b67e-faccb60fea14\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c35d6b-mw9n9" Oct 30 12:23:20.486480 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:20.486447 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d570f102-c33d-4383-b67e-faccb60fea14-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c35d6b-mw9n9\" (UID: \"d570f102-c33d-4383-b67e-faccb60fea14\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c35d6b-mw9n9" Oct 30 12:23:20.488053 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:20.488028 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d570f102-c33d-4383-b67e-faccb60fea14-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c35d6b-mw9n9\" (UID: \"d570f102-c33d-4383-b67e-faccb60fea14\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c35d6b-mw9n9" Oct 30 12:23:20.507965 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:20.507942 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnnzh\" (UniqueName: \"kubernetes.io/projected/d570f102-c33d-4383-b67e-faccb60fea14-kube-api-access-lnnzh\") pod \"maas-api-revoke-keys-e2e-api-a-c35d6b-mw9n9\" (UID: \"d570f102-c33d-4383-b67e-faccb60fea14\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c35d6b-mw9n9" Oct 30 12:23:20.588648 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:20.588583 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c35d6b-mw9n9" Oct 30 12:23:20.897813 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:20.897778 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c35d6b-mw9n9"] Oct 30 12:23:20.899309 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:23:20.899276 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd570f102_c33d_4383_b67e_faccb60fea14.slice/crio-70d0ae8c83fe7b07189326a00c27ed760e706a6fb2d8a566130d15457d2d407c WatchSource:0}: Error finding container 70d0ae8c83fe7b07189326a00c27ed760e706a6fb2d8a566130d15457d2d407c: Status 404 returned error can't find the container with id 70d0ae8c83fe7b07189326a00c27ed760e706a6fb2d8a566130d15457d2d407c Oct 30 12:23:21.237654 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:21.237610 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c35d6b-mw9n9" event={"ID":"d570f102-c33d-4383-b67e-faccb60fea14","Type":"ContainerStarted","Data":"186188cd6e795b7d461111050eaaf79462673726469dc83b0d99238b72fb5eae"} Oct 30 12:23:21.237795 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:21.237688 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c35d6b-mw9n9" event={"ID":"d570f102-c33d-4383-b67e-faccb60fea14","Type":"ContainerStarted","Data":"70d0ae8c83fe7b07189326a00c27ed760e706a6fb2d8a566130d15457d2d407c"} Oct 30 12:23:21.276726 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:21.276674 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c35d6b-mw9n9" podStartSLOduration=1.276660435 podStartE2EDuration="1.276660435s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:23:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:23:21.274756239 +0000 UTC m=+1757.789184988" watchObservedRunningTime="2026-10-30 12:23:21.276660435 +0000 UTC m=+1757.791089179" Oct 30 12:23:22.246576 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:22.246540 4908 generic.go:356] "Generic (PLEG): container finished" podID="d570f102-c33d-4383-b67e-faccb60fea14" containerID="186188cd6e795b7d461111050eaaf79462673726469dc83b0d99238b72fb5eae" exitCode=0 Oct 30 12:23:22.246969 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:22.246626 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c35d6b-mw9n9" event={"ID":"d570f102-c33d-4383-b67e-faccb60fea14","Type":"ContainerDied","Data":"186188cd6e795b7d461111050eaaf79462673726469dc83b0d99238b72fb5eae"} Oct 30 12:23:23.415534 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:23.415512 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c35d6b-mw9n9" Oct 30 12:23:23.524521 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:23.524448 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d570f102-c33d-4383-b67e-faccb60fea14-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d570f102-c33d-4383-b67e-faccb60fea14-maas-api-xks-ca\") pod \"d570f102-c33d-4383-b67e-faccb60fea14\" (UID: \"d570f102-c33d-4383-b67e-faccb60fea14\") " Oct 30 12:23:23.524685 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:23.524550 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d570f102-c33d-4383-b67e-faccb60fea14-kube-api-access-lnnzh\" (UniqueName: \"kubernetes.io/projected/d570f102-c33d-4383-b67e-faccb60fea14-kube-api-access-lnnzh\") pod \"d570f102-c33d-4383-b67e-faccb60fea14\" (UID: \"d570f102-c33d-4383-b67e-faccb60fea14\") " Oct 30 12:23:23.524685 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:23.524575 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d570f102-c33d-4383-b67e-faccb60fea14-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d570f102-c33d-4383-b67e-faccb60fea14-maas-api-openshift-ca\") pod \"d570f102-c33d-4383-b67e-faccb60fea14\" (UID: \"d570f102-c33d-4383-b67e-faccb60fea14\") " Oct 30 12:23:23.524949 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:23.524915 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d570f102-c33d-4383-b67e-faccb60fea14-maas-api-openshift-ca" pod "d570f102-c33d-4383-b67e-faccb60fea14" (UID: "d570f102-c33d-4383-b67e-faccb60fea14"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:23:23.525096 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:23.525083 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d570f102-c33d-4383-b67e-faccb60fea14-maas-api-openshift-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:23:23.526583 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:23.526562 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d570f102-c33d-4383-b67e-faccb60fea14-maas-api-xks-ca" pod "d570f102-c33d-4383-b67e-faccb60fea14" (UID: "d570f102-c33d-4383-b67e-faccb60fea14"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:23:23.526673 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:23.526658 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d570f102-c33d-4383-b67e-faccb60fea14-kube-api-access-lnnzh" pod "d570f102-c33d-4383-b67e-faccb60fea14" (UID: "d570f102-c33d-4383-b67e-faccb60fea14"). InnerVolumeSpecName "kube-api-access-lnnzh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:23:23.626402 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:23.626377 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lnnzh\" (UniqueName: \"kubernetes.io/projected/d570f102-c33d-4383-b67e-faccb60fea14-kube-api-access-lnnzh\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:23:23.626402 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:23.626401 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d570f102-c33d-4383-b67e-faccb60fea14-maas-api-xks-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:23:24.279683 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:24.279637 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c35d6b-mw9n9" Oct 30 12:23:24.279897 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:24.279641 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c35d6b-mw9n9" event={"ID":"d570f102-c33d-4383-b67e-faccb60fea14","Type":"ContainerDied","Data":"70d0ae8c83fe7b07189326a00c27ed760e706a6fb2d8a566130d15457d2d407c"} Oct 30 12:23:24.279897 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:24.279855 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="70d0ae8c83fe7b07189326a00c27ed760e706a6fb2d8a566130d15457d2d407c" Oct 30 12:23:25.039348 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:25.039314 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4988034a-6w927"] Oct 30 12:23:25.043849 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:25.043812 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4988034a-6w927"] Oct 30 12:23:25.606579 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:25.606548 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="261a158f-13b4-4c82-883d-46f2f71cc6e8" path="/var/lib/kubelet/pods/261a158f-13b4-4c82-883d-46f2f71cc6e8/volumes" Oct 30 12:23:28.037402 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:28.037373 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89356cf2-wzcrk"] Oct 30 12:23:28.048129 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:28.048104 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89356cf2-wzcrk"] Oct 30 12:23:29.606137 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:29.606104 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2c42610-d2fb-467b-9c70-a7afc6e3047b" path="/var/lib/kubelet/pods/a2c42610-d2fb-467b-9c70-a7afc6e3047b/volumes" Oct 30 12:23:31.338051 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:31.338025 4908 generic.go:356] "Generic (PLEG): container finished" podID="1398a43d-5477-4f46-bfad-6e8dba0af13a" containerID="9c365a87066ee631e6338d735b5c66f694d8922952730d349d4e3dad7448b0ac" exitCode=1 Oct 30 12:23:31.338330 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:31.338096 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-lnqqz" event={"ID":"1398a43d-5477-4f46-bfad-6e8dba0af13a","Type":"ContainerDied","Data":"9c365a87066ee631e6338d735b5c66f694d8922952730d349d4e3dad7448b0ac"} Oct 30 12:23:31.338425 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:31.338411 4908 scope.go:122] "RemoveContainer" containerID="9c365a87066ee631e6338d735b5c66f694d8922952730d349d4e3dad7448b0ac" Oct 30 12:23:31.362781 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:31.362759 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Oct 30 12:23:32.348656 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:32.348622 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-lnqqz" event={"ID":"1398a43d-5477-4f46-bfad-6e8dba0af13a","Type":"ContainerStarted","Data":"9a6776b602b417761a0a529e9d58cb92806a1975d3a15cf348403af6296566d2"} Oct 30 12:23:32.349045 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:32.348878 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-lnqqz" Oct 30 12:23:38.028845 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:38.028791 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a08284d4-jrgdb"] Oct 30 12:23:38.046122 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:38.046083 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a08284d4-jrgdb"] Oct 30 12:23:39.605543 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:39.605509 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bec895b3-a116-43e4-8465-735a56c63b51" path="/var/lib/kubelet/pods/bec895b3-a116-43e4-8465-735a56c63b51/volumes" Oct 30 12:23:42.029139 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:42.029108 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0452683a-mwvss"] Oct 30 12:23:42.040013 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:42.039982 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0452683a-mwvss"] Oct 30 12:23:43.357793 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:43.357762 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-lnqqz" Oct 30 12:23:43.610703 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:43.610632 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e059d9e-06a0-4f7b-9b54-faa87582e184" path="/var/lib/kubelet/pods/4e059d9e-06a0-4f7b-9b54-faa87582e184/volumes" Oct 30 12:23:44.043198 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:44.043160 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-779a2a29-xbdqx"] Oct 30 12:23:44.053437 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:44.053413 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-779a2a29-xbdqx"] Oct 30 12:23:45.605611 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:45.605578 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99920224-1d67-4af4-90a0-ebd548e4f618" path="/var/lib/kubelet/pods/99920224-1d67-4af4-90a0-ebd548e4f618/volumes" Oct 30 12:23:49.029681 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:49.029645 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20f2699e-ljxv4"] Oct 30 12:23:49.033839 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:49.033801 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20f2699e-ljxv4"] Oct 30 12:23:49.609302 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:49.609262 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1122f4cb-4c3f-41e6-b9fb-c2079a9e208c" path="/var/lib/kubelet/pods/1122f4cb-4c3f-41e6-b9fb-c2079a9e208c/volumes" Oct 30 12:23:49.684926 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:23:49.684889 4908 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vn6vv" containerName="extractor" Oct 30 12:23:50.982720 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:50.982689 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9wwfc"] Oct 30 12:23:54.028871 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:54.028840 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de045378-pcjbr"] Oct 30 12:23:54.039671 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:54.039643 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de045378-pcjbr"] Oct 30 12:23:55.607709 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:55.607675 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3054913-cac1-435a-ac9a-1b06e9434c35" path="/var/lib/kubelet/pods/b3054913-cac1-435a-ac9a-1b06e9434c35/volumes" Oct 30 12:23:57.031292 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:57.031257 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9b22a7d-dtk4r"] Oct 30 12:23:57.043156 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:57.043125 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9b22a7d-dtk4r"] Oct 30 12:23:57.163311 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:57.163275 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ebc6bb-ngzj5"] Oct 30 12:23:57.167618 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:57.167591 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ebc6bb-ngzj5"] Oct 30 12:23:57.608505 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:57.608469 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d5b74c2-a2b1-4e10-a9e2-7690fd383724" path="/var/lib/kubelet/pods/5d5b74c2-a2b1-4e10-a9e2-7690fd383724/volumes" Oct 30 12:23:57.609503 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:23:57.609462 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1afad88-b907-4719-ab94-8588ae5e24b2" path="/var/lib/kubelet/pods/c1afad88-b907-4719-ab94-8588ae5e24b2/volumes" Oct 30 12:24:04.028943 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:04.028908 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21b8dc27-bptpx"] Oct 30 12:24:04.039787 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:04.039759 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21b8dc27-bptpx"] Oct 30 12:24:04.280442 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:04.280357 4908 scope.go:122] "RemoveContainer" containerID="eb58bacec1879baa5a58f1d00649db8641536014bbbc9cd317dd1e536be5921a" Oct 30 12:24:04.292482 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:04.292458 4908 scope.go:122] "RemoveContainer" containerID="b761ddeb3c274aa262e1ca9f2cb0e56b0c23c933494d282c6c244d81a33a6319" Oct 30 12:24:04.541344 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:04.541228 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-ebc6bb-openshift-default-84d66c85b7-ssbvj"] Oct 30 12:24:04.541872 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:04.541783 4908 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-ebc6bb-openshift-default-84d66c85b7-ssbvj" podUID="cd7c01ae-47be-43dd-93d3-c4702d08ef42" containerName="istio-proxy" containerID="cri-o://fd5af407cd770b40e16d7568d6c178d5626a7b9ee02663acd3acd3121470de38" gracePeriod=30 Oct 30 12:24:05.046677 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:05.046648 4908 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-plczt_a70ee11c-c20e-4a4f-8476-ac5eb02816d0/console-operator/3.log" Oct 30 12:24:05.052671 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:05.052648 4908 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b8bxz_644e19b3-10ad-4145-9ed3-595d19ed9b07/ovn-acl-logging/0.log" Oct 30 12:24:05.607157 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:05.607126 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3e63d4d-0257-41a0-9b25-24a4c7514b95" path="/var/lib/kubelet/pods/d3e63d4d-0257-41a0-9b25-24a4c7514b95/volumes" Oct 30 12:24:05.802461 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:05.802426 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-060f41f2-openshift-default-5f86797c8f-7vcrc"] Oct 30 12:24:05.802952 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:05.802890 4908 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-060f41f2-openshift-default-5f86797c8f-7vcrc" podUID="ca257393-2ed9-42fa-a81a-ce6001a4ac3b" containerName="istio-proxy" containerID="cri-o://76ef52ef0419d17a6526ba1b2fdf3da59239cb587f03613f3a4a46e2115d3c1d" gracePeriod=30 Oct 30 12:24:06.229445 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:06.229410 4908 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-plczt_a70ee11c-c20e-4a4f-8476-ac5eb02816d0/console-operator/3.log" Oct 30 12:24:06.234721 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:06.234699 4908 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b8bxz_644e19b3-10ad-4145-9ed3-595d19ed9b07/ovn-acl-logging/0.log" Oct 30 12:24:09.759455 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:09.759413 4908 generic.go:356] "Generic (PLEG): container finished" podID="cd7c01ae-47be-43dd-93d3-c4702d08ef42" containerID="fd5af407cd770b40e16d7568d6c178d5626a7b9ee02663acd3acd3121470de38" exitCode=0 Oct 30 12:24:09.760338 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:09.759496 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ebc6bb-openshift-default-84d66c85b7-ssbvj" event={"ID":"cd7c01ae-47be-43dd-93d3-c4702d08ef42","Type":"ContainerDied","Data":"fd5af407cd770b40e16d7568d6c178d5626a7b9ee02663acd3acd3121470de38"} Oct 30 12:24:09.760338 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:09.759551 4908 scope.go:122] "RemoveContainer" containerID="ff15f396b5ac70fd1d2a8dea2c2f98be7f0a9d96dd278f371bc1db4f595ccbbd" Oct 30 12:24:10.479569 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:10.479539 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-ebc6bb-openshift-default-84d66c85b7-ssbvj" Oct 30 12:24:10.610817 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:10.610722 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cd7c01ae-47be-43dd-93d3-c4702d08ef42-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd7c01ae-47be-43dd-93d3-c4702d08ef42-istio-podinfo\") pod \"cd7c01ae-47be-43dd-93d3-c4702d08ef42\" (UID: \"cd7c01ae-47be-43dd-93d3-c4702d08ef42\") " Oct 30 12:24:10.611487 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:10.611102 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd7c01ae-47be-43dd-93d3-c4702d08ef42-istio-token\" (UniqueName: \"kubernetes.io/projected/cd7c01ae-47be-43dd-93d3-c4702d08ef42-istio-token\") pod \"cd7c01ae-47be-43dd-93d3-c4702d08ef42\" (UID: \"cd7c01ae-47be-43dd-93d3-c4702d08ef42\") " Oct 30 12:24:10.611607 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:10.611558 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd7c01ae-47be-43dd-93d3-c4702d08ef42-kube-api-access-rmz8s\" (UniqueName: \"kubernetes.io/projected/cd7c01ae-47be-43dd-93d3-c4702d08ef42-kube-api-access-rmz8s\") pod \"cd7c01ae-47be-43dd-93d3-c4702d08ef42\" (UID: \"cd7c01ae-47be-43dd-93d3-c4702d08ef42\") " Oct 30 12:24:10.611668 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:10.611635 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd7c01ae-47be-43dd-93d3-c4702d08ef42-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd7c01ae-47be-43dd-93d3-c4702d08ef42-istiod-ca-cert\") pod \"cd7c01ae-47be-43dd-93d3-c4702d08ef42\" (UID: \"cd7c01ae-47be-43dd-93d3-c4702d08ef42\") " Oct 30 12:24:10.611723 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:10.611670 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd7c01ae-47be-43dd-93d3-c4702d08ef42-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd7c01ae-47be-43dd-93d3-c4702d08ef42-workload-socket\") pod \"cd7c01ae-47be-43dd-93d3-c4702d08ef42\" (UID: \"cd7c01ae-47be-43dd-93d3-c4702d08ef42\") " Oct 30 12:24:10.611723 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:10.611713 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd7c01ae-47be-43dd-93d3-c4702d08ef42-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd7c01ae-47be-43dd-93d3-c4702d08ef42-istio-envoy\") pod \"cd7c01ae-47be-43dd-93d3-c4702d08ef42\" (UID: \"cd7c01ae-47be-43dd-93d3-c4702d08ef42\") " Oct 30 12:24:10.611868 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:10.611807 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd7c01ae-47be-43dd-93d3-c4702d08ef42-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd7c01ae-47be-43dd-93d3-c4702d08ef42-istio-data\") pod \"cd7c01ae-47be-43dd-93d3-c4702d08ef42\" (UID: \"cd7c01ae-47be-43dd-93d3-c4702d08ef42\") " Oct 30 12:24:10.611933 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:10.611862 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd7c01ae-47be-43dd-93d3-c4702d08ef42-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd7c01ae-47be-43dd-93d3-c4702d08ef42-workload-certs\") pod \"cd7c01ae-47be-43dd-93d3-c4702d08ef42\" (UID: \"cd7c01ae-47be-43dd-93d3-c4702d08ef42\") " Oct 30 12:24:10.611984 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:10.611956 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd7c01ae-47be-43dd-93d3-c4702d08ef42-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd7c01ae-47be-43dd-93d3-c4702d08ef42-credential-socket\") pod \"cd7c01ae-47be-43dd-93d3-c4702d08ef42\" (UID: \"cd7c01ae-47be-43dd-93d3-c4702d08ef42\") " Oct 30 12:24:10.612890 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:10.612862 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd7c01ae-47be-43dd-93d3-c4702d08ef42-credential-socket" pod "cd7c01ae-47be-43dd-93d3-c4702d08ef42" (UID: "cd7c01ae-47be-43dd-93d3-c4702d08ef42"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:24:10.613197 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:10.613178 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd7c01ae-47be-43dd-93d3-c4702d08ef42-workload-socket" pod "cd7c01ae-47be-43dd-93d3-c4702d08ef42" (UID: "cd7c01ae-47be-43dd-93d3-c4702d08ef42"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:24:10.614100 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:10.614076 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd7c01ae-47be-43dd-93d3-c4702d08ef42-istiod-ca-cert" pod "cd7c01ae-47be-43dd-93d3-c4702d08ef42" (UID: "cd7c01ae-47be-43dd-93d3-c4702d08ef42"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:24:10.614504 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:10.614484 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd7c01ae-47be-43dd-93d3-c4702d08ef42-istio-data" pod "cd7c01ae-47be-43dd-93d3-c4702d08ef42" (UID: "cd7c01ae-47be-43dd-93d3-c4702d08ef42"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:24:10.615889 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:10.615861 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd7c01ae-47be-43dd-93d3-c4702d08ef42-workload-certs" pod "cd7c01ae-47be-43dd-93d3-c4702d08ef42" (UID: "cd7c01ae-47be-43dd-93d3-c4702d08ef42"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:24:10.616001 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:10.615970 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd7c01ae-47be-43dd-93d3-c4702d08ef42-kube-api-access-rmz8s" pod "cd7c01ae-47be-43dd-93d3-c4702d08ef42" (UID: "cd7c01ae-47be-43dd-93d3-c4702d08ef42"). InnerVolumeSpecName "kube-api-access-rmz8s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:24:10.618474 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:10.618440 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd7c01ae-47be-43dd-93d3-c4702d08ef42-istio-envoy" pod "cd7c01ae-47be-43dd-93d3-c4702d08ef42" (UID: "cd7c01ae-47be-43dd-93d3-c4702d08ef42"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:24:10.619792 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:10.619768 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cd7c01ae-47be-43dd-93d3-c4702d08ef42-istio-podinfo" pod "cd7c01ae-47be-43dd-93d3-c4702d08ef42" (UID: "cd7c01ae-47be-43dd-93d3-c4702d08ef42"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 12:24:10.619998 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:10.619973 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd7c01ae-47be-43dd-93d3-c4702d08ef42-istio-token" pod "cd7c01ae-47be-43dd-93d3-c4702d08ef42" (UID: "cd7c01ae-47be-43dd-93d3-c4702d08ef42"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:24:10.648684 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:10.648658 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c35d6b-mw9n9"] Oct 30 12:24:10.653010 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:10.652989 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c35d6b-mw9n9"] Oct 30 12:24:10.666420 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:10.666396 4908 patch_prober.go:29] interesting pod/e2e-mt-060f41f2-openshift-default-5f86797c8f-7vcrc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 12:24:10.666588 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:10.666564 4908 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-060f41f2-openshift-default-5f86797c8f-7vcrc" podUID="ca257393-2ed9-42fa-a81a-ce6001a4ac3b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 12:24:10.717761 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:10.717720 4908 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd7c01ae-47be-43dd-93d3-c4702d08ef42-istio-data\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:24:10.717761 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:10.717760 4908 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd7c01ae-47be-43dd-93d3-c4702d08ef42-workload-certs\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:24:10.718076 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:10.717771 4908 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd7c01ae-47be-43dd-93d3-c4702d08ef42-credential-socket\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:24:10.718076 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:10.717784 4908 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd7c01ae-47be-43dd-93d3-c4702d08ef42-istio-podinfo\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:24:10.718076 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:10.717796 4908 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd7c01ae-47be-43dd-93d3-c4702d08ef42-istio-token\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:24:10.718076 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:10.717807 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rmz8s\" (UniqueName: \"kubernetes.io/projected/cd7c01ae-47be-43dd-93d3-c4702d08ef42-kube-api-access-rmz8s\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:24:10.718076 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:10.717818 4908 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd7c01ae-47be-43dd-93d3-c4702d08ef42-istiod-ca-cert\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:24:10.718076 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:10.717845 4908 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd7c01ae-47be-43dd-93d3-c4702d08ef42-workload-socket\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:24:10.718076 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:10.717857 4908 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd7c01ae-47be-43dd-93d3-c4702d08ef42-istio-envoy\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:24:10.775054 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:10.775027 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-ebc6bb-openshift-default-84d66c85b7-ssbvj" Oct 30 12:24:10.775669 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:10.775060 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ebc6bb-openshift-default-84d66c85b7-ssbvj" event={"ID":"cd7c01ae-47be-43dd-93d3-c4702d08ef42","Type":"ContainerDied","Data":"4f20024605c3e48a934abd83019fa003c078d470786520cfbcc64b5adbcd8481"} Oct 30 12:24:10.775669 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:10.775132 4908 scope.go:122] "RemoveContainer" containerID="fd5af407cd770b40e16d7568d6c178d5626a7b9ee02663acd3acd3121470de38" Oct 30 12:24:10.827299 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:10.827269 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-ebc6bb-openshift-default-84d66c85b7-ssbvj"] Oct 30 12:24:10.831016 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:10.830990 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-ebc6bb-openshift-default-84d66c85b7-ssbvj"] Oct 30 12:24:11.030425 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:11.030398 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-799d024f-6gftm"] Oct 30 12:24:11.034941 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:11.034908 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-799d024f-6gftm"] Oct 30 12:24:11.144848 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:11.133210 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-060f41f2-openshift-default-5f86797c8f-7vcrc" Oct 30 12:24:11.224613 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:11.224544 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ca257393-2ed9-42fa-a81a-ce6001a4ac3b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca257393-2ed9-42fa-a81a-ce6001a4ac3b-istio-podinfo\") pod \"ca257393-2ed9-42fa-a81a-ce6001a4ac3b\" (UID: \"ca257393-2ed9-42fa-a81a-ce6001a4ac3b\") " Oct 30 12:24:11.224613 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:11.224606 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca257393-2ed9-42fa-a81a-ce6001a4ac3b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca257393-2ed9-42fa-a81a-ce6001a4ac3b-workload-socket\") pod \"ca257393-2ed9-42fa-a81a-ce6001a4ac3b\" (UID: \"ca257393-2ed9-42fa-a81a-ce6001a4ac3b\") " Oct 30 12:24:11.224844 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:11.224631 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca257393-2ed9-42fa-a81a-ce6001a4ac3b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca257393-2ed9-42fa-a81a-ce6001a4ac3b-credential-socket\") pod \"ca257393-2ed9-42fa-a81a-ce6001a4ac3b\" (UID: \"ca257393-2ed9-42fa-a81a-ce6001a4ac3b\") " Oct 30 12:24:11.224844 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:11.224665 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca257393-2ed9-42fa-a81a-ce6001a4ac3b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca257393-2ed9-42fa-a81a-ce6001a4ac3b-workload-certs\") pod \"ca257393-2ed9-42fa-a81a-ce6001a4ac3b\" (UID: \"ca257393-2ed9-42fa-a81a-ce6001a4ac3b\") " Oct 30 12:24:11.224844 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:11.224704 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca257393-2ed9-42fa-a81a-ce6001a4ac3b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca257393-2ed9-42fa-a81a-ce6001a4ac3b-istiod-ca-cert\") pod \"ca257393-2ed9-42fa-a81a-ce6001a4ac3b\" (UID: \"ca257393-2ed9-42fa-a81a-ce6001a4ac3b\") " Oct 30 12:24:11.224844 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:11.224733 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca257393-2ed9-42fa-a81a-ce6001a4ac3b-kube-api-access-5xphm\" (UniqueName: \"kubernetes.io/projected/ca257393-2ed9-42fa-a81a-ce6001a4ac3b-kube-api-access-5xphm\") pod \"ca257393-2ed9-42fa-a81a-ce6001a4ac3b\" (UID: \"ca257393-2ed9-42fa-a81a-ce6001a4ac3b\") " Oct 30 12:24:11.224844 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:11.224780 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca257393-2ed9-42fa-a81a-ce6001a4ac3b-istio-token\" (UniqueName: \"kubernetes.io/projected/ca257393-2ed9-42fa-a81a-ce6001a4ac3b-istio-token\") pod \"ca257393-2ed9-42fa-a81a-ce6001a4ac3b\" (UID: \"ca257393-2ed9-42fa-a81a-ce6001a4ac3b\") " Oct 30 12:24:11.225117 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:11.224924 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca257393-2ed9-42fa-a81a-ce6001a4ac3b-workload-socket" pod "ca257393-2ed9-42fa-a81a-ce6001a4ac3b" (UID: "ca257393-2ed9-42fa-a81a-ce6001a4ac3b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:24:11.225117 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:11.224930 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca257393-2ed9-42fa-a81a-ce6001a4ac3b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca257393-2ed9-42fa-a81a-ce6001a4ac3b-istio-data\") pod \"ca257393-2ed9-42fa-a81a-ce6001a4ac3b\" (UID: \"ca257393-2ed9-42fa-a81a-ce6001a4ac3b\") " Oct 30 12:24:11.225117 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:11.225027 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca257393-2ed9-42fa-a81a-ce6001a4ac3b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca257393-2ed9-42fa-a81a-ce6001a4ac3b-istio-envoy\") pod \"ca257393-2ed9-42fa-a81a-ce6001a4ac3b\" (UID: \"ca257393-2ed9-42fa-a81a-ce6001a4ac3b\") " Oct 30 12:24:11.225117 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:11.225045 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca257393-2ed9-42fa-a81a-ce6001a4ac3b-workload-certs" pod "ca257393-2ed9-42fa-a81a-ce6001a4ac3b" (UID: "ca257393-2ed9-42fa-a81a-ce6001a4ac3b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:24:11.225325 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:11.225164 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca257393-2ed9-42fa-a81a-ce6001a4ac3b-credential-socket" pod "ca257393-2ed9-42fa-a81a-ce6001a4ac3b" (UID: "ca257393-2ed9-42fa-a81a-ce6001a4ac3b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:24:11.225325 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:11.225180 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca257393-2ed9-42fa-a81a-ce6001a4ac3b-istio-data" pod "ca257393-2ed9-42fa-a81a-ce6001a4ac3b" (UID: "ca257393-2ed9-42fa-a81a-ce6001a4ac3b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:24:11.225325 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:11.225263 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca257393-2ed9-42fa-a81a-ce6001a4ac3b-istiod-ca-cert" pod "ca257393-2ed9-42fa-a81a-ce6001a4ac3b" (UID: "ca257393-2ed9-42fa-a81a-ce6001a4ac3b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:24:11.225670 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:11.225648 4908 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca257393-2ed9-42fa-a81a-ce6001a4ac3b-workload-socket\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:24:11.225746 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:11.225675 4908 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca257393-2ed9-42fa-a81a-ce6001a4ac3b-credential-socket\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:24:11.225746 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:11.225692 4908 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca257393-2ed9-42fa-a81a-ce6001a4ac3b-workload-certs\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:24:11.225746 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:11.225705 4908 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca257393-2ed9-42fa-a81a-ce6001a4ac3b-istiod-ca-cert\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:24:11.225746 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:11.225718 4908 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca257393-2ed9-42fa-a81a-ce6001a4ac3b-istio-data\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:24:11.227137 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:11.227111 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca257393-2ed9-42fa-a81a-ce6001a4ac3b-istio-envoy" pod "ca257393-2ed9-42fa-a81a-ce6001a4ac3b" (UID: "ca257393-2ed9-42fa-a81a-ce6001a4ac3b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:24:11.227137 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:11.227128 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca257393-2ed9-42fa-a81a-ce6001a4ac3b-istio-token" pod "ca257393-2ed9-42fa-a81a-ce6001a4ac3b" (UID: "ca257393-2ed9-42fa-a81a-ce6001a4ac3b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:24:11.227432 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:11.227408 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ca257393-2ed9-42fa-a81a-ce6001a4ac3b-istio-podinfo" pod "ca257393-2ed9-42fa-a81a-ce6001a4ac3b" (UID: "ca257393-2ed9-42fa-a81a-ce6001a4ac3b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 12:24:11.227432 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:11.227419 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca257393-2ed9-42fa-a81a-ce6001a4ac3b-kube-api-access-5xphm" pod "ca257393-2ed9-42fa-a81a-ce6001a4ac3b" (UID: "ca257393-2ed9-42fa-a81a-ce6001a4ac3b"). InnerVolumeSpecName "kube-api-access-5xphm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:24:11.326543 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:11.326510 4908 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca257393-2ed9-42fa-a81a-ce6001a4ac3b-istio-podinfo\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:24:11.326543 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:11.326539 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xphm\" (UniqueName: \"kubernetes.io/projected/ca257393-2ed9-42fa-a81a-ce6001a4ac3b-kube-api-access-5xphm\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:24:11.326702 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:11.326553 4908 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca257393-2ed9-42fa-a81a-ce6001a4ac3b-istio-token\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:24:11.326702 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:11.326564 4908 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca257393-2ed9-42fa-a81a-ce6001a4ac3b-istio-envoy\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:24:11.608180 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:11.608147 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6276351f-7c08-4c6d-b841-1d0e259c3fb8" path="/var/lib/kubelet/pods/6276351f-7c08-4c6d-b841-1d0e259c3fb8/volumes" Oct 30 12:24:11.608642 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:11.608627 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd7c01ae-47be-43dd-93d3-c4702d08ef42" path="/var/lib/kubelet/pods/cd7c01ae-47be-43dd-93d3-c4702d08ef42/volumes" Oct 30 12:24:11.609061 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:11.609046 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d570f102-c33d-4383-b67e-faccb60fea14" path="/var/lib/kubelet/pods/d570f102-c33d-4383-b67e-faccb60fea14/volumes" Oct 30 12:24:11.784085 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:11.784049 4908 generic.go:356] "Generic (PLEG): container finished" podID="ca257393-2ed9-42fa-a81a-ce6001a4ac3b" containerID="76ef52ef0419d17a6526ba1b2fdf3da59239cb587f03613f3a4a46e2115d3c1d" exitCode=0 Oct 30 12:24:11.784501 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:11.784094 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-060f41f2-openshift-default-5f86797c8f-7vcrc" Oct 30 12:24:11.784501 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:11.784140 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-060f41f2-openshift-default-5f86797c8f-7vcrc" event={"ID":"ca257393-2ed9-42fa-a81a-ce6001a4ac3b","Type":"ContainerDied","Data":"76ef52ef0419d17a6526ba1b2fdf3da59239cb587f03613f3a4a46e2115d3c1d"} Oct 30 12:24:11.784501 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:11.784204 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-060f41f2-openshift-default-5f86797c8f-7vcrc" event={"ID":"ca257393-2ed9-42fa-a81a-ce6001a4ac3b","Type":"ContainerDied","Data":"9e03c1db8f721feb91500749ed6510ee44463fc39ee75db34a7b374c085b7a23"} Oct 30 12:24:11.784501 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:11.784238 4908 scope.go:122] "RemoveContainer" containerID="76ef52ef0419d17a6526ba1b2fdf3da59239cb587f03613f3a4a46e2115d3c1d" Oct 30 12:24:11.796611 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:11.796590 4908 scope.go:122] "RemoveContainer" containerID="76ef52ef0419d17a6526ba1b2fdf3da59239cb587f03613f3a4a46e2115d3c1d" Oct 30 12:24:11.796963 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:24:11.796934 4908 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76ef52ef0419d17a6526ba1b2fdf3da59239cb587f03613f3a4a46e2115d3c1d\": container with ID starting with 76ef52ef0419d17a6526ba1b2fdf3da59239cb587f03613f3a4a46e2115d3c1d not found: ID does not exist" containerID="76ef52ef0419d17a6526ba1b2fdf3da59239cb587f03613f3a4a46e2115d3c1d" Oct 30 12:24:11.797092 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:11.796973 4908 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76ef52ef0419d17a6526ba1b2fdf3da59239cb587f03613f3a4a46e2115d3c1d"} err="failed to get container status \"76ef52ef0419d17a6526ba1b2fdf3da59239cb587f03613f3a4a46e2115d3c1d\": rpc error: code = NotFound desc = could not find container \"76ef52ef0419d17a6526ba1b2fdf3da59239cb587f03613f3a4a46e2115d3c1d\": container with ID starting with 76ef52ef0419d17a6526ba1b2fdf3da59239cb587f03613f3a4a46e2115d3c1d not found: ID does not exist" Oct 30 12:24:11.834911 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:11.834886 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-060f41f2-openshift-default-5f86797c8f-7vcrc"] Oct 30 12:24:11.845416 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:11.845396 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-060f41f2-openshift-default-5f86797c8f-7vcrc"] Oct 30 12:24:13.608424 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:13.608390 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca257393-2ed9-42fa-a81a-ce6001a4ac3b" path="/var/lib/kubelet/pods/ca257393-2ed9-42fa-a81a-ce6001a4ac3b/volumes" Oct 30 12:24:15.030741 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:15.030702 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e57de56-tns89"] Oct 30 12:24:15.043066 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:15.043031 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e57de56-tns89"] Oct 30 12:24:15.606794 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:15.606763 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21d19651-5c90-47ba-acf5-5004aa8e11f5" path="/var/lib/kubelet/pods/21d19651-5c90-47ba-acf5-5004aa8e11f5/volumes" Oct 30 12:24:18.377943 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:18.377902 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8f2898-j8lbz"] Oct 30 12:24:18.378725 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:18.378707 4908 cpu_manager.go:414] "containerMap: removing container" podUID="d570f102-c33d-4383-b67e-faccb60fea14" containerName="revoke-keys" Oct 30 12:24:18.378773 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:18.378728 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d570f102-c33d-4383-b67e-faccb60fea14" containerName="revoke-keys" Oct 30 12:24:18.378773 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:18.378770 4908 cpu_manager.go:414] "containerMap: removing container" podUID="ca257393-2ed9-42fa-a81a-ce6001a4ac3b" containerName="istio-proxy" Oct 30 12:24:18.378862 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:18.378778 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca257393-2ed9-42fa-a81a-ce6001a4ac3b" containerName="istio-proxy" Oct 30 12:24:18.378862 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:18.378815 4908 cpu_manager.go:414] "containerMap: removing container" podUID="cd7c01ae-47be-43dd-93d3-c4702d08ef42" containerName="istio-proxy" Oct 30 12:24:18.378862 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:18.378823 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd7c01ae-47be-43dd-93d3-c4702d08ef42" containerName="istio-proxy" Oct 30 12:24:18.378947 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:18.378862 4908 cpu_manager.go:414] "containerMap: removing container" podUID="cd7c01ae-47be-43dd-93d3-c4702d08ef42" containerName="istio-proxy" Oct 30 12:24:18.378947 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:18.378869 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd7c01ae-47be-43dd-93d3-c4702d08ef42" containerName="istio-proxy" Oct 30 12:24:18.379025 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:18.379013 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="d570f102-c33d-4383-b67e-faccb60fea14" containerName="revoke-keys" Oct 30 12:24:18.379060 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:18.379032 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd7c01ae-47be-43dd-93d3-c4702d08ef42" containerName="istio-proxy" Oct 30 12:24:18.379060 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:18.379043 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca257393-2ed9-42fa-a81a-ce6001a4ac3b" containerName="istio-proxy" Oct 30 12:24:18.379060 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:18.379053 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd7c01ae-47be-43dd-93d3-c4702d08ef42" containerName="istio-proxy" Oct 30 12:24:18.384270 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:18.384245 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8f2898-j8lbz" Oct 30 12:24:18.399952 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:18.399923 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8f2898-j8lbz"] Oct 30 12:24:18.404571 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:18.404545 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 12:24:18.407529 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:18.405192 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 12:24:18.445802 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:18.445768 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 12:24:18.445993 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:18.445871 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s75hq\"" Oct 30 12:24:18.538849 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:18.538805 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed5ccf16-364f-49c7-b4da-3d82850f94ce-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-8f2898-j8lbz\" (UID: \"ed5ccf16-364f-49c7-b4da-3d82850f94ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8f2898-j8lbz" Oct 30 12:24:18.539022 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:18.538874 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ds4xd\" (UniqueName: \"kubernetes.io/projected/ed5ccf16-364f-49c7-b4da-3d82850f94ce-kube-api-access-ds4xd\") pod \"maas-api-revoke-keys-e2e-api-b-8f2898-j8lbz\" (UID: \"ed5ccf16-364f-49c7-b4da-3d82850f94ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8f2898-j8lbz" Oct 30 12:24:18.539022 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:18.538995 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed5ccf16-364f-49c7-b4da-3d82850f94ce-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-8f2898-j8lbz\" (UID: \"ed5ccf16-364f-49c7-b4da-3d82850f94ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8f2898-j8lbz" Oct 30 12:24:18.640393 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:18.640314 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed5ccf16-364f-49c7-b4da-3d82850f94ce-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-8f2898-j8lbz\" (UID: \"ed5ccf16-364f-49c7-b4da-3d82850f94ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8f2898-j8lbz" Oct 30 12:24:18.640393 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:18.640357 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ds4xd\" (UniqueName: \"kubernetes.io/projected/ed5ccf16-364f-49c7-b4da-3d82850f94ce-kube-api-access-ds4xd\") pod \"maas-api-revoke-keys-e2e-api-b-8f2898-j8lbz\" (UID: \"ed5ccf16-364f-49c7-b4da-3d82850f94ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8f2898-j8lbz" Oct 30 12:24:18.640585 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:18.640405 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed5ccf16-364f-49c7-b4da-3d82850f94ce-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-8f2898-j8lbz\" (UID: \"ed5ccf16-364f-49c7-b4da-3d82850f94ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8f2898-j8lbz" Oct 30 12:24:18.641096 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:18.641074 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed5ccf16-364f-49c7-b4da-3d82850f94ce-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-8f2898-j8lbz\" (UID: \"ed5ccf16-364f-49c7-b4da-3d82850f94ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8f2898-j8lbz" Oct 30 12:24:18.642640 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:18.642619 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed5ccf16-364f-49c7-b4da-3d82850f94ce-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-8f2898-j8lbz\" (UID: \"ed5ccf16-364f-49c7-b4da-3d82850f94ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8f2898-j8lbz" Oct 30 12:24:18.667287 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:18.667262 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ds4xd\" (UniqueName: \"kubernetes.io/projected/ed5ccf16-364f-49c7-b4da-3d82850f94ce-kube-api-access-ds4xd\") pod \"maas-api-revoke-keys-e2e-api-b-8f2898-j8lbz\" (UID: \"ed5ccf16-364f-49c7-b4da-3d82850f94ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8f2898-j8lbz" Oct 30 12:24:18.704961 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:18.704940 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8f2898-j8lbz" Oct 30 12:24:19.014287 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:19.014261 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8f2898-j8lbz"] Oct 30 12:24:19.016720 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:24:19.016695 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded5ccf16_364f_49c7_b4da_3d82850f94ce.slice/crio-806ea2d75b7667f42e2b2de9d33f1e247bb4eea93026fd17872c938af56b40a0 WatchSource:0}: Error finding container 806ea2d75b7667f42e2b2de9d33f1e247bb4eea93026fd17872c938af56b40a0: Status 404 returned error can't find the container with id 806ea2d75b7667f42e2b2de9d33f1e247bb4eea93026fd17872c938af56b40a0 Oct 30 12:24:19.685336 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:24:19.685247 4908 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vn6vv" containerName="extractor" Oct 30 12:24:19.868095 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:19.868061 4908 generic.go:356] "Generic (PLEG): container finished" podID="ed5ccf16-364f-49c7-b4da-3d82850f94ce" containerID="43088f232fe3bfc5ec3ae83f893d8ef64ccb12cc2e6689ae4466e7b9cf67d996" exitCode=0 Oct 30 12:24:19.868251 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:19.868146 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8f2898-j8lbz" event={"ID":"ed5ccf16-364f-49c7-b4da-3d82850f94ce","Type":"ContainerDied","Data":"43088f232fe3bfc5ec3ae83f893d8ef64ccb12cc2e6689ae4466e7b9cf67d996"} Oct 30 12:24:19.868251 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:19.868209 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8f2898-j8lbz" event={"ID":"ed5ccf16-364f-49c7-b4da-3d82850f94ce","Type":"ContainerStarted","Data":"806ea2d75b7667f42e2b2de9d33f1e247bb4eea93026fd17872c938af56b40a0"} Oct 30 12:24:21.045687 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:21.045663 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8f2898-j8lbz" Oct 30 12:24:21.172695 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:21.172616 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed5ccf16-364f-49c7-b4da-3d82850f94ce-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed5ccf16-364f-49c7-b4da-3d82850f94ce-maas-api-openshift-ca\") pod \"ed5ccf16-364f-49c7-b4da-3d82850f94ce\" (UID: \"ed5ccf16-364f-49c7-b4da-3d82850f94ce\") " Oct 30 12:24:21.172882 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:21.172707 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed5ccf16-364f-49c7-b4da-3d82850f94ce-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed5ccf16-364f-49c7-b4da-3d82850f94ce-maas-api-xks-ca\") pod \"ed5ccf16-364f-49c7-b4da-3d82850f94ce\" (UID: \"ed5ccf16-364f-49c7-b4da-3d82850f94ce\") " Oct 30 12:24:21.172882 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:21.172756 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed5ccf16-364f-49c7-b4da-3d82850f94ce-kube-api-access-ds4xd\" (UniqueName: \"kubernetes.io/projected/ed5ccf16-364f-49c7-b4da-3d82850f94ce-kube-api-access-ds4xd\") pod \"ed5ccf16-364f-49c7-b4da-3d82850f94ce\" (UID: \"ed5ccf16-364f-49c7-b4da-3d82850f94ce\") " Oct 30 12:24:21.173066 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:21.173044 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed5ccf16-364f-49c7-b4da-3d82850f94ce-maas-api-openshift-ca" pod "ed5ccf16-364f-49c7-b4da-3d82850f94ce" (UID: "ed5ccf16-364f-49c7-b4da-3d82850f94ce"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:24:21.173304 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:21.173286 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed5ccf16-364f-49c7-b4da-3d82850f94ce-maas-api-openshift-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:24:21.175328 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:21.175300 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed5ccf16-364f-49c7-b4da-3d82850f94ce-maas-api-xks-ca" pod "ed5ccf16-364f-49c7-b4da-3d82850f94ce" (UID: "ed5ccf16-364f-49c7-b4da-3d82850f94ce"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:24:21.175436 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:21.175361 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed5ccf16-364f-49c7-b4da-3d82850f94ce-kube-api-access-ds4xd" pod "ed5ccf16-364f-49c7-b4da-3d82850f94ce" (UID: "ed5ccf16-364f-49c7-b4da-3d82850f94ce"). InnerVolumeSpecName "kube-api-access-ds4xd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:24:21.274904 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:21.274876 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed5ccf16-364f-49c7-b4da-3d82850f94ce-maas-api-xks-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:24:21.274904 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:21.274904 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ds4xd\" (UniqueName: \"kubernetes.io/projected/ed5ccf16-364f-49c7-b4da-3d82850f94ce-kube-api-access-ds4xd\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:24:21.888303 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:21.888085 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8f2898-j8lbz" Oct 30 12:24:21.888303 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:21.888114 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8f2898-j8lbz" event={"ID":"ed5ccf16-364f-49c7-b4da-3d82850f94ce","Type":"ContainerDied","Data":"806ea2d75b7667f42e2b2de9d33f1e247bb4eea93026fd17872c938af56b40a0"} Oct 30 12:24:21.888303 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:21.888165 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="806ea2d75b7667f42e2b2de9d33f1e247bb4eea93026fd17872c938af56b40a0" Oct 30 12:24:23.272084 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:23.272042 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9wwfc"] Oct 30 12:24:31.027854 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:31.027801 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eacb4712-5xgh4"] Oct 30 12:24:31.038480 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:31.038454 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eacb4712-5xgh4"] Oct 30 12:24:31.609220 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:31.609185 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82946bf7-7840-4bc5-b599-04b3939f1f53" path="/var/lib/kubelet/pods/82946bf7-7840-4bc5-b599-04b3939f1f53/volumes" Oct 30 12:24:34.033774 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:34.033738 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f9ae990-bmc72"] Oct 30 12:24:34.044534 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:34.044494 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f9ae990-bmc72"] Oct 30 12:24:35.609147 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:35.609113 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae8121a7-b304-4a56-8ecf-8d3bfdeee659" path="/var/lib/kubelet/pods/ae8121a7-b304-4a56-8ecf-8d3bfdeee659/volumes" Oct 30 12:24:42.641704 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:42.641665 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-aef61e-openshift-default-554c4fbd5f-t6k2v"] Oct 30 12:24:42.642421 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:42.642398 4908 cpu_manager.go:414] "containerMap: removing container" podUID="ed5ccf16-364f-49c7-b4da-3d82850f94ce" containerName="revoke-keys" Oct 30 12:24:42.642421 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:42.642422 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed5ccf16-364f-49c7-b4da-3d82850f94ce" containerName="revoke-keys" Oct 30 12:24:42.642657 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:42.642641 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed5ccf16-364f-49c7-b4da-3d82850f94ce" containerName="revoke-keys" Oct 30 12:24:42.648176 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:42.648152 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-aef61e-openshift-default-554c4fbd5f-t6k2v" Oct 30 12:24:42.665001 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:42.664972 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-aef61e-openshift-default-dockercfg-rcknf\"" Oct 30 12:24:42.686430 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:42.686404 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-aef61e-openshift-default-554c4fbd5f-t6k2v"] Oct 30 12:24:42.688399 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:42.688368 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abd92788-b8eb-4998-b22a-0ecae4a3a81d-istiod-ca-cert\") pod \"e2e-api-b-aef61e-openshift-default-554c4fbd5f-t6k2v\" (UID: \"abd92788-b8eb-4998-b22a-0ecae4a3a81d\") " pod="openshift-ingress/e2e-api-b-aef61e-openshift-default-554c4fbd5f-t6k2v" Oct 30 12:24:42.688520 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:42.688414 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abd92788-b8eb-4998-b22a-0ecae4a3a81d-istio-token\") pod \"e2e-api-b-aef61e-openshift-default-554c4fbd5f-t6k2v\" (UID: \"abd92788-b8eb-4998-b22a-0ecae4a3a81d\") " pod="openshift-ingress/e2e-api-b-aef61e-openshift-default-554c4fbd5f-t6k2v" Oct 30 12:24:42.688520 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:42.688492 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfjwq\" (UniqueName: \"kubernetes.io/projected/abd92788-b8eb-4998-b22a-0ecae4a3a81d-kube-api-access-pfjwq\") pod \"e2e-api-b-aef61e-openshift-default-554c4fbd5f-t6k2v\" (UID: \"abd92788-b8eb-4998-b22a-0ecae4a3a81d\") " pod="openshift-ingress/e2e-api-b-aef61e-openshift-default-554c4fbd5f-t6k2v" Oct 30 12:24:42.688647 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:42.688575 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abd92788-b8eb-4998-b22a-0ecae4a3a81d-istio-podinfo\") pod \"e2e-api-b-aef61e-openshift-default-554c4fbd5f-t6k2v\" (UID: \"abd92788-b8eb-4998-b22a-0ecae4a3a81d\") " pod="openshift-ingress/e2e-api-b-aef61e-openshift-default-554c4fbd5f-t6k2v" Oct 30 12:24:42.688723 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:42.688708 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abd92788-b8eb-4998-b22a-0ecae4a3a81d-workload-certs\") pod \"e2e-api-b-aef61e-openshift-default-554c4fbd5f-t6k2v\" (UID: \"abd92788-b8eb-4998-b22a-0ecae4a3a81d\") " pod="openshift-ingress/e2e-api-b-aef61e-openshift-default-554c4fbd5f-t6k2v" Oct 30 12:24:42.688780 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:42.688746 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abd92788-b8eb-4998-b22a-0ecae4a3a81d-workload-socket\") pod \"e2e-api-b-aef61e-openshift-default-554c4fbd5f-t6k2v\" (UID: \"abd92788-b8eb-4998-b22a-0ecae4a3a81d\") " pod="openshift-ingress/e2e-api-b-aef61e-openshift-default-554c4fbd5f-t6k2v" Oct 30 12:24:42.688872 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:42.688783 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abd92788-b8eb-4998-b22a-0ecae4a3a81d-istio-data\") pod \"e2e-api-b-aef61e-openshift-default-554c4fbd5f-t6k2v\" (UID: \"abd92788-b8eb-4998-b22a-0ecae4a3a81d\") " pod="openshift-ingress/e2e-api-b-aef61e-openshift-default-554c4fbd5f-t6k2v" Oct 30 12:24:42.688872 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:42.688856 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abd92788-b8eb-4998-b22a-0ecae4a3a81d-credential-socket\") pod \"e2e-api-b-aef61e-openshift-default-554c4fbd5f-t6k2v\" (UID: \"abd92788-b8eb-4998-b22a-0ecae4a3a81d\") " pod="openshift-ingress/e2e-api-b-aef61e-openshift-default-554c4fbd5f-t6k2v" Oct 30 12:24:42.689025 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:42.689007 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abd92788-b8eb-4998-b22a-0ecae4a3a81d-istio-envoy\") pod \"e2e-api-b-aef61e-openshift-default-554c4fbd5f-t6k2v\" (UID: \"abd92788-b8eb-4998-b22a-0ecae4a3a81d\") " pod="openshift-ingress/e2e-api-b-aef61e-openshift-default-554c4fbd5f-t6k2v" Oct 30 12:24:42.801093 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:42.790887 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abd92788-b8eb-4998-b22a-0ecae4a3a81d-istio-envoy\") pod \"e2e-api-b-aef61e-openshift-default-554c4fbd5f-t6k2v\" (UID: \"abd92788-b8eb-4998-b22a-0ecae4a3a81d\") " pod="openshift-ingress/e2e-api-b-aef61e-openshift-default-554c4fbd5f-t6k2v" Oct 30 12:24:42.801854 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:42.801768 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abd92788-b8eb-4998-b22a-0ecae4a3a81d-istiod-ca-cert\") pod \"e2e-api-b-aef61e-openshift-default-554c4fbd5f-t6k2v\" (UID: \"abd92788-b8eb-4998-b22a-0ecae4a3a81d\") " pod="openshift-ingress/e2e-api-b-aef61e-openshift-default-554c4fbd5f-t6k2v" Oct 30 12:24:42.802851 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:42.802813 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abd92788-b8eb-4998-b22a-0ecae4a3a81d-istio-token\") pod \"e2e-api-b-aef61e-openshift-default-554c4fbd5f-t6k2v\" (UID: \"abd92788-b8eb-4998-b22a-0ecae4a3a81d\") " pod="openshift-ingress/e2e-api-b-aef61e-openshift-default-554c4fbd5f-t6k2v" Oct 30 12:24:42.803414 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:42.803392 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfjwq\" (UniqueName: \"kubernetes.io/projected/abd92788-b8eb-4998-b22a-0ecae4a3a81d-kube-api-access-pfjwq\") pod \"e2e-api-b-aef61e-openshift-default-554c4fbd5f-t6k2v\" (UID: \"abd92788-b8eb-4998-b22a-0ecae4a3a81d\") " pod="openshift-ingress/e2e-api-b-aef61e-openshift-default-554c4fbd5f-t6k2v" Oct 30 12:24:42.803588 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:42.803569 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abd92788-b8eb-4998-b22a-0ecae4a3a81d-istio-podinfo\") pod \"e2e-api-b-aef61e-openshift-default-554c4fbd5f-t6k2v\" (UID: \"abd92788-b8eb-4998-b22a-0ecae4a3a81d\") " pod="openshift-ingress/e2e-api-b-aef61e-openshift-default-554c4fbd5f-t6k2v" Oct 30 12:24:42.803915 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:42.803895 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abd92788-b8eb-4998-b22a-0ecae4a3a81d-workload-certs\") pod \"e2e-api-b-aef61e-openshift-default-554c4fbd5f-t6k2v\" (UID: \"abd92788-b8eb-4998-b22a-0ecae4a3a81d\") " pod="openshift-ingress/e2e-api-b-aef61e-openshift-default-554c4fbd5f-t6k2v" Oct 30 12:24:42.832350 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:42.804229 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abd92788-b8eb-4998-b22a-0ecae4a3a81d-istiod-ca-cert\") pod \"e2e-api-b-aef61e-openshift-default-554c4fbd5f-t6k2v\" (UID: \"abd92788-b8eb-4998-b22a-0ecae4a3a81d\") " pod="openshift-ingress/e2e-api-b-aef61e-openshift-default-554c4fbd5f-t6k2v" Oct 30 12:24:42.832587 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:42.809418 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abd92788-b8eb-4998-b22a-0ecae4a3a81d-workload-certs\") pod \"e2e-api-b-aef61e-openshift-default-554c4fbd5f-t6k2v\" (UID: \"abd92788-b8eb-4998-b22a-0ecae4a3a81d\") " pod="openshift-ingress/e2e-api-b-aef61e-openshift-default-554c4fbd5f-t6k2v" Oct 30 12:24:42.836905 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:42.833195 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abd92788-b8eb-4998-b22a-0ecae4a3a81d-workload-socket\") pod \"e2e-api-b-aef61e-openshift-default-554c4fbd5f-t6k2v\" (UID: \"abd92788-b8eb-4998-b22a-0ecae4a3a81d\") " pod="openshift-ingress/e2e-api-b-aef61e-openshift-default-554c4fbd5f-t6k2v" Oct 30 12:24:42.837217 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:42.833490 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abd92788-b8eb-4998-b22a-0ecae4a3a81d-workload-socket\") pod \"e2e-api-b-aef61e-openshift-default-554c4fbd5f-t6k2v\" (UID: \"abd92788-b8eb-4998-b22a-0ecae4a3a81d\") " pod="openshift-ingress/e2e-api-b-aef61e-openshift-default-554c4fbd5f-t6k2v" Oct 30 12:24:42.841899 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:42.841864 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abd92788-b8eb-4998-b22a-0ecae4a3a81d-istio-data\") pod \"e2e-api-b-aef61e-openshift-default-554c4fbd5f-t6k2v\" (UID: \"abd92788-b8eb-4998-b22a-0ecae4a3a81d\") " pod="openshift-ingress/e2e-api-b-aef61e-openshift-default-554c4fbd5f-t6k2v" Oct 30 12:24:42.842134 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:42.842113 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abd92788-b8eb-4998-b22a-0ecae4a3a81d-credential-socket\") pod \"e2e-api-b-aef61e-openshift-default-554c4fbd5f-t6k2v\" (UID: \"abd92788-b8eb-4998-b22a-0ecae4a3a81d\") " pod="openshift-ingress/e2e-api-b-aef61e-openshift-default-554c4fbd5f-t6k2v" Oct 30 12:24:42.842431 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:42.842129 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abd92788-b8eb-4998-b22a-0ecae4a3a81d-istio-data\") pod \"e2e-api-b-aef61e-openshift-default-554c4fbd5f-t6k2v\" (UID: \"abd92788-b8eb-4998-b22a-0ecae4a3a81d\") " pod="openshift-ingress/e2e-api-b-aef61e-openshift-default-554c4fbd5f-t6k2v" Oct 30 12:24:42.842575 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:42.842546 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abd92788-b8eb-4998-b22a-0ecae4a3a81d-credential-socket\") pod \"e2e-api-b-aef61e-openshift-default-554c4fbd5f-t6k2v\" (UID: \"abd92788-b8eb-4998-b22a-0ecae4a3a81d\") " pod="openshift-ingress/e2e-api-b-aef61e-openshift-default-554c4fbd5f-t6k2v" Oct 30 12:24:42.850548 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:42.850497 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfjwq\" (UniqueName: \"kubernetes.io/projected/abd92788-b8eb-4998-b22a-0ecae4a3a81d-kube-api-access-pfjwq\") pod \"e2e-api-b-aef61e-openshift-default-554c4fbd5f-t6k2v\" (UID: \"abd92788-b8eb-4998-b22a-0ecae4a3a81d\") " pod="openshift-ingress/e2e-api-b-aef61e-openshift-default-554c4fbd5f-t6k2v" Oct 30 12:24:42.858871 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:42.855156 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abd92788-b8eb-4998-b22a-0ecae4a3a81d-istio-envoy\") pod \"e2e-api-b-aef61e-openshift-default-554c4fbd5f-t6k2v\" (UID: \"abd92788-b8eb-4998-b22a-0ecae4a3a81d\") " pod="openshift-ingress/e2e-api-b-aef61e-openshift-default-554c4fbd5f-t6k2v" Oct 30 12:24:42.859466 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:42.859434 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abd92788-b8eb-4998-b22a-0ecae4a3a81d-istio-token\") pod \"e2e-api-b-aef61e-openshift-default-554c4fbd5f-t6k2v\" (UID: \"abd92788-b8eb-4998-b22a-0ecae4a3a81d\") " pod="openshift-ingress/e2e-api-b-aef61e-openshift-default-554c4fbd5f-t6k2v" Oct 30 12:24:42.893897 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:42.886597 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abd92788-b8eb-4998-b22a-0ecae4a3a81d-istio-podinfo\") pod \"e2e-api-b-aef61e-openshift-default-554c4fbd5f-t6k2v\" (UID: \"abd92788-b8eb-4998-b22a-0ecae4a3a81d\") " pod="openshift-ingress/e2e-api-b-aef61e-openshift-default-554c4fbd5f-t6k2v" Oct 30 12:24:42.971851 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:42.971159 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-aef61e-openshift-default-554c4fbd5f-t6k2v" Oct 30 12:24:43.369395 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:43.369371 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-aef61e-openshift-default-554c4fbd5f-t6k2v"] Oct 30 12:24:43.372199 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:24:43.372171 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabd92788_b8eb_4998_b22a_0ecae4a3a81d.slice/crio-584f7263bb6440899dc2649bc97960e86db28a34c6555075b8d1fda7d8aee945 WatchSource:0}: Error finding container 584f7263bb6440899dc2649bc97960e86db28a34c6555075b8d1fda7d8aee945: Status 404 returned error can't find the container with id 584f7263bb6440899dc2649bc97960e86db28a34c6555075b8d1fda7d8aee945 Oct 30 12:24:43.374554 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:43.374524 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:24:43.374627 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:43.374587 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:24:43.374627 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:43.374618 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:24:44.182762 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:44.182733 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-aef61e-openshift-default-554c4fbd5f-t6k2v" event={"ID":"abd92788-b8eb-4998-b22a-0ecae4a3a81d","Type":"ContainerStarted","Data":"804a9abea774a102a224c5637f7b80700ce2809fdbb641118c49325f9bf9d925"} Oct 30 12:24:44.183151 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:44.182784 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-aef61e-openshift-default-554c4fbd5f-t6k2v" event={"ID":"abd92788-b8eb-4998-b22a-0ecae4a3a81d","Type":"ContainerStarted","Data":"584f7263bb6440899dc2649bc97960e86db28a34c6555075b8d1fda7d8aee945"} Oct 30 12:24:44.240298 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:44.240252 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-aef61e-openshift-default-554c4fbd5f-t6k2v" podStartSLOduration=2.240238947 podStartE2EDuration="2.240238947s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:24:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:24:44.225513729 +0000 UTC m=+1840.739942472" watchObservedRunningTime="2026-10-30 12:24:44.240238947 +0000 UTC m=+1840.754667691" Oct 30 12:24:44.972302 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:44.972270 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-aef61e-openshift-default-554c4fbd5f-t6k2v" Oct 30 12:24:44.977862 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:44.977813 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-aef61e-openshift-default-554c4fbd5f-t6k2v" Oct 30 12:24:45.192330 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:45.192298 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-aef61e-openshift-default-554c4fbd5f-t6k2v" Oct 30 12:24:45.193311 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:45.193287 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-aef61e-openshift-default-554c4fbd5f-t6k2v" Oct 30 12:24:49.021132 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:49.021046 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8f2898-j8lbz"] Oct 30 12:24:49.031445 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:49.031421 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8f2898-j8lbz"] Oct 30 12:24:49.607997 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:49.607961 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed5ccf16-364f-49c7-b4da-3d82850f94ce" path="/var/lib/kubelet/pods/ed5ccf16-364f-49c7-b4da-3d82850f94ce/volumes" Oct 30 12:24:49.682773 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:24:49.682737 4908 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vn6vv" containerName="extractor" Oct 30 12:24:52.883692 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:24:52.883660 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9wwfc"] Oct 30 12:25:04.331398 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:04.331367 4908 scope.go:122] "RemoveContainer" containerID="95a5994b61fc34f72c35f50c4385b0222fcb8847253b68a61d85da07f7c408eb" Oct 30 12:25:04.341978 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:04.341946 4908 scope.go:122] "RemoveContainer" containerID="7f50075bac1a3f4ace0e8eea604ad0390fa4a214083d11cba98917a12221c6fa" Oct 30 12:25:04.350920 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:04.350900 4908 scope.go:122] "RemoveContainer" containerID="2c06cc1e52f1692f36a5dab13acb200cb1eea4f00385b5e76eabd9b49b6b93e7" Oct 30 12:25:04.364974 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:04.364950 4908 scope.go:122] "RemoveContainer" containerID="916e69059bde9f089ac49330da767098b619fef80f78c4c2a2026664b18ac9d6" Oct 30 12:25:04.377203 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:04.377183 4908 scope.go:122] "RemoveContainer" containerID="868234781b4bc2ab281698b7268bb2addf2ff4f61df2054615cb08783be6850f" Oct 30 12:25:04.388256 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:04.388224 4908 scope.go:122] "RemoveContainer" containerID="0d3ae493403b988ef3f8fdcaa54318423bb1557314f08a321c1f5963ffdc0816" Oct 30 12:25:04.398342 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:04.398325 4908 scope.go:122] "RemoveContainer" containerID="d54656eee5ee5e4a34afa9ea69ecabd4b0dddc1e3c276194982eb98bda987926" Oct 30 12:25:04.407785 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:04.407759 4908 scope.go:122] "RemoveContainer" containerID="4d7d2a67f0ecf7f482747df45b6ba4027a052def3d7f387574b2f04505d348e3" Oct 30 12:25:04.416213 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:04.416198 4908 scope.go:122] "RemoveContainer" containerID="6439df20b81d3959712b21227dfdb0cfc0afb68492590eb1382a29fb774fff9d" Oct 30 12:25:04.424256 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:04.424234 4908 scope.go:122] "RemoveContainer" containerID="e2d5fa0c2c585d90d93e16fc54b74572f523df0ff806e81cda2b3ed654358617" Oct 30 12:25:04.432366 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:04.432350 4908 scope.go:122] "RemoveContainer" containerID="a997ecd54d51424760c9b9401b2efe6c09f44bd4377b72bbd5f09adadd20b9ee" Oct 30 12:25:05.207979 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:05.207948 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-hbpxv"] Oct 30 12:25:05.208665 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:05.208238 4908 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-hbpxv" podUID="e36676bf-3529-43a7-b65f-ddb63f3b9278" containerName="istio-proxy" containerID="cri-o://e27cf3a01e8bd3563f2b72a43bfb22f9db8711899c13825047573c48bcdbf8d9" gracePeriod=30 Oct 30 12:25:05.223096 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:05.218018 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-pzdcj"] Oct 30 12:25:05.223096 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:05.218242 4908 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-pzdcj" podUID="bd702b8b-1d62-41b9-8fbf-1cb4aebdaa71" containerName="istio-proxy" containerID="cri-o://1145f457ca80476d2590cbefa8c8c19e55242d855919b7bdfe576b5855a4495c" gracePeriod=30 Oct 30 12:25:05.228873 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:05.227387 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rx4tf"] Oct 30 12:25:05.228873 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:05.227613 4908 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rx4tf" podUID="76c80954-caf1-42b7-a812-fa52488f2e1b" containerName="istio-proxy" containerID="cri-o://9fc1dc78272c01ea0b0a3ea409879f085f1c4fe92405803a6c3e0879507206b2" gracePeriod=30 Oct 30 12:25:10.425114 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:10.425064 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-hbpxv" Oct 30 12:25:10.446920 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:10.445267 4908 generic.go:356] "Generic (PLEG): container finished" podID="76c80954-caf1-42b7-a812-fa52488f2e1b" containerID="9fc1dc78272c01ea0b0a3ea409879f085f1c4fe92405803a6c3e0879507206b2" exitCode=0 Oct 30 12:25:10.446920 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:10.445445 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rx4tf" event={"ID":"76c80954-caf1-42b7-a812-fa52488f2e1b","Type":"ContainerDied","Data":"9fc1dc78272c01ea0b0a3ea409879f085f1c4fe92405803a6c3e0879507206b2"} Oct 30 12:25:10.447252 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:10.447228 4908 generic.go:356] "Generic (PLEG): container finished" podID="e36676bf-3529-43a7-b65f-ddb63f3b9278" containerID="e27cf3a01e8bd3563f2b72a43bfb22f9db8711899c13825047573c48bcdbf8d9" exitCode=0 Oct 30 12:25:10.447380 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:10.447362 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-hbpxv" event={"ID":"e36676bf-3529-43a7-b65f-ddb63f3b9278","Type":"ContainerDied","Data":"e27cf3a01e8bd3563f2b72a43bfb22f9db8711899c13825047573c48bcdbf8d9"} Oct 30 12:25:10.447446 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:10.447407 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-hbpxv" event={"ID":"e36676bf-3529-43a7-b65f-ddb63f3b9278","Type":"ContainerDied","Data":"4981c804ef11b5a353f31b3736310360e91fa62518e7e76937b1a76a65b9c035"} Oct 30 12:25:10.447446 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:10.447437 4908 scope.go:122] "RemoveContainer" containerID="e27cf3a01e8bd3563f2b72a43bfb22f9db8711899c13825047573c48bcdbf8d9" Oct 30 12:25:10.447625 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:10.447608 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-hbpxv" Oct 30 12:25:10.451426 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:10.451403 4908 generic.go:356] "Generic (PLEG): container finished" podID="bd702b8b-1d62-41b9-8fbf-1cb4aebdaa71" containerID="1145f457ca80476d2590cbefa8c8c19e55242d855919b7bdfe576b5855a4495c" exitCode=0 Oct 30 12:25:10.451535 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:10.451448 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-pzdcj" event={"ID":"bd702b8b-1d62-41b9-8fbf-1cb4aebdaa71","Type":"ContainerDied","Data":"1145f457ca80476d2590cbefa8c8c19e55242d855919b7bdfe576b5855a4495c"} Oct 30 12:25:10.461589 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:10.461568 4908 scope.go:122] "RemoveContainer" containerID="e27cf3a01e8bd3563f2b72a43bfb22f9db8711899c13825047573c48bcdbf8d9" Oct 30 12:25:10.461870 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:25:10.461823 4908 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e27cf3a01e8bd3563f2b72a43bfb22f9db8711899c13825047573c48bcdbf8d9\": container with ID starting with e27cf3a01e8bd3563f2b72a43bfb22f9db8711899c13825047573c48bcdbf8d9 not found: ID does not exist" containerID="e27cf3a01e8bd3563f2b72a43bfb22f9db8711899c13825047573c48bcdbf8d9" Oct 30 12:25:10.461956 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:10.461881 4908 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e27cf3a01e8bd3563f2b72a43bfb22f9db8711899c13825047573c48bcdbf8d9"} err="failed to get container status \"e27cf3a01e8bd3563f2b72a43bfb22f9db8711899c13825047573c48bcdbf8d9\": rpc error: code = NotFound desc = could not find container \"e27cf3a01e8bd3563f2b72a43bfb22f9db8711899c13825047573c48bcdbf8d9\": container with ID starting with e27cf3a01e8bd3563f2b72a43bfb22f9db8711899c13825047573c48bcdbf8d9 not found: ID does not exist" Oct 30 12:25:10.499140 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:10.498665 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e36676bf-3529-43a7-b65f-ddb63f3b9278-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e36676bf-3529-43a7-b65f-ddb63f3b9278-workload-socket\") pod \"e36676bf-3529-43a7-b65f-ddb63f3b9278\" (UID: \"e36676bf-3529-43a7-b65f-ddb63f3b9278\") " Oct 30 12:25:10.499140 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:10.498796 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e36676bf-3529-43a7-b65f-ddb63f3b9278-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e36676bf-3529-43a7-b65f-ddb63f3b9278-istio-envoy\") pod \"e36676bf-3529-43a7-b65f-ddb63f3b9278\" (UID: \"e36676bf-3529-43a7-b65f-ddb63f3b9278\") " Oct 30 12:25:10.499140 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:10.498849 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e36676bf-3529-43a7-b65f-ddb63f3b9278-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e36676bf-3529-43a7-b65f-ddb63f3b9278-istio-data\") pod \"e36676bf-3529-43a7-b65f-ddb63f3b9278\" (UID: \"e36676bf-3529-43a7-b65f-ddb63f3b9278\") " Oct 30 12:25:10.499140 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:10.498898 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e36676bf-3529-43a7-b65f-ddb63f3b9278-istio-token\" (UniqueName: \"kubernetes.io/projected/e36676bf-3529-43a7-b65f-ddb63f3b9278-istio-token\") pod \"e36676bf-3529-43a7-b65f-ddb63f3b9278\" (UID: \"e36676bf-3529-43a7-b65f-ddb63f3b9278\") " Oct 30 12:25:10.499414 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:10.499154 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e36676bf-3529-43a7-b65f-ddb63f3b9278-kube-api-access-m7d4j\" (UniqueName: \"kubernetes.io/projected/e36676bf-3529-43a7-b65f-ddb63f3b9278-kube-api-access-m7d4j\") pod \"e36676bf-3529-43a7-b65f-ddb63f3b9278\" (UID: \"e36676bf-3529-43a7-b65f-ddb63f3b9278\") " Oct 30 12:25:10.499414 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:10.499317 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e36676bf-3529-43a7-b65f-ddb63f3b9278-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e36676bf-3529-43a7-b65f-ddb63f3b9278-istio-podinfo\") pod \"e36676bf-3529-43a7-b65f-ddb63f3b9278\" (UID: \"e36676bf-3529-43a7-b65f-ddb63f3b9278\") " Oct 30 12:25:10.499414 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:10.499373 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e36676bf-3529-43a7-b65f-ddb63f3b9278-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e36676bf-3529-43a7-b65f-ddb63f3b9278-credential-socket\") pod \"e36676bf-3529-43a7-b65f-ddb63f3b9278\" (UID: \"e36676bf-3529-43a7-b65f-ddb63f3b9278\") " Oct 30 12:25:10.499591 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:10.499415 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e36676bf-3529-43a7-b65f-ddb63f3b9278-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e36676bf-3529-43a7-b65f-ddb63f3b9278-workload-certs\") pod \"e36676bf-3529-43a7-b65f-ddb63f3b9278\" (UID: \"e36676bf-3529-43a7-b65f-ddb63f3b9278\") " Oct 30 12:25:10.499591 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:10.499469 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e36676bf-3529-43a7-b65f-ddb63f3b9278-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e36676bf-3529-43a7-b65f-ddb63f3b9278-istiod-ca-cert\") pod \"e36676bf-3529-43a7-b65f-ddb63f3b9278\" (UID: \"e36676bf-3529-43a7-b65f-ddb63f3b9278\") " Oct 30 12:25:10.502993 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:10.500420 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e36676bf-3529-43a7-b65f-ddb63f3b9278-istiod-ca-cert" pod "e36676bf-3529-43a7-b65f-ddb63f3b9278" (UID: "e36676bf-3529-43a7-b65f-ddb63f3b9278"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:25:10.502993 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:10.500690 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e36676bf-3529-43a7-b65f-ddb63f3b9278-workload-socket" pod "e36676bf-3529-43a7-b65f-ddb63f3b9278" (UID: "e36676bf-3529-43a7-b65f-ddb63f3b9278"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:25:10.502993 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:10.501746 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e36676bf-3529-43a7-b65f-ddb63f3b9278-istio-data" pod "e36676bf-3529-43a7-b65f-ddb63f3b9278" (UID: "e36676bf-3529-43a7-b65f-ddb63f3b9278"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:25:10.511868 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:10.510659 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e36676bf-3529-43a7-b65f-ddb63f3b9278-istio-envoy" pod "e36676bf-3529-43a7-b65f-ddb63f3b9278" (UID: "e36676bf-3529-43a7-b65f-ddb63f3b9278"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:25:10.513926 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:10.512839 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e36676bf-3529-43a7-b65f-ddb63f3b9278-workload-certs" pod "e36676bf-3529-43a7-b65f-ddb63f3b9278" (UID: "e36676bf-3529-43a7-b65f-ddb63f3b9278"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:25:10.513926 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:10.512907 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e36676bf-3529-43a7-b65f-ddb63f3b9278-credential-socket" pod "e36676bf-3529-43a7-b65f-ddb63f3b9278" (UID: "e36676bf-3529-43a7-b65f-ddb63f3b9278"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:25:10.515649 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:10.514465 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e36676bf-3529-43a7-b65f-ddb63f3b9278-istio-token" pod "e36676bf-3529-43a7-b65f-ddb63f3b9278" (UID: "e36676bf-3529-43a7-b65f-ddb63f3b9278"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:25:10.515649 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:10.514514 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e36676bf-3529-43a7-b65f-ddb63f3b9278-kube-api-access-m7d4j" pod "e36676bf-3529-43a7-b65f-ddb63f3b9278" (UID: "e36676bf-3529-43a7-b65f-ddb63f3b9278"). InnerVolumeSpecName "kube-api-access-m7d4j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:25:10.515649 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:10.515218 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e36676bf-3529-43a7-b65f-ddb63f3b9278-istio-podinfo" pod "e36676bf-3529-43a7-b65f-ddb63f3b9278" (UID: "e36676bf-3529-43a7-b65f-ddb63f3b9278"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 12:25:10.552060 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:10.552036 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-pzdcj" Oct 30 12:25:10.585156 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:10.585137 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rx4tf" Oct 30 12:25:10.601240 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:10.601208 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd702b8b-1d62-41b9-8fbf-1cb4aebdaa71-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd702b8b-1d62-41b9-8fbf-1cb4aebdaa71-istiod-ca-cert\") pod \"bd702b8b-1d62-41b9-8fbf-1cb4aebdaa71\" (UID: \"bd702b8b-1d62-41b9-8fbf-1cb4aebdaa71\") " Oct 30 12:25:10.601406 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:10.601354 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd702b8b-1d62-41b9-8fbf-1cb4aebdaa71-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd702b8b-1d62-41b9-8fbf-1cb4aebdaa71-workload-certs\") pod \"bd702b8b-1d62-41b9-8fbf-1cb4aebdaa71\" (UID: \"bd702b8b-1d62-41b9-8fbf-1cb4aebdaa71\") " Oct 30 12:25:10.601680 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:10.601654 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd702b8b-1d62-41b9-8fbf-1cb4aebdaa71-istiod-ca-cert" pod "bd702b8b-1d62-41b9-8fbf-1cb4aebdaa71" (UID: "bd702b8b-1d62-41b9-8fbf-1cb4aebdaa71"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:25:10.601777 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:10.601672 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd702b8b-1d62-41b9-8fbf-1cb4aebdaa71-workload-certs" pod "bd702b8b-1d62-41b9-8fbf-1cb4aebdaa71" (UID: "bd702b8b-1d62-41b9-8fbf-1cb4aebdaa71"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:25:10.602364 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:10.601388 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bd702b8b-1d62-41b9-8fbf-1cb4aebdaa71-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd702b8b-1d62-41b9-8fbf-1cb4aebdaa71-istio-podinfo\") pod \"bd702b8b-1d62-41b9-8fbf-1cb4aebdaa71\" (UID: \"bd702b8b-1d62-41b9-8fbf-1cb4aebdaa71\") " Oct 30 12:25:10.602551 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:10.602529 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd702b8b-1d62-41b9-8fbf-1cb4aebdaa71-credential-socket" pod "bd702b8b-1d62-41b9-8fbf-1cb4aebdaa71" (UID: "bd702b8b-1d62-41b9-8fbf-1cb4aebdaa71"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:25:10.602630 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:10.602589 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd702b8b-1d62-41b9-8fbf-1cb4aebdaa71-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd702b8b-1d62-41b9-8fbf-1cb4aebdaa71-credential-socket\") pod \"bd702b8b-1d62-41b9-8fbf-1cb4aebdaa71\" (UID: \"bd702b8b-1d62-41b9-8fbf-1cb4aebdaa71\") " Oct 30 12:25:10.602691 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:10.602676 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd702b8b-1d62-41b9-8fbf-1cb4aebdaa71-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd702b8b-1d62-41b9-8fbf-1cb4aebdaa71-workload-socket\") pod \"bd702b8b-1d62-41b9-8fbf-1cb4aebdaa71\" (UID: \"bd702b8b-1d62-41b9-8fbf-1cb4aebdaa71\") " Oct 30 12:25:10.603105 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:10.603061 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd702b8b-1d62-41b9-8fbf-1cb4aebdaa71-workload-socket" pod "bd702b8b-1d62-41b9-8fbf-1cb4aebdaa71" (UID: "bd702b8b-1d62-41b9-8fbf-1cb4aebdaa71"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:25:10.603188 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:10.603128 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd702b8b-1d62-41b9-8fbf-1cb4aebdaa71-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd702b8b-1d62-41b9-8fbf-1cb4aebdaa71-istio-envoy\") pod \"bd702b8b-1d62-41b9-8fbf-1cb4aebdaa71\" (UID: \"bd702b8b-1d62-41b9-8fbf-1cb4aebdaa71\") " Oct 30 12:25:10.603515 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:10.603490 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd702b8b-1d62-41b9-8fbf-1cb4aebdaa71-kube-api-access-s7tzl\" (UniqueName: \"kubernetes.io/projected/bd702b8b-1d62-41b9-8fbf-1cb4aebdaa71-kube-api-access-s7tzl\") pod \"bd702b8b-1d62-41b9-8fbf-1cb4aebdaa71\" (UID: \"bd702b8b-1d62-41b9-8fbf-1cb4aebdaa71\") " Oct 30 12:25:10.603912 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:10.603875 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd702b8b-1d62-41b9-8fbf-1cb4aebdaa71-istio-token\" (UniqueName: \"kubernetes.io/projected/bd702b8b-1d62-41b9-8fbf-1cb4aebdaa71-istio-token\") pod \"bd702b8b-1d62-41b9-8fbf-1cb4aebdaa71\" (UID: \"bd702b8b-1d62-41b9-8fbf-1cb4aebdaa71\") " Oct 30 12:25:10.604205 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:10.604183 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd702b8b-1d62-41b9-8fbf-1cb4aebdaa71-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd702b8b-1d62-41b9-8fbf-1cb4aebdaa71-istio-data\") pod \"bd702b8b-1d62-41b9-8fbf-1cb4aebdaa71\" (UID: \"bd702b8b-1d62-41b9-8fbf-1cb4aebdaa71\") " Oct 30 12:25:10.604822 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:10.604781 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7d4j\" (UniqueName: \"kubernetes.io/projected/e36676bf-3529-43a7-b65f-ddb63f3b9278-kube-api-access-m7d4j\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:25:10.604822 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:10.604812 4908 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd702b8b-1d62-41b9-8fbf-1cb4aebdaa71-istiod-ca-cert\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:25:10.604999 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:10.604842 4908 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e36676bf-3529-43a7-b65f-ddb63f3b9278-istio-podinfo\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:25:10.604999 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:10.604858 4908 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e36676bf-3529-43a7-b65f-ddb63f3b9278-credential-socket\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:25:10.604999 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:10.604872 4908 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e36676bf-3529-43a7-b65f-ddb63f3b9278-workload-certs\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:25:10.604999 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:10.604907 4908 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e36676bf-3529-43a7-b65f-ddb63f3b9278-istiod-ca-cert\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:25:10.604999 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:10.604920 4908 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd702b8b-1d62-41b9-8fbf-1cb4aebdaa71-workload-certs\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:25:10.604999 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:10.604933 4908 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e36676bf-3529-43a7-b65f-ddb63f3b9278-workload-socket\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:25:10.604999 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:10.604949 4908 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e36676bf-3529-43a7-b65f-ddb63f3b9278-istio-envoy\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:25:10.604999 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:10.604976 4908 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e36676bf-3529-43a7-b65f-ddb63f3b9278-istio-data\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:25:10.604999 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:10.604989 4908 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd702b8b-1d62-41b9-8fbf-1cb4aebdaa71-credential-socket\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:25:10.604999 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:10.605003 4908 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e36676bf-3529-43a7-b65f-ddb63f3b9278-istio-token\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:25:10.605483 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:10.605016 4908 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd702b8b-1d62-41b9-8fbf-1cb4aebdaa71-workload-socket\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:25:10.606588 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:10.606565 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd702b8b-1d62-41b9-8fbf-1cb4aebdaa71-istio-data" pod "bd702b8b-1d62-41b9-8fbf-1cb4aebdaa71" (UID: "bd702b8b-1d62-41b9-8fbf-1cb4aebdaa71"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:25:10.607193 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:10.607157 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd702b8b-1d62-41b9-8fbf-1cb4aebdaa71-istio-envoy" pod "bd702b8b-1d62-41b9-8fbf-1cb4aebdaa71" (UID: "bd702b8b-1d62-41b9-8fbf-1cb4aebdaa71"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:25:10.607359 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:10.607221 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bd702b8b-1d62-41b9-8fbf-1cb4aebdaa71-istio-podinfo" pod "bd702b8b-1d62-41b9-8fbf-1cb4aebdaa71" (UID: "bd702b8b-1d62-41b9-8fbf-1cb4aebdaa71"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 12:25:10.608155 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:10.608132 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd702b8b-1d62-41b9-8fbf-1cb4aebdaa71-istio-token" pod "bd702b8b-1d62-41b9-8fbf-1cb4aebdaa71" (UID: "bd702b8b-1d62-41b9-8fbf-1cb4aebdaa71"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:25:10.611092 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:10.611064 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd702b8b-1d62-41b9-8fbf-1cb4aebdaa71-kube-api-access-s7tzl" pod "bd702b8b-1d62-41b9-8fbf-1cb4aebdaa71" (UID: "bd702b8b-1d62-41b9-8fbf-1cb4aebdaa71"). InnerVolumeSpecName "kube-api-access-s7tzl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:25:10.706070 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:10.706028 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76c80954-caf1-42b7-a812-fa52488f2e1b-istio-token\" (UniqueName: \"kubernetes.io/projected/76c80954-caf1-42b7-a812-fa52488f2e1b-istio-token\") pod \"76c80954-caf1-42b7-a812-fa52488f2e1b\" (UID: \"76c80954-caf1-42b7-a812-fa52488f2e1b\") " Oct 30 12:25:10.706212 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:10.706138 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76c80954-caf1-42b7-a812-fa52488f2e1b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76c80954-caf1-42b7-a812-fa52488f2e1b-credential-socket\") pod \"76c80954-caf1-42b7-a812-fa52488f2e1b\" (UID: \"76c80954-caf1-42b7-a812-fa52488f2e1b\") " Oct 30 12:25:10.706212 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:10.706206 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76c80954-caf1-42b7-a812-fa52488f2e1b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76c80954-caf1-42b7-a812-fa52488f2e1b-istiod-ca-cert\") pod \"76c80954-caf1-42b7-a812-fa52488f2e1b\" (UID: \"76c80954-caf1-42b7-a812-fa52488f2e1b\") " Oct 30 12:25:10.706304 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:10.706259 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76c80954-caf1-42b7-a812-fa52488f2e1b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76c80954-caf1-42b7-a812-fa52488f2e1b-istio-data\") pod \"76c80954-caf1-42b7-a812-fa52488f2e1b\" (UID: \"76c80954-caf1-42b7-a812-fa52488f2e1b\") " Oct 30 12:25:10.706419 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:10.706390 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76c80954-caf1-42b7-a812-fa52488f2e1b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76c80954-caf1-42b7-a812-fa52488f2e1b-istio-envoy\") pod \"76c80954-caf1-42b7-a812-fa52488f2e1b\" (UID: \"76c80954-caf1-42b7-a812-fa52488f2e1b\") " Oct 30 12:25:10.706485 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:10.706433 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76c80954-caf1-42b7-a812-fa52488f2e1b-credential-socket" pod "76c80954-caf1-42b7-a812-fa52488f2e1b" (UID: "76c80954-caf1-42b7-a812-fa52488f2e1b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:25:10.706539 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:10.706492 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/76c80954-caf1-42b7-a812-fa52488f2e1b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76c80954-caf1-42b7-a812-fa52488f2e1b-istio-podinfo\") pod \"76c80954-caf1-42b7-a812-fa52488f2e1b\" (UID: \"76c80954-caf1-42b7-a812-fa52488f2e1b\") " Oct 30 12:25:10.706539 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:10.706524 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76c80954-caf1-42b7-a812-fa52488f2e1b-istio-data" pod "76c80954-caf1-42b7-a812-fa52488f2e1b" (UID: "76c80954-caf1-42b7-a812-fa52488f2e1b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:25:10.706539 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:10.706533 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76c80954-caf1-42b7-a812-fa52488f2e1b-kube-api-access-4dm5l\" (UniqueName: \"kubernetes.io/projected/76c80954-caf1-42b7-a812-fa52488f2e1b-kube-api-access-4dm5l\") pod \"76c80954-caf1-42b7-a812-fa52488f2e1b\" (UID: \"76c80954-caf1-42b7-a812-fa52488f2e1b\") " Oct 30 12:25:10.706687 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:10.706574 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76c80954-caf1-42b7-a812-fa52488f2e1b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76c80954-caf1-42b7-a812-fa52488f2e1b-workload-socket\") pod \"76c80954-caf1-42b7-a812-fa52488f2e1b\" (UID: \"76c80954-caf1-42b7-a812-fa52488f2e1b\") " Oct 30 12:25:10.706687 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:10.706642 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76c80954-caf1-42b7-a812-fa52488f2e1b-istiod-ca-cert" pod "76c80954-caf1-42b7-a812-fa52488f2e1b" (UID: "76c80954-caf1-42b7-a812-fa52488f2e1b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:25:10.706793 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:10.706728 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76c80954-caf1-42b7-a812-fa52488f2e1b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76c80954-caf1-42b7-a812-fa52488f2e1b-workload-certs\") pod \"76c80954-caf1-42b7-a812-fa52488f2e1b\" (UID: \"76c80954-caf1-42b7-a812-fa52488f2e1b\") " Oct 30 12:25:10.707441 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:10.707363 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76c80954-caf1-42b7-a812-fa52488f2e1b-workload-certs" pod "76c80954-caf1-42b7-a812-fa52488f2e1b" (UID: "76c80954-caf1-42b7-a812-fa52488f2e1b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:25:10.707441 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:10.707399 4908 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76c80954-caf1-42b7-a812-fa52488f2e1b-credential-socket\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:25:10.707441 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:10.707422 4908 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd702b8b-1d62-41b9-8fbf-1cb4aebdaa71-istio-podinfo\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:25:10.707441 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:10.707437 4908 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76c80954-caf1-42b7-a812-fa52488f2e1b-istiod-ca-cert\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:25:10.707717 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:10.707452 4908 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76c80954-caf1-42b7-a812-fa52488f2e1b-istio-data\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:25:10.707717 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:10.707466 4908 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd702b8b-1d62-41b9-8fbf-1cb4aebdaa71-istio-envoy\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:25:10.707717 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:10.707478 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7tzl\" (UniqueName: \"kubernetes.io/projected/bd702b8b-1d62-41b9-8fbf-1cb4aebdaa71-kube-api-access-s7tzl\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:25:10.707717 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:10.707492 4908 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd702b8b-1d62-41b9-8fbf-1cb4aebdaa71-istio-token\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:25:10.707717 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:10.707517 4908 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd702b8b-1d62-41b9-8fbf-1cb4aebdaa71-istio-data\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:25:10.708066 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:10.708033 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76c80954-caf1-42b7-a812-fa52488f2e1b-workload-socket" pod "76c80954-caf1-42b7-a812-fa52488f2e1b" (UID: "76c80954-caf1-42b7-a812-fa52488f2e1b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:25:10.708731 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:10.708710 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76c80954-caf1-42b7-a812-fa52488f2e1b-istio-token" pod "76c80954-caf1-42b7-a812-fa52488f2e1b" (UID: "76c80954-caf1-42b7-a812-fa52488f2e1b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:25:10.708818 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:10.708712 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76c80954-caf1-42b7-a812-fa52488f2e1b-istio-envoy" pod "76c80954-caf1-42b7-a812-fa52488f2e1b" (UID: "76c80954-caf1-42b7-a812-fa52488f2e1b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:25:10.708951 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:10.708935 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76c80954-caf1-42b7-a812-fa52488f2e1b-kube-api-access-4dm5l" pod "76c80954-caf1-42b7-a812-fa52488f2e1b" (UID: "76c80954-caf1-42b7-a812-fa52488f2e1b"). InnerVolumeSpecName "kube-api-access-4dm5l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:25:10.708989 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:10.708973 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/76c80954-caf1-42b7-a812-fa52488f2e1b-istio-podinfo" pod "76c80954-caf1-42b7-a812-fa52488f2e1b" (UID: "76c80954-caf1-42b7-a812-fa52488f2e1b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 12:25:10.797956 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:10.797928 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-hbpxv"] Oct 30 12:25:10.803864 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:10.803795 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-hbpxv"] Oct 30 12:25:10.810626 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:10.810566 4908 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76c80954-caf1-42b7-a812-fa52488f2e1b-workload-certs\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:25:10.810626 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:10.810597 4908 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76c80954-caf1-42b7-a812-fa52488f2e1b-istio-token\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:25:10.812871 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:10.812790 4908 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76c80954-caf1-42b7-a812-fa52488f2e1b-istio-envoy\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:25:10.813029 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:10.812819 4908 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76c80954-caf1-42b7-a812-fa52488f2e1b-istio-podinfo\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:25:10.813164 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:10.813008 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4dm5l\" (UniqueName: \"kubernetes.io/projected/76c80954-caf1-42b7-a812-fa52488f2e1b-kube-api-access-4dm5l\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:25:10.813164 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:10.813145 4908 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76c80954-caf1-42b7-a812-fa52488f2e1b-workload-socket\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:25:11.462771 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:11.462741 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rx4tf" Oct 30 12:25:11.463761 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:11.462746 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rx4tf" event={"ID":"76c80954-caf1-42b7-a812-fa52488f2e1b","Type":"ContainerDied","Data":"3bc88771ac88beba4f4b4f3f3f7e88f8a6c5c64c86d01c364d1ec73930d64e79"} Oct 30 12:25:11.463908 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:11.463800 4908 scope.go:122] "RemoveContainer" containerID="9fc1dc78272c01ea0b0a3ea409879f085f1c4fe92405803a6c3e0879507206b2" Oct 30 12:25:11.467798 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:11.467768 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-pzdcj" event={"ID":"bd702b8b-1d62-41b9-8fbf-1cb4aebdaa71","Type":"ContainerDied","Data":"6edff75f619ef4b909e91711444b0cbdd227a0cc28874c0142d60044b04e36a2"} Oct 30 12:25:11.468398 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:11.467907 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-pzdcj" Oct 30 12:25:11.478996 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:11.478903 4908 scope.go:122] "RemoveContainer" containerID="1145f457ca80476d2590cbefa8c8c19e55242d855919b7bdfe576b5855a4495c" Oct 30 12:25:11.529352 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:11.529328 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-pzdcj"] Oct 30 12:25:11.540030 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:11.540009 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-pzdcj"] Oct 30 12:25:11.574020 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:11.573990 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rx4tf"] Oct 30 12:25:11.579761 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:11.579737 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rx4tf"] Oct 30 12:25:11.610119 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:11.610096 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76c80954-caf1-42b7-a812-fa52488f2e1b" path="/var/lib/kubelet/pods/76c80954-caf1-42b7-a812-fa52488f2e1b/volumes" Oct 30 12:25:11.610576 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:11.610557 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd702b8b-1d62-41b9-8fbf-1cb4aebdaa71" path="/var/lib/kubelet/pods/bd702b8b-1d62-41b9-8fbf-1cb4aebdaa71/volumes" Oct 30 12:25:11.611174 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:11.611156 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e36676bf-3529-43a7-b65f-ddb63f3b9278" path="/var/lib/kubelet/pods/e36676bf-3529-43a7-b65f-ddb63f3b9278/volumes" Oct 30 12:25:13.031483 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:13.031445 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e447a5b1-crx2n"] Oct 30 12:25:13.042052 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:13.042026 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e447a5b1-crx2n"] Oct 30 12:25:13.607570 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:13.607537 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71f7eeee-de32-4636-a9d1-314b6046c3a3" path="/var/lib/kubelet/pods/71f7eeee-de32-4636-a9d1-314b6046c3a3/volumes" Oct 30 12:25:19.684099 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:25:19.684064 4908 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vn6vv" containerName="extractor" Oct 30 12:25:24.028821 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:24.028783 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a04a4d4c-4jxqh"] Oct 30 12:25:24.039255 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:24.039228 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a04a4d4c-4jxqh"] Oct 30 12:25:25.609024 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:25.608992 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="479b056c-a79f-489b-8e05-10bddcb95072" path="/var/lib/kubelet/pods/479b056c-a79f-489b-8e05-10bddcb95072/volumes" Oct 30 12:25:27.482132 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:27.482091 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9wwfc"] Oct 30 12:25:33.298140 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:33.298104 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-951ad1-566fb4ff59-wmdq5"] Oct 30 12:25:33.299472 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:33.299448 4908 cpu_manager.go:414] "containerMap: removing container" podUID="e36676bf-3529-43a7-b65f-ddb63f3b9278" containerName="istio-proxy" Oct 30 12:25:33.299613 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:33.299602 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e36676bf-3529-43a7-b65f-ddb63f3b9278" containerName="istio-proxy" Oct 30 12:25:33.299699 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:33.299692 4908 cpu_manager.go:414] "containerMap: removing container" podUID="76c80954-caf1-42b7-a812-fa52488f2e1b" containerName="istio-proxy" Oct 30 12:25:33.299747 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:33.299740 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76c80954-caf1-42b7-a812-fa52488f2e1b" containerName="istio-proxy" Oct 30 12:25:33.299798 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:33.299791 4908 cpu_manager.go:414] "containerMap: removing container" podUID="bd702b8b-1d62-41b9-8fbf-1cb4aebdaa71" containerName="istio-proxy" Oct 30 12:25:33.299871 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:33.299860 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd702b8b-1d62-41b9-8fbf-1cb4aebdaa71" containerName="istio-proxy" Oct 30 12:25:33.300039 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:33.300031 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="e36676bf-3529-43a7-b65f-ddb63f3b9278" containerName="istio-proxy" Oct 30 12:25:33.300110 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:33.300101 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="76c80954-caf1-42b7-a812-fa52488f2e1b" containerName="istio-proxy" Oct 30 12:25:33.300170 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:33.300164 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd702b8b-1d62-41b9-8fbf-1cb4aebdaa71" containerName="istio-proxy" Oct 30 12:25:33.304192 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:33.304168 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-951ad1-566fb4ff59-wmdq5" Oct 30 12:25:33.310963 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:33.309111 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-951ad1-566fb4ff59-wmdq5"] Oct 30 12:25:33.318071 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:33.318050 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-951ad1-dockercfg-nd6t7\"" Oct 30 12:25:33.331893 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:33.331870 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-951ad1\"" Oct 30 12:25:33.404866 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:33.397143 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zt467\" (UniqueName: \"kubernetes.io/projected/3af18c4f-4306-43d2-a584-18b4353c7651-kube-api-access-zt467\") pod \"payload-pre-processing-e2e-ipp-a-951ad1-566fb4ff59-wmdq5\" (UID: \"3af18c4f-4306-43d2-a584-18b4353c7651\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-951ad1-566fb4ff59-wmdq5" Oct 30 12:25:33.406216 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:33.406028 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3af18c4f-4306-43d2-a584-18b4353c7651-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-951ad1-566fb4ff59-wmdq5\" (UID: \"3af18c4f-4306-43d2-a584-18b4353c7651\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-951ad1-566fb4ff59-wmdq5" Oct 30 12:25:33.508921 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:33.508794 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zt467\" (UniqueName: \"kubernetes.io/projected/3af18c4f-4306-43d2-a584-18b4353c7651-kube-api-access-zt467\") pod \"payload-pre-processing-e2e-ipp-a-951ad1-566fb4ff59-wmdq5\" (UID: \"3af18c4f-4306-43d2-a584-18b4353c7651\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-951ad1-566fb4ff59-wmdq5" Oct 30 12:25:33.509210 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:33.509186 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3af18c4f-4306-43d2-a584-18b4353c7651-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-951ad1-566fb4ff59-wmdq5\" (UID: \"3af18c4f-4306-43d2-a584-18b4353c7651\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-951ad1-566fb4ff59-wmdq5" Oct 30 12:25:33.510664 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:33.510504 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3af18c4f-4306-43d2-a584-18b4353c7651-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-951ad1-566fb4ff59-wmdq5\" (UID: \"3af18c4f-4306-43d2-a584-18b4353c7651\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-951ad1-566fb4ff59-wmdq5" Oct 30 12:25:33.531535 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:33.531508 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zt467\" (UniqueName: \"kubernetes.io/projected/3af18c4f-4306-43d2-a584-18b4353c7651-kube-api-access-zt467\") pod \"payload-pre-processing-e2e-ipp-a-951ad1-566fb4ff59-wmdq5\" (UID: \"3af18c4f-4306-43d2-a584-18b4353c7651\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-951ad1-566fb4ff59-wmdq5" Oct 30 12:25:33.615555 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:33.615463 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-922643-9zzrf"] Oct 30 12:25:33.627921 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:33.627885 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-922643-9zzrf"] Oct 30 12:25:33.628100 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:33.627993 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-922643-9zzrf" Oct 30 12:25:33.628912 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:33.628888 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-951ad1-566fb4ff59-wmdq5" Oct 30 12:25:33.641556 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:33.641535 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 12:25:33.641691 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:33.641672 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 12:25:33.641765 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:33.641750 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s75hq\"" Oct 30 12:25:33.641942 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:33.641926 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 12:25:33.717841 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:33.717802 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8312b5a5-4d8d-4c54-b9fb-85228cd75c1f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-922643-9zzrf\" (UID: \"8312b5a5-4d8d-4c54-b9fb-85228cd75c1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-922643-9zzrf" Oct 30 12:25:33.717998 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:33.717884 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8312b5a5-4d8d-4c54-b9fb-85228cd75c1f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-922643-9zzrf\" (UID: \"8312b5a5-4d8d-4c54-b9fb-85228cd75c1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-922643-9zzrf" Oct 30 12:25:33.718087 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:33.718039 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knprc\" (UniqueName: \"kubernetes.io/projected/8312b5a5-4d8d-4c54-b9fb-85228cd75c1f-kube-api-access-knprc\") pod \"maas-api-revoke-keys-e2e-api-a-922643-9zzrf\" (UID: \"8312b5a5-4d8d-4c54-b9fb-85228cd75c1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-922643-9zzrf" Oct 30 12:25:33.725157 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:33.725130 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-951ad1-7d788c86d-tmc8p"] Oct 30 12:25:33.729913 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:33.729569 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-951ad1-7d788c86d-tmc8p" Oct 30 12:25:33.751820 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:33.751787 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-951ad1-7d788c86d-tmc8p"] Oct 30 12:25:33.819412 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:33.819372 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knprc\" (UniqueName: \"kubernetes.io/projected/8312b5a5-4d8d-4c54-b9fb-85228cd75c1f-kube-api-access-knprc\") pod \"maas-api-revoke-keys-e2e-api-a-922643-9zzrf\" (UID: \"8312b5a5-4d8d-4c54-b9fb-85228cd75c1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-922643-9zzrf" Oct 30 12:25:33.819607 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:33.819515 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14f93ee4-d2c7-456f-94c3-e1378e226d62-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-951ad1-7d788c86d-tmc8p\" (UID: \"14f93ee4-d2c7-456f-94c3-e1378e226d62\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-951ad1-7d788c86d-tmc8p" Oct 30 12:25:33.819607 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:33.819595 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8312b5a5-4d8d-4c54-b9fb-85228cd75c1f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-922643-9zzrf\" (UID: \"8312b5a5-4d8d-4c54-b9fb-85228cd75c1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-922643-9zzrf" Oct 30 12:25:33.819721 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:33.819623 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2vvk\" (UniqueName: \"kubernetes.io/projected/14f93ee4-d2c7-456f-94c3-e1378e226d62-kube-api-access-k2vvk\") pod \"payload-processing-e2e-ipp-a-951ad1-7d788c86d-tmc8p\" (UID: \"14f93ee4-d2c7-456f-94c3-e1378e226d62\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-951ad1-7d788c86d-tmc8p" Oct 30 12:25:33.819721 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:33.819713 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8312b5a5-4d8d-4c54-b9fb-85228cd75c1f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-922643-9zzrf\" (UID: \"8312b5a5-4d8d-4c54-b9fb-85228cd75c1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-922643-9zzrf" Oct 30 12:25:33.820550 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:33.820522 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8312b5a5-4d8d-4c54-b9fb-85228cd75c1f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-922643-9zzrf\" (UID: \"8312b5a5-4d8d-4c54-b9fb-85228cd75c1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-922643-9zzrf" Oct 30 12:25:33.822210 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:33.822174 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8312b5a5-4d8d-4c54-b9fb-85228cd75c1f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-922643-9zzrf\" (UID: \"8312b5a5-4d8d-4c54-b9fb-85228cd75c1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-922643-9zzrf" Oct 30 12:25:33.841630 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:33.841608 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-knprc\" (UniqueName: \"kubernetes.io/projected/8312b5a5-4d8d-4c54-b9fb-85228cd75c1f-kube-api-access-knprc\") pod \"maas-api-revoke-keys-e2e-api-a-922643-9zzrf\" (UID: \"8312b5a5-4d8d-4c54-b9fb-85228cd75c1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-922643-9zzrf" Oct 30 12:25:33.920540 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:33.920451 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14f93ee4-d2c7-456f-94c3-e1378e226d62-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-951ad1-7d788c86d-tmc8p\" (UID: \"14f93ee4-d2c7-456f-94c3-e1378e226d62\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-951ad1-7d788c86d-tmc8p" Oct 30 12:25:33.920540 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:33.920516 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2vvk\" (UniqueName: \"kubernetes.io/projected/14f93ee4-d2c7-456f-94c3-e1378e226d62-kube-api-access-k2vvk\") pod \"payload-processing-e2e-ipp-a-951ad1-7d788c86d-tmc8p\" (UID: \"14f93ee4-d2c7-456f-94c3-e1378e226d62\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-951ad1-7d788c86d-tmc8p" Oct 30 12:25:33.921234 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:33.921213 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14f93ee4-d2c7-456f-94c3-e1378e226d62-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-951ad1-7d788c86d-tmc8p\" (UID: \"14f93ee4-d2c7-456f-94c3-e1378e226d62\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-951ad1-7d788c86d-tmc8p" Oct 30 12:25:33.942416 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:33.942389 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2vvk\" (UniqueName: \"kubernetes.io/projected/14f93ee4-d2c7-456f-94c3-e1378e226d62-kube-api-access-k2vvk\") pod \"payload-processing-e2e-ipp-a-951ad1-7d788c86d-tmc8p\" (UID: \"14f93ee4-d2c7-456f-94c3-e1378e226d62\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-951ad1-7d788c86d-tmc8p" Oct 30 12:25:33.948296 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:33.948253 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-922643-9zzrf" Oct 30 12:25:33.981328 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:33.981301 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-951ad1-566fb4ff59-wmdq5"] Oct 30 12:25:33.992076 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:33.992055 4908 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 12:25:34.054309 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:34.054131 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-951ad1-7d788c86d-tmc8p" Oct 30 12:25:34.265192 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:34.265166 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-922643-9zzrf"] Oct 30 12:25:34.267476 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:25:34.267441 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8312b5a5_4d8d_4c54_b9fb_85228cd75c1f.slice/crio-9dcfa18a61ed7a094cf944297bd6b36b0ceb130008097b8f9dc52fcc086dd108 WatchSource:0}: Error finding container 9dcfa18a61ed7a094cf944297bd6b36b0ceb130008097b8f9dc52fcc086dd108: Status 404 returned error can't find the container with id 9dcfa18a61ed7a094cf944297bd6b36b0ceb130008097b8f9dc52fcc086dd108 Oct 30 12:25:34.371110 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:34.371000 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-951ad1-7d788c86d-tmc8p"] Oct 30 12:25:34.462028 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:25:34.461979 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14f93ee4_d2c7_456f_94c3_e1378e226d62.slice/crio-10387b6a373338b08d1774785e680c76baf1ccd52762296e7a0316734127fe5e WatchSource:0}: Error finding container 10387b6a373338b08d1774785e680c76baf1ccd52762296e7a0316734127fe5e: Status 404 returned error can't find the container with id 10387b6a373338b08d1774785e680c76baf1ccd52762296e7a0316734127fe5e Oct 30 12:25:34.785140 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:34.785091 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-951ad1-7d788c86d-tmc8p" event={"ID":"14f93ee4-d2c7-456f-94c3-e1378e226d62","Type":"ContainerStarted","Data":"10387b6a373338b08d1774785e680c76baf1ccd52762296e7a0316734127fe5e"} Oct 30 12:25:34.788438 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:34.788371 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-922643-9zzrf" event={"ID":"8312b5a5-4d8d-4c54-b9fb-85228cd75c1f","Type":"ContainerStarted","Data":"9dcfa18a61ed7a094cf944297bd6b36b0ceb130008097b8f9dc52fcc086dd108"} Oct 30 12:25:34.791523 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:34.791499 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-951ad1-566fb4ff59-wmdq5" event={"ID":"3af18c4f-4306-43d2-a584-18b4353c7651","Type":"ContainerStarted","Data":"661de7523bbfe0e9530dfa0b4275eb66f0ced4788b6c6357b475f9e80403babf"} Oct 30 12:25:35.804265 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:35.804212 4908 generic.go:356] "Generic (PLEG): container finished" podID="8312b5a5-4d8d-4c54-b9fb-85228cd75c1f" containerID="25c1a2c0ba09560ee93bbd92687c64163cfd6faada412c983043583bcd9e221e" exitCode=0 Oct 30 12:25:35.804660 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:35.804277 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-922643-9zzrf" event={"ID":"8312b5a5-4d8d-4c54-b9fb-85228cd75c1f","Type":"ContainerDied","Data":"25c1a2c0ba09560ee93bbd92687c64163cfd6faada412c983043583bcd9e221e"} Oct 30 12:25:36.814611 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:36.814574 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-951ad1-566fb4ff59-wmdq5" event={"ID":"3af18c4f-4306-43d2-a584-18b4353c7651","Type":"ContainerStarted","Data":"fa3b20ed73007427c8ff25d7c73f55782f8d7c89b886ce928f651a93aa0de19f"} Oct 30 12:25:36.815078 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:36.814671 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-951ad1-566fb4ff59-wmdq5" Oct 30 12:25:36.865084 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:36.865028 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-951ad1-566fb4ff59-wmdq5" podStartSLOduration=1.276073296 podStartE2EDuration="3.865008404s" totalImagesPullingTime="2.588935108s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:25:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:25:36.852383 +0000 UTC m=+1893.366811746" watchObservedRunningTime="2026-10-30 12:25:36.865008404 +0000 UTC m=+1893.379437151" Oct 30 12:25:37.045098 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:37.045066 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-922643-9zzrf" Oct 30 12:25:37.190915 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:37.190090 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8312b5a5-4d8d-4c54-b9fb-85228cd75c1f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8312b5a5-4d8d-4c54-b9fb-85228cd75c1f-maas-api-openshift-ca\") pod \"8312b5a5-4d8d-4c54-b9fb-85228cd75c1f\" (UID: \"8312b5a5-4d8d-4c54-b9fb-85228cd75c1f\") " Oct 30 12:25:37.190915 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:37.190140 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8312b5a5-4d8d-4c54-b9fb-85228cd75c1f-kube-api-access-knprc\" (UniqueName: \"kubernetes.io/projected/8312b5a5-4d8d-4c54-b9fb-85228cd75c1f-kube-api-access-knprc\") pod \"8312b5a5-4d8d-4c54-b9fb-85228cd75c1f\" (UID: \"8312b5a5-4d8d-4c54-b9fb-85228cd75c1f\") " Oct 30 12:25:37.190915 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:37.190198 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8312b5a5-4d8d-4c54-b9fb-85228cd75c1f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8312b5a5-4d8d-4c54-b9fb-85228cd75c1f-maas-api-xks-ca\") pod \"8312b5a5-4d8d-4c54-b9fb-85228cd75c1f\" (UID: \"8312b5a5-4d8d-4c54-b9fb-85228cd75c1f\") " Oct 30 12:25:37.190915 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:37.190512 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8312b5a5-4d8d-4c54-b9fb-85228cd75c1f-maas-api-openshift-ca" pod "8312b5a5-4d8d-4c54-b9fb-85228cd75c1f" (UID: "8312b5a5-4d8d-4c54-b9fb-85228cd75c1f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:25:37.191220 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:37.190972 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8312b5a5-4d8d-4c54-b9fb-85228cd75c1f-maas-api-openshift-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:25:37.194866 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:37.193026 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8312b5a5-4d8d-4c54-b9fb-85228cd75c1f-maas-api-xks-ca" pod "8312b5a5-4d8d-4c54-b9fb-85228cd75c1f" (UID: "8312b5a5-4d8d-4c54-b9fb-85228cd75c1f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:25:37.194866 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:37.194451 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8312b5a5-4d8d-4c54-b9fb-85228cd75c1f-kube-api-access-knprc" pod "8312b5a5-4d8d-4c54-b9fb-85228cd75c1f" (UID: "8312b5a5-4d8d-4c54-b9fb-85228cd75c1f"). InnerVolumeSpecName "kube-api-access-knprc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:25:37.292556 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:37.292519 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-knprc\" (UniqueName: \"kubernetes.io/projected/8312b5a5-4d8d-4c54-b9fb-85228cd75c1f-kube-api-access-knprc\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:25:37.292556 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:37.292552 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8312b5a5-4d8d-4c54-b9fb-85228cd75c1f-maas-api-xks-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:25:37.828981 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:37.828945 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-922643-9zzrf" Oct 30 12:25:37.829434 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:37.829043 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-922643-9zzrf" event={"ID":"8312b5a5-4d8d-4c54-b9fb-85228cd75c1f","Type":"ContainerDied","Data":"9dcfa18a61ed7a094cf944297bd6b36b0ceb130008097b8f9dc52fcc086dd108"} Oct 30 12:25:37.829434 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:37.829100 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9dcfa18a61ed7a094cf944297bd6b36b0ceb130008097b8f9dc52fcc086dd108" Oct 30 12:25:37.832794 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:37.832766 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-951ad1-7d788c86d-tmc8p" event={"ID":"14f93ee4-d2c7-456f-94c3-e1378e226d62","Type":"ContainerStarted","Data":"ebeaab5500901574a7567efe0c39fe85b6c896fb8e8fde261c7b7d7593e3935e"} Oct 30 12:25:37.832981 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:37.832947 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-951ad1-7d788c86d-tmc8p" Oct 30 12:25:37.936814 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:37.936707 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-951ad1-7d788c86d-tmc8p" podStartSLOduration=2.543682757 podStartE2EDuration="4.936690016s" totalImagesPullingTime="2.393007259s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:25:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:25:37.93220614 +0000 UTC m=+1894.446634903" watchObservedRunningTime="2026-10-30 12:25:37.936690016 +0000 UTC m=+1894.451118762" Oct 30 12:25:38.843648 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:38.843612 4908 generic.go:356] "Generic (PLEG): container finished" podID="abd92788-b8eb-4998-b22a-0ecae4a3a81d" containerID="804a9abea774a102a224c5637f7b80700ce2809fdbb641118c49325f9bf9d925" exitCode=0 Oct 30 12:25:38.844212 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:38.843688 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-aef61e-openshift-default-554c4fbd5f-t6k2v" event={"ID":"abd92788-b8eb-4998-b22a-0ecae4a3a81d","Type":"ContainerDied","Data":"804a9abea774a102a224c5637f7b80700ce2809fdbb641118c49325f9bf9d925"} Oct 30 12:25:38.844290 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:38.844268 4908 scope.go:122] "RemoveContainer" containerID="804a9abea774a102a224c5637f7b80700ce2809fdbb641118c49325f9bf9d925" Oct 30 12:25:38.845692 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:38.845664 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:25:38.845770 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:38.845731 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:25:38.845770 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:38.845761 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:25:39.853871 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:39.853551 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-aef61e-openshift-default-554c4fbd5f-t6k2v" event={"ID":"abd92788-b8eb-4998-b22a-0ecae4a3a81d","Type":"ContainerStarted","Data":"fba29ea422c3624ed5423da382a3419476a57d13d01741d8b7a0669e791da9c9"} Oct 30 12:25:39.984066 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:39.982891 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-aef61e-openshift-default-554c4fbd5f-t6k2v" Oct 30 12:25:39.984066 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:39.982972 4908 patch_prober.go:29] interesting pod/e2e-api-b-aef61e-openshift-default-554c4fbd5f-t6k2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Oct 30 12:25:39.984066 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:39.983027 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-aef61e-openshift-default-554c4fbd5f-t6k2v" podUID="abd92788-b8eb-4998-b22a-0ecae4a3a81d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Oct 30 12:25:40.973850 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:40.973195 4908 patch_prober.go:29] interesting pod/e2e-api-b-aef61e-openshift-default-554c4fbd5f-t6k2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Oct 30 12:25:40.973850 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:40.973248 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-aef61e-openshift-default-554c4fbd5f-t6k2v" podUID="abd92788-b8eb-4998-b22a-0ecae4a3a81d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Oct 30 12:25:41.978298 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:41.978258 4908 patch_prober.go:29] interesting pod/e2e-api-b-aef61e-openshift-default-554c4fbd5f-t6k2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Oct 30 12:25:41.978928 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:41.978894 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-aef61e-openshift-default-554c4fbd5f-t6k2v" podUID="abd92788-b8eb-4998-b22a-0ecae4a3a81d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Oct 30 12:25:42.975249 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:42.975209 4908 patch_prober.go:29] interesting pod/e2e-api-b-aef61e-openshift-default-554c4fbd5f-t6k2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Oct 30 12:25:42.975555 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:42.975527 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-aef61e-openshift-default-554c4fbd5f-t6k2v" podUID="abd92788-b8eb-4998-b22a-0ecae4a3a81d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Oct 30 12:25:43.977957 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:43.977923 4908 patch_prober.go:29] interesting pod/e2e-api-b-aef61e-openshift-default-554c4fbd5f-t6k2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Oct 30 12:25:43.978530 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:43.978499 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-aef61e-openshift-default-554c4fbd5f-t6k2v" podUID="abd92788-b8eb-4998-b22a-0ecae4a3a81d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Oct 30 12:25:45.085277 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:45.085242 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-aef61e-openshift-default-554c4fbd5f-t6k2v" Oct 30 12:25:45.085779 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:45.085738 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-aef61e-openshift-default-554c4fbd5f-t6k2v" Oct 30 12:25:45.086966 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:45.086937 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-aef61e-openshift-default-554c4fbd5f-t6k2v" Oct 30 12:25:47.834591 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:47.834561 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-951ad1-566fb4ff59-wmdq5" Oct 30 12:25:48.846516 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:48.846488 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-951ad1-7d788c86d-tmc8p" Oct 30 12:25:49.684765 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:25:49.684731 4908 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vn6vv" containerName="extractor" Oct 30 12:25:51.236036 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:51.235996 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9wwfc"] Oct 30 12:25:55.051574 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:55.051533 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef80232e-6rnpp"] Oct 30 12:25:55.064187 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:55.064152 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef80232e-6rnpp"] Oct 30 12:25:55.607236 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:25:55.607192 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53e329b8-a4e4-4ced-8c81-d141b31076c3" path="/var/lib/kubelet/pods/53e329b8-a4e4-4ced-8c81-d141b31076c3/volumes" Oct 30 12:26:04.538808 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:26:04.538772 4908 scope.go:122] "RemoveContainer" containerID="05de84f1b1c4cd5cb4f3fba550466a3e586f3ac60fa195501ee7b613756b9503" Oct 30 12:26:04.550311 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:26:04.550288 4908 scope.go:122] "RemoveContainer" containerID="4434116c2ca20b1f86d50f26d6f26123623105046510f0c5c2eae5b4f45417ec" Oct 30 12:26:04.560446 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:26:04.560424 4908 scope.go:122] "RemoveContainer" containerID="4481120c8d3e36ddd0dd41a1a29ebaec0b22f0fd5d8ffa3a0214c1b26f040fe7" Oct 30 12:26:04.570012 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:26:04.569974 4908 scope.go:122] "RemoveContainer" containerID="015d2e03956594024194f11affa0ded6923d588575dc88e6a9740203b15b23d0" Oct 30 12:26:19.684685 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:26:19.684595 4908 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vn6vv" containerName="extractor" Oct 30 12:26:22.118669 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:26:22.118633 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-922643-9zzrf"] Oct 30 12:26:22.121877 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:26:22.121855 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-922643-9zzrf"] Oct 30 12:26:23.606954 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:26:23.606918 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8312b5a5-4d8d-4c54-b9fb-85228cd75c1f" path="/var/lib/kubelet/pods/8312b5a5-4d8d-4c54-b9fb-85228cd75c1f/volumes" Oct 30 12:26:24.836198 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:26:24.836150 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9wwfc"] Oct 30 12:26:30.018218 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:26:30.018179 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-aef61e-qlj8f"] Oct 30 12:26:30.018768 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:26:30.018754 4908 cpu_manager.go:414] "containerMap: removing container" podUID="8312b5a5-4d8d-4c54-b9fb-85228cd75c1f" containerName="revoke-keys" Oct 30 12:26:30.018819 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:26:30.018769 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8312b5a5-4d8d-4c54-b9fb-85228cd75c1f" containerName="revoke-keys" Oct 30 12:26:30.018914 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:26:30.018901 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="8312b5a5-4d8d-4c54-b9fb-85228cd75c1f" containerName="revoke-keys" Oct 30 12:26:30.022306 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:26:30.022290 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-aef61e-qlj8f" Oct 30 12:26:30.055572 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:26:30.055545 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-aef61e-qlj8f"] Oct 30 12:26:30.067129 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:26:30.067108 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 12:26:30.067283 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:26:30.067121 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 12:26:30.067283 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:26:30.067154 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s75hq\"" Oct 30 12:26:30.067473 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:26:30.067199 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 12:26:30.081902 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:26:30.081879 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97e125c7-4dbe-467b-82af-028e217f7bba-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-aef61e-qlj8f\" (UID: \"97e125c7-4dbe-467b-82af-028e217f7bba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-aef61e-qlj8f" Oct 30 12:26:30.082024 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:26:30.081927 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97e125c7-4dbe-467b-82af-028e217f7bba-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-aef61e-qlj8f\" (UID: \"97e125c7-4dbe-467b-82af-028e217f7bba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-aef61e-qlj8f" Oct 30 12:26:30.082173 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:26:30.082149 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tp752\" (UniqueName: \"kubernetes.io/projected/97e125c7-4dbe-467b-82af-028e217f7bba-kube-api-access-tp752\") pod \"maas-api-revoke-keys-e2e-api-b-aef61e-qlj8f\" (UID: \"97e125c7-4dbe-467b-82af-028e217f7bba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-aef61e-qlj8f" Oct 30 12:26:30.183753 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:26:30.183729 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tp752\" (UniqueName: \"kubernetes.io/projected/97e125c7-4dbe-467b-82af-028e217f7bba-kube-api-access-tp752\") pod \"maas-api-revoke-keys-e2e-api-b-aef61e-qlj8f\" (UID: \"97e125c7-4dbe-467b-82af-028e217f7bba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-aef61e-qlj8f" Oct 30 12:26:30.183918 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:26:30.183889 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97e125c7-4dbe-467b-82af-028e217f7bba-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-aef61e-qlj8f\" (UID: \"97e125c7-4dbe-467b-82af-028e217f7bba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-aef61e-qlj8f" Oct 30 12:26:30.183994 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:26:30.183930 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97e125c7-4dbe-467b-82af-028e217f7bba-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-aef61e-qlj8f\" (UID: \"97e125c7-4dbe-467b-82af-028e217f7bba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-aef61e-qlj8f" Oct 30 12:26:30.184772 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:26:30.184747 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97e125c7-4dbe-467b-82af-028e217f7bba-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-aef61e-qlj8f\" (UID: \"97e125c7-4dbe-467b-82af-028e217f7bba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-aef61e-qlj8f" Oct 30 12:26:30.186382 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:26:30.186357 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97e125c7-4dbe-467b-82af-028e217f7bba-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-aef61e-qlj8f\" (UID: \"97e125c7-4dbe-467b-82af-028e217f7bba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-aef61e-qlj8f" Oct 30 12:26:30.205668 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:26:30.205640 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tp752\" (UniqueName: \"kubernetes.io/projected/97e125c7-4dbe-467b-82af-028e217f7bba-kube-api-access-tp752\") pod \"maas-api-revoke-keys-e2e-api-b-aef61e-qlj8f\" (UID: \"97e125c7-4dbe-467b-82af-028e217f7bba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-aef61e-qlj8f" Oct 30 12:26:30.335591 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:26:30.335533 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-aef61e-qlj8f" Oct 30 12:26:30.638802 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:26:30.638776 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-aef61e-qlj8f"] Oct 30 12:26:30.641843 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:26:30.641804 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97e125c7_4dbe_467b_82af_028e217f7bba.slice/crio-4f94eec8d4df44949e09c2569e6e0f624f7d9c5d5f5baf860c49aa0d5c4a472a WatchSource:0}: Error finding container 4f94eec8d4df44949e09c2569e6e0f624f7d9c5d5f5baf860c49aa0d5c4a472a: Status 404 returned error can't find the container with id 4f94eec8d4df44949e09c2569e6e0f624f7d9c5d5f5baf860c49aa0d5c4a472a Oct 30 12:26:31.477008 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:26:31.476971 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-aef61e-qlj8f" event={"ID":"97e125c7-4dbe-467b-82af-028e217f7bba","Type":"ContainerStarted","Data":"8108a5ce3adfd825095c9d98c6daf227dc11bd586371ce2746bb559968f799f1"} Oct 30 12:26:31.477396 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:26:31.477067 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-aef61e-qlj8f" event={"ID":"97e125c7-4dbe-467b-82af-028e217f7bba","Type":"ContainerStarted","Data":"4f94eec8d4df44949e09c2569e6e0f624f7d9c5d5f5baf860c49aa0d5c4a472a"} Oct 30 12:26:31.536783 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:26:31.536732 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-aef61e-qlj8f" podStartSLOduration=2.5367202129999997 podStartE2EDuration="2.536720213s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:26:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:26:31.534231247 +0000 UTC m=+1948.048659993" watchObservedRunningTime="2026-10-30 12:26:31.536720213 +0000 UTC m=+1948.051148957" Oct 30 12:26:32.496850 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:26:32.495900 4908 generic.go:356] "Generic (PLEG): container finished" podID="97e125c7-4dbe-467b-82af-028e217f7bba" containerID="8108a5ce3adfd825095c9d98c6daf227dc11bd586371ce2746bb559968f799f1" exitCode=0 Oct 30 12:26:32.496850 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:26:32.496073 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-aef61e-qlj8f" event={"ID":"97e125c7-4dbe-467b-82af-028e217f7bba","Type":"ContainerDied","Data":"8108a5ce3adfd825095c9d98c6daf227dc11bd586371ce2746bb559968f799f1"} Oct 30 12:26:33.720670 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:26:33.720637 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-aef61e-qlj8f" Oct 30 12:26:33.819052 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:26:33.818962 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97e125c7-4dbe-467b-82af-028e217f7bba-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97e125c7-4dbe-467b-82af-028e217f7bba-maas-api-openshift-ca\") pod \"97e125c7-4dbe-467b-82af-028e217f7bba\" (UID: \"97e125c7-4dbe-467b-82af-028e217f7bba\") " Oct 30 12:26:33.819218 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:26:33.819085 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97e125c7-4dbe-467b-82af-028e217f7bba-kube-api-access-tp752\" (UniqueName: \"kubernetes.io/projected/97e125c7-4dbe-467b-82af-028e217f7bba-kube-api-access-tp752\") pod \"97e125c7-4dbe-467b-82af-028e217f7bba\" (UID: \"97e125c7-4dbe-467b-82af-028e217f7bba\") " Oct 30 12:26:33.819218 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:26:33.819183 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/97e125c7-4dbe-467b-82af-028e217f7bba-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97e125c7-4dbe-467b-82af-028e217f7bba-maas-api-xks-ca\") pod \"97e125c7-4dbe-467b-82af-028e217f7bba\" (UID: \"97e125c7-4dbe-467b-82af-028e217f7bba\") " Oct 30 12:26:33.819346 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:26:33.819305 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97e125c7-4dbe-467b-82af-028e217f7bba-maas-api-openshift-ca" pod "97e125c7-4dbe-467b-82af-028e217f7bba" (UID: "97e125c7-4dbe-467b-82af-028e217f7bba"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:26:33.819633 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:26:33.819611 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97e125c7-4dbe-467b-82af-028e217f7bba-maas-api-openshift-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:26:33.821214 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:26:33.821191 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97e125c7-4dbe-467b-82af-028e217f7bba-maas-api-xks-ca" pod "97e125c7-4dbe-467b-82af-028e217f7bba" (UID: "97e125c7-4dbe-467b-82af-028e217f7bba"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:26:33.821298 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:26:33.821230 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97e125c7-4dbe-467b-82af-028e217f7bba-kube-api-access-tp752" pod "97e125c7-4dbe-467b-82af-028e217f7bba" (UID: "97e125c7-4dbe-467b-82af-028e217f7bba"). InnerVolumeSpecName "kube-api-access-tp752". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:26:33.921137 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:26:33.921110 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tp752\" (UniqueName: \"kubernetes.io/projected/97e125c7-4dbe-467b-82af-028e217f7bba-kube-api-access-tp752\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:26:33.921250 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:26:33.921139 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97e125c7-4dbe-467b-82af-028e217f7bba-maas-api-xks-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:26:34.514966 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:26:34.514934 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-aef61e-qlj8f" Oct 30 12:26:34.514966 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:26:34.514957 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-aef61e-qlj8f" event={"ID":"97e125c7-4dbe-467b-82af-028e217f7bba","Type":"ContainerDied","Data":"4f94eec8d4df44949e09c2569e6e0f624f7d9c5d5f5baf860c49aa0d5c4a472a"} Oct 30 12:26:34.515226 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:26:34.515008 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4f94eec8d4df44949e09c2569e6e0f624f7d9c5d5f5baf860c49aa0d5c4a472a" Oct 30 12:26:48.007857 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:26:48.007423 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9wwfc"] Oct 30 12:26:49.690855 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:26:49.690767 4908 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vn6vv" containerName="extractor" Oct 30 12:26:52.039533 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:26:52.039497 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb601c3c-2w42x"] Oct 30 12:26:52.049515 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:26:52.049483 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb601c3c-2w42x"] Oct 30 12:26:53.049867 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:26:53.049839 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a476f81-4hcqx"] Oct 30 12:26:53.061001 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:26:53.060961 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a476f81-4hcqx"] Oct 30 12:26:53.606091 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:26:53.606060 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e1fbc3b-d000-48eb-b78e-b5b74d952dad" path="/var/lib/kubelet/pods/5e1fbc3b-d000-48eb-b78e-b5b74d952dad/volumes" Oct 30 12:26:53.606521 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:26:53.606506 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd0f25d5-fa88-41a5-88c7-c787799b40b3" path="/var/lib/kubelet/pods/dd0f25d5-fa88-41a5-88c7-c787799b40b3/volumes" Oct 30 12:27:04.628362 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:04.628330 4908 scope.go:122] "RemoveContainer" containerID="05d9d1e688bd95421cb442d5eae1a7345069394957ae6bc622bbcf8c78f53921" Oct 30 12:27:04.638914 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:04.638890 4908 scope.go:122] "RemoveContainer" containerID="17f8e796b274bbbe570e8163d5d003f6b963b3c0f8b47123d7a208011d25332f" Oct 30 12:27:04.649766 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:04.649734 4908 scope.go:122] "RemoveContainer" containerID="61c5899fd11c9c752ab32ac7a53fbdc834f5954fd4bbedd6e606f0fdbad41b56" Oct 30 12:27:09.615845 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:09.615807 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-431bf6-77bf6f85ff-fbcs2"] Oct 30 12:27:09.616354 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:09.616339 4908 cpu_manager.go:414] "containerMap: removing container" podUID="97e125c7-4dbe-467b-82af-028e217f7bba" containerName="revoke-keys" Oct 30 12:27:09.616400 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:09.616355 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97e125c7-4dbe-467b-82af-028e217f7bba" containerName="revoke-keys" Oct 30 12:27:09.616486 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:09.616477 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="97e125c7-4dbe-467b-82af-028e217f7bba" containerName="revoke-keys" Oct 30 12:27:09.621570 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:09.621550 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-431bf6-77bf6f85ff-fbcs2" Oct 30 12:27:09.639298 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:09.639278 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-431bf6-77bf6f85ff-fbcs2"] Oct 30 12:27:09.645724 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:09.645704 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-431bf6-dockercfg-ph8cp\"" Oct 30 12:27:09.646185 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:09.646165 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-431bf6\"" Oct 30 12:27:09.696489 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:09.696463 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctv7k\" (UniqueName: \"kubernetes.io/projected/c7b40c65-f1b9-4996-8ba2-23f34045cab5-kube-api-access-ctv7k\") pod \"payload-pre-processing-e2e-ipp-b-431bf6-77bf6f85ff-fbcs2\" (UID: \"c7b40c65-f1b9-4996-8ba2-23f34045cab5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-431bf6-77bf6f85ff-fbcs2" Oct 30 12:27:09.696684 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:09.696664 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7b40c65-f1b9-4996-8ba2-23f34045cab5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-431bf6-77bf6f85ff-fbcs2\" (UID: \"c7b40c65-f1b9-4996-8ba2-23f34045cab5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-431bf6-77bf6f85ff-fbcs2" Oct 30 12:27:09.797601 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:09.797555 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7b40c65-f1b9-4996-8ba2-23f34045cab5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-431bf6-77bf6f85ff-fbcs2\" (UID: \"c7b40c65-f1b9-4996-8ba2-23f34045cab5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-431bf6-77bf6f85ff-fbcs2" Oct 30 12:27:09.819091 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:09.819055 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctv7k\" (UniqueName: \"kubernetes.io/projected/c7b40c65-f1b9-4996-8ba2-23f34045cab5-kube-api-access-ctv7k\") pod \"payload-pre-processing-e2e-ipp-b-431bf6-77bf6f85ff-fbcs2\" (UID: \"c7b40c65-f1b9-4996-8ba2-23f34045cab5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-431bf6-77bf6f85ff-fbcs2" Oct 30 12:27:09.819697 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:09.798758 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7b40c65-f1b9-4996-8ba2-23f34045cab5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-431bf6-77bf6f85ff-fbcs2\" (UID: \"c7b40c65-f1b9-4996-8ba2-23f34045cab5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-431bf6-77bf6f85ff-fbcs2" Oct 30 12:27:09.857016 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:09.856978 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctv7k\" (UniqueName: \"kubernetes.io/projected/c7b40c65-f1b9-4996-8ba2-23f34045cab5-kube-api-access-ctv7k\") pod \"payload-pre-processing-e2e-ipp-b-431bf6-77bf6f85ff-fbcs2\" (UID: \"c7b40c65-f1b9-4996-8ba2-23f34045cab5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-431bf6-77bf6f85ff-fbcs2" Oct 30 12:27:09.880936 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:09.880843 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-431bf6-6f845d7bb8-rcc6w"] Oct 30 12:27:09.891210 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:09.891171 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-431bf6-6f845d7bb8-rcc6w" Oct 30 12:27:09.908323 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:09.908293 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-431bf6-6f845d7bb8-rcc6w"] Oct 30 12:27:09.935261 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:09.935230 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-431bf6-77bf6f85ff-fbcs2" Oct 30 12:27:10.022890 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:10.022783 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nslz2\" (UniqueName: \"kubernetes.io/projected/3104798d-8219-4972-b98f-2e30238ebbcc-kube-api-access-nslz2\") pod \"payload-processing-e2e-ipp-b-431bf6-6f845d7bb8-rcc6w\" (UID: \"3104798d-8219-4972-b98f-2e30238ebbcc\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-431bf6-6f845d7bb8-rcc6w" Oct 30 12:27:10.023107 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:10.023083 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3104798d-8219-4972-b98f-2e30238ebbcc-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-431bf6-6f845d7bb8-rcc6w\" (UID: \"3104798d-8219-4972-b98f-2e30238ebbcc\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-431bf6-6f845d7bb8-rcc6w" Oct 30 12:27:10.124960 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:10.124927 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3104798d-8219-4972-b98f-2e30238ebbcc-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-431bf6-6f845d7bb8-rcc6w\" (UID: \"3104798d-8219-4972-b98f-2e30238ebbcc\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-431bf6-6f845d7bb8-rcc6w" Oct 30 12:27:10.125104 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:10.125027 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nslz2\" (UniqueName: \"kubernetes.io/projected/3104798d-8219-4972-b98f-2e30238ebbcc-kube-api-access-nslz2\") pod \"payload-processing-e2e-ipp-b-431bf6-6f845d7bb8-rcc6w\" (UID: \"3104798d-8219-4972-b98f-2e30238ebbcc\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-431bf6-6f845d7bb8-rcc6w" Oct 30 12:27:10.125634 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:10.125613 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3104798d-8219-4972-b98f-2e30238ebbcc-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-431bf6-6f845d7bb8-rcc6w\" (UID: \"3104798d-8219-4972-b98f-2e30238ebbcc\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-431bf6-6f845d7bb8-rcc6w" Oct 30 12:27:10.147263 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:10.147204 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nslz2\" (UniqueName: \"kubernetes.io/projected/3104798d-8219-4972-b98f-2e30238ebbcc-kube-api-access-nslz2\") pod \"payload-processing-e2e-ipp-b-431bf6-6f845d7bb8-rcc6w\" (UID: \"3104798d-8219-4972-b98f-2e30238ebbcc\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-431bf6-6f845d7bb8-rcc6w" Oct 30 12:27:10.225244 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:10.225216 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-431bf6-6f845d7bb8-rcc6w" Oct 30 12:27:10.256382 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:10.256357 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-431bf6-77bf6f85ff-fbcs2"] Oct 30 12:27:10.257110 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:27:10.257086 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7b40c65_f1b9_4996_8ba2_23f34045cab5.slice/crio-51e0ad972288a9439a69358ed8267174cebe49d73550bfc6354c42915f6e0c9f WatchSource:0}: Error finding container 51e0ad972288a9439a69358ed8267174cebe49d73550bfc6354c42915f6e0c9f: Status 404 returned error can't find the container with id 51e0ad972288a9439a69358ed8267174cebe49d73550bfc6354c42915f6e0c9f Oct 30 12:27:10.558488 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:10.558451 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-431bf6-6f845d7bb8-rcc6w"] Oct 30 12:27:10.563197 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:27:10.563164 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3104798d_8219_4972_b98f_2e30238ebbcc.slice/crio-1f105faa23f329240d179c0a16cdb65336500249a2d09057aad1cd9eee34cf64 WatchSource:0}: Error finding container 1f105faa23f329240d179c0a16cdb65336500249a2d09057aad1cd9eee34cf64: Status 404 returned error can't find the container with id 1f105faa23f329240d179c0a16cdb65336500249a2d09057aad1cd9eee34cf64 Oct 30 12:27:10.955784 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:10.955682 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-431bf6-6f845d7bb8-rcc6w" event={"ID":"3104798d-8219-4972-b98f-2e30238ebbcc","Type":"ContainerStarted","Data":"3a51cd4f48dd6e1507d326847bf5a5599e46a0070b5f2685ebab413a1b8352e1"} Oct 30 12:27:10.955784 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:10.955749 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-431bf6-6f845d7bb8-rcc6w" event={"ID":"3104798d-8219-4972-b98f-2e30238ebbcc","Type":"ContainerStarted","Data":"1f105faa23f329240d179c0a16cdb65336500249a2d09057aad1cd9eee34cf64"} Oct 30 12:27:10.955784 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:10.955778 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-431bf6-6f845d7bb8-rcc6w" Oct 30 12:27:10.957584 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:10.957549 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-431bf6-77bf6f85ff-fbcs2" event={"ID":"c7b40c65-f1b9-4996-8ba2-23f34045cab5","Type":"ContainerStarted","Data":"80905f0efb6ec840e7806d027183fdda10473a801324f5304329652de2d22961"} Oct 30 12:27:10.957690 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:10.957597 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-431bf6-77bf6f85ff-fbcs2" event={"ID":"c7b40c65-f1b9-4996-8ba2-23f34045cab5","Type":"ContainerStarted","Data":"51e0ad972288a9439a69358ed8267174cebe49d73550bfc6354c42915f6e0c9f"} Oct 30 12:27:10.957690 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:10.957624 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-431bf6-77bf6f85ff-fbcs2" Oct 30 12:27:10.999046 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:10.998987 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-431bf6-6f845d7bb8-rcc6w" podStartSLOduration=1.998970687 podStartE2EDuration="1.998970687s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:27:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:27:10.996170887 +0000 UTC m=+1987.510599681" watchObservedRunningTime="2026-10-30 12:27:10.998970687 +0000 UTC m=+1987.513399432" Oct 30 12:27:11.038920 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:11.038873 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-431bf6-77bf6f85ff-fbcs2" podStartSLOduration=2.038856745 podStartE2EDuration="2.038856745s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:27:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:27:11.038145702 +0000 UTC m=+1987.552574438" watchObservedRunningTime="2026-10-30 12:27:11.038856745 +0000 UTC m=+1987.553285508" Oct 30 12:27:13.460990 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:13.460947 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-aef61e-qlj8f"] Oct 30 12:27:13.470952 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:13.470911 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-aef61e-qlj8f"] Oct 30 12:27:13.607492 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:13.607461 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97e125c7-4dbe-467b-82af-028e217f7bba" path="/var/lib/kubelet/pods/97e125c7-4dbe-467b-82af-028e217f7bba/volumes" Oct 30 12:27:19.683817 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:27:19.683780 4908 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vn6vv" containerName="extractor" Oct 30 12:27:20.515238 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:20.515200 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-aef61e-openshift-default-554c4fbd5f-t6k2v"] Oct 30 12:27:20.515504 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:20.515461 4908 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-aef61e-openshift-default-554c4fbd5f-t6k2v" podUID="abd92788-b8eb-4998-b22a-0ecae4a3a81d" containerName="istio-proxy" containerID="cri-o://fba29ea422c3624ed5423da382a3419476a57d13d01741d8b7a0669e791da9c9" gracePeriod=30 Oct 30 12:27:21.967668 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:21.967637 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-431bf6-6f845d7bb8-rcc6w" Oct 30 12:27:21.968156 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:21.967725 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-431bf6-77bf6f85ff-fbcs2" Oct 30 12:27:23.821706 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:23.821664 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9wwfc"] Oct 30 12:27:25.998899 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:25.998633 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-aef61e-openshift-default-554c4fbd5f-t6k2v" Oct 30 12:27:26.095271 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:26.093988 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abd92788-b8eb-4998-b22a-0ecae4a3a81d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abd92788-b8eb-4998-b22a-0ecae4a3a81d-istio-data\") pod \"abd92788-b8eb-4998-b22a-0ecae4a3a81d\" (UID: \"abd92788-b8eb-4998-b22a-0ecae4a3a81d\") " Oct 30 12:27:26.095271 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:26.094074 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abd92788-b8eb-4998-b22a-0ecae4a3a81d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abd92788-b8eb-4998-b22a-0ecae4a3a81d-istio-envoy\") pod \"abd92788-b8eb-4998-b22a-0ecae4a3a81d\" (UID: \"abd92788-b8eb-4998-b22a-0ecae4a3a81d\") " Oct 30 12:27:26.095271 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:26.094123 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abd92788-b8eb-4998-b22a-0ecae4a3a81d-istio-token\" (UniqueName: \"kubernetes.io/projected/abd92788-b8eb-4998-b22a-0ecae4a3a81d-istio-token\") pod \"abd92788-b8eb-4998-b22a-0ecae4a3a81d\" (UID: \"abd92788-b8eb-4998-b22a-0ecae4a3a81d\") " Oct 30 12:27:26.095271 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:26.094156 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abd92788-b8eb-4998-b22a-0ecae4a3a81d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abd92788-b8eb-4998-b22a-0ecae4a3a81d-workload-certs\") pod \"abd92788-b8eb-4998-b22a-0ecae4a3a81d\" (UID: \"abd92788-b8eb-4998-b22a-0ecae4a3a81d\") " Oct 30 12:27:26.095271 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:26.094262 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abd92788-b8eb-4998-b22a-0ecae4a3a81d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abd92788-b8eb-4998-b22a-0ecae4a3a81d-workload-socket\") pod \"abd92788-b8eb-4998-b22a-0ecae4a3a81d\" (UID: \"abd92788-b8eb-4998-b22a-0ecae4a3a81d\") " Oct 30 12:27:26.095271 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:26.094297 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abd92788-b8eb-4998-b22a-0ecae4a3a81d-kube-api-access-pfjwq\" (UniqueName: \"kubernetes.io/projected/abd92788-b8eb-4998-b22a-0ecae4a3a81d-kube-api-access-pfjwq\") pod \"abd92788-b8eb-4998-b22a-0ecae4a3a81d\" (UID: \"abd92788-b8eb-4998-b22a-0ecae4a3a81d\") " Oct 30 12:27:26.095271 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:26.094402 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/abd92788-b8eb-4998-b22a-0ecae4a3a81d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abd92788-b8eb-4998-b22a-0ecae4a3a81d-istio-podinfo\") pod \"abd92788-b8eb-4998-b22a-0ecae4a3a81d\" (UID: \"abd92788-b8eb-4998-b22a-0ecae4a3a81d\") " Oct 30 12:27:26.095271 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:26.094445 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/abd92788-b8eb-4998-b22a-0ecae4a3a81d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abd92788-b8eb-4998-b22a-0ecae4a3a81d-istiod-ca-cert\") pod \"abd92788-b8eb-4998-b22a-0ecae4a3a81d\" (UID: \"abd92788-b8eb-4998-b22a-0ecae4a3a81d\") " Oct 30 12:27:26.095271 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:26.094494 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abd92788-b8eb-4998-b22a-0ecae4a3a81d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abd92788-b8eb-4998-b22a-0ecae4a3a81d-credential-socket\") pod \"abd92788-b8eb-4998-b22a-0ecae4a3a81d\" (UID: \"abd92788-b8eb-4998-b22a-0ecae4a3a81d\") " Oct 30 12:27:26.095943 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:26.095393 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abd92788-b8eb-4998-b22a-0ecae4a3a81d-workload-certs" pod "abd92788-b8eb-4998-b22a-0ecae4a3a81d" (UID: "abd92788-b8eb-4998-b22a-0ecae4a3a81d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:27:26.095943 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:26.095591 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abd92788-b8eb-4998-b22a-0ecae4a3a81d-istio-data" pod "abd92788-b8eb-4998-b22a-0ecae4a3a81d" (UID: "abd92788-b8eb-4998-b22a-0ecae4a3a81d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:27:26.096059 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:26.095943 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abd92788-b8eb-4998-b22a-0ecae4a3a81d-credential-socket" pod "abd92788-b8eb-4998-b22a-0ecae4a3a81d" (UID: "abd92788-b8eb-4998-b22a-0ecae4a3a81d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:27:26.096351 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:26.096158 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abd92788-b8eb-4998-b22a-0ecae4a3a81d-workload-socket" pod "abd92788-b8eb-4998-b22a-0ecae4a3a81d" (UID: "abd92788-b8eb-4998-b22a-0ecae4a3a81d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:27:26.096533 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:26.096478 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abd92788-b8eb-4998-b22a-0ecae4a3a81d-istiod-ca-cert" pod "abd92788-b8eb-4998-b22a-0ecae4a3a81d" (UID: "abd92788-b8eb-4998-b22a-0ecae4a3a81d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:27:26.099295 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:26.099268 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abd92788-b8eb-4998-b22a-0ecae4a3a81d-istio-token" pod "abd92788-b8eb-4998-b22a-0ecae4a3a81d" (UID: "abd92788-b8eb-4998-b22a-0ecae4a3a81d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:27:26.101933 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:26.101910 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abd92788-b8eb-4998-b22a-0ecae4a3a81d-kube-api-access-pfjwq" pod "abd92788-b8eb-4998-b22a-0ecae4a3a81d" (UID: "abd92788-b8eb-4998-b22a-0ecae4a3a81d"). InnerVolumeSpecName "kube-api-access-pfjwq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:27:26.103723 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:26.103704 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abd92788-b8eb-4998-b22a-0ecae4a3a81d-istio-envoy" pod "abd92788-b8eb-4998-b22a-0ecae4a3a81d" (UID: "abd92788-b8eb-4998-b22a-0ecae4a3a81d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:27:26.103850 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:26.103723 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/abd92788-b8eb-4998-b22a-0ecae4a3a81d-istio-podinfo" pod "abd92788-b8eb-4998-b22a-0ecae4a3a81d" (UID: "abd92788-b8eb-4998-b22a-0ecae4a3a81d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 12:27:26.196292 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:26.196253 4908 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abd92788-b8eb-4998-b22a-0ecae4a3a81d-istio-token\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:27:26.196292 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:26.196289 4908 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abd92788-b8eb-4998-b22a-0ecae4a3a81d-workload-certs\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:27:26.196589 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:26.196304 4908 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abd92788-b8eb-4998-b22a-0ecae4a3a81d-workload-socket\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:27:26.196589 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:26.196318 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pfjwq\" (UniqueName: \"kubernetes.io/projected/abd92788-b8eb-4998-b22a-0ecae4a3a81d-kube-api-access-pfjwq\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:27:26.196589 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:26.196332 4908 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abd92788-b8eb-4998-b22a-0ecae4a3a81d-istio-podinfo\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:27:26.196589 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:26.196345 4908 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abd92788-b8eb-4998-b22a-0ecae4a3a81d-istiod-ca-cert\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:27:26.196589 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:26.196358 4908 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abd92788-b8eb-4998-b22a-0ecae4a3a81d-credential-socket\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:27:26.196589 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:26.196382 4908 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abd92788-b8eb-4998-b22a-0ecae4a3a81d-istio-data\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:27:26.196589 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:26.196396 4908 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abd92788-b8eb-4998-b22a-0ecae4a3a81d-istio-envoy\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:27:26.201203 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:26.201171 4908 generic.go:356] "Generic (PLEG): container finished" podID="abd92788-b8eb-4998-b22a-0ecae4a3a81d" containerID="fba29ea422c3624ed5423da382a3419476a57d13d01741d8b7a0669e791da9c9" exitCode=0 Oct 30 12:27:26.201356 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:26.201214 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-aef61e-openshift-default-554c4fbd5f-t6k2v" Oct 30 12:27:26.201356 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:26.201221 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-aef61e-openshift-default-554c4fbd5f-t6k2v" event={"ID":"abd92788-b8eb-4998-b22a-0ecae4a3a81d","Type":"ContainerDied","Data":"fba29ea422c3624ed5423da382a3419476a57d13d01741d8b7a0669e791da9c9"} Oct 30 12:27:26.201356 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:26.201287 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-aef61e-openshift-default-554c4fbd5f-t6k2v" event={"ID":"abd92788-b8eb-4998-b22a-0ecae4a3a81d","Type":"ContainerDied","Data":"584f7263bb6440899dc2649bc97960e86db28a34c6555075b8d1fda7d8aee945"} Oct 30 12:27:26.201356 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:26.201349 4908 scope.go:122] "RemoveContainer" containerID="fba29ea422c3624ed5423da382a3419476a57d13d01741d8b7a0669e791da9c9" Oct 30 12:27:26.216913 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:26.216895 4908 scope.go:122] "RemoveContainer" containerID="804a9abea774a102a224c5637f7b80700ce2809fdbb641118c49325f9bf9d925" Oct 30 12:27:26.226757 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:26.226719 4908 scope.go:122] "RemoveContainer" containerID="fba29ea422c3624ed5423da382a3419476a57d13d01741d8b7a0669e791da9c9" Oct 30 12:27:26.227127 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:27:26.227099 4908 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fba29ea422c3624ed5423da382a3419476a57d13d01741d8b7a0669e791da9c9\": container with ID starting with fba29ea422c3624ed5423da382a3419476a57d13d01741d8b7a0669e791da9c9 not found: ID does not exist" containerID="fba29ea422c3624ed5423da382a3419476a57d13d01741d8b7a0669e791da9c9" Oct 30 12:27:26.227205 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:26.227140 4908 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fba29ea422c3624ed5423da382a3419476a57d13d01741d8b7a0669e791da9c9"} err="failed to get container status \"fba29ea422c3624ed5423da382a3419476a57d13d01741d8b7a0669e791da9c9\": rpc error: code = NotFound desc = could not find container \"fba29ea422c3624ed5423da382a3419476a57d13d01741d8b7a0669e791da9c9\": container with ID starting with fba29ea422c3624ed5423da382a3419476a57d13d01741d8b7a0669e791da9c9 not found: ID does not exist" Oct 30 12:27:26.227205 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:26.227164 4908 scope.go:122] "RemoveContainer" containerID="804a9abea774a102a224c5637f7b80700ce2809fdbb641118c49325f9bf9d925" Oct 30 12:27:26.227463 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:27:26.227436 4908 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"804a9abea774a102a224c5637f7b80700ce2809fdbb641118c49325f9bf9d925\": container with ID starting with 804a9abea774a102a224c5637f7b80700ce2809fdbb641118c49325f9bf9d925 not found: ID does not exist" containerID="804a9abea774a102a224c5637f7b80700ce2809fdbb641118c49325f9bf9d925" Oct 30 12:27:26.227528 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:26.227470 4908 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"804a9abea774a102a224c5637f7b80700ce2809fdbb641118c49325f9bf9d925"} err="failed to get container status \"804a9abea774a102a224c5637f7b80700ce2809fdbb641118c49325f9bf9d925\": rpc error: code = NotFound desc = could not find container \"804a9abea774a102a224c5637f7b80700ce2809fdbb641118c49325f9bf9d925\": container with ID starting with 804a9abea774a102a224c5637f7b80700ce2809fdbb641118c49325f9bf9d925 not found: ID does not exist" Oct 30 12:27:26.252230 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:26.252207 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-aef61e-openshift-default-554c4fbd5f-t6k2v"] Oct 30 12:27:26.268238 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:26.268203 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-aef61e-openshift-default-554c4fbd5f-t6k2v"] Oct 30 12:27:27.609026 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:27.608995 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="abd92788-b8eb-4998-b22a-0ecae4a3a81d" path="/var/lib/kubelet/pods/abd92788-b8eb-4998-b22a-0ecae4a3a81d/volumes" Oct 30 12:27:28.447397 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:28.447355 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7e2db2-hkczk"] Oct 30 12:27:28.448358 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:28.448248 4908 cpu_manager.go:414] "containerMap: removing container" podUID="abd92788-b8eb-4998-b22a-0ecae4a3a81d" containerName="istio-proxy" Oct 30 12:27:28.448358 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:28.448271 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abd92788-b8eb-4998-b22a-0ecae4a3a81d" containerName="istio-proxy" Oct 30 12:27:28.448543 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:28.448469 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="abd92788-b8eb-4998-b22a-0ecae4a3a81d" containerName="istio-proxy" Oct 30 12:27:28.448543 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:28.448487 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="abd92788-b8eb-4998-b22a-0ecae4a3a81d" containerName="istio-proxy" Oct 30 12:27:28.452384 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:28.452352 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7e2db2-hkczk" Oct 30 12:27:28.555902 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:28.485086 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7e2db2-hkczk"] Oct 30 12:27:28.573964 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:28.518183 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s75hq\"" Oct 30 12:27:28.644844 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:28.518240 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 12:27:28.644844 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:28.518271 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 12:27:28.644844 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:28.518331 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 12:27:28.644844 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:28.548193 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06ee92ce-63a6-4bad-b7c8-9da2252a175b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-7e2db2-hkczk\" (UID: \"06ee92ce-63a6-4bad-b7c8-9da2252a175b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7e2db2-hkczk" Oct 30 12:27:28.644844 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:28.636804 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dl7tw\" (UniqueName: \"kubernetes.io/projected/06ee92ce-63a6-4bad-b7c8-9da2252a175b-kube-api-access-dl7tw\") pod \"maas-api-revoke-keys-e2e-ipp-a-7e2db2-hkczk\" (UID: \"06ee92ce-63a6-4bad-b7c8-9da2252a175b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7e2db2-hkczk" Oct 30 12:27:28.644844 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:28.636861 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06ee92ce-63a6-4bad-b7c8-9da2252a175b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-7e2db2-hkczk\" (UID: \"06ee92ce-63a6-4bad-b7c8-9da2252a175b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7e2db2-hkczk" Oct 30 12:27:28.737735 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:28.737697 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dl7tw\" (UniqueName: \"kubernetes.io/projected/06ee92ce-63a6-4bad-b7c8-9da2252a175b-kube-api-access-dl7tw\") pod \"maas-api-revoke-keys-e2e-ipp-a-7e2db2-hkczk\" (UID: \"06ee92ce-63a6-4bad-b7c8-9da2252a175b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7e2db2-hkczk" Oct 30 12:27:28.737947 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:28.737749 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06ee92ce-63a6-4bad-b7c8-9da2252a175b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-7e2db2-hkczk\" (UID: \"06ee92ce-63a6-4bad-b7c8-9da2252a175b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7e2db2-hkczk" Oct 30 12:27:28.738116 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:28.738060 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06ee92ce-63a6-4bad-b7c8-9da2252a175b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-7e2db2-hkczk\" (UID: \"06ee92ce-63a6-4bad-b7c8-9da2252a175b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7e2db2-hkczk" Oct 30 12:27:28.738523 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:28.738505 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06ee92ce-63a6-4bad-b7c8-9da2252a175b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-7e2db2-hkczk\" (UID: \"06ee92ce-63a6-4bad-b7c8-9da2252a175b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7e2db2-hkczk" Oct 30 12:27:28.740390 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:28.740369 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06ee92ce-63a6-4bad-b7c8-9da2252a175b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-7e2db2-hkczk\" (UID: \"06ee92ce-63a6-4bad-b7c8-9da2252a175b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7e2db2-hkczk" Oct 30 12:27:28.760130 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:28.760098 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dl7tw\" (UniqueName: \"kubernetes.io/projected/06ee92ce-63a6-4bad-b7c8-9da2252a175b-kube-api-access-dl7tw\") pod \"maas-api-revoke-keys-e2e-ipp-a-7e2db2-hkczk\" (UID: \"06ee92ce-63a6-4bad-b7c8-9da2252a175b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7e2db2-hkczk" Oct 30 12:27:28.946802 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:28.946771 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7e2db2-hkczk" Oct 30 12:27:29.300953 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:29.300855 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7e2db2-hkczk"] Oct 30 12:27:29.303555 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:27:29.303527 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06ee92ce_63a6_4bad_b7c8_9da2252a175b.slice/crio-4b604ec5bb85257bad57ce225a3a4ee78ed270acb2a3c50fa06fd533e3816691 WatchSource:0}: Error finding container 4b604ec5bb85257bad57ce225a3a4ee78ed270acb2a3c50fa06fd533e3816691: Status 404 returned error can't find the container with id 4b604ec5bb85257bad57ce225a3a4ee78ed270acb2a3c50fa06fd533e3816691 Oct 30 12:27:30.241575 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:30.241538 4908 generic.go:356] "Generic (PLEG): container finished" podID="06ee92ce-63a6-4bad-b7c8-9da2252a175b" containerID="5113b55b6db549ba7f8b132585f80665e0806f06f066153003d4e26e12d6f037" exitCode=0 Oct 30 12:27:30.241934 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:30.241616 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7e2db2-hkczk" event={"ID":"06ee92ce-63a6-4bad-b7c8-9da2252a175b","Type":"ContainerDied","Data":"5113b55b6db549ba7f8b132585f80665e0806f06f066153003d4e26e12d6f037"} Oct 30 12:27:30.241934 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:30.241698 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7e2db2-hkczk" event={"ID":"06ee92ce-63a6-4bad-b7c8-9da2252a175b","Type":"ContainerStarted","Data":"4b604ec5bb85257bad57ce225a3a4ee78ed270acb2a3c50fa06fd533e3816691"} Oct 30 12:27:31.432024 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:31.431992 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7e2db2-hkczk" Oct 30 12:27:31.481537 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:31.481503 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06ee92ce-63a6-4bad-b7c8-9da2252a175b-kube-api-access-dl7tw\" (UniqueName: \"kubernetes.io/projected/06ee92ce-63a6-4bad-b7c8-9da2252a175b-kube-api-access-dl7tw\") pod \"06ee92ce-63a6-4bad-b7c8-9da2252a175b\" (UID: \"06ee92ce-63a6-4bad-b7c8-9da2252a175b\") " Oct 30 12:27:31.481700 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:31.481635 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/06ee92ce-63a6-4bad-b7c8-9da2252a175b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06ee92ce-63a6-4bad-b7c8-9da2252a175b-maas-api-xks-ca\") pod \"06ee92ce-63a6-4bad-b7c8-9da2252a175b\" (UID: \"06ee92ce-63a6-4bad-b7c8-9da2252a175b\") " Oct 30 12:27:31.481809 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:31.481789 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06ee92ce-63a6-4bad-b7c8-9da2252a175b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06ee92ce-63a6-4bad-b7c8-9da2252a175b-maas-api-openshift-ca\") pod \"06ee92ce-63a6-4bad-b7c8-9da2252a175b\" (UID: \"06ee92ce-63a6-4bad-b7c8-9da2252a175b\") " Oct 30 12:27:31.483468 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:31.483369 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06ee92ce-63a6-4bad-b7c8-9da2252a175b-maas-api-openshift-ca" pod "06ee92ce-63a6-4bad-b7c8-9da2252a175b" (UID: "06ee92ce-63a6-4bad-b7c8-9da2252a175b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:27:31.486559 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:31.486531 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06ee92ce-63a6-4bad-b7c8-9da2252a175b-maas-api-xks-ca" pod "06ee92ce-63a6-4bad-b7c8-9da2252a175b" (UID: "06ee92ce-63a6-4bad-b7c8-9da2252a175b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:27:31.486823 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:31.486796 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06ee92ce-63a6-4bad-b7c8-9da2252a175b-kube-api-access-dl7tw" pod "06ee92ce-63a6-4bad-b7c8-9da2252a175b" (UID: "06ee92ce-63a6-4bad-b7c8-9da2252a175b"). InnerVolumeSpecName "kube-api-access-dl7tw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:27:31.583603 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:31.583522 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06ee92ce-63a6-4bad-b7c8-9da2252a175b-maas-api-xks-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:27:31.583603 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:31.583548 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06ee92ce-63a6-4bad-b7c8-9da2252a175b-maas-api-openshift-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:27:31.583603 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:31.583558 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dl7tw\" (UniqueName: \"kubernetes.io/projected/06ee92ce-63a6-4bad-b7c8-9da2252a175b-kube-api-access-dl7tw\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:27:32.258305 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:32.258275 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7e2db2-hkczk" Oct 30 12:27:32.258493 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:32.258309 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7e2db2-hkczk" event={"ID":"06ee92ce-63a6-4bad-b7c8-9da2252a175b","Type":"ContainerDied","Data":"4b604ec5bb85257bad57ce225a3a4ee78ed270acb2a3c50fa06fd533e3816691"} Oct 30 12:27:32.258493 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:32.258367 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4b604ec5bb85257bad57ce225a3a4ee78ed270acb2a3c50fa06fd533e3816691" Oct 30 12:27:34.784785 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:34.784747 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-951ad1-q75kc"] Oct 30 12:27:34.786727 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:34.786687 4908 cpu_manager.go:414] "containerMap: removing container" podUID="abd92788-b8eb-4998-b22a-0ecae4a3a81d" containerName="istio-proxy" Oct 30 12:27:34.786727 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:34.786733 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abd92788-b8eb-4998-b22a-0ecae4a3a81d" containerName="istio-proxy" Oct 30 12:27:34.786935 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:34.786851 4908 cpu_manager.go:414] "containerMap: removing container" podUID="06ee92ce-63a6-4bad-b7c8-9da2252a175b" containerName="revoke-keys" Oct 30 12:27:34.786935 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:34.786864 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06ee92ce-63a6-4bad-b7c8-9da2252a175b" containerName="revoke-keys" Oct 30 12:27:34.787157 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:34.787135 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="06ee92ce-63a6-4bad-b7c8-9da2252a175b" containerName="revoke-keys" Oct 30 12:27:34.817696 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:34.817668 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-951ad1-q75kc" Oct 30 12:27:34.848852 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:34.836914 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s75hq\"" Oct 30 12:27:34.885990 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:34.852578 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 12:27:34.885990 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:34.852797 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 12:27:34.885990 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:34.852946 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 12:27:34.937918 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:34.937767 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-951ad1-q75kc"] Oct 30 12:27:34.959924 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:34.959892 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvh9x\" (UniqueName: \"kubernetes.io/projected/1a82c7fc-062d-42ff-90ba-868c74f5f97d-kube-api-access-fvh9x\") pod \"maas-api-revoke-keys-e2e-ipp-a-951ad1-q75kc\" (UID: \"1a82c7fc-062d-42ff-90ba-868c74f5f97d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-951ad1-q75kc" Oct 30 12:27:34.960121 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:34.959944 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a82c7fc-062d-42ff-90ba-868c74f5f97d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-951ad1-q75kc\" (UID: \"1a82c7fc-062d-42ff-90ba-868c74f5f97d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-951ad1-q75kc" Oct 30 12:27:34.960192 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:34.960134 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a82c7fc-062d-42ff-90ba-868c74f5f97d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-951ad1-q75kc\" (UID: \"1a82c7fc-062d-42ff-90ba-868c74f5f97d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-951ad1-q75kc" Oct 30 12:27:35.061750 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:35.061678 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a82c7fc-062d-42ff-90ba-868c74f5f97d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-951ad1-q75kc\" (UID: \"1a82c7fc-062d-42ff-90ba-868c74f5f97d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-951ad1-q75kc" Oct 30 12:27:35.061929 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:35.061766 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvh9x\" (UniqueName: \"kubernetes.io/projected/1a82c7fc-062d-42ff-90ba-868c74f5f97d-kube-api-access-fvh9x\") pod \"maas-api-revoke-keys-e2e-ipp-a-951ad1-q75kc\" (UID: \"1a82c7fc-062d-42ff-90ba-868c74f5f97d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-951ad1-q75kc" Oct 30 12:27:35.061929 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:35.061792 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a82c7fc-062d-42ff-90ba-868c74f5f97d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-951ad1-q75kc\" (UID: \"1a82c7fc-062d-42ff-90ba-868c74f5f97d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-951ad1-q75kc" Oct 30 12:27:35.063295 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:35.063265 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a82c7fc-062d-42ff-90ba-868c74f5f97d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-951ad1-q75kc\" (UID: \"1a82c7fc-062d-42ff-90ba-868c74f5f97d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-951ad1-q75kc" Oct 30 12:27:35.064701 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:35.064676 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a82c7fc-062d-42ff-90ba-868c74f5f97d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-951ad1-q75kc\" (UID: \"1a82c7fc-062d-42ff-90ba-868c74f5f97d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-951ad1-q75kc" Oct 30 12:27:35.083730 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:35.083699 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvh9x\" (UniqueName: \"kubernetes.io/projected/1a82c7fc-062d-42ff-90ba-868c74f5f97d-kube-api-access-fvh9x\") pod \"maas-api-revoke-keys-e2e-ipp-a-951ad1-q75kc\" (UID: \"1a82c7fc-062d-42ff-90ba-868c74f5f97d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-951ad1-q75kc" Oct 30 12:27:35.270965 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:35.270932 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-951ad1-q75kc" Oct 30 12:27:35.586424 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:35.586395 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-951ad1-q75kc"] Oct 30 12:27:35.588403 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:27:35.588373 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a82c7fc_062d_42ff_90ba_868c74f5f97d.slice/crio-00548a9c44c086b81a5f44fe57e88c04f77e2a3d2cb5ecceadf6053de03bb70c WatchSource:0}: Error finding container 00548a9c44c086b81a5f44fe57e88c04f77e2a3d2cb5ecceadf6053de03bb70c: Status 404 returned error can't find the container with id 00548a9c44c086b81a5f44fe57e88c04f77e2a3d2cb5ecceadf6053de03bb70c Oct 30 12:27:36.289928 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:36.289888 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-951ad1-q75kc" event={"ID":"1a82c7fc-062d-42ff-90ba-868c74f5f97d","Type":"ContainerStarted","Data":"d5c465473f67485ed98ced3fdacdaa9f07b2b028baba5d69a2d5042af9ee7018"} Oct 30 12:27:36.290402 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:36.289946 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-951ad1-q75kc" event={"ID":"1a82c7fc-062d-42ff-90ba-868c74f5f97d","Type":"ContainerStarted","Data":"00548a9c44c086b81a5f44fe57e88c04f77e2a3d2cb5ecceadf6053de03bb70c"} Oct 30 12:27:36.332592 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:36.332536 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-951ad1-q75kc" podStartSLOduration=2.332523112 podStartE2EDuration="2.332523112s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:27:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:27:36.329790593 +0000 UTC m=+2012.844219340" watchObservedRunningTime="2026-10-30 12:27:36.332523112 +0000 UTC m=+2012.846951856" Oct 30 12:27:37.302321 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:37.302283 4908 generic.go:356] "Generic (PLEG): container finished" podID="1a82c7fc-062d-42ff-90ba-868c74f5f97d" containerID="d5c465473f67485ed98ced3fdacdaa9f07b2b028baba5d69a2d5042af9ee7018" exitCode=0 Oct 30 12:27:37.302724 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:37.302354 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-951ad1-q75kc" event={"ID":"1a82c7fc-062d-42ff-90ba-868c74f5f97d","Type":"ContainerDied","Data":"d5c465473f67485ed98ced3fdacdaa9f07b2b028baba5d69a2d5042af9ee7018"} Oct 30 12:27:38.460878 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:38.460854 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-951ad1-q75kc" Oct 30 12:27:38.530713 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:38.530680 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a82c7fc-062d-42ff-90ba-868c74f5f97d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a82c7fc-062d-42ff-90ba-868c74f5f97d-maas-api-openshift-ca\") pod \"1a82c7fc-062d-42ff-90ba-868c74f5f97d\" (UID: \"1a82c7fc-062d-42ff-90ba-868c74f5f97d\") " Oct 30 12:27:38.530910 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:38.530876 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a82c7fc-062d-42ff-90ba-868c74f5f97d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a82c7fc-062d-42ff-90ba-868c74f5f97d-maas-api-xks-ca\") pod \"1a82c7fc-062d-42ff-90ba-868c74f5f97d\" (UID: \"1a82c7fc-062d-42ff-90ba-868c74f5f97d\") " Oct 30 12:27:38.531044 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:38.531023 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a82c7fc-062d-42ff-90ba-868c74f5f97d-kube-api-access-fvh9x\" (UniqueName: \"kubernetes.io/projected/1a82c7fc-062d-42ff-90ba-868c74f5f97d-kube-api-access-fvh9x\") pod \"1a82c7fc-062d-42ff-90ba-868c74f5f97d\" (UID: \"1a82c7fc-062d-42ff-90ba-868c74f5f97d\") " Oct 30 12:27:38.531188 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:38.531080 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a82c7fc-062d-42ff-90ba-868c74f5f97d-maas-api-openshift-ca" pod "1a82c7fc-062d-42ff-90ba-868c74f5f97d" (UID: "1a82c7fc-062d-42ff-90ba-868c74f5f97d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:27:38.531590 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:38.531561 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a82c7fc-062d-42ff-90ba-868c74f5f97d-maas-api-openshift-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:27:38.535243 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:38.535213 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a82c7fc-062d-42ff-90ba-868c74f5f97d-kube-api-access-fvh9x" pod "1a82c7fc-062d-42ff-90ba-868c74f5f97d" (UID: "1a82c7fc-062d-42ff-90ba-868c74f5f97d"). InnerVolumeSpecName "kube-api-access-fvh9x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:27:38.544427 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:38.544405 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a82c7fc-062d-42ff-90ba-868c74f5f97d-maas-api-xks-ca" pod "1a82c7fc-062d-42ff-90ba-868c74f5f97d" (UID: "1a82c7fc-062d-42ff-90ba-868c74f5f97d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:27:38.632874 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:38.632774 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a82c7fc-062d-42ff-90ba-868c74f5f97d-maas-api-xks-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:27:38.632874 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:38.632802 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvh9x\" (UniqueName: \"kubernetes.io/projected/1a82c7fc-062d-42ff-90ba-868c74f5f97d-kube-api-access-fvh9x\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:27:39.318078 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:39.318049 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-951ad1-q75kc" Oct 30 12:27:39.318254 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:39.318083 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-951ad1-q75kc" event={"ID":"1a82c7fc-062d-42ff-90ba-868c74f5f97d","Type":"ContainerDied","Data":"00548a9c44c086b81a5f44fe57e88c04f77e2a3d2cb5ecceadf6053de03bb70c"} Oct 30 12:27:39.318254 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:39.318127 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="00548a9c44c086b81a5f44fe57e88c04f77e2a3d2cb5ecceadf6053de03bb70c" Oct 30 12:27:43.018494 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:43.018459 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-pk78c"] Oct 30 12:27:43.018977 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:43.018714 4908 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-pk78c" podUID="714b47ac-9d7f-422f-a676-08be901b4216" containerName="istio-proxy" containerID="cri-o://34fd5eed27439bb58111269fac7ed1ab03f468eb330ed99a8593e6b33321ca95" gracePeriod=30 Oct 30 12:27:43.029779 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:43.029743 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-b2fvc"] Oct 30 12:27:43.030024 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:43.029984 4908 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-b2fvc" podUID="486f71e9-0730-4dbc-b456-31a57942cd2b" containerName="istio-proxy" containerID="cri-o://4fa9870ea7935d8bf00699f570c5eecdb4bb51681c5f9fc446be527510b3b10a" gracePeriod=30 Oct 30 12:27:47.517760 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:47.517679 4908 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-b2fvc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 12:27:47.517760 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:47.517747 4908 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-b2fvc" podUID="486f71e9-0730-4dbc-b456-31a57942cd2b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 12:27:47.781798 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:47.781707 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-951ad1-7d788c86d-tmc8p"] Oct 30 12:27:47.782016 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:47.781965 4908 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-951ad1-7d788c86d-tmc8p" podUID="14f93ee4-d2c7-456f-94c3-e1378e226d62" containerName="payload-processing" containerID="cri-o://ebeaab5500901574a7567efe0c39fe85b6c896fb8e8fde261c7b7d7593e3935e" gracePeriod=30 Oct 30 12:27:47.814865 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:47.814809 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-951ad1-566fb4ff59-wmdq5"] Oct 30 12:27:47.815127 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:47.815101 4908 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-951ad1-566fb4ff59-wmdq5" podUID="3af18c4f-4306-43d2-a584-18b4353c7651" containerName="payload-pre-processing" containerID="cri-o://fa3b20ed73007427c8ff25d7c73f55782f8d7c89b886ce928f651a93aa0de19f" gracePeriod=30 Oct 30 12:27:47.833748 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:47.833714 4908 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-951ad1-566fb4ff59-wmdq5" podUID="3af18c4f-4306-43d2-a584-18b4353c7651" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.132.0.130:9004: connect: connection refused" Oct 30 12:27:48.317801 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:48.317711 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-951ad1-7d788c86d-tmc8p" Oct 30 12:27:48.320286 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:48.320215 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-951ad1-566fb4ff59-wmdq5" Oct 30 12:27:48.393692 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:48.393525 4908 generic.go:356] "Generic (PLEG): container finished" podID="714b47ac-9d7f-422f-a676-08be901b4216" containerID="34fd5eed27439bb58111269fac7ed1ab03f468eb330ed99a8593e6b33321ca95" exitCode=0 Oct 30 12:27:48.393692 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:48.393617 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-pk78c" event={"ID":"714b47ac-9d7f-422f-a676-08be901b4216","Type":"ContainerDied","Data":"34fd5eed27439bb58111269fac7ed1ab03f468eb330ed99a8593e6b33321ca95"} Oct 30 12:27:48.395678 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:48.395651 4908 generic.go:356] "Generic (PLEG): container finished" podID="3af18c4f-4306-43d2-a584-18b4353c7651" containerID="fa3b20ed73007427c8ff25d7c73f55782f8d7c89b886ce928f651a93aa0de19f" exitCode=0 Oct 30 12:27:48.396099 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:48.395922 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-951ad1-566fb4ff59-wmdq5" event={"ID":"3af18c4f-4306-43d2-a584-18b4353c7651","Type":"ContainerDied","Data":"fa3b20ed73007427c8ff25d7c73f55782f8d7c89b886ce928f651a93aa0de19f"} Oct 30 12:27:48.396099 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:48.395954 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-951ad1-566fb4ff59-wmdq5" Oct 30 12:27:48.396099 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:48.395980 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-951ad1-566fb4ff59-wmdq5" event={"ID":"3af18c4f-4306-43d2-a584-18b4353c7651","Type":"ContainerDied","Data":"661de7523bbfe0e9530dfa0b4275eb66f0ced4788b6c6357b475f9e80403babf"} Oct 30 12:27:48.396099 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:48.396011 4908 scope.go:122] "RemoveContainer" containerID="fa3b20ed73007427c8ff25d7c73f55782f8d7c89b886ce928f651a93aa0de19f" Oct 30 12:27:48.401468 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:48.401441 4908 generic.go:356] "Generic (PLEG): container finished" podID="14f93ee4-d2c7-456f-94c3-e1378e226d62" containerID="ebeaab5500901574a7567efe0c39fe85b6c896fb8e8fde261c7b7d7593e3935e" exitCode=0 Oct 30 12:27:48.401700 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:48.401682 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-951ad1-7d788c86d-tmc8p" event={"ID":"14f93ee4-d2c7-456f-94c3-e1378e226d62","Type":"ContainerDied","Data":"ebeaab5500901574a7567efe0c39fe85b6c896fb8e8fde261c7b7d7593e3935e"} Oct 30 12:27:48.401887 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:48.401864 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-951ad1-7d788c86d-tmc8p" event={"ID":"14f93ee4-d2c7-456f-94c3-e1378e226d62","Type":"ContainerDied","Data":"10387b6a373338b08d1774785e680c76baf1ccd52762296e7a0316734127fe5e"} Oct 30 12:27:48.402081 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:48.402068 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-951ad1-7d788c86d-tmc8p" Oct 30 12:27:48.404726 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:48.404511 4908 generic.go:356] "Generic (PLEG): container finished" podID="486f71e9-0730-4dbc-b456-31a57942cd2b" containerID="4fa9870ea7935d8bf00699f570c5eecdb4bb51681c5f9fc446be527510b3b10a" exitCode=0 Oct 30 12:27:48.404726 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:48.404622 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-b2fvc" event={"ID":"486f71e9-0730-4dbc-b456-31a57942cd2b","Type":"ContainerDied","Data":"4fa9870ea7935d8bf00699f570c5eecdb4bb51681c5f9fc446be527510b3b10a"} Oct 30 12:27:48.416987 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:48.416721 4908 scope.go:122] "RemoveContainer" containerID="fa3b20ed73007427c8ff25d7c73f55782f8d7c89b886ce928f651a93aa0de19f" Oct 30 12:27:48.417536 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:27:48.417186 4908 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa3b20ed73007427c8ff25d7c73f55782f8d7c89b886ce928f651a93aa0de19f\": container with ID starting with fa3b20ed73007427c8ff25d7c73f55782f8d7c89b886ce928f651a93aa0de19f not found: ID does not exist" containerID="fa3b20ed73007427c8ff25d7c73f55782f8d7c89b886ce928f651a93aa0de19f" Oct 30 12:27:48.417536 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:48.417222 4908 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa3b20ed73007427c8ff25d7c73f55782f8d7c89b886ce928f651a93aa0de19f"} err="failed to get container status \"fa3b20ed73007427c8ff25d7c73f55782f8d7c89b886ce928f651a93aa0de19f\": rpc error: code = NotFound desc = could not find container \"fa3b20ed73007427c8ff25d7c73f55782f8d7c89b886ce928f651a93aa0de19f\": container with ID starting with fa3b20ed73007427c8ff25d7c73f55782f8d7c89b886ce928f651a93aa0de19f not found: ID does not exist" Oct 30 12:27:48.417536 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:48.417244 4908 scope.go:122] "RemoveContainer" containerID="ebeaab5500901574a7567efe0c39fe85b6c896fb8e8fde261c7b7d7593e3935e" Oct 30 12:27:48.436541 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:48.436518 4908 scope.go:122] "RemoveContainer" containerID="ebeaab5500901574a7567efe0c39fe85b6c896fb8e8fde261c7b7d7593e3935e" Oct 30 12:27:48.436845 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:27:48.436811 4908 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebeaab5500901574a7567efe0c39fe85b6c896fb8e8fde261c7b7d7593e3935e\": container with ID starting with ebeaab5500901574a7567efe0c39fe85b6c896fb8e8fde261c7b7d7593e3935e not found: ID does not exist" containerID="ebeaab5500901574a7567efe0c39fe85b6c896fb8e8fde261c7b7d7593e3935e" Oct 30 12:27:48.436905 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:48.436855 4908 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebeaab5500901574a7567efe0c39fe85b6c896fb8e8fde261c7b7d7593e3935e"} err="failed to get container status \"ebeaab5500901574a7567efe0c39fe85b6c896fb8e8fde261c7b7d7593e3935e\": rpc error: code = NotFound desc = could not find container \"ebeaab5500901574a7567efe0c39fe85b6c896fb8e8fde261c7b7d7593e3935e\": container with ID starting with ebeaab5500901574a7567efe0c39fe85b6c896fb8e8fde261c7b7d7593e3935e not found: ID does not exist" Oct 30 12:27:48.459214 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:48.459185 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3af18c4f-4306-43d2-a584-18b4353c7651-kube-api-access-zt467\" (UniqueName: \"kubernetes.io/projected/3af18c4f-4306-43d2-a584-18b4353c7651-kube-api-access-zt467\") pod \"3af18c4f-4306-43d2-a584-18b4353c7651\" (UID: \"3af18c4f-4306-43d2-a584-18b4353c7651\") " Oct 30 12:27:48.459486 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:48.459466 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3af18c4f-4306-43d2-a584-18b4353c7651-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3af18c4f-4306-43d2-a584-18b4353c7651-plugins-config-volume\") pod \"3af18c4f-4306-43d2-a584-18b4353c7651\" (UID: \"3af18c4f-4306-43d2-a584-18b4353c7651\") " Oct 30 12:27:48.459649 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:48.459632 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14f93ee4-d2c7-456f-94c3-e1378e226d62-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14f93ee4-d2c7-456f-94c3-e1378e226d62-plugins-config-volume\") pod \"14f93ee4-d2c7-456f-94c3-e1378e226d62\" (UID: \"14f93ee4-d2c7-456f-94c3-e1378e226d62\") " Oct 30 12:27:48.459753 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:48.459740 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14f93ee4-d2c7-456f-94c3-e1378e226d62-kube-api-access-k2vvk\" (UniqueName: \"kubernetes.io/projected/14f93ee4-d2c7-456f-94c3-e1378e226d62-kube-api-access-k2vvk\") pod \"14f93ee4-d2c7-456f-94c3-e1378e226d62\" (UID: \"14f93ee4-d2c7-456f-94c3-e1378e226d62\") " Oct 30 12:27:48.461489 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:48.461449 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14f93ee4-d2c7-456f-94c3-e1378e226d62-plugins-config-volume" pod "14f93ee4-d2c7-456f-94c3-e1378e226d62" (UID: "14f93ee4-d2c7-456f-94c3-e1378e226d62"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:27:48.461926 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:48.461907 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-pk78c" Oct 30 12:27:48.462506 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:48.462480 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3af18c4f-4306-43d2-a584-18b4353c7651-plugins-config-volume" pod "3af18c4f-4306-43d2-a584-18b4353c7651" (UID: "3af18c4f-4306-43d2-a584-18b4353c7651"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:27:48.466888 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:48.466863 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3af18c4f-4306-43d2-a584-18b4353c7651-kube-api-access-zt467" pod "3af18c4f-4306-43d2-a584-18b4353c7651" (UID: "3af18c4f-4306-43d2-a584-18b4353c7651"). InnerVolumeSpecName "kube-api-access-zt467". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:27:48.476333 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:48.476312 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14f93ee4-d2c7-456f-94c3-e1378e226d62-kube-api-access-k2vvk" pod "14f93ee4-d2c7-456f-94c3-e1378e226d62" (UID: "14f93ee4-d2c7-456f-94c3-e1378e226d62"). InnerVolumeSpecName "kube-api-access-k2vvk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:27:48.508375 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:48.507624 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-b2fvc" Oct 30 12:27:48.560944 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:48.560915 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/714b47ac-9d7f-422f-a676-08be901b4216-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/714b47ac-9d7f-422f-a676-08be901b4216-workload-socket\") pod \"714b47ac-9d7f-422f-a676-08be901b4216\" (UID: \"714b47ac-9d7f-422f-a676-08be901b4216\") " Oct 30 12:27:48.560944 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:48.560947 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/714b47ac-9d7f-422f-a676-08be901b4216-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/714b47ac-9d7f-422f-a676-08be901b4216-credential-socket\") pod \"714b47ac-9d7f-422f-a676-08be901b4216\" (UID: \"714b47ac-9d7f-422f-a676-08be901b4216\") " Oct 30 12:27:48.561474 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:48.561081 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/714b47ac-9d7f-422f-a676-08be901b4216-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/714b47ac-9d7f-422f-a676-08be901b4216-istio-envoy\") pod \"714b47ac-9d7f-422f-a676-08be901b4216\" (UID: \"714b47ac-9d7f-422f-a676-08be901b4216\") " Oct 30 12:27:48.561474 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:48.561159 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/714b47ac-9d7f-422f-a676-08be901b4216-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/714b47ac-9d7f-422f-a676-08be901b4216-istiod-ca-cert\") pod \"714b47ac-9d7f-422f-a676-08be901b4216\" (UID: \"714b47ac-9d7f-422f-a676-08be901b4216\") " Oct 30 12:27:48.561474 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:48.561225 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/714b47ac-9d7f-422f-a676-08be901b4216-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/714b47ac-9d7f-422f-a676-08be901b4216-workload-certs\") pod \"714b47ac-9d7f-422f-a676-08be901b4216\" (UID: \"714b47ac-9d7f-422f-a676-08be901b4216\") " Oct 30 12:27:48.561474 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:48.561258 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/714b47ac-9d7f-422f-a676-08be901b4216-credential-socket" pod "714b47ac-9d7f-422f-a676-08be901b4216" (UID: "714b47ac-9d7f-422f-a676-08be901b4216"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:27:48.561474 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:48.561297 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/714b47ac-9d7f-422f-a676-08be901b4216-istio-token\" (UniqueName: \"kubernetes.io/projected/714b47ac-9d7f-422f-a676-08be901b4216-istio-token\") pod \"714b47ac-9d7f-422f-a676-08be901b4216\" (UID: \"714b47ac-9d7f-422f-a676-08be901b4216\") " Oct 30 12:27:48.561474 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:48.561327 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/714b47ac-9d7f-422f-a676-08be901b4216-workload-socket" pod "714b47ac-9d7f-422f-a676-08be901b4216" (UID: "714b47ac-9d7f-422f-a676-08be901b4216"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:27:48.561474 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:48.561339 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/714b47ac-9d7f-422f-a676-08be901b4216-kube-api-access-cvcrl\" (UniqueName: \"kubernetes.io/projected/714b47ac-9d7f-422f-a676-08be901b4216-kube-api-access-cvcrl\") pod \"714b47ac-9d7f-422f-a676-08be901b4216\" (UID: \"714b47ac-9d7f-422f-a676-08be901b4216\") " Oct 30 12:27:48.561474 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:48.561406 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/714b47ac-9d7f-422f-a676-08be901b4216-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/714b47ac-9d7f-422f-a676-08be901b4216-istio-data\") pod \"714b47ac-9d7f-422f-a676-08be901b4216\" (UID: \"714b47ac-9d7f-422f-a676-08be901b4216\") " Oct 30 12:27:48.561926 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:48.561498 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/714b47ac-9d7f-422f-a676-08be901b4216-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/714b47ac-9d7f-422f-a676-08be901b4216-istio-podinfo\") pod \"714b47ac-9d7f-422f-a676-08be901b4216\" (UID: \"714b47ac-9d7f-422f-a676-08be901b4216\") " Oct 30 12:27:48.561926 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:48.561565 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/714b47ac-9d7f-422f-a676-08be901b4216-istiod-ca-cert" pod "714b47ac-9d7f-422f-a676-08be901b4216" (UID: "714b47ac-9d7f-422f-a676-08be901b4216"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:27:48.562052 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:48.561982 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/714b47ac-9d7f-422f-a676-08be901b4216-istio-data" pod "714b47ac-9d7f-422f-a676-08be901b4216" (UID: "714b47ac-9d7f-422f-a676-08be901b4216"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:27:48.562143 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:48.562123 4908 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/714b47ac-9d7f-422f-a676-08be901b4216-istiod-ca-cert\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:27:48.562236 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:48.562184 4908 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14f93ee4-d2c7-456f-94c3-e1378e226d62-plugins-config-volume\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:27:48.562236 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:48.562226 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2vvk\" (UniqueName: \"kubernetes.io/projected/14f93ee4-d2c7-456f-94c3-e1378e226d62-kube-api-access-k2vvk\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:27:48.562438 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:48.562241 4908 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/714b47ac-9d7f-422f-a676-08be901b4216-istio-data\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:27:48.562438 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:48.562256 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zt467\" (UniqueName: \"kubernetes.io/projected/3af18c4f-4306-43d2-a584-18b4353c7651-kube-api-access-zt467\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:27:48.562595 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:48.562273 4908 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/714b47ac-9d7f-422f-a676-08be901b4216-workload-socket\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:27:48.562595 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:48.562568 4908 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/714b47ac-9d7f-422f-a676-08be901b4216-credential-socket\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:27:48.562595 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:48.562590 4908 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3af18c4f-4306-43d2-a584-18b4353c7651-plugins-config-volume\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:27:48.566107 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:48.565768 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/714b47ac-9d7f-422f-a676-08be901b4216-workload-certs" pod "714b47ac-9d7f-422f-a676-08be901b4216" (UID: "714b47ac-9d7f-422f-a676-08be901b4216"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:27:48.566107 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:48.565885 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/714b47ac-9d7f-422f-a676-08be901b4216-istio-podinfo" pod "714b47ac-9d7f-422f-a676-08be901b4216" (UID: "714b47ac-9d7f-422f-a676-08be901b4216"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 12:27:48.566107 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:48.566000 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/714b47ac-9d7f-422f-a676-08be901b4216-kube-api-access-cvcrl" pod "714b47ac-9d7f-422f-a676-08be901b4216" (UID: "714b47ac-9d7f-422f-a676-08be901b4216"). InnerVolumeSpecName "kube-api-access-cvcrl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:27:48.568231 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:48.568076 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/714b47ac-9d7f-422f-a676-08be901b4216-istio-envoy" pod "714b47ac-9d7f-422f-a676-08be901b4216" (UID: "714b47ac-9d7f-422f-a676-08be901b4216"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:27:48.570436 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:48.570410 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/714b47ac-9d7f-422f-a676-08be901b4216-istio-token" pod "714b47ac-9d7f-422f-a676-08be901b4216" (UID: "714b47ac-9d7f-422f-a676-08be901b4216"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:27:48.619176 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:48.619138 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-951ad1-q75kc"] Oct 30 12:27:48.623773 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:48.623750 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-951ad1-q75kc"] Oct 30 12:27:48.663676 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:48.663647 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/486f71e9-0730-4dbc-b456-31a57942cd2b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/486f71e9-0730-4dbc-b456-31a57942cd2b-istio-data\") pod \"486f71e9-0730-4dbc-b456-31a57942cd2b\" (UID: \"486f71e9-0730-4dbc-b456-31a57942cd2b\") " Oct 30 12:27:48.663816 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:48.663682 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/486f71e9-0730-4dbc-b456-31a57942cd2b-istio-token\" (UniqueName: \"kubernetes.io/projected/486f71e9-0730-4dbc-b456-31a57942cd2b-istio-token\") pod \"486f71e9-0730-4dbc-b456-31a57942cd2b\" (UID: \"486f71e9-0730-4dbc-b456-31a57942cd2b\") " Oct 30 12:27:48.663816 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:48.663736 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/486f71e9-0730-4dbc-b456-31a57942cd2b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/486f71e9-0730-4dbc-b456-31a57942cd2b-workload-certs\") pod \"486f71e9-0730-4dbc-b456-31a57942cd2b\" (UID: \"486f71e9-0730-4dbc-b456-31a57942cd2b\") " Oct 30 12:27:48.663816 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:48.663775 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/486f71e9-0730-4dbc-b456-31a57942cd2b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/486f71e9-0730-4dbc-b456-31a57942cd2b-istiod-ca-cert\") pod \"486f71e9-0730-4dbc-b456-31a57942cd2b\" (UID: \"486f71e9-0730-4dbc-b456-31a57942cd2b\") " Oct 30 12:27:48.663816 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:48.663806 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/486f71e9-0730-4dbc-b456-31a57942cd2b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/486f71e9-0730-4dbc-b456-31a57942cd2b-credential-socket\") pod \"486f71e9-0730-4dbc-b456-31a57942cd2b\" (UID: \"486f71e9-0730-4dbc-b456-31a57942cd2b\") " Oct 30 12:27:48.664058 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:48.663901 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/486f71e9-0730-4dbc-b456-31a57942cd2b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/486f71e9-0730-4dbc-b456-31a57942cd2b-istio-podinfo\") pod \"486f71e9-0730-4dbc-b456-31a57942cd2b\" (UID: \"486f71e9-0730-4dbc-b456-31a57942cd2b\") " Oct 30 12:27:48.664058 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:48.663985 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/486f71e9-0730-4dbc-b456-31a57942cd2b-kube-api-access-2h4wr\" (UniqueName: \"kubernetes.io/projected/486f71e9-0730-4dbc-b456-31a57942cd2b-kube-api-access-2h4wr\") pod \"486f71e9-0730-4dbc-b456-31a57942cd2b\" (UID: \"486f71e9-0730-4dbc-b456-31a57942cd2b\") " Oct 30 12:27:48.664058 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:48.664003 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/486f71e9-0730-4dbc-b456-31a57942cd2b-istio-data" pod "486f71e9-0730-4dbc-b456-31a57942cd2b" (UID: "486f71e9-0730-4dbc-b456-31a57942cd2b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:27:48.664058 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:48.664030 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/486f71e9-0730-4dbc-b456-31a57942cd2b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/486f71e9-0730-4dbc-b456-31a57942cd2b-istio-envoy\") pod \"486f71e9-0730-4dbc-b456-31a57942cd2b\" (UID: \"486f71e9-0730-4dbc-b456-31a57942cd2b\") " Oct 30 12:27:48.664265 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:48.664077 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/486f71e9-0730-4dbc-b456-31a57942cd2b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/486f71e9-0730-4dbc-b456-31a57942cd2b-workload-socket\") pod \"486f71e9-0730-4dbc-b456-31a57942cd2b\" (UID: \"486f71e9-0730-4dbc-b456-31a57942cd2b\") " Oct 30 12:27:48.664265 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:48.664164 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/486f71e9-0730-4dbc-b456-31a57942cd2b-credential-socket" pod "486f71e9-0730-4dbc-b456-31a57942cd2b" (UID: "486f71e9-0730-4dbc-b456-31a57942cd2b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:27:48.664629 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:48.664605 4908 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/486f71e9-0730-4dbc-b456-31a57942cd2b-istio-data\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:27:48.664732 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:48.664635 4908 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/714b47ac-9d7f-422f-a676-08be901b4216-istio-envoy\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:27:48.664732 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:48.664649 4908 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/486f71e9-0730-4dbc-b456-31a57942cd2b-credential-socket\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:27:48.664732 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:48.664664 4908 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/714b47ac-9d7f-422f-a676-08be901b4216-workload-certs\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:27:48.664732 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:48.664679 4908 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/714b47ac-9d7f-422f-a676-08be901b4216-istio-token\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:27:48.664732 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:48.664693 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvcrl\" (UniqueName: \"kubernetes.io/projected/714b47ac-9d7f-422f-a676-08be901b4216-kube-api-access-cvcrl\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:27:48.664732 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:48.664708 4908 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/714b47ac-9d7f-422f-a676-08be901b4216-istio-podinfo\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:27:48.665050 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:48.664917 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/486f71e9-0730-4dbc-b456-31a57942cd2b-workload-socket" pod "486f71e9-0730-4dbc-b456-31a57942cd2b" (UID: "486f71e9-0730-4dbc-b456-31a57942cd2b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:27:48.666072 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:48.666048 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/486f71e9-0730-4dbc-b456-31a57942cd2b-workload-certs" pod "486f71e9-0730-4dbc-b456-31a57942cd2b" (UID: "486f71e9-0730-4dbc-b456-31a57942cd2b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:27:48.666222 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:48.666190 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/486f71e9-0730-4dbc-b456-31a57942cd2b-istiod-ca-cert" pod "486f71e9-0730-4dbc-b456-31a57942cd2b" (UID: "486f71e9-0730-4dbc-b456-31a57942cd2b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:27:48.666602 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:48.666502 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/486f71e9-0730-4dbc-b456-31a57942cd2b-istio-token" pod "486f71e9-0730-4dbc-b456-31a57942cd2b" (UID: "486f71e9-0730-4dbc-b456-31a57942cd2b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:27:48.666602 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:48.666549 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/486f71e9-0730-4dbc-b456-31a57942cd2b-istio-podinfo" pod "486f71e9-0730-4dbc-b456-31a57942cd2b" (UID: "486f71e9-0730-4dbc-b456-31a57942cd2b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 12:27:48.666991 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:48.666961 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/486f71e9-0730-4dbc-b456-31a57942cd2b-istio-envoy" pod "486f71e9-0730-4dbc-b456-31a57942cd2b" (UID: "486f71e9-0730-4dbc-b456-31a57942cd2b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:27:48.668025 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:48.667999 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/486f71e9-0730-4dbc-b456-31a57942cd2b-kube-api-access-2h4wr" pod "486f71e9-0730-4dbc-b456-31a57942cd2b" (UID: "486f71e9-0730-4dbc-b456-31a57942cd2b"). InnerVolumeSpecName "kube-api-access-2h4wr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:27:48.747942 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:48.747903 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-951ad1-566fb4ff59-wmdq5"] Oct 30 12:27:48.751129 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:48.751101 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-951ad1-566fb4ff59-wmdq5"] Oct 30 12:27:48.765958 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:48.765935 4908 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/486f71e9-0730-4dbc-b456-31a57942cd2b-istiod-ca-cert\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:27:48.765958 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:48.765958 4908 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/486f71e9-0730-4dbc-b456-31a57942cd2b-istio-podinfo\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:27:48.766084 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:48.765968 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2h4wr\" (UniqueName: \"kubernetes.io/projected/486f71e9-0730-4dbc-b456-31a57942cd2b-kube-api-access-2h4wr\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:27:48.766084 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:48.765978 4908 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/486f71e9-0730-4dbc-b456-31a57942cd2b-istio-envoy\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:27:48.766084 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:48.765987 4908 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/486f71e9-0730-4dbc-b456-31a57942cd2b-workload-socket\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:27:48.766084 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:48.765995 4908 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/486f71e9-0730-4dbc-b456-31a57942cd2b-istio-token\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:27:48.766084 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:48.766003 4908 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/486f71e9-0730-4dbc-b456-31a57942cd2b-workload-certs\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:27:48.773883 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:48.773863 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-951ad1-7d788c86d-tmc8p"] Oct 30 12:27:48.779765 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:48.779744 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-951ad1-7d788c86d-tmc8p"] Oct 30 12:27:49.416973 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:49.416943 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-b2fvc" Oct 30 12:27:49.416973 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:49.416962 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-b2fvc" event={"ID":"486f71e9-0730-4dbc-b456-31a57942cd2b","Type":"ContainerDied","Data":"b440307840bfc54c22b28d6fe927555a02b4b60cbeb6d89287fa96f0195787fe"} Oct 30 12:27:49.417244 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:49.417059 4908 scope.go:122] "RemoveContainer" containerID="4fa9870ea7935d8bf00699f570c5eecdb4bb51681c5f9fc446be527510b3b10a" Oct 30 12:27:49.418507 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:49.418486 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-pk78c" Oct 30 12:27:49.418621 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:49.418497 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-pk78c" event={"ID":"714b47ac-9d7f-422f-a676-08be901b4216","Type":"ContainerDied","Data":"ef82112fbeea6cff10d2534850fc1aaec7bdc607e736f8617e939305579e4cb6"} Oct 30 12:27:49.430312 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:49.430287 4908 scope.go:122] "RemoveContainer" containerID="34fd5eed27439bb58111269fac7ed1ab03f468eb330ed99a8593e6b33321ca95" Oct 30 12:27:49.480205 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:49.480177 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-pk78c"] Oct 30 12:27:49.489469 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:49.489444 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-pk78c"] Oct 30 12:27:49.522721 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:49.522694 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-b2fvc"] Oct 30 12:27:49.533424 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:49.533392 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-b2fvc"] Oct 30 12:27:49.606592 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:49.606553 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14f93ee4-d2c7-456f-94c3-e1378e226d62" path="/var/lib/kubelet/pods/14f93ee4-d2c7-456f-94c3-e1378e226d62/volumes" Oct 30 12:27:49.607171 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:49.607153 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a82c7fc-062d-42ff-90ba-868c74f5f97d" path="/var/lib/kubelet/pods/1a82c7fc-062d-42ff-90ba-868c74f5f97d/volumes" Oct 30 12:27:49.607659 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:49.607640 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3af18c4f-4306-43d2-a584-18b4353c7651" path="/var/lib/kubelet/pods/3af18c4f-4306-43d2-a584-18b4353c7651/volumes" Oct 30 12:27:49.608351 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:49.608334 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="486f71e9-0730-4dbc-b456-31a57942cd2b" path="/var/lib/kubelet/pods/486f71e9-0730-4dbc-b456-31a57942cd2b/volumes" Oct 30 12:27:49.608878 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:49.608859 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="714b47ac-9d7f-422f-a676-08be901b4216" path="/var/lib/kubelet/pods/714b47ac-9d7f-422f-a676-08be901b4216/volumes" Oct 30 12:27:49.683620 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:27:49.683551 4908 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vn6vv" containerName="extractor" Oct 30 12:27:53.662856 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:53.662691 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-096834-bf586ccff-wmtjz"] Oct 30 12:27:53.681969 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:53.681874 4908 cpu_manager.go:414] "containerMap: removing container" podUID="486f71e9-0730-4dbc-b456-31a57942cd2b" containerName="istio-proxy" Oct 30 12:27:53.681969 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:53.681960 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="486f71e9-0730-4dbc-b456-31a57942cd2b" containerName="istio-proxy" Oct 30 12:27:53.682219 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:53.681984 4908 cpu_manager.go:414] "containerMap: removing container" podUID="714b47ac-9d7f-422f-a676-08be901b4216" containerName="istio-proxy" Oct 30 12:27:53.682219 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:53.681992 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="714b47ac-9d7f-422f-a676-08be901b4216" containerName="istio-proxy" Oct 30 12:27:53.682219 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:53.682016 4908 cpu_manager.go:414] "containerMap: removing container" podUID="1a82c7fc-062d-42ff-90ba-868c74f5f97d" containerName="revoke-keys" Oct 30 12:27:53.682219 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:53.682024 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a82c7fc-062d-42ff-90ba-868c74f5f97d" containerName="revoke-keys" Oct 30 12:27:53.682219 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:53.682040 4908 cpu_manager.go:414] "containerMap: removing container" podUID="14f93ee4-d2c7-456f-94c3-e1378e226d62" containerName="payload-processing" Oct 30 12:27:53.682219 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:53.682049 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14f93ee4-d2c7-456f-94c3-e1378e226d62" containerName="payload-processing" Oct 30 12:27:53.682219 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:53.682088 4908 cpu_manager.go:414] "containerMap: removing container" podUID="3af18c4f-4306-43d2-a584-18b4353c7651" containerName="payload-pre-processing" Oct 30 12:27:53.682219 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:53.682096 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3af18c4f-4306-43d2-a584-18b4353c7651" containerName="payload-pre-processing" Oct 30 12:27:53.682699 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:53.682233 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="486f71e9-0730-4dbc-b456-31a57942cd2b" containerName="istio-proxy" Oct 30 12:27:53.682699 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:53.682246 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="714b47ac-9d7f-422f-a676-08be901b4216" containerName="istio-proxy" Oct 30 12:27:53.682699 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:53.682261 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="3af18c4f-4306-43d2-a584-18b4353c7651" containerName="payload-pre-processing" Oct 30 12:27:53.682699 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:53.682270 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a82c7fc-062d-42ff-90ba-868c74f5f97d" containerName="revoke-keys" Oct 30 12:27:53.682699 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:53.682281 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="14f93ee4-d2c7-456f-94c3-e1378e226d62" containerName="payload-processing" Oct 30 12:27:53.712795 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:53.712760 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-096834-bf586ccff-wmtjz"] Oct 30 12:27:53.717642 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:53.717008 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-096834-bf586ccff-wmtjz" Oct 30 12:27:53.731758 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:53.731728 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-096834\"" Oct 30 12:27:53.743155 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:53.743086 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-096834-dockercfg-bbfsb\"" Oct 30 12:27:53.813987 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:53.813955 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-096834-d65c9bd64-ff7gb"] Oct 30 12:27:53.817955 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:53.817936 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-096834-d65c9bd64-ff7gb" Oct 30 12:27:53.831073 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:53.831051 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf02c237-e2af-4e5e-b761-bec08016a5aa-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-096834-bf586ccff-wmtjz\" (UID: \"cf02c237-e2af-4e5e-b761-bec08016a5aa\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-096834-bf586ccff-wmtjz" Oct 30 12:27:53.831167 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:53.831089 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hs7zd\" (UniqueName: \"kubernetes.io/projected/cf02c237-e2af-4e5e-b761-bec08016a5aa-kube-api-access-hs7zd\") pod \"payload-pre-processing-e2e-ipp-b-096834-bf586ccff-wmtjz\" (UID: \"cf02c237-e2af-4e5e-b761-bec08016a5aa\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-096834-bf586ccff-wmtjz" Oct 30 12:27:53.835940 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:53.835919 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-096834-d65c9bd64-ff7gb"] Oct 30 12:27:53.932621 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:53.932542 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a52e7b73-f327-4842-8012-279998b7a861-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-096834-d65c9bd64-ff7gb\" (UID: \"a52e7b73-f327-4842-8012-279998b7a861\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-096834-d65c9bd64-ff7gb" Oct 30 12:27:53.932758 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:53.932627 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf02c237-e2af-4e5e-b761-bec08016a5aa-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-096834-bf586ccff-wmtjz\" (UID: \"cf02c237-e2af-4e5e-b761-bec08016a5aa\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-096834-bf586ccff-wmtjz" Oct 30 12:27:53.932758 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:53.932673 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hs7zd\" (UniqueName: \"kubernetes.io/projected/cf02c237-e2af-4e5e-b761-bec08016a5aa-kube-api-access-hs7zd\") pod \"payload-pre-processing-e2e-ipp-b-096834-bf586ccff-wmtjz\" (UID: \"cf02c237-e2af-4e5e-b761-bec08016a5aa\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-096834-bf586ccff-wmtjz" Oct 30 12:27:53.932865 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:53.932837 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwnr9\" (UniqueName: \"kubernetes.io/projected/a52e7b73-f327-4842-8012-279998b7a861-kube-api-access-xwnr9\") pod \"payload-processing-e2e-ipp-b-096834-d65c9bd64-ff7gb\" (UID: \"a52e7b73-f327-4842-8012-279998b7a861\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-096834-d65c9bd64-ff7gb" Oct 30 12:27:53.933357 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:53.933340 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf02c237-e2af-4e5e-b761-bec08016a5aa-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-096834-bf586ccff-wmtjz\" (UID: \"cf02c237-e2af-4e5e-b761-bec08016a5aa\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-096834-bf586ccff-wmtjz" Oct 30 12:27:53.954819 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:53.954790 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hs7zd\" (UniqueName: \"kubernetes.io/projected/cf02c237-e2af-4e5e-b761-bec08016a5aa-kube-api-access-hs7zd\") pod \"payload-pre-processing-e2e-ipp-b-096834-bf586ccff-wmtjz\" (UID: \"cf02c237-e2af-4e5e-b761-bec08016a5aa\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-096834-bf586ccff-wmtjz" Oct 30 12:27:54.034136 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:54.034103 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwnr9\" (UniqueName: \"kubernetes.io/projected/a52e7b73-f327-4842-8012-279998b7a861-kube-api-access-xwnr9\") pod \"payload-processing-e2e-ipp-b-096834-d65c9bd64-ff7gb\" (UID: \"a52e7b73-f327-4842-8012-279998b7a861\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-096834-d65c9bd64-ff7gb" Oct 30 12:27:54.034320 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:54.034300 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a52e7b73-f327-4842-8012-279998b7a861-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-096834-d65c9bd64-ff7gb\" (UID: \"a52e7b73-f327-4842-8012-279998b7a861\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-096834-d65c9bd64-ff7gb" Oct 30 12:27:54.035021 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:54.034996 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a52e7b73-f327-4842-8012-279998b7a861-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-096834-d65c9bd64-ff7gb\" (UID: \"a52e7b73-f327-4842-8012-279998b7a861\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-096834-d65c9bd64-ff7gb" Oct 30 12:27:54.048891 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:54.048865 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-096834-bf586ccff-wmtjz" Oct 30 12:27:54.056390 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:54.056370 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwnr9\" (UniqueName: \"kubernetes.io/projected/a52e7b73-f327-4842-8012-279998b7a861-kube-api-access-xwnr9\") pod \"payload-processing-e2e-ipp-b-096834-d65c9bd64-ff7gb\" (UID: \"a52e7b73-f327-4842-8012-279998b7a861\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-096834-d65c9bd64-ff7gb" Oct 30 12:27:54.131894 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:54.131861 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-096834-d65c9bd64-ff7gb" Oct 30 12:27:54.359114 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:54.359073 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-096834-bf586ccff-wmtjz"] Oct 30 12:27:54.375239 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:27:54.374329 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf02c237_e2af_4e5e_b761_bec08016a5aa.slice/crio-89853de68380caf9b5e09798536db641ed101f6db02cce47eb846b435d95b75f WatchSource:0}: Error finding container 89853de68380caf9b5e09798536db641ed101f6db02cce47eb846b435d95b75f: Status 404 returned error can't find the container with id 89853de68380caf9b5e09798536db641ed101f6db02cce47eb846b435d95b75f Oct 30 12:27:54.462094 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:54.462066 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-096834-d65c9bd64-ff7gb"] Oct 30 12:27:54.487888 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:54.487235 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-096834-bf586ccff-wmtjz" event={"ID":"cf02c237-e2af-4e5e-b761-bec08016a5aa","Type":"ContainerStarted","Data":"da86daf985877ccf80cf8953bb8085d79618df6702a5e1edf207917ab7a9efe9"} Oct 30 12:27:54.487888 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:54.487359 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-096834-bf586ccff-wmtjz" event={"ID":"cf02c237-e2af-4e5e-b761-bec08016a5aa","Type":"ContainerStarted","Data":"89853de68380caf9b5e09798536db641ed101f6db02cce47eb846b435d95b75f"} Oct 30 12:27:54.487888 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:54.487399 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-096834-bf586ccff-wmtjz" Oct 30 12:27:54.541050 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:54.540956 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-096834-bf586ccff-wmtjz" podStartSLOduration=1.540936223 podStartE2EDuration="1.540936223s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:27:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:27:54.537851662 +0000 UTC m=+2031.052280432" watchObservedRunningTime="2026-10-30 12:27:54.540936223 +0000 UTC m=+2031.055364978" Oct 30 12:27:55.501915 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:55.501877 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-096834-d65c9bd64-ff7gb" event={"ID":"a52e7b73-f327-4842-8012-279998b7a861","Type":"ContainerStarted","Data":"6740d4e13336de1bded4dd04f00d32bb77c9ed90da8c7bb9c9c9d8f4d732df2e"} Oct 30 12:27:55.502316 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:55.501928 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-096834-d65c9bd64-ff7gb" event={"ID":"a52e7b73-f327-4842-8012-279998b7a861","Type":"ContainerStarted","Data":"1d4b9565571ac4697a312fcfecc4f1f25675e708c00d86bbb7517ca476122686"} Oct 30 12:27:55.502316 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:55.501979 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-096834-d65c9bd64-ff7gb" Oct 30 12:27:55.543904 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:55.540165 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-096834-d65c9bd64-ff7gb" podStartSLOduration=2.540147827 podStartE2EDuration="2.540147827s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:27:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:27:55.539729321 +0000 UTC m=+2032.054158067" watchObservedRunningTime="2026-10-30 12:27:55.540147827 +0000 UTC m=+2032.054576606" Oct 30 12:27:56.825705 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:56.825674 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-266486-rxh9b"] Oct 30 12:27:56.829906 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:56.829887 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-266486-rxh9b" Oct 30 12:27:56.844631 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:56.844614 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 12:27:56.855558 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:56.855533 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 12:27:56.858983 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:56.858962 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-266486-rxh9b"] Oct 30 12:27:56.876869 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:56.875536 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0842d602-f75a-4cb2-88ca-798ded2da818-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-266486-rxh9b\" (UID: \"0842d602-f75a-4cb2-88ca-798ded2da818\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-266486-rxh9b" Oct 30 12:27:56.876869 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:56.875646 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjzkr\" (UniqueName: \"kubernetes.io/projected/0842d602-f75a-4cb2-88ca-798ded2da818-kube-api-access-kjzkr\") pod \"maas-api-revoke-keys-e2e-ipp-b-266486-rxh9b\" (UID: \"0842d602-f75a-4cb2-88ca-798ded2da818\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-266486-rxh9b" Oct 30 12:27:56.876869 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:56.875741 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0842d602-f75a-4cb2-88ca-798ded2da818-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-266486-rxh9b\" (UID: \"0842d602-f75a-4cb2-88ca-798ded2da818\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-266486-rxh9b" Oct 30 12:27:56.912228 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:56.912204 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 12:27:56.912315 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:56.912282 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s75hq\"" Oct 30 12:27:56.976559 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:56.976532 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0842d602-f75a-4cb2-88ca-798ded2da818-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-266486-rxh9b\" (UID: \"0842d602-f75a-4cb2-88ca-798ded2da818\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-266486-rxh9b" Oct 30 12:27:56.976716 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:56.976692 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjzkr\" (UniqueName: \"kubernetes.io/projected/0842d602-f75a-4cb2-88ca-798ded2da818-kube-api-access-kjzkr\") pod \"maas-api-revoke-keys-e2e-ipp-b-266486-rxh9b\" (UID: \"0842d602-f75a-4cb2-88ca-798ded2da818\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-266486-rxh9b" Oct 30 12:27:56.976819 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:56.976803 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0842d602-f75a-4cb2-88ca-798ded2da818-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-266486-rxh9b\" (UID: \"0842d602-f75a-4cb2-88ca-798ded2da818\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-266486-rxh9b" Oct 30 12:27:56.977250 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:56.977229 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0842d602-f75a-4cb2-88ca-798ded2da818-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-266486-rxh9b\" (UID: \"0842d602-f75a-4cb2-88ca-798ded2da818\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-266486-rxh9b" Oct 30 12:27:56.979274 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:56.979247 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0842d602-f75a-4cb2-88ca-798ded2da818-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-266486-rxh9b\" (UID: \"0842d602-f75a-4cb2-88ca-798ded2da818\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-266486-rxh9b" Oct 30 12:27:56.998923 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:56.998898 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjzkr\" (UniqueName: \"kubernetes.io/projected/0842d602-f75a-4cb2-88ca-798ded2da818-kube-api-access-kjzkr\") pod \"maas-api-revoke-keys-e2e-ipp-b-266486-rxh9b\" (UID: \"0842d602-f75a-4cb2-88ca-798ded2da818\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-266486-rxh9b" Oct 30 12:27:57.143231 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:57.143158 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-266486-rxh9b" Oct 30 12:27:57.455195 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:57.455044 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-266486-rxh9b"] Oct 30 12:27:57.458066 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:27:57.458036 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0842d602_f75a_4cb2_88ca_798ded2da818.slice/crio-fe2316af31352e543dcf6f92a81fd895f4e5aeda8a98ba6a39858bad794c1119 WatchSource:0}: Error finding container fe2316af31352e543dcf6f92a81fd895f4e5aeda8a98ba6a39858bad794c1119: Status 404 returned error can't find the container with id fe2316af31352e543dcf6f92a81fd895f4e5aeda8a98ba6a39858bad794c1119 Oct 30 12:27:57.533060 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:57.533030 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-266486-rxh9b" event={"ID":"0842d602-f75a-4cb2-88ca-798ded2da818","Type":"ContainerStarted","Data":"fe2316af31352e543dcf6f92a81fd895f4e5aeda8a98ba6a39858bad794c1119"} Oct 30 12:27:58.541804 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:58.541720 4908 generic.go:356] "Generic (PLEG): container finished" podID="0842d602-f75a-4cb2-88ca-798ded2da818" containerID="e27fdac5987b431a869150937e48f5b2d65f4e7bf4aa1568302dae692a7ca88d" exitCode=0 Oct 30 12:27:58.542236 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:58.541801 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-266486-rxh9b" event={"ID":"0842d602-f75a-4cb2-88ca-798ded2da818","Type":"ContainerDied","Data":"e27fdac5987b431a869150937e48f5b2d65f4e7bf4aa1568302dae692a7ca88d"} Oct 30 12:27:59.404129 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:59.404083 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9wwfc"] Oct 30 12:27:59.459638 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:59.459608 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7e2db2-hkczk"] Oct 30 12:27:59.462793 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:59.462767 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7e2db2-hkczk"] Oct 30 12:27:59.628852 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:59.626495 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06ee92ce-63a6-4bad-b7c8-9da2252a175b" path="/var/lib/kubelet/pods/06ee92ce-63a6-4bad-b7c8-9da2252a175b/volumes" Oct 30 12:27:59.752217 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:59.752197 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-266486-rxh9b" Oct 30 12:27:59.832587 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:59.832532 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0842d602-f75a-4cb2-88ca-798ded2da818-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0842d602-f75a-4cb2-88ca-798ded2da818-maas-api-openshift-ca\") pod \"0842d602-f75a-4cb2-88ca-798ded2da818\" (UID: \"0842d602-f75a-4cb2-88ca-798ded2da818\") " Oct 30 12:27:59.832755 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:59.832735 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0842d602-f75a-4cb2-88ca-798ded2da818-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0842d602-f75a-4cb2-88ca-798ded2da818-maas-api-xks-ca\") pod \"0842d602-f75a-4cb2-88ca-798ded2da818\" (UID: \"0842d602-f75a-4cb2-88ca-798ded2da818\") " Oct 30 12:27:59.832856 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:59.832774 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0842d602-f75a-4cb2-88ca-798ded2da818-kube-api-access-kjzkr\" (UniqueName: \"kubernetes.io/projected/0842d602-f75a-4cb2-88ca-798ded2da818-kube-api-access-kjzkr\") pod \"0842d602-f75a-4cb2-88ca-798ded2da818\" (UID: \"0842d602-f75a-4cb2-88ca-798ded2da818\") " Oct 30 12:27:59.832979 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:59.832955 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0842d602-f75a-4cb2-88ca-798ded2da818-maas-api-openshift-ca" pod "0842d602-f75a-4cb2-88ca-798ded2da818" (UID: "0842d602-f75a-4cb2-88ca-798ded2da818"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:27:59.833436 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:59.833409 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0842d602-f75a-4cb2-88ca-798ded2da818-maas-api-openshift-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:27:59.834900 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:59.834867 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0842d602-f75a-4cb2-88ca-798ded2da818-kube-api-access-kjzkr" pod "0842d602-f75a-4cb2-88ca-798ded2da818" (UID: "0842d602-f75a-4cb2-88ca-798ded2da818"). InnerVolumeSpecName "kube-api-access-kjzkr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:27:59.834900 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:59.834870 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0842d602-f75a-4cb2-88ca-798ded2da818-maas-api-xks-ca" pod "0842d602-f75a-4cb2-88ca-798ded2da818" (UID: "0842d602-f75a-4cb2-88ca-798ded2da818"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:27:59.934949 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:59.934919 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0842d602-f75a-4cb2-88ca-798ded2da818-maas-api-xks-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:27:59.934949 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:27:59.934944 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kjzkr\" (UniqueName: \"kubernetes.io/projected/0842d602-f75a-4cb2-88ca-798ded2da818-kube-api-access-kjzkr\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:28:00.557662 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:00.557581 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-266486-rxh9b" Oct 30 12:28:00.557662 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:00.557611 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-266486-rxh9b" event={"ID":"0842d602-f75a-4cb2-88ca-798ded2da818","Type":"ContainerDied","Data":"fe2316af31352e543dcf6f92a81fd895f4e5aeda8a98ba6a39858bad794c1119"} Oct 30 12:28:00.557662 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:00.557658 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fe2316af31352e543dcf6f92a81fd895f4e5aeda8a98ba6a39858bad794c1119" Oct 30 12:28:03.144895 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:03.141706 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5c57af-6d8484d895-78hm2"] Oct 30 12:28:03.144895 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:03.142607 4908 cpu_manager.go:414] "containerMap: removing container" podUID="0842d602-f75a-4cb2-88ca-798ded2da818" containerName="revoke-keys" Oct 30 12:28:03.144895 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:03.142627 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0842d602-f75a-4cb2-88ca-798ded2da818" containerName="revoke-keys" Oct 30 12:28:03.144895 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:03.142810 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="0842d602-f75a-4cb2-88ca-798ded2da818" containerName="revoke-keys" Oct 30 12:28:03.186860 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:03.185773 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5c57af-6d8484d895-78hm2"] Oct 30 12:28:03.186860 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:03.185946 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5c57af-6d8484d895-78hm2" Oct 30 12:28:03.205276 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:03.201028 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-5c57af\"" Oct 30 12:28:03.205276 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:03.201216 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-5c57af-dockercfg-f4q6b\"" Oct 30 12:28:03.293475 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:03.293365 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xqn9\" (UniqueName: \"kubernetes.io/projected/3834e8b4-8ff0-43cf-8aef-65de733483e6-kube-api-access-9xqn9\") pod \"payload-pre-processing-e2e-ipp-b-5c57af-6d8484d895-78hm2\" (UID: \"3834e8b4-8ff0-43cf-8aef-65de733483e6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5c57af-6d8484d895-78hm2" Oct 30 12:28:03.293673 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:03.293524 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3834e8b4-8ff0-43cf-8aef-65de733483e6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-5c57af-6d8484d895-78hm2\" (UID: \"3834e8b4-8ff0-43cf-8aef-65de733483e6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5c57af-6d8484d895-78hm2" Oct 30 12:28:03.395349 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:03.395249 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xqn9\" (UniqueName: \"kubernetes.io/projected/3834e8b4-8ff0-43cf-8aef-65de733483e6-kube-api-access-9xqn9\") pod \"payload-pre-processing-e2e-ipp-b-5c57af-6d8484d895-78hm2\" (UID: \"3834e8b4-8ff0-43cf-8aef-65de733483e6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5c57af-6d8484d895-78hm2" Oct 30 12:28:03.395536 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:03.395364 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3834e8b4-8ff0-43cf-8aef-65de733483e6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-5c57af-6d8484d895-78hm2\" (UID: \"3834e8b4-8ff0-43cf-8aef-65de733483e6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5c57af-6d8484d895-78hm2" Oct 30 12:28:03.396294 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:03.396268 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3834e8b4-8ff0-43cf-8aef-65de733483e6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-5c57af-6d8484d895-78hm2\" (UID: \"3834e8b4-8ff0-43cf-8aef-65de733483e6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5c57af-6d8484d895-78hm2" Oct 30 12:28:03.417652 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:03.417618 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xqn9\" (UniqueName: \"kubernetes.io/projected/3834e8b4-8ff0-43cf-8aef-65de733483e6-kube-api-access-9xqn9\") pod \"payload-pre-processing-e2e-ipp-b-5c57af-6d8484d895-78hm2\" (UID: \"3834e8b4-8ff0-43cf-8aef-65de733483e6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5c57af-6d8484d895-78hm2" Oct 30 12:28:03.541173 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:03.541126 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5c57af-6d8484d895-78hm2" Oct 30 12:28:03.870199 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:03.870167 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5c57af-6d8484d895-78hm2"] Oct 30 12:28:03.870948 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:28:03.870916 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3834e8b4_8ff0_43cf_8aef_65de733483e6.slice/crio-d22da190f9ae04b1da96e1beda0b522e173515fb11b854721d4fd1d3773cf592 WatchSource:0}: Error finding container d22da190f9ae04b1da96e1beda0b522e173515fb11b854721d4fd1d3773cf592: Status 404 returned error can't find the container with id d22da190f9ae04b1da96e1beda0b522e173515fb11b854721d4fd1d3773cf592 Oct 30 12:28:04.603076 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:04.603040 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5c57af-6d8484d895-78hm2" event={"ID":"3834e8b4-8ff0-43cf-8aef-65de733483e6","Type":"ContainerStarted","Data":"f728842bc6344419396dbfd3e985df1140741aa79af14bc68134404da6b00f1d"} Oct 30 12:28:04.603564 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:04.603088 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5c57af-6d8484d895-78hm2" event={"ID":"3834e8b4-8ff0-43cf-8aef-65de733483e6","Type":"ContainerStarted","Data":"d22da190f9ae04b1da96e1beda0b522e173515fb11b854721d4fd1d3773cf592"} Oct 30 12:28:04.603683 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:04.603660 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5c57af-6d8484d895-78hm2" Oct 30 12:28:04.642087 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:04.642046 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5c57af-6d8484d895-78hm2" podStartSLOduration=1.6420337649999999 podStartE2EDuration="1.642033765s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:28:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:28:04.641412797 +0000 UTC m=+2041.155841542" watchObservedRunningTime="2026-10-30 12:28:04.642033765 +0000 UTC m=+2041.156462510" Oct 30 12:28:04.685124 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:04.685090 4908 scope.go:122] "RemoveContainer" containerID="0dfb5696ff79c2d508333f8b27b3103dca71d2128003a0ec721c09a90047bd13" Oct 30 12:28:04.698000 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:04.697986 4908 scope.go:122] "RemoveContainer" containerID="fb52762311b20e114a5fa60f4e035c67ab461ec406f7fdb9f7b426eb2a76dbf1" Oct 30 12:28:04.709664 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:04.709635 4908 scope.go:122] "RemoveContainer" containerID="a023ce7cf4600d83691a751af80028cad5e0e527798d47da82d1649261808a87" Oct 30 12:28:04.718580 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:04.718562 4908 scope.go:122] "RemoveContainer" containerID="4bdc6be23f7f4ec21790d10426ebf1f6bf812654532836d9d133ccab32866834" Oct 30 12:28:04.726970 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:04.726950 4908 scope.go:122] "RemoveContainer" containerID="ffbb95821eba745fa706df06b2824b5d0eabdb354866bcdf034ce8404d648c7c" Oct 30 12:28:05.503708 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:05.503682 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-096834-bf586ccff-wmtjz" Oct 30 12:28:07.535621 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:07.535594 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-096834-d65c9bd64-ff7gb" Oct 30 12:28:07.580902 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:07.575530 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-431bf6-b4fdn"] Oct 30 12:28:07.581516 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:07.581494 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-431bf6-b4fdn" Oct 30 12:28:07.601239 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:07.601214 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 12:28:07.601477 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:07.601462 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 12:28:07.621866 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:07.621809 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-431bf6-b4fdn"] Oct 30 12:28:07.643658 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:07.643634 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 12:28:07.643804 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:07.643706 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s75hq\"" Oct 30 12:28:07.657517 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:07.657484 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d65a27b-b005-4877-9bba-e6d9da019f3a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-431bf6-b4fdn\" (UID: \"7d65a27b-b005-4877-9bba-e6d9da019f3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-431bf6-b4fdn" Oct 30 12:28:07.657641 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:07.657541 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsjpv\" (UniqueName: \"kubernetes.io/projected/7d65a27b-b005-4877-9bba-e6d9da019f3a-kube-api-access-lsjpv\") pod \"maas-api-revoke-keys-e2e-ipp-b-431bf6-b4fdn\" (UID: \"7d65a27b-b005-4877-9bba-e6d9da019f3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-431bf6-b4fdn" Oct 30 12:28:07.657641 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:07.657592 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d65a27b-b005-4877-9bba-e6d9da019f3a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-431bf6-b4fdn\" (UID: \"7d65a27b-b005-4877-9bba-e6d9da019f3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-431bf6-b4fdn" Oct 30 12:28:07.758780 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:07.758744 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d65a27b-b005-4877-9bba-e6d9da019f3a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-431bf6-b4fdn\" (UID: \"7d65a27b-b005-4877-9bba-e6d9da019f3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-431bf6-b4fdn" Oct 30 12:28:07.758780 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:07.758786 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsjpv\" (UniqueName: \"kubernetes.io/projected/7d65a27b-b005-4877-9bba-e6d9da019f3a-kube-api-access-lsjpv\") pod \"maas-api-revoke-keys-e2e-ipp-b-431bf6-b4fdn\" (UID: \"7d65a27b-b005-4877-9bba-e6d9da019f3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-431bf6-b4fdn" Oct 30 12:28:07.759030 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:07.758818 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d65a27b-b005-4877-9bba-e6d9da019f3a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-431bf6-b4fdn\" (UID: \"7d65a27b-b005-4877-9bba-e6d9da019f3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-431bf6-b4fdn" Oct 30 12:28:07.759669 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:07.759647 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d65a27b-b005-4877-9bba-e6d9da019f3a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-431bf6-b4fdn\" (UID: \"7d65a27b-b005-4877-9bba-e6d9da019f3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-431bf6-b4fdn" Oct 30 12:28:07.761437 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:07.761406 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d65a27b-b005-4877-9bba-e6d9da019f3a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-431bf6-b4fdn\" (UID: \"7d65a27b-b005-4877-9bba-e6d9da019f3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-431bf6-b4fdn" Oct 30 12:28:07.781217 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:07.781190 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsjpv\" (UniqueName: \"kubernetes.io/projected/7d65a27b-b005-4877-9bba-e6d9da019f3a-kube-api-access-lsjpv\") pod \"maas-api-revoke-keys-e2e-ipp-b-431bf6-b4fdn\" (UID: \"7d65a27b-b005-4877-9bba-e6d9da019f3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-431bf6-b4fdn" Oct 30 12:28:07.903312 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:07.903239 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-431bf6-b4fdn" Oct 30 12:28:08.209446 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:08.209422 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-431bf6-b4fdn"] Oct 30 12:28:08.211722 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:28:08.211697 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d65a27b_b005_4877_9bba_e6d9da019f3a.slice/crio-e3f9eea3c5647141518cd5599ee65fbb46fc892dd48eb2cd6d7c06fa8a0a35b2 WatchSource:0}: Error finding container e3f9eea3c5647141518cd5599ee65fbb46fc892dd48eb2cd6d7c06fa8a0a35b2: Status 404 returned error can't find the container with id e3f9eea3c5647141518cd5599ee65fbb46fc892dd48eb2cd6d7c06fa8a0a35b2 Oct 30 12:28:08.643511 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:08.643473 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-431bf6-b4fdn" event={"ID":"7d65a27b-b005-4877-9bba-e6d9da019f3a","Type":"ContainerStarted","Data":"707421c0567db4bb146230e62929c3806e97fdfce9a03f1fed1dd9cdb867f52c"} Oct 30 12:28:08.644230 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:08.643535 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-431bf6-b4fdn" event={"ID":"7d65a27b-b005-4877-9bba-e6d9da019f3a","Type":"ContainerStarted","Data":"e3f9eea3c5647141518cd5599ee65fbb46fc892dd48eb2cd6d7c06fa8a0a35b2"} Oct 30 12:28:08.682714 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:08.682661 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-431bf6-b4fdn" podStartSLOduration=1.6826479330000002 podStartE2EDuration="1.682647933s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:28:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:28:08.681393627 +0000 UTC m=+2045.195822373" watchObservedRunningTime="2026-10-30 12:28:08.682647933 +0000 UTC m=+2045.197076672" Oct 30 12:28:09.044231 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:09.044197 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-876549-k52kr"] Oct 30 12:28:09.048568 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:09.048550 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-876549-k52kr" Oct 30 12:28:09.066384 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:09.066358 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-876549-k52kr"] Oct 30 12:28:09.178775 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:09.178745 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbctx\" (UniqueName: \"kubernetes.io/projected/cf0cf743-759d-41e6-b405-8b221eb41c2e-kube-api-access-zbctx\") pod \"maas-api-revoke-keys-e2e-ipp-a-876549-k52kr\" (UID: \"cf0cf743-759d-41e6-b405-8b221eb41c2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-876549-k52kr" Oct 30 12:28:09.178936 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:09.178897 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf0cf743-759d-41e6-b405-8b221eb41c2e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-876549-k52kr\" (UID: \"cf0cf743-759d-41e6-b405-8b221eb41c2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-876549-k52kr" Oct 30 12:28:09.179004 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:09.178962 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf0cf743-759d-41e6-b405-8b221eb41c2e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-876549-k52kr\" (UID: \"cf0cf743-759d-41e6-b405-8b221eb41c2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-876549-k52kr" Oct 30 12:28:09.280337 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:09.280303 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbctx\" (UniqueName: \"kubernetes.io/projected/cf0cf743-759d-41e6-b405-8b221eb41c2e-kube-api-access-zbctx\") pod \"maas-api-revoke-keys-e2e-ipp-a-876549-k52kr\" (UID: \"cf0cf743-759d-41e6-b405-8b221eb41c2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-876549-k52kr" Oct 30 12:28:09.280530 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:09.280366 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf0cf743-759d-41e6-b405-8b221eb41c2e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-876549-k52kr\" (UID: \"cf0cf743-759d-41e6-b405-8b221eb41c2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-876549-k52kr" Oct 30 12:28:09.280530 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:09.280420 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf0cf743-759d-41e6-b405-8b221eb41c2e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-876549-k52kr\" (UID: \"cf0cf743-759d-41e6-b405-8b221eb41c2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-876549-k52kr" Oct 30 12:28:09.281150 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:09.281125 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf0cf743-759d-41e6-b405-8b221eb41c2e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-876549-k52kr\" (UID: \"cf0cf743-759d-41e6-b405-8b221eb41c2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-876549-k52kr" Oct 30 12:28:09.282869 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:09.282848 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf0cf743-759d-41e6-b405-8b221eb41c2e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-876549-k52kr\" (UID: \"cf0cf743-759d-41e6-b405-8b221eb41c2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-876549-k52kr" Oct 30 12:28:09.302642 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:09.302581 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbctx\" (UniqueName: \"kubernetes.io/projected/cf0cf743-759d-41e6-b405-8b221eb41c2e-kube-api-access-zbctx\") pod \"maas-api-revoke-keys-e2e-ipp-a-876549-k52kr\" (UID: \"cf0cf743-759d-41e6-b405-8b221eb41c2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-876549-k52kr" Oct 30 12:28:09.362726 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:09.362703 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-876549-k52kr" Oct 30 12:28:09.653543 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:09.653470 4908 generic.go:356] "Generic (PLEG): container finished" podID="7d65a27b-b005-4877-9bba-e6d9da019f3a" containerID="707421c0567db4bb146230e62929c3806e97fdfce9a03f1fed1dd9cdb867f52c" exitCode=0 Oct 30 12:28:09.653543 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:09.653508 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-431bf6-b4fdn" event={"ID":"7d65a27b-b005-4877-9bba-e6d9da019f3a","Type":"ContainerDied","Data":"707421c0567db4bb146230e62929c3806e97fdfce9a03f1fed1dd9cdb867f52c"} Oct 30 12:28:09.666299 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:09.666276 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-876549-k52kr"] Oct 30 12:28:09.668738 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:28:09.668711 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf0cf743_759d_41e6_b405_8b221eb41c2e.slice/crio-2c34c2edbf1c22a8d17f9fe2bca71617a735809b51259d52fbaef28af6ea5ea7 WatchSource:0}: Error finding container 2c34c2edbf1c22a8d17f9fe2bca71617a735809b51259d52fbaef28af6ea5ea7: Status 404 returned error can't find the container with id 2c34c2edbf1c22a8d17f9fe2bca71617a735809b51259d52fbaef28af6ea5ea7 Oct 30 12:28:10.034220 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:10.034192 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93eb20b4-hmhp2"] Oct 30 12:28:10.037984 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:10.037958 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93eb20b4-hmhp2"] Oct 30 12:28:10.662606 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:10.662574 4908 generic.go:356] "Generic (PLEG): container finished" podID="cf0cf743-759d-41e6-b405-8b221eb41c2e" containerID="82f91830fc682e19f2aa68ab350ae2d4c1f778741183968de927afd0517578bc" exitCode=0 Oct 30 12:28:10.663088 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:10.662658 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-876549-k52kr" event={"ID":"cf0cf743-759d-41e6-b405-8b221eb41c2e","Type":"ContainerDied","Data":"82f91830fc682e19f2aa68ab350ae2d4c1f778741183968de927afd0517578bc"} Oct 30 12:28:10.663088 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:10.662705 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-876549-k52kr" event={"ID":"cf0cf743-759d-41e6-b405-8b221eb41c2e","Type":"ContainerStarted","Data":"2c34c2edbf1c22a8d17f9fe2bca71617a735809b51259d52fbaef28af6ea5ea7"} Oct 30 12:28:10.810381 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:10.810354 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-431bf6-b4fdn" Oct 30 12:28:10.900245 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:10.900214 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d65a27b-b005-4877-9bba-e6d9da019f3a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d65a27b-b005-4877-9bba-e6d9da019f3a-maas-api-xks-ca\") pod \"7d65a27b-b005-4877-9bba-e6d9da019f3a\" (UID: \"7d65a27b-b005-4877-9bba-e6d9da019f3a\") " Oct 30 12:28:10.900494 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:10.900473 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d65a27b-b005-4877-9bba-e6d9da019f3a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d65a27b-b005-4877-9bba-e6d9da019f3a-maas-api-openshift-ca\") pod \"7d65a27b-b005-4877-9bba-e6d9da019f3a\" (UID: \"7d65a27b-b005-4877-9bba-e6d9da019f3a\") " Oct 30 12:28:10.900581 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:10.900558 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d65a27b-b005-4877-9bba-e6d9da019f3a-kube-api-access-lsjpv\" (UniqueName: \"kubernetes.io/projected/7d65a27b-b005-4877-9bba-e6d9da019f3a-kube-api-access-lsjpv\") pod \"7d65a27b-b005-4877-9bba-e6d9da019f3a\" (UID: \"7d65a27b-b005-4877-9bba-e6d9da019f3a\") " Oct 30 12:28:10.900996 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:10.900967 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d65a27b-b005-4877-9bba-e6d9da019f3a-maas-api-openshift-ca" pod "7d65a27b-b005-4877-9bba-e6d9da019f3a" (UID: "7d65a27b-b005-4877-9bba-e6d9da019f3a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:28:10.901522 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:10.901496 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d65a27b-b005-4877-9bba-e6d9da019f3a-maas-api-openshift-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:28:10.902588 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:10.902567 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d65a27b-b005-4877-9bba-e6d9da019f3a-maas-api-xks-ca" pod "7d65a27b-b005-4877-9bba-e6d9da019f3a" (UID: "7d65a27b-b005-4877-9bba-e6d9da019f3a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:28:10.903043 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:10.903023 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d65a27b-b005-4877-9bba-e6d9da019f3a-kube-api-access-lsjpv" pod "7d65a27b-b005-4877-9bba-e6d9da019f3a" (UID: "7d65a27b-b005-4877-9bba-e6d9da019f3a"). InnerVolumeSpecName "kube-api-access-lsjpv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:28:11.002340 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:11.002314 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lsjpv\" (UniqueName: \"kubernetes.io/projected/7d65a27b-b005-4877-9bba-e6d9da019f3a-kube-api-access-lsjpv\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:28:11.002340 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:11.002340 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d65a27b-b005-4877-9bba-e6d9da019f3a-maas-api-xks-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:28:11.607336 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:11.607300 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96537d4e-4868-47ee-bb61-f7b2d2f3708c" path="/var/lib/kubelet/pods/96537d4e-4868-47ee-bb61-f7b2d2f3708c/volumes" Oct 30 12:28:11.671179 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:11.671153 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-431bf6-b4fdn" Oct 30 12:28:11.671597 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:11.671193 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-431bf6-b4fdn" event={"ID":"7d65a27b-b005-4877-9bba-e6d9da019f3a","Type":"ContainerDied","Data":"e3f9eea3c5647141518cd5599ee65fbb46fc892dd48eb2cd6d7c06fa8a0a35b2"} Oct 30 12:28:11.671597 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:11.671241 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e3f9eea3c5647141518cd5599ee65fbb46fc892dd48eb2cd6d7c06fa8a0a35b2" Oct 30 12:28:11.829152 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:11.829127 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-876549-k52kr" Oct 30 12:28:11.913040 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:11.912946 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf0cf743-759d-41e6-b405-8b221eb41c2e-kube-api-access-zbctx\" (UniqueName: \"kubernetes.io/projected/cf0cf743-759d-41e6-b405-8b221eb41c2e-kube-api-access-zbctx\") pod \"cf0cf743-759d-41e6-b405-8b221eb41c2e\" (UID: \"cf0cf743-759d-41e6-b405-8b221eb41c2e\") " Oct 30 12:28:11.913040 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:11.913015 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf0cf743-759d-41e6-b405-8b221eb41c2e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf0cf743-759d-41e6-b405-8b221eb41c2e-maas-api-openshift-ca\") pod \"cf0cf743-759d-41e6-b405-8b221eb41c2e\" (UID: \"cf0cf743-759d-41e6-b405-8b221eb41c2e\") " Oct 30 12:28:11.913261 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:11.913138 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf0cf743-759d-41e6-b405-8b221eb41c2e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf0cf743-759d-41e6-b405-8b221eb41c2e-maas-api-xks-ca\") pod \"cf0cf743-759d-41e6-b405-8b221eb41c2e\" (UID: \"cf0cf743-759d-41e6-b405-8b221eb41c2e\") " Oct 30 12:28:11.913762 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:11.913714 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf0cf743-759d-41e6-b405-8b221eb41c2e-maas-api-openshift-ca" pod "cf0cf743-759d-41e6-b405-8b221eb41c2e" (UID: "cf0cf743-759d-41e6-b405-8b221eb41c2e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:28:11.916923 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:11.916883 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf0cf743-759d-41e6-b405-8b221eb41c2e-kube-api-access-zbctx" pod "cf0cf743-759d-41e6-b405-8b221eb41c2e" (UID: "cf0cf743-759d-41e6-b405-8b221eb41c2e"). InnerVolumeSpecName "kube-api-access-zbctx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:28:11.918013 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:11.917977 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf0cf743-759d-41e6-b405-8b221eb41c2e-maas-api-xks-ca" pod "cf0cf743-759d-41e6-b405-8b221eb41c2e" (UID: "cf0cf743-759d-41e6-b405-8b221eb41c2e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:28:12.014781 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:12.014745 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zbctx\" (UniqueName: \"kubernetes.io/projected/cf0cf743-759d-41e6-b405-8b221eb41c2e-kube-api-access-zbctx\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:28:12.014781 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:12.014777 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf0cf743-759d-41e6-b405-8b221eb41c2e-maas-api-openshift-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:28:12.014781 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:12.014788 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf0cf743-759d-41e6-b405-8b221eb41c2e-maas-api-xks-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:28:12.679515 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:12.679489 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-876549-k52kr" Oct 30 12:28:12.679962 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:12.679489 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-876549-k52kr" event={"ID":"cf0cf743-759d-41e6-b405-8b221eb41c2e","Type":"ContainerDied","Data":"2c34c2edbf1c22a8d17f9fe2bca71617a735809b51259d52fbaef28af6ea5ea7"} Oct 30 12:28:12.680036 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:12.679949 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2c34c2edbf1c22a8d17f9fe2bca71617a735809b51259d52fbaef28af6ea5ea7" Oct 30 12:28:14.231918 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:14.231883 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-266486-rxh9b"] Oct 30 12:28:14.242458 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:14.242425 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-266486-rxh9b"] Oct 30 12:28:15.606300 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:15.606265 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0842d602-f75a-4cb2-88ca-798ded2da818" path="/var/lib/kubelet/pods/0842d602-f75a-4cb2-88ca-798ded2da818/volumes" Oct 30 12:28:16.626351 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:16.626324 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5c57af-6d8484d895-78hm2" Oct 30 12:28:19.685331 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:28:19.685282 4908 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vn6vv" containerName="extractor" Oct 30 12:28:22.272673 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:22.272632 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-876549-k52kr"] Oct 30 12:28:22.283309 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:22.283272 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-876549-k52kr"] Oct 30 12:28:23.629120 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:23.629084 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf0cf743-759d-41e6-b405-8b221eb41c2e" path="/var/lib/kubelet/pods/cf0cf743-759d-41e6-b405-8b221eb41c2e/volumes" Oct 30 12:28:24.181847 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:24.174539 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9wwfc"] Oct 30 12:28:30.788789 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:30.788745 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-431bf6-6f845d7bb8-rcc6w"] Oct 30 12:28:30.789340 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:30.789013 4908 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-431bf6-6f845d7bb8-rcc6w" podUID="3104798d-8219-4972-b98f-2e30238ebbcc" containerName="payload-processing" containerID="cri-o://3a51cd4f48dd6e1507d326847bf5a5599e46a0070b5f2685ebab413a1b8352e1" gracePeriod=30 Oct 30 12:28:30.820999 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:30.820966 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-431bf6-77bf6f85ff-fbcs2"] Oct 30 12:28:30.821224 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:30.821196 4908 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-431bf6-77bf6f85ff-fbcs2" podUID="c7b40c65-f1b9-4996-8ba2-23f34045cab5" containerName="payload-pre-processing" containerID="cri-o://80905f0efb6ec840e7806d027183fdda10473a801324f5304329652de2d22961" gracePeriod=30 Oct 30 12:28:30.972844 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:30.969821 4908 generic.go:356] "Generic (PLEG): container finished" podID="3104798d-8219-4972-b98f-2e30238ebbcc" containerID="3a51cd4f48dd6e1507d326847bf5a5599e46a0070b5f2685ebab413a1b8352e1" exitCode=0 Oct 30 12:28:30.972844 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:30.970129 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-431bf6-6f845d7bb8-rcc6w" event={"ID":"3104798d-8219-4972-b98f-2e30238ebbcc","Type":"ContainerDied","Data":"3a51cd4f48dd6e1507d326847bf5a5599e46a0070b5f2685ebab413a1b8352e1"} Oct 30 12:28:31.342444 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:31.342423 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-431bf6-6f845d7bb8-rcc6w" Oct 30 12:28:31.346331 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:31.346275 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-431bf6-77bf6f85ff-fbcs2" Oct 30 12:28:31.360214 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:31.360191 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-096834-zz2cf"] Oct 30 12:28:31.360757 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:31.360742 4908 cpu_manager.go:414] "containerMap: removing container" podUID="3104798d-8219-4972-b98f-2e30238ebbcc" containerName="payload-processing" Oct 30 12:28:31.360820 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:31.360758 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3104798d-8219-4972-b98f-2e30238ebbcc" containerName="payload-processing" Oct 30 12:28:31.360820 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:31.360767 4908 cpu_manager.go:414] "containerMap: removing container" podUID="cf0cf743-759d-41e6-b405-8b221eb41c2e" containerName="revoke-keys" Oct 30 12:28:31.360820 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:31.360773 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf0cf743-759d-41e6-b405-8b221eb41c2e" containerName="revoke-keys" Oct 30 12:28:31.360820 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:31.360791 4908 cpu_manager.go:414] "containerMap: removing container" podUID="7d65a27b-b005-4877-9bba-e6d9da019f3a" containerName="revoke-keys" Oct 30 12:28:31.360820 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:31.360795 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d65a27b-b005-4877-9bba-e6d9da019f3a" containerName="revoke-keys" Oct 30 12:28:31.360820 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:31.360805 4908 cpu_manager.go:414] "containerMap: removing container" podUID="c7b40c65-f1b9-4996-8ba2-23f34045cab5" containerName="payload-pre-processing" Oct 30 12:28:31.360820 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:31.360811 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7b40c65-f1b9-4996-8ba2-23f34045cab5" containerName="payload-pre-processing" Oct 30 12:28:31.361055 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:31.360924 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7b40c65-f1b9-4996-8ba2-23f34045cab5" containerName="payload-pre-processing" Oct 30 12:28:31.361055 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:31.360935 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="3104798d-8219-4972-b98f-2e30238ebbcc" containerName="payload-processing" Oct 30 12:28:31.361055 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:31.360941 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf0cf743-759d-41e6-b405-8b221eb41c2e" containerName="revoke-keys" Oct 30 12:28:31.361055 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:31.360950 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d65a27b-b005-4877-9bba-e6d9da019f3a" containerName="revoke-keys" Oct 30 12:28:31.364412 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:31.364393 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-096834-zz2cf" Oct 30 12:28:31.378059 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:31.378039 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 12:28:31.392876 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:31.388876 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-096834-zz2cf"] Oct 30 12:28:31.446857 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:31.440720 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 12:28:31.446857 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:31.440881 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s75hq\"" Oct 30 12:28:31.484551 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:31.478040 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 12:28:31.484551 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:31.479592 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7b40c65-f1b9-4996-8ba2-23f34045cab5-kube-api-access-ctv7k\" (UniqueName: \"kubernetes.io/projected/c7b40c65-f1b9-4996-8ba2-23f34045cab5-kube-api-access-ctv7k\") pod \"c7b40c65-f1b9-4996-8ba2-23f34045cab5\" (UID: \"c7b40c65-f1b9-4996-8ba2-23f34045cab5\") " Oct 30 12:28:31.484551 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:31.479659 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3104798d-8219-4972-b98f-2e30238ebbcc-kube-api-access-nslz2\" (UniqueName: \"kubernetes.io/projected/3104798d-8219-4972-b98f-2e30238ebbcc-kube-api-access-nslz2\") pod \"3104798d-8219-4972-b98f-2e30238ebbcc\" (UID: \"3104798d-8219-4972-b98f-2e30238ebbcc\") " Oct 30 12:28:31.484551 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:31.479901 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3104798d-8219-4972-b98f-2e30238ebbcc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3104798d-8219-4972-b98f-2e30238ebbcc-plugins-config-volume\") pod \"3104798d-8219-4972-b98f-2e30238ebbcc\" (UID: \"3104798d-8219-4972-b98f-2e30238ebbcc\") " Oct 30 12:28:31.484551 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:31.480036 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7b40c65-f1b9-4996-8ba2-23f34045cab5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7b40c65-f1b9-4996-8ba2-23f34045cab5-plugins-config-volume\") pod \"c7b40c65-f1b9-4996-8ba2-23f34045cab5\" (UID: \"c7b40c65-f1b9-4996-8ba2-23f34045cab5\") " Oct 30 12:28:31.484551 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:31.480426 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0553b7b5-ebee-4582-b160-6994f7fdc051-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-096834-zz2cf\" (UID: \"0553b7b5-ebee-4582-b160-6994f7fdc051\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-096834-zz2cf" Oct 30 12:28:31.484551 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:31.480704 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0553b7b5-ebee-4582-b160-6994f7fdc051-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-096834-zz2cf\" (UID: \"0553b7b5-ebee-4582-b160-6994f7fdc051\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-096834-zz2cf" Oct 30 12:28:31.484551 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:31.480740 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28njx\" (UniqueName: \"kubernetes.io/projected/0553b7b5-ebee-4582-b160-6994f7fdc051-kube-api-access-28njx\") pod \"maas-api-revoke-keys-e2e-ipp-b-096834-zz2cf\" (UID: \"0553b7b5-ebee-4582-b160-6994f7fdc051\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-096834-zz2cf" Oct 30 12:28:31.484551 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:31.481868 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3104798d-8219-4972-b98f-2e30238ebbcc-plugins-config-volume" pod "3104798d-8219-4972-b98f-2e30238ebbcc" (UID: "3104798d-8219-4972-b98f-2e30238ebbcc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:28:31.484551 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:31.482671 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7b40c65-f1b9-4996-8ba2-23f34045cab5-plugins-config-volume" pod "c7b40c65-f1b9-4996-8ba2-23f34045cab5" (UID: "c7b40c65-f1b9-4996-8ba2-23f34045cab5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:28:31.512877 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:31.507535 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7b40c65-f1b9-4996-8ba2-23f34045cab5-kube-api-access-ctv7k" pod "c7b40c65-f1b9-4996-8ba2-23f34045cab5" (UID: "c7b40c65-f1b9-4996-8ba2-23f34045cab5"). InnerVolumeSpecName "kube-api-access-ctv7k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:28:31.520299 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:31.519160 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3104798d-8219-4972-b98f-2e30238ebbcc-kube-api-access-nslz2" pod "3104798d-8219-4972-b98f-2e30238ebbcc" (UID: "3104798d-8219-4972-b98f-2e30238ebbcc"). InnerVolumeSpecName "kube-api-access-nslz2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:28:31.582307 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:31.581676 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0553b7b5-ebee-4582-b160-6994f7fdc051-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-096834-zz2cf\" (UID: \"0553b7b5-ebee-4582-b160-6994f7fdc051\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-096834-zz2cf" Oct 30 12:28:31.582307 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:31.581908 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0553b7b5-ebee-4582-b160-6994f7fdc051-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-096834-zz2cf\" (UID: \"0553b7b5-ebee-4582-b160-6994f7fdc051\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-096834-zz2cf" Oct 30 12:28:31.582307 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:31.581942 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28njx\" (UniqueName: \"kubernetes.io/projected/0553b7b5-ebee-4582-b160-6994f7fdc051-kube-api-access-28njx\") pod \"maas-api-revoke-keys-e2e-ipp-b-096834-zz2cf\" (UID: \"0553b7b5-ebee-4582-b160-6994f7fdc051\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-096834-zz2cf" Oct 30 12:28:31.582307 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:31.582066 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctv7k\" (UniqueName: \"kubernetes.io/projected/c7b40c65-f1b9-4996-8ba2-23f34045cab5-kube-api-access-ctv7k\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:28:31.582307 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:31.582082 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nslz2\" (UniqueName: \"kubernetes.io/projected/3104798d-8219-4972-b98f-2e30238ebbcc-kube-api-access-nslz2\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:28:31.582307 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:31.582096 4908 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3104798d-8219-4972-b98f-2e30238ebbcc-plugins-config-volume\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:28:31.582307 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:31.582110 4908 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7b40c65-f1b9-4996-8ba2-23f34045cab5-plugins-config-volume\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:28:31.582869 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:31.582822 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0553b7b5-ebee-4582-b160-6994f7fdc051-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-096834-zz2cf\" (UID: \"0553b7b5-ebee-4582-b160-6994f7fdc051\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-096834-zz2cf" Oct 30 12:28:31.584916 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:31.584890 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0553b7b5-ebee-4582-b160-6994f7fdc051-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-096834-zz2cf\" (UID: \"0553b7b5-ebee-4582-b160-6994f7fdc051\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-096834-zz2cf" Oct 30 12:28:31.604630 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:31.604606 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28njx\" (UniqueName: \"kubernetes.io/projected/0553b7b5-ebee-4582-b160-6994f7fdc051-kube-api-access-28njx\") pod \"maas-api-revoke-keys-e2e-ipp-b-096834-zz2cf\" (UID: \"0553b7b5-ebee-4582-b160-6994f7fdc051\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-096834-zz2cf" Oct 30 12:28:31.679024 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:31.678934 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-096834-zz2cf" Oct 30 12:28:31.933187 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:31.933088 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-431bf6-b4fdn"] Oct 30 12:28:31.938699 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:31.938185 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-431bf6-b4fdn"] Oct 30 12:28:31.980682 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:31.980646 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-431bf6-6f845d7bb8-rcc6w" Oct 30 12:28:31.980971 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:31.980643 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-431bf6-6f845d7bb8-rcc6w" event={"ID":"3104798d-8219-4972-b98f-2e30238ebbcc","Type":"ContainerDied","Data":"1f105faa23f329240d179c0a16cdb65336500249a2d09057aad1cd9eee34cf64"} Oct 30 12:28:31.980971 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:31.980815 4908 scope.go:122] "RemoveContainer" containerID="3a51cd4f48dd6e1507d326847bf5a5599e46a0070b5f2685ebab413a1b8352e1" Oct 30 12:28:31.983663 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:31.983549 4908 generic.go:356] "Generic (PLEG): container finished" podID="c7b40c65-f1b9-4996-8ba2-23f34045cab5" containerID="80905f0efb6ec840e7806d027183fdda10473a801324f5304329652de2d22961" exitCode=0 Oct 30 12:28:31.983877 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:31.983856 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-431bf6-77bf6f85ff-fbcs2" event={"ID":"c7b40c65-f1b9-4996-8ba2-23f34045cab5","Type":"ContainerDied","Data":"80905f0efb6ec840e7806d027183fdda10473a801324f5304329652de2d22961"} Oct 30 12:28:31.983964 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:31.983904 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-431bf6-77bf6f85ff-fbcs2" event={"ID":"c7b40c65-f1b9-4996-8ba2-23f34045cab5","Type":"ContainerDied","Data":"51e0ad972288a9439a69358ed8267174cebe49d73550bfc6354c42915f6e0c9f"} Oct 30 12:28:31.984056 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:31.984042 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-431bf6-77bf6f85ff-fbcs2" Oct 30 12:28:31.990416 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:28:31.990386 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0553b7b5_ebee_4582_b160_6994f7fdc051.slice/crio-9254191856afd11f7a173ae6a65324c71858f9d3de6b1415727f95c0f4d57cd9 WatchSource:0}: Error finding container 9254191856afd11f7a173ae6a65324c71858f9d3de6b1415727f95c0f4d57cd9: Status 404 returned error can't find the container with id 9254191856afd11f7a173ae6a65324c71858f9d3de6b1415727f95c0f4d57cd9 Oct 30 12:28:31.995667 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:31.995642 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-096834-zz2cf"] Oct 30 12:28:31.995747 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:31.995740 4908 scope.go:122] "RemoveContainer" containerID="80905f0efb6ec840e7806d027183fdda10473a801324f5304329652de2d22961" Oct 30 12:28:32.013407 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:32.013386 4908 scope.go:122] "RemoveContainer" containerID="80905f0efb6ec840e7806d027183fdda10473a801324f5304329652de2d22961" Oct 30 12:28:32.013748 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:28:32.013722 4908 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80905f0efb6ec840e7806d027183fdda10473a801324f5304329652de2d22961\": container with ID starting with 80905f0efb6ec840e7806d027183fdda10473a801324f5304329652de2d22961 not found: ID does not exist" containerID="80905f0efb6ec840e7806d027183fdda10473a801324f5304329652de2d22961" Oct 30 12:28:32.013811 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:32.013759 4908 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80905f0efb6ec840e7806d027183fdda10473a801324f5304329652de2d22961"} err="failed to get container status \"80905f0efb6ec840e7806d027183fdda10473a801324f5304329652de2d22961\": rpc error: code = NotFound desc = could not find container \"80905f0efb6ec840e7806d027183fdda10473a801324f5304329652de2d22961\": container with ID starting with 80905f0efb6ec840e7806d027183fdda10473a801324f5304329652de2d22961 not found: ID does not exist" Oct 30 12:28:32.043049 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:32.042990 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-431bf6-6f845d7bb8-rcc6w"] Oct 30 12:28:32.048187 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:32.048152 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-431bf6-6f845d7bb8-rcc6w"] Oct 30 12:28:32.098034 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:32.098000 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-431bf6-77bf6f85ff-fbcs2"] Oct 30 12:28:32.102014 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:32.101984 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-431bf6-77bf6f85ff-fbcs2"] Oct 30 12:28:32.998456 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:32.998412 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-096834-zz2cf" event={"ID":"0553b7b5-ebee-4582-b160-6994f7fdc051","Type":"ContainerStarted","Data":"55ce4b63e49998770c3873f059cf32fd6309f5b9925105333d0c8a92306fac87"} Oct 30 12:28:32.998981 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:32.998475 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-096834-zz2cf" event={"ID":"0553b7b5-ebee-4582-b160-6994f7fdc051","Type":"ContainerStarted","Data":"9254191856afd11f7a173ae6a65324c71858f9d3de6b1415727f95c0f4d57cd9"} Oct 30 12:28:33.037616 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:33.037559 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-096834-zz2cf" podStartSLOduration=2.037545082 podStartE2EDuration="2.037545082s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:28:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:28:33.036382024 +0000 UTC m=+2069.550810768" watchObservedRunningTime="2026-10-30 12:28:33.037545082 +0000 UTC m=+2069.551973823" Oct 30 12:28:33.605901 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:33.605872 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3104798d-8219-4972-b98f-2e30238ebbcc" path="/var/lib/kubelet/pods/3104798d-8219-4972-b98f-2e30238ebbcc/volumes" Oct 30 12:28:33.606304 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:33.606289 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d65a27b-b005-4877-9bba-e6d9da019f3a" path="/var/lib/kubelet/pods/7d65a27b-b005-4877-9bba-e6d9da019f3a/volumes" Oct 30 12:28:33.606642 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:33.606630 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7b40c65-f1b9-4996-8ba2-23f34045cab5" path="/var/lib/kubelet/pods/c7b40c65-f1b9-4996-8ba2-23f34045cab5/volumes" Oct 30 12:28:34.037849 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:34.036937 4908 generic.go:356] "Generic (PLEG): container finished" podID="0553b7b5-ebee-4582-b160-6994f7fdc051" containerID="55ce4b63e49998770c3873f059cf32fd6309f5b9925105333d0c8a92306fac87" exitCode=0 Oct 30 12:28:34.037849 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:34.037266 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-096834-zz2cf" event={"ID":"0553b7b5-ebee-4582-b160-6994f7fdc051","Type":"ContainerDied","Data":"55ce4b63e49998770c3873f059cf32fd6309f5b9925105333d0c8a92306fac87"} Oct 30 12:28:35.204969 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:35.204946 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-096834-zz2cf" Oct 30 12:28:35.362237 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:35.362156 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0553b7b5-ebee-4582-b160-6994f7fdc051-kube-api-access-28njx\" (UniqueName: \"kubernetes.io/projected/0553b7b5-ebee-4582-b160-6994f7fdc051-kube-api-access-28njx\") pod \"0553b7b5-ebee-4582-b160-6994f7fdc051\" (UID: \"0553b7b5-ebee-4582-b160-6994f7fdc051\") " Oct 30 12:28:35.362391 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:35.362289 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0553b7b5-ebee-4582-b160-6994f7fdc051-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0553b7b5-ebee-4582-b160-6994f7fdc051-maas-api-xks-ca\") pod \"0553b7b5-ebee-4582-b160-6994f7fdc051\" (UID: \"0553b7b5-ebee-4582-b160-6994f7fdc051\") " Oct 30 12:28:35.362391 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:35.362315 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0553b7b5-ebee-4582-b160-6994f7fdc051-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0553b7b5-ebee-4582-b160-6994f7fdc051-maas-api-openshift-ca\") pod \"0553b7b5-ebee-4582-b160-6994f7fdc051\" (UID: \"0553b7b5-ebee-4582-b160-6994f7fdc051\") " Oct 30 12:28:35.362771 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:35.362744 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0553b7b5-ebee-4582-b160-6994f7fdc051-maas-api-openshift-ca" pod "0553b7b5-ebee-4582-b160-6994f7fdc051" (UID: "0553b7b5-ebee-4582-b160-6994f7fdc051"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:28:35.362882 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:35.362816 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0553b7b5-ebee-4582-b160-6994f7fdc051-maas-api-openshift-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:28:35.364341 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:35.364314 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0553b7b5-ebee-4582-b160-6994f7fdc051-maas-api-xks-ca" pod "0553b7b5-ebee-4582-b160-6994f7fdc051" (UID: "0553b7b5-ebee-4582-b160-6994f7fdc051"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:28:35.364441 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:35.364406 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0553b7b5-ebee-4582-b160-6994f7fdc051-kube-api-access-28njx" pod "0553b7b5-ebee-4582-b160-6994f7fdc051" (UID: "0553b7b5-ebee-4582-b160-6994f7fdc051"). InnerVolumeSpecName "kube-api-access-28njx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:28:35.464319 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:35.464286 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28njx\" (UniqueName: \"kubernetes.io/projected/0553b7b5-ebee-4582-b160-6994f7fdc051-kube-api-access-28njx\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:28:35.464319 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:35.464316 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0553b7b5-ebee-4582-b160-6994f7fdc051-maas-api-xks-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:28:36.071783 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:36.071734 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-096834-zz2cf" event={"ID":"0553b7b5-ebee-4582-b160-6994f7fdc051","Type":"ContainerDied","Data":"9254191856afd11f7a173ae6a65324c71858f9d3de6b1415727f95c0f4d57cd9"} Oct 30 12:28:36.072011 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:36.071749 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-096834-zz2cf" Oct 30 12:28:36.072011 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:36.071871 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9254191856afd11f7a173ae6a65324c71858f9d3de6b1415727f95c0f4d57cd9" Oct 30 12:28:46.078191 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:28:46.078156 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9wwfc"] Oct 30 12:28:49.684111 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:28:49.684058 4908 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vn6vv" containerName="extractor" Oct 30 12:29:01.340418 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:01.340376 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-096834-d65c9bd64-ff7gb"] Oct 30 12:29:01.340937 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:01.340579 4908 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-096834-d65c9bd64-ff7gb" podUID="a52e7b73-f327-4842-8012-279998b7a861" containerName="payload-processing" containerID="cri-o://6740d4e13336de1bded4dd04f00d32bb77c9ed90da8c7bb9c9c9d8f4d732df2e" gracePeriod=30 Oct 30 12:29:01.360038 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:01.359875 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-096834-bf586ccff-wmtjz"] Oct 30 12:29:01.360182 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:01.360106 4908 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-096834-bf586ccff-wmtjz" podUID="cf02c237-e2af-4e5e-b761-bec08016a5aa" containerName="payload-pre-processing" containerID="cri-o://da86daf985877ccf80cf8953bb8085d79618df6702a5e1edf207917ab7a9efe9" gracePeriod=30 Oct 30 12:29:01.837854 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:01.829282 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-096834-d65c9bd64-ff7gb" Oct 30 12:29:01.856887 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:01.849753 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-096834-bf586ccff-wmtjz" Oct 30 12:29:01.962326 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:01.962229 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf02c237-e2af-4e5e-b761-bec08016a5aa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf02c237-e2af-4e5e-b761-bec08016a5aa-plugins-config-volume\") pod \"cf02c237-e2af-4e5e-b761-bec08016a5aa\" (UID: \"cf02c237-e2af-4e5e-b761-bec08016a5aa\") " Oct 30 12:29:01.962499 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:01.962351 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a52e7b73-f327-4842-8012-279998b7a861-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a52e7b73-f327-4842-8012-279998b7a861-plugins-config-volume\") pod \"a52e7b73-f327-4842-8012-279998b7a861\" (UID: \"a52e7b73-f327-4842-8012-279998b7a861\") " Oct 30 12:29:01.962574 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:01.962530 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a52e7b73-f327-4842-8012-279998b7a861-kube-api-access-xwnr9\" (UniqueName: \"kubernetes.io/projected/a52e7b73-f327-4842-8012-279998b7a861-kube-api-access-xwnr9\") pod \"a52e7b73-f327-4842-8012-279998b7a861\" (UID: \"a52e7b73-f327-4842-8012-279998b7a861\") " Oct 30 12:29:01.962634 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:01.962573 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf02c237-e2af-4e5e-b761-bec08016a5aa-kube-api-access-hs7zd\" (UniqueName: \"kubernetes.io/projected/cf02c237-e2af-4e5e-b761-bec08016a5aa-kube-api-access-hs7zd\") pod \"cf02c237-e2af-4e5e-b761-bec08016a5aa\" (UID: \"cf02c237-e2af-4e5e-b761-bec08016a5aa\") " Oct 30 12:29:01.963533 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:01.963501 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a52e7b73-f327-4842-8012-279998b7a861-plugins-config-volume" pod "a52e7b73-f327-4842-8012-279998b7a861" (UID: "a52e7b73-f327-4842-8012-279998b7a861"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:29:01.963643 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:01.963503 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf02c237-e2af-4e5e-b761-bec08016a5aa-plugins-config-volume" pod "cf02c237-e2af-4e5e-b761-bec08016a5aa" (UID: "cf02c237-e2af-4e5e-b761-bec08016a5aa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:29:01.965586 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:01.965565 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a52e7b73-f327-4842-8012-279998b7a861-kube-api-access-xwnr9" pod "a52e7b73-f327-4842-8012-279998b7a861" (UID: "a52e7b73-f327-4842-8012-279998b7a861"). InnerVolumeSpecName "kube-api-access-xwnr9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:29:01.966807 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:01.966782 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf02c237-e2af-4e5e-b761-bec08016a5aa-kube-api-access-hs7zd" pod "cf02c237-e2af-4e5e-b761-bec08016a5aa" (UID: "cf02c237-e2af-4e5e-b761-bec08016a5aa"). InnerVolumeSpecName "kube-api-access-hs7zd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:29:02.064247 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:02.064214 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwnr9\" (UniqueName: \"kubernetes.io/projected/a52e7b73-f327-4842-8012-279998b7a861-kube-api-access-xwnr9\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:29:02.064247 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:02.064245 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hs7zd\" (UniqueName: \"kubernetes.io/projected/cf02c237-e2af-4e5e-b761-bec08016a5aa-kube-api-access-hs7zd\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:29:02.064247 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:02.064256 4908 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf02c237-e2af-4e5e-b761-bec08016a5aa-plugins-config-volume\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:29:02.064479 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:02.064265 4908 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a52e7b73-f327-4842-8012-279998b7a861-plugins-config-volume\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:29:02.187713 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:02.187675 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-096834-zz2cf"] Oct 30 12:29:02.190994 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:02.190966 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-096834-zz2cf"] Oct 30 12:29:02.287337 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:02.286986 4908 generic.go:356] "Generic (PLEG): container finished" podID="a52e7b73-f327-4842-8012-279998b7a861" containerID="6740d4e13336de1bded4dd04f00d32bb77c9ed90da8c7bb9c9c9d8f4d732df2e" exitCode=0 Oct 30 12:29:02.287337 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:02.287083 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-096834-d65c9bd64-ff7gb" event={"ID":"a52e7b73-f327-4842-8012-279998b7a861","Type":"ContainerDied","Data":"6740d4e13336de1bded4dd04f00d32bb77c9ed90da8c7bb9c9c9d8f4d732df2e"} Oct 30 12:29:02.287337 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:02.287131 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-096834-d65c9bd64-ff7gb" event={"ID":"a52e7b73-f327-4842-8012-279998b7a861","Type":"ContainerDied","Data":"1d4b9565571ac4697a312fcfecc4f1f25675e708c00d86bbb7517ca476122686"} Oct 30 12:29:02.287337 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:02.287159 4908 scope.go:122] "RemoveContainer" containerID="6740d4e13336de1bded4dd04f00d32bb77c9ed90da8c7bb9c9c9d8f4d732df2e" Oct 30 12:29:02.287720 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:02.287381 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-096834-d65c9bd64-ff7gb" Oct 30 12:29:02.296500 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:02.295426 4908 generic.go:356] "Generic (PLEG): container finished" podID="cf02c237-e2af-4e5e-b761-bec08016a5aa" containerID="da86daf985877ccf80cf8953bb8085d79618df6702a5e1edf207917ab7a9efe9" exitCode=0 Oct 30 12:29:02.296500 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:02.295549 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-096834-bf586ccff-wmtjz" event={"ID":"cf02c237-e2af-4e5e-b761-bec08016a5aa","Type":"ContainerDied","Data":"da86daf985877ccf80cf8953bb8085d79618df6702a5e1edf207917ab7a9efe9"} Oct 30 12:29:02.296500 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:02.295597 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-096834-bf586ccff-wmtjz" event={"ID":"cf02c237-e2af-4e5e-b761-bec08016a5aa","Type":"ContainerDied","Data":"89853de68380caf9b5e09798536db641ed101f6db02cce47eb846b435d95b75f"} Oct 30 12:29:02.296500 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:02.295664 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-096834-bf586ccff-wmtjz" Oct 30 12:29:02.312792 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:02.312688 4908 scope.go:122] "RemoveContainer" containerID="6740d4e13336de1bded4dd04f00d32bb77c9ed90da8c7bb9c9c9d8f4d732df2e" Oct 30 12:29:02.315247 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:29:02.315217 4908 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6740d4e13336de1bded4dd04f00d32bb77c9ed90da8c7bb9c9c9d8f4d732df2e\": container with ID starting with 6740d4e13336de1bded4dd04f00d32bb77c9ed90da8c7bb9c9c9d8f4d732df2e not found: ID does not exist" containerID="6740d4e13336de1bded4dd04f00d32bb77c9ed90da8c7bb9c9c9d8f4d732df2e" Oct 30 12:29:02.315385 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:02.315253 4908 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6740d4e13336de1bded4dd04f00d32bb77c9ed90da8c7bb9c9c9d8f4d732df2e"} err="failed to get container status \"6740d4e13336de1bded4dd04f00d32bb77c9ed90da8c7bb9c9c9d8f4d732df2e\": rpc error: code = NotFound desc = could not find container \"6740d4e13336de1bded4dd04f00d32bb77c9ed90da8c7bb9c9c9d8f4d732df2e\": container with ID starting with 6740d4e13336de1bded4dd04f00d32bb77c9ed90da8c7bb9c9c9d8f4d732df2e not found: ID does not exist" Oct 30 12:29:02.315385 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:02.315271 4908 scope.go:122] "RemoveContainer" containerID="da86daf985877ccf80cf8953bb8085d79618df6702a5e1edf207917ab7a9efe9" Oct 30 12:29:02.338886 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:02.338800 4908 scope.go:122] "RemoveContainer" containerID="da86daf985877ccf80cf8953bb8085d79618df6702a5e1edf207917ab7a9efe9" Oct 30 12:29:02.339247 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:29:02.339221 4908 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da86daf985877ccf80cf8953bb8085d79618df6702a5e1edf207917ab7a9efe9\": container with ID starting with da86daf985877ccf80cf8953bb8085d79618df6702a5e1edf207917ab7a9efe9 not found: ID does not exist" containerID="da86daf985877ccf80cf8953bb8085d79618df6702a5e1edf207917ab7a9efe9" Oct 30 12:29:02.339322 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:02.339258 4908 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da86daf985877ccf80cf8953bb8085d79618df6702a5e1edf207917ab7a9efe9"} err="failed to get container status \"da86daf985877ccf80cf8953bb8085d79618df6702a5e1edf207917ab7a9efe9\": rpc error: code = NotFound desc = could not find container \"da86daf985877ccf80cf8953bb8085d79618df6702a5e1edf207917ab7a9efe9\": container with ID starting with da86daf985877ccf80cf8953bb8085d79618df6702a5e1edf207917ab7a9efe9 not found: ID does not exist" Oct 30 12:29:02.374861 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:02.373634 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-096834-bf586ccff-wmtjz"] Oct 30 12:29:02.379502 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:02.379466 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-096834-bf586ccff-wmtjz"] Oct 30 12:29:02.448233 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:02.448199 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-096834-d65c9bd64-ff7gb"] Oct 30 12:29:02.451072 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:02.451052 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-096834-d65c9bd64-ff7gb"] Oct 30 12:29:03.608423 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:03.608393 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0553b7b5-ebee-4582-b160-6994f7fdc051" path="/var/lib/kubelet/pods/0553b7b5-ebee-4582-b160-6994f7fdc051/volumes" Oct 30 12:29:03.608798 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:03.608787 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a52e7b73-f327-4842-8012-279998b7a861" path="/var/lib/kubelet/pods/a52e7b73-f327-4842-8012-279998b7a861/volumes" Oct 30 12:29:03.609197 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:03.609181 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf02c237-e2af-4e5e-b761-bec08016a5aa" path="/var/lib/kubelet/pods/cf02c237-e2af-4e5e-b761-bec08016a5aa/volumes" Oct 30 12:29:04.835376 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:04.835344 4908 scope.go:122] "RemoveContainer" containerID="76fdb648ad6498eaf1bab8ceb6f4a9bb82c58a32bd22c98840c00a43eba4de55" Oct 30 12:29:04.845213 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:04.845193 4908 scope.go:122] "RemoveContainer" containerID="c7bb35db053aab00a2954af1791efee329d653d43d7980f7ac1c8720b1e373f3" Oct 30 12:29:04.854306 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:04.854289 4908 scope.go:122] "RemoveContainer" containerID="d68e004e94124997d6f806a99f1b267fd18ca6574f077d1ed1b5b8996a873f16" Oct 30 12:29:05.103579 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:05.103493 4908 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-plczt_a70ee11c-c20e-4a4f-8476-ac5eb02816d0/console-operator/3.log" Oct 30 12:29:05.109130 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:05.109109 4908 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b8bxz_644e19b3-10ad-4145-9ed3-595d19ed9b07/ovn-acl-logging/0.log" Oct 30 12:29:06.279351 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:06.279320 4908 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-plczt_a70ee11c-c20e-4a4f-8476-ac5eb02816d0/console-operator/3.log" Oct 30 12:29:06.285130 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:06.285106 4908 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b8bxz_644e19b3-10ad-4145-9ed3-595d19ed9b07/ovn-acl-logging/0.log" Oct 30 12:29:13.427114 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:13.426807 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-c44eee-openshift-default-6758d9545-k5l44"] Oct 30 12:29:13.427818 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:13.427609 4908 cpu_manager.go:414] "containerMap: removing container" podUID="0553b7b5-ebee-4582-b160-6994f7fdc051" containerName="revoke-keys" Oct 30 12:29:13.427818 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:13.427632 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0553b7b5-ebee-4582-b160-6994f7fdc051" containerName="revoke-keys" Oct 30 12:29:13.427818 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:13.427663 4908 cpu_manager.go:414] "containerMap: removing container" podUID="cf02c237-e2af-4e5e-b761-bec08016a5aa" containerName="payload-pre-processing" Oct 30 12:29:13.427818 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:13.427670 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf02c237-e2af-4e5e-b761-bec08016a5aa" containerName="payload-pre-processing" Oct 30 12:29:13.427818 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:13.427698 4908 cpu_manager.go:414] "containerMap: removing container" podUID="a52e7b73-f327-4842-8012-279998b7a861" containerName="payload-processing" Oct 30 12:29:13.427818 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:13.427706 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a52e7b73-f327-4842-8012-279998b7a861" containerName="payload-processing" Oct 30 12:29:13.428210 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:13.427885 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="0553b7b5-ebee-4582-b160-6994f7fdc051" containerName="revoke-keys" Oct 30 12:29:13.428210 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:13.427909 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf02c237-e2af-4e5e-b761-bec08016a5aa" containerName="payload-pre-processing" Oct 30 12:29:13.428210 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:13.427922 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="a52e7b73-f327-4842-8012-279998b7a861" containerName="payload-processing" Oct 30 12:29:13.433330 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:13.433307 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-c44eee-openshift-default-6758d9545-k5l44" Oct 30 12:29:13.463790 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:13.463189 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-c44eee-openshift-default-dockercfg-gx9xg\"" Oct 30 12:29:13.467005 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:13.466977 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-c44eee-openshift-default-6758d9545-k5l44"] Oct 30 12:29:13.516411 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:13.516374 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcefbe36-2c1b-43bf-aec5-4e2def37ef97-istiod-ca-cert\") pod \"e2e-praxis-a-c44eee-openshift-default-6758d9545-k5l44\" (UID: \"fcefbe36-2c1b-43bf-aec5-4e2def37ef97\") " pod="openshift-ingress/e2e-praxis-a-c44eee-openshift-default-6758d9545-k5l44" Oct 30 12:29:13.516592 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:13.516479 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcefbe36-2c1b-43bf-aec5-4e2def37ef97-workload-socket\") pod \"e2e-praxis-a-c44eee-openshift-default-6758d9545-k5l44\" (UID: \"fcefbe36-2c1b-43bf-aec5-4e2def37ef97\") " pod="openshift-ingress/e2e-praxis-a-c44eee-openshift-default-6758d9545-k5l44" Oct 30 12:29:13.516592 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:13.516531 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcefbe36-2c1b-43bf-aec5-4e2def37ef97-istio-podinfo\") pod \"e2e-praxis-a-c44eee-openshift-default-6758d9545-k5l44\" (UID: \"fcefbe36-2c1b-43bf-aec5-4e2def37ef97\") " pod="openshift-ingress/e2e-praxis-a-c44eee-openshift-default-6758d9545-k5l44" Oct 30 12:29:13.516718 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:13.516635 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fcefbe36-2c1b-43bf-aec5-4e2def37ef97-istio-token\") pod \"e2e-praxis-a-c44eee-openshift-default-6758d9545-k5l44\" (UID: \"fcefbe36-2c1b-43bf-aec5-4e2def37ef97\") " pod="openshift-ingress/e2e-praxis-a-c44eee-openshift-default-6758d9545-k5l44" Oct 30 12:29:13.516718 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:13.516674 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjdlk\" (UniqueName: \"kubernetes.io/projected/fcefbe36-2c1b-43bf-aec5-4e2def37ef97-kube-api-access-tjdlk\") pod \"e2e-praxis-a-c44eee-openshift-default-6758d9545-k5l44\" (UID: \"fcefbe36-2c1b-43bf-aec5-4e2def37ef97\") " pod="openshift-ingress/e2e-praxis-a-c44eee-openshift-default-6758d9545-k5l44" Oct 30 12:29:13.516818 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:13.516717 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcefbe36-2c1b-43bf-aec5-4e2def37ef97-istio-data\") pod \"e2e-praxis-a-c44eee-openshift-default-6758d9545-k5l44\" (UID: \"fcefbe36-2c1b-43bf-aec5-4e2def37ef97\") " pod="openshift-ingress/e2e-praxis-a-c44eee-openshift-default-6758d9545-k5l44" Oct 30 12:29:13.516818 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:13.516770 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcefbe36-2c1b-43bf-aec5-4e2def37ef97-credential-socket\") pod \"e2e-praxis-a-c44eee-openshift-default-6758d9545-k5l44\" (UID: \"fcefbe36-2c1b-43bf-aec5-4e2def37ef97\") " pod="openshift-ingress/e2e-praxis-a-c44eee-openshift-default-6758d9545-k5l44" Oct 30 12:29:13.516818 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:13.516796 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcefbe36-2c1b-43bf-aec5-4e2def37ef97-workload-certs\") pod \"e2e-praxis-a-c44eee-openshift-default-6758d9545-k5l44\" (UID: \"fcefbe36-2c1b-43bf-aec5-4e2def37ef97\") " pod="openshift-ingress/e2e-praxis-a-c44eee-openshift-default-6758d9545-k5l44" Oct 30 12:29:13.516991 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:13.516907 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcefbe36-2c1b-43bf-aec5-4e2def37ef97-istio-envoy\") pod \"e2e-praxis-a-c44eee-openshift-default-6758d9545-k5l44\" (UID: \"fcefbe36-2c1b-43bf-aec5-4e2def37ef97\") " pod="openshift-ingress/e2e-praxis-a-c44eee-openshift-default-6758d9545-k5l44" Oct 30 12:29:13.618378 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:13.618345 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fcefbe36-2c1b-43bf-aec5-4e2def37ef97-istio-token\") pod \"e2e-praxis-a-c44eee-openshift-default-6758d9545-k5l44\" (UID: \"fcefbe36-2c1b-43bf-aec5-4e2def37ef97\") " pod="openshift-ingress/e2e-praxis-a-c44eee-openshift-default-6758d9545-k5l44" Oct 30 12:29:13.618603 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:13.618577 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjdlk\" (UniqueName: \"kubernetes.io/projected/fcefbe36-2c1b-43bf-aec5-4e2def37ef97-kube-api-access-tjdlk\") pod \"e2e-praxis-a-c44eee-openshift-default-6758d9545-k5l44\" (UID: \"fcefbe36-2c1b-43bf-aec5-4e2def37ef97\") " pod="openshift-ingress/e2e-praxis-a-c44eee-openshift-default-6758d9545-k5l44" Oct 30 12:29:13.618677 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:13.618635 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcefbe36-2c1b-43bf-aec5-4e2def37ef97-istio-data\") pod \"e2e-praxis-a-c44eee-openshift-default-6758d9545-k5l44\" (UID: \"fcefbe36-2c1b-43bf-aec5-4e2def37ef97\") " pod="openshift-ingress/e2e-praxis-a-c44eee-openshift-default-6758d9545-k5l44" Oct 30 12:29:13.618735 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:13.618694 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcefbe36-2c1b-43bf-aec5-4e2def37ef97-credential-socket\") pod \"e2e-praxis-a-c44eee-openshift-default-6758d9545-k5l44\" (UID: \"fcefbe36-2c1b-43bf-aec5-4e2def37ef97\") " pod="openshift-ingress/e2e-praxis-a-c44eee-openshift-default-6758d9545-k5l44" Oct 30 12:29:13.618802 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:13.618730 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcefbe36-2c1b-43bf-aec5-4e2def37ef97-workload-certs\") pod \"e2e-praxis-a-c44eee-openshift-default-6758d9545-k5l44\" (UID: \"fcefbe36-2c1b-43bf-aec5-4e2def37ef97\") " pod="openshift-ingress/e2e-praxis-a-c44eee-openshift-default-6758d9545-k5l44" Oct 30 12:29:13.618897 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:13.618807 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcefbe36-2c1b-43bf-aec5-4e2def37ef97-istio-envoy\") pod \"e2e-praxis-a-c44eee-openshift-default-6758d9545-k5l44\" (UID: \"fcefbe36-2c1b-43bf-aec5-4e2def37ef97\") " pod="openshift-ingress/e2e-praxis-a-c44eee-openshift-default-6758d9545-k5l44" Oct 30 12:29:13.618897 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:13.618884 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcefbe36-2c1b-43bf-aec5-4e2def37ef97-istiod-ca-cert\") pod \"e2e-praxis-a-c44eee-openshift-default-6758d9545-k5l44\" (UID: \"fcefbe36-2c1b-43bf-aec5-4e2def37ef97\") " pod="openshift-ingress/e2e-praxis-a-c44eee-openshift-default-6758d9545-k5l44" Oct 30 12:29:13.619009 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:13.618971 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcefbe36-2c1b-43bf-aec5-4e2def37ef97-workload-socket\") pod \"e2e-praxis-a-c44eee-openshift-default-6758d9545-k5l44\" (UID: \"fcefbe36-2c1b-43bf-aec5-4e2def37ef97\") " pod="openshift-ingress/e2e-praxis-a-c44eee-openshift-default-6758d9545-k5l44" Oct 30 12:29:13.619060 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:13.619013 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcefbe36-2c1b-43bf-aec5-4e2def37ef97-istio-podinfo\") pod \"e2e-praxis-a-c44eee-openshift-default-6758d9545-k5l44\" (UID: \"fcefbe36-2c1b-43bf-aec5-4e2def37ef97\") " pod="openshift-ingress/e2e-praxis-a-c44eee-openshift-default-6758d9545-k5l44" Oct 30 12:29:13.620170 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:13.619757 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcefbe36-2c1b-43bf-aec5-4e2def37ef97-credential-socket\") pod \"e2e-praxis-a-c44eee-openshift-default-6758d9545-k5l44\" (UID: \"fcefbe36-2c1b-43bf-aec5-4e2def37ef97\") " pod="openshift-ingress/e2e-praxis-a-c44eee-openshift-default-6758d9545-k5l44" Oct 30 12:29:13.620170 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:13.620076 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcefbe36-2c1b-43bf-aec5-4e2def37ef97-istio-data\") pod \"e2e-praxis-a-c44eee-openshift-default-6758d9545-k5l44\" (UID: \"fcefbe36-2c1b-43bf-aec5-4e2def37ef97\") " pod="openshift-ingress/e2e-praxis-a-c44eee-openshift-default-6758d9545-k5l44" Oct 30 12:29:13.620558 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:13.620231 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcefbe36-2c1b-43bf-aec5-4e2def37ef97-workload-certs\") pod \"e2e-praxis-a-c44eee-openshift-default-6758d9545-k5l44\" (UID: \"fcefbe36-2c1b-43bf-aec5-4e2def37ef97\") " pod="openshift-ingress/e2e-praxis-a-c44eee-openshift-default-6758d9545-k5l44" Oct 30 12:29:13.622363 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:13.620994 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcefbe36-2c1b-43bf-aec5-4e2def37ef97-istiod-ca-cert\") pod \"e2e-praxis-a-c44eee-openshift-default-6758d9545-k5l44\" (UID: \"fcefbe36-2c1b-43bf-aec5-4e2def37ef97\") " pod="openshift-ingress/e2e-praxis-a-c44eee-openshift-default-6758d9545-k5l44" Oct 30 12:29:13.622666 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:13.622621 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcefbe36-2c1b-43bf-aec5-4e2def37ef97-workload-socket\") pod \"e2e-praxis-a-c44eee-openshift-default-6758d9545-k5l44\" (UID: \"fcefbe36-2c1b-43bf-aec5-4e2def37ef97\") " pod="openshift-ingress/e2e-praxis-a-c44eee-openshift-default-6758d9545-k5l44" Oct 30 12:29:13.623239 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:13.623221 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcefbe36-2c1b-43bf-aec5-4e2def37ef97-istio-podinfo\") pod \"e2e-praxis-a-c44eee-openshift-default-6758d9545-k5l44\" (UID: \"fcefbe36-2c1b-43bf-aec5-4e2def37ef97\") " pod="openshift-ingress/e2e-praxis-a-c44eee-openshift-default-6758d9545-k5l44" Oct 30 12:29:13.634696 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:13.630802 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcefbe36-2c1b-43bf-aec5-4e2def37ef97-istio-envoy\") pod \"e2e-praxis-a-c44eee-openshift-default-6758d9545-k5l44\" (UID: \"fcefbe36-2c1b-43bf-aec5-4e2def37ef97\") " pod="openshift-ingress/e2e-praxis-a-c44eee-openshift-default-6758d9545-k5l44" Oct 30 12:29:13.644956 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:13.644896 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjdlk\" (UniqueName: \"kubernetes.io/projected/fcefbe36-2c1b-43bf-aec5-4e2def37ef97-kube-api-access-tjdlk\") pod \"e2e-praxis-a-c44eee-openshift-default-6758d9545-k5l44\" (UID: \"fcefbe36-2c1b-43bf-aec5-4e2def37ef97\") " pod="openshift-ingress/e2e-praxis-a-c44eee-openshift-default-6758d9545-k5l44" Oct 30 12:29:13.648465 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:13.648417 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fcefbe36-2c1b-43bf-aec5-4e2def37ef97-istio-token\") pod \"e2e-praxis-a-c44eee-openshift-default-6758d9545-k5l44\" (UID: \"fcefbe36-2c1b-43bf-aec5-4e2def37ef97\") " pod="openshift-ingress/e2e-praxis-a-c44eee-openshift-default-6758d9545-k5l44" Oct 30 12:29:13.748312 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:13.748229 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-c44eee-openshift-default-6758d9545-k5l44" Oct 30 12:29:14.126050 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:14.125201 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-c44eee-openshift-default-6758d9545-k5l44"] Oct 30 12:29:14.128735 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:29:14.128709 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcefbe36_2c1b_43bf_aec5_4e2def37ef97.slice/crio-0975b689c4da48002bde7bbfc79624d329b14dc985db7a2348b82efe7f1d30fd WatchSource:0}: Error finding container 0975b689c4da48002bde7bbfc79624d329b14dc985db7a2348b82efe7f1d30fd: Status 404 returned error can't find the container with id 0975b689c4da48002bde7bbfc79624d329b14dc985db7a2348b82efe7f1d30fd Oct 30 12:29:14.131085 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:14.131041 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:29:14.131193 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:14.131115 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:29:14.131193 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:14.131150 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:29:14.391105 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:14.390277 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-c44eee-openshift-default-6758d9545-k5l44" event={"ID":"fcefbe36-2c1b-43bf-aec5-4e2def37ef97","Type":"ContainerStarted","Data":"a83eb4ce815d8daf3833499153def814577c34c6a9b1c3b022fa405292eda2c4"} Oct 30 12:29:14.391105 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:14.390354 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-c44eee-openshift-default-6758d9545-k5l44" event={"ID":"fcefbe36-2c1b-43bf-aec5-4e2def37ef97","Type":"ContainerStarted","Data":"0975b689c4da48002bde7bbfc79624d329b14dc985db7a2348b82efe7f1d30fd"} Oct 30 12:29:14.458247 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:14.458173 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-c44eee-openshift-default-6758d9545-k5l44" podStartSLOduration=1.4581548500000001 podStartE2EDuration="1.45815485s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:29:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:29:14.433461157 +0000 UTC m=+2110.947889905" watchObservedRunningTime="2026-10-30 12:29:14.45815485 +0000 UTC m=+2110.972583592" Oct 30 12:29:14.748722 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:14.748690 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-c44eee-openshift-default-6758d9545-k5l44" Oct 30 12:29:15.332870 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:15.332817 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-ec236e-openshift-default-f6575b7df-kwhwr"] Oct 30 12:29:15.338146 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:15.338123 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-ec236e-openshift-default-f6575b7df-kwhwr" Oct 30 12:29:15.349202 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:15.349175 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-ec236e-openshift-default-f6575b7df-kwhwr"] Oct 30 12:29:15.358178 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:15.358075 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-ec236e-openshift-default-dockercfg-b5586\"" Oct 30 12:29:15.442852 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:15.441696 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ea7d109-957f-4d4f-9897-6eb86c402601-credential-socket\") pod \"e2e-praxis-b-ec236e-openshift-default-f6575b7df-kwhwr\" (UID: \"5ea7d109-957f-4d4f-9897-6eb86c402601\") " pod="openshift-ingress/e2e-praxis-b-ec236e-openshift-default-f6575b7df-kwhwr" Oct 30 12:29:15.442852 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:15.441749 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ea7d109-957f-4d4f-9897-6eb86c402601-istio-token\") pod \"e2e-praxis-b-ec236e-openshift-default-f6575b7df-kwhwr\" (UID: \"5ea7d109-957f-4d4f-9897-6eb86c402601\") " pod="openshift-ingress/e2e-praxis-b-ec236e-openshift-default-f6575b7df-kwhwr" Oct 30 12:29:15.442852 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:15.441821 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ea7d109-957f-4d4f-9897-6eb86c402601-workload-certs\") pod \"e2e-praxis-b-ec236e-openshift-default-f6575b7df-kwhwr\" (UID: \"5ea7d109-957f-4d4f-9897-6eb86c402601\") " pod="openshift-ingress/e2e-praxis-b-ec236e-openshift-default-f6575b7df-kwhwr" Oct 30 12:29:15.443182 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:15.442921 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ea7d109-957f-4d4f-9897-6eb86c402601-istio-envoy\") pod \"e2e-praxis-b-ec236e-openshift-default-f6575b7df-kwhwr\" (UID: \"5ea7d109-957f-4d4f-9897-6eb86c402601\") " pod="openshift-ingress/e2e-praxis-b-ec236e-openshift-default-f6575b7df-kwhwr" Oct 30 12:29:15.443182 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:15.442973 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ea7d109-957f-4d4f-9897-6eb86c402601-istio-data\") pod \"e2e-praxis-b-ec236e-openshift-default-f6575b7df-kwhwr\" (UID: \"5ea7d109-957f-4d4f-9897-6eb86c402601\") " pod="openshift-ingress/e2e-praxis-b-ec236e-openshift-default-f6575b7df-kwhwr" Oct 30 12:29:15.443182 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:15.443049 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptdcf\" (UniqueName: \"kubernetes.io/projected/5ea7d109-957f-4d4f-9897-6eb86c402601-kube-api-access-ptdcf\") pod \"e2e-praxis-b-ec236e-openshift-default-f6575b7df-kwhwr\" (UID: \"5ea7d109-957f-4d4f-9897-6eb86c402601\") " pod="openshift-ingress/e2e-praxis-b-ec236e-openshift-default-f6575b7df-kwhwr" Oct 30 12:29:15.443182 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:15.443090 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ea7d109-957f-4d4f-9897-6eb86c402601-workload-socket\") pod \"e2e-praxis-b-ec236e-openshift-default-f6575b7df-kwhwr\" (UID: \"5ea7d109-957f-4d4f-9897-6eb86c402601\") " pod="openshift-ingress/e2e-praxis-b-ec236e-openshift-default-f6575b7df-kwhwr" Oct 30 12:29:15.443182 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:15.443114 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ea7d109-957f-4d4f-9897-6eb86c402601-istiod-ca-cert\") pod \"e2e-praxis-b-ec236e-openshift-default-f6575b7df-kwhwr\" (UID: \"5ea7d109-957f-4d4f-9897-6eb86c402601\") " pod="openshift-ingress/e2e-praxis-b-ec236e-openshift-default-f6575b7df-kwhwr" Oct 30 12:29:15.443182 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:15.443160 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ea7d109-957f-4d4f-9897-6eb86c402601-istio-podinfo\") pod \"e2e-praxis-b-ec236e-openshift-default-f6575b7df-kwhwr\" (UID: \"5ea7d109-957f-4d4f-9897-6eb86c402601\") " pod="openshift-ingress/e2e-praxis-b-ec236e-openshift-default-f6575b7df-kwhwr" Oct 30 12:29:15.560900 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:15.547709 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ea7d109-957f-4d4f-9897-6eb86c402601-istio-podinfo\") pod \"e2e-praxis-b-ec236e-openshift-default-f6575b7df-kwhwr\" (UID: \"5ea7d109-957f-4d4f-9897-6eb86c402601\") " pod="openshift-ingress/e2e-praxis-b-ec236e-openshift-default-f6575b7df-kwhwr" Oct 30 12:29:15.560900 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:15.547795 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ea7d109-957f-4d4f-9897-6eb86c402601-credential-socket\") pod \"e2e-praxis-b-ec236e-openshift-default-f6575b7df-kwhwr\" (UID: \"5ea7d109-957f-4d4f-9897-6eb86c402601\") " pod="openshift-ingress/e2e-praxis-b-ec236e-openshift-default-f6575b7df-kwhwr" Oct 30 12:29:15.560900 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:15.547855 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ea7d109-957f-4d4f-9897-6eb86c402601-istio-token\") pod \"e2e-praxis-b-ec236e-openshift-default-f6575b7df-kwhwr\" (UID: \"5ea7d109-957f-4d4f-9897-6eb86c402601\") " pod="openshift-ingress/e2e-praxis-b-ec236e-openshift-default-f6575b7df-kwhwr" Oct 30 12:29:15.560900 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:15.547934 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ea7d109-957f-4d4f-9897-6eb86c402601-workload-certs\") pod \"e2e-praxis-b-ec236e-openshift-default-f6575b7df-kwhwr\" (UID: \"5ea7d109-957f-4d4f-9897-6eb86c402601\") " pod="openshift-ingress/e2e-praxis-b-ec236e-openshift-default-f6575b7df-kwhwr" Oct 30 12:29:15.560900 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:15.548213 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ea7d109-957f-4d4f-9897-6eb86c402601-istio-envoy\") pod \"e2e-praxis-b-ec236e-openshift-default-f6575b7df-kwhwr\" (UID: \"5ea7d109-957f-4d4f-9897-6eb86c402601\") " pod="openshift-ingress/e2e-praxis-b-ec236e-openshift-default-f6575b7df-kwhwr" Oct 30 12:29:15.560900 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:15.548254 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ea7d109-957f-4d4f-9897-6eb86c402601-istio-data\") pod \"e2e-praxis-b-ec236e-openshift-default-f6575b7df-kwhwr\" (UID: \"5ea7d109-957f-4d4f-9897-6eb86c402601\") " pod="openshift-ingress/e2e-praxis-b-ec236e-openshift-default-f6575b7df-kwhwr" Oct 30 12:29:15.560900 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:15.548312 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptdcf\" (UniqueName: \"kubernetes.io/projected/5ea7d109-957f-4d4f-9897-6eb86c402601-kube-api-access-ptdcf\") pod \"e2e-praxis-b-ec236e-openshift-default-f6575b7df-kwhwr\" (UID: \"5ea7d109-957f-4d4f-9897-6eb86c402601\") " pod="openshift-ingress/e2e-praxis-b-ec236e-openshift-default-f6575b7df-kwhwr" Oct 30 12:29:15.560900 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:15.548349 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ea7d109-957f-4d4f-9897-6eb86c402601-workload-socket\") pod \"e2e-praxis-b-ec236e-openshift-default-f6575b7df-kwhwr\" (UID: \"5ea7d109-957f-4d4f-9897-6eb86c402601\") " pod="openshift-ingress/e2e-praxis-b-ec236e-openshift-default-f6575b7df-kwhwr" Oct 30 12:29:15.560900 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:15.548373 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ea7d109-957f-4d4f-9897-6eb86c402601-istiod-ca-cert\") pod \"e2e-praxis-b-ec236e-openshift-default-f6575b7df-kwhwr\" (UID: \"5ea7d109-957f-4d4f-9897-6eb86c402601\") " pod="openshift-ingress/e2e-praxis-b-ec236e-openshift-default-f6575b7df-kwhwr" Oct 30 12:29:15.560900 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:15.549199 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ea7d109-957f-4d4f-9897-6eb86c402601-istiod-ca-cert\") pod \"e2e-praxis-b-ec236e-openshift-default-f6575b7df-kwhwr\" (UID: \"5ea7d109-957f-4d4f-9897-6eb86c402601\") " pod="openshift-ingress/e2e-praxis-b-ec236e-openshift-default-f6575b7df-kwhwr" Oct 30 12:29:15.560900 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:15.550183 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ea7d109-957f-4d4f-9897-6eb86c402601-workload-certs\") pod \"e2e-praxis-b-ec236e-openshift-default-f6575b7df-kwhwr\" (UID: \"5ea7d109-957f-4d4f-9897-6eb86c402601\") " pod="openshift-ingress/e2e-praxis-b-ec236e-openshift-default-f6575b7df-kwhwr" Oct 30 12:29:15.560900 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:15.550524 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ea7d109-957f-4d4f-9897-6eb86c402601-workload-socket\") pod \"e2e-praxis-b-ec236e-openshift-default-f6575b7df-kwhwr\" (UID: \"5ea7d109-957f-4d4f-9897-6eb86c402601\") " pod="openshift-ingress/e2e-praxis-b-ec236e-openshift-default-f6575b7df-kwhwr" Oct 30 12:29:15.560900 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:15.550679 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ea7d109-957f-4d4f-9897-6eb86c402601-credential-socket\") pod \"e2e-praxis-b-ec236e-openshift-default-f6575b7df-kwhwr\" (UID: \"5ea7d109-957f-4d4f-9897-6eb86c402601\") " pod="openshift-ingress/e2e-praxis-b-ec236e-openshift-default-f6575b7df-kwhwr" Oct 30 12:29:15.560900 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:15.550901 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ea7d109-957f-4d4f-9897-6eb86c402601-istio-data\") pod \"e2e-praxis-b-ec236e-openshift-default-f6575b7df-kwhwr\" (UID: \"5ea7d109-957f-4d4f-9897-6eb86c402601\") " pod="openshift-ingress/e2e-praxis-b-ec236e-openshift-default-f6575b7df-kwhwr" Oct 30 12:29:15.595921 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:15.587424 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ea7d109-957f-4d4f-9897-6eb86c402601-istio-podinfo\") pod \"e2e-praxis-b-ec236e-openshift-default-f6575b7df-kwhwr\" (UID: \"5ea7d109-957f-4d4f-9897-6eb86c402601\") " pod="openshift-ingress/e2e-praxis-b-ec236e-openshift-default-f6575b7df-kwhwr" Oct 30 12:29:15.595921 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:15.592068 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptdcf\" (UniqueName: \"kubernetes.io/projected/5ea7d109-957f-4d4f-9897-6eb86c402601-kube-api-access-ptdcf\") pod \"e2e-praxis-b-ec236e-openshift-default-f6575b7df-kwhwr\" (UID: \"5ea7d109-957f-4d4f-9897-6eb86c402601\") " pod="openshift-ingress/e2e-praxis-b-ec236e-openshift-default-f6575b7df-kwhwr" Oct 30 12:29:15.605856 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:15.605439 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ea7d109-957f-4d4f-9897-6eb86c402601-istio-envoy\") pod \"e2e-praxis-b-ec236e-openshift-default-f6575b7df-kwhwr\" (UID: \"5ea7d109-957f-4d4f-9897-6eb86c402601\") " pod="openshift-ingress/e2e-praxis-b-ec236e-openshift-default-f6575b7df-kwhwr" Oct 30 12:29:15.608884 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:15.606301 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ea7d109-957f-4d4f-9897-6eb86c402601-istio-token\") pod \"e2e-praxis-b-ec236e-openshift-default-f6575b7df-kwhwr\" (UID: \"5ea7d109-957f-4d4f-9897-6eb86c402601\") " pod="openshift-ingress/e2e-praxis-b-ec236e-openshift-default-f6575b7df-kwhwr" Oct 30 12:29:15.666965 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:15.666922 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-ec236e-openshift-default-f6575b7df-kwhwr" Oct 30 12:29:15.760174 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:15.759707 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-c44eee-openshift-default-6758d9545-k5l44" Oct 30 12:29:16.026939 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:16.026908 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-ec236e-openshift-default-f6575b7df-kwhwr"] Oct 30 12:29:16.027574 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:29:16.027546 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ea7d109_957f_4d4f_9897_6eb86c402601.slice/crio-ff5a52763ef94c689d6ef44284e3c86b24f90a5b83cfa8919d54b76ccf793ca4 WatchSource:0}: Error finding container ff5a52763ef94c689d6ef44284e3c86b24f90a5b83cfa8919d54b76ccf793ca4: Status 404 returned error can't find the container with id ff5a52763ef94c689d6ef44284e3c86b24f90a5b83cfa8919d54b76ccf793ca4 Oct 30 12:29:16.030239 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:16.030206 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:29:16.030322 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:16.030274 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:29:16.030322 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:16.030304 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:29:16.409634 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:16.409535 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-ec236e-openshift-default-f6575b7df-kwhwr" event={"ID":"5ea7d109-957f-4d4f-9897-6eb86c402601","Type":"ContainerStarted","Data":"5129d25b769d5c0f940748bcb1555021168905a268bd4b3c31a2c6fba52703e1"} Oct 30 12:29:16.409634 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:16.409602 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-ec236e-openshift-default-f6575b7df-kwhwr" event={"ID":"5ea7d109-957f-4d4f-9897-6eb86c402601","Type":"ContainerStarted","Data":"ff5a52763ef94c689d6ef44284e3c86b24f90a5b83cfa8919d54b76ccf793ca4"} Oct 30 12:29:16.410179 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:16.410111 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-c44eee-openshift-default-6758d9545-k5l44" Oct 30 12:29:16.410776 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:16.410758 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-c44eee-openshift-default-6758d9545-k5l44" Oct 30 12:29:16.476525 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:16.476470 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-ec236e-openshift-default-f6575b7df-kwhwr" podStartSLOduration=1.476451033 podStartE2EDuration="1.476451033s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:29:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:29:16.462623654 +0000 UTC m=+2112.977052411" watchObservedRunningTime="2026-10-30 12:29:16.476451033 +0000 UTC m=+2112.990879778" Oct 30 12:29:16.667950 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:16.667865 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-ec236e-openshift-default-f6575b7df-kwhwr" Oct 30 12:29:17.674665 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:17.674631 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-ec236e-openshift-default-f6575b7df-kwhwr" Oct 30 12:29:18.433169 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:18.433125 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-ec236e-openshift-default-f6575b7df-kwhwr" Oct 30 12:29:18.434089 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:18.434069 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-ec236e-openshift-default-f6575b7df-kwhwr" Oct 30 12:29:19.684847 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:29:19.684310 4908 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vn6vv" containerName="extractor" Oct 30 12:29:24.529331 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:24.529285 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9wwfc"] Oct 30 12:29:26.601656 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:26.601623 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zvsx7"] Oct 30 12:29:26.602080 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:26.601822 4908 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zvsx7" podUID="06ac4d9d-7718-4f9d-a5b0-99047a324a3c" containerName="istio-proxy" containerID="cri-o://b3332897c0fc28be62327823a761e65ea6d776c719ba72bdfc4b5a4975da7baa" gracePeriod=30 Oct 30 12:29:31.870749 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:31.870690 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zvsx7" Oct 30 12:29:31.984924 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:31.984893 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06ac4d9d-7718-4f9d-a5b0-99047a324a3c-kube-api-access-zstw5\" (UniqueName: \"kubernetes.io/projected/06ac4d9d-7718-4f9d-a5b0-99047a324a3c-kube-api-access-zstw5\") pod \"06ac4d9d-7718-4f9d-a5b0-99047a324a3c\" (UID: \"06ac4d9d-7718-4f9d-a5b0-99047a324a3c\") " Oct 30 12:29:31.985096 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:31.984946 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06ac4d9d-7718-4f9d-a5b0-99047a324a3c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06ac4d9d-7718-4f9d-a5b0-99047a324a3c-istiod-ca-cert\") pod \"06ac4d9d-7718-4f9d-a5b0-99047a324a3c\" (UID: \"06ac4d9d-7718-4f9d-a5b0-99047a324a3c\") " Oct 30 12:29:31.985096 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:31.984972 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06ac4d9d-7718-4f9d-a5b0-99047a324a3c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06ac4d9d-7718-4f9d-a5b0-99047a324a3c-workload-certs\") pod \"06ac4d9d-7718-4f9d-a5b0-99047a324a3c\" (UID: \"06ac4d9d-7718-4f9d-a5b0-99047a324a3c\") " Oct 30 12:29:31.985096 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:31.985003 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/06ac4d9d-7718-4f9d-a5b0-99047a324a3c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06ac4d9d-7718-4f9d-a5b0-99047a324a3c-istio-podinfo\") pod \"06ac4d9d-7718-4f9d-a5b0-99047a324a3c\" (UID: \"06ac4d9d-7718-4f9d-a5b0-99047a324a3c\") " Oct 30 12:29:31.985096 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:31.985075 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06ac4d9d-7718-4f9d-a5b0-99047a324a3c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06ac4d9d-7718-4f9d-a5b0-99047a324a3c-workload-socket\") pod \"06ac4d9d-7718-4f9d-a5b0-99047a324a3c\" (UID: \"06ac4d9d-7718-4f9d-a5b0-99047a324a3c\") " Oct 30 12:29:31.985305 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:31.985136 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06ac4d9d-7718-4f9d-a5b0-99047a324a3c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06ac4d9d-7718-4f9d-a5b0-99047a324a3c-istio-data\") pod \"06ac4d9d-7718-4f9d-a5b0-99047a324a3c\" (UID: \"06ac4d9d-7718-4f9d-a5b0-99047a324a3c\") " Oct 30 12:29:31.985305 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:31.985207 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06ac4d9d-7718-4f9d-a5b0-99047a324a3c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06ac4d9d-7718-4f9d-a5b0-99047a324a3c-istio-envoy\") pod \"06ac4d9d-7718-4f9d-a5b0-99047a324a3c\" (UID: \"06ac4d9d-7718-4f9d-a5b0-99047a324a3c\") " Oct 30 12:29:31.985305 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:31.985248 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06ac4d9d-7718-4f9d-a5b0-99047a324a3c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06ac4d9d-7718-4f9d-a5b0-99047a324a3c-credential-socket\") pod \"06ac4d9d-7718-4f9d-a5b0-99047a324a3c\" (UID: \"06ac4d9d-7718-4f9d-a5b0-99047a324a3c\") " Oct 30 12:29:31.985457 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:31.985308 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06ac4d9d-7718-4f9d-a5b0-99047a324a3c-istio-token\" (UniqueName: \"kubernetes.io/projected/06ac4d9d-7718-4f9d-a5b0-99047a324a3c-istio-token\") pod \"06ac4d9d-7718-4f9d-a5b0-99047a324a3c\" (UID: \"06ac4d9d-7718-4f9d-a5b0-99047a324a3c\") " Oct 30 12:29:31.985457 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:31.985315 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06ac4d9d-7718-4f9d-a5b0-99047a324a3c-workload-certs" pod "06ac4d9d-7718-4f9d-a5b0-99047a324a3c" (UID: "06ac4d9d-7718-4f9d-a5b0-99047a324a3c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:29:31.985457 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:31.985396 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06ac4d9d-7718-4f9d-a5b0-99047a324a3c-workload-socket" pod "06ac4d9d-7718-4f9d-a5b0-99047a324a3c" (UID: "06ac4d9d-7718-4f9d-a5b0-99047a324a3c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:29:31.985457 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:31.985402 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06ac4d9d-7718-4f9d-a5b0-99047a324a3c-istiod-ca-cert" pod "06ac4d9d-7718-4f9d-a5b0-99047a324a3c" (UID: "06ac4d9d-7718-4f9d-a5b0-99047a324a3c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:29:31.985663 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:31.985557 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06ac4d9d-7718-4f9d-a5b0-99047a324a3c-credential-socket" pod "06ac4d9d-7718-4f9d-a5b0-99047a324a3c" (UID: "06ac4d9d-7718-4f9d-a5b0-99047a324a3c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:29:31.985663 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:31.985568 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06ac4d9d-7718-4f9d-a5b0-99047a324a3c-istio-data" pod "06ac4d9d-7718-4f9d-a5b0-99047a324a3c" (UID: "06ac4d9d-7718-4f9d-a5b0-99047a324a3c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:29:31.985997 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:31.985969 4908 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06ac4d9d-7718-4f9d-a5b0-99047a324a3c-istiod-ca-cert\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:29:31.986098 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:31.986002 4908 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06ac4d9d-7718-4f9d-a5b0-99047a324a3c-workload-certs\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:29:31.986098 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:31.986015 4908 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06ac4d9d-7718-4f9d-a5b0-99047a324a3c-workload-socket\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:29:31.986098 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:31.986028 4908 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06ac4d9d-7718-4f9d-a5b0-99047a324a3c-istio-data\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:29:31.986098 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:31.986042 4908 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06ac4d9d-7718-4f9d-a5b0-99047a324a3c-credential-socket\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:29:31.987616 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:31.987588 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06ac4d9d-7718-4f9d-a5b0-99047a324a3c-istio-envoy" pod "06ac4d9d-7718-4f9d-a5b0-99047a324a3c" (UID: "06ac4d9d-7718-4f9d-a5b0-99047a324a3c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:29:31.987616 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:31.987600 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06ac4d9d-7718-4f9d-a5b0-99047a324a3c-istio-token" pod "06ac4d9d-7718-4f9d-a5b0-99047a324a3c" (UID: "06ac4d9d-7718-4f9d-a5b0-99047a324a3c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:29:31.987910 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:31.987889 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06ac4d9d-7718-4f9d-a5b0-99047a324a3c-kube-api-access-zstw5" pod "06ac4d9d-7718-4f9d-a5b0-99047a324a3c" (UID: "06ac4d9d-7718-4f9d-a5b0-99047a324a3c"). InnerVolumeSpecName "kube-api-access-zstw5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:29:31.987989 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:31.987911 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/06ac4d9d-7718-4f9d-a5b0-99047a324a3c-istio-podinfo" pod "06ac4d9d-7718-4f9d-a5b0-99047a324a3c" (UID: "06ac4d9d-7718-4f9d-a5b0-99047a324a3c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 12:29:32.087448 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:32.087414 4908 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06ac4d9d-7718-4f9d-a5b0-99047a324a3c-istio-envoy\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:29:32.087448 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:32.087448 4908 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06ac4d9d-7718-4f9d-a5b0-99047a324a3c-istio-token\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:29:32.087647 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:32.087465 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zstw5\" (UniqueName: \"kubernetes.io/projected/06ac4d9d-7718-4f9d-a5b0-99047a324a3c-kube-api-access-zstw5\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:29:32.087647 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:32.087479 4908 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06ac4d9d-7718-4f9d-a5b0-99047a324a3c-istio-podinfo\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:29:32.641193 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:32.641156 4908 generic.go:356] "Generic (PLEG): container finished" podID="06ac4d9d-7718-4f9d-a5b0-99047a324a3c" containerID="b3332897c0fc28be62327823a761e65ea6d776c719ba72bdfc4b5a4975da7baa" exitCode=0 Oct 30 12:29:32.641193 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:32.641197 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zvsx7" event={"ID":"06ac4d9d-7718-4f9d-a5b0-99047a324a3c","Type":"ContainerDied","Data":"b3332897c0fc28be62327823a761e65ea6d776c719ba72bdfc4b5a4975da7baa"} Oct 30 12:29:32.641407 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:32.641217 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zvsx7" Oct 30 12:29:32.641407 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:32.641256 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zvsx7" event={"ID":"06ac4d9d-7718-4f9d-a5b0-99047a324a3c","Type":"ContainerDied","Data":"bf928e7e5d64fbb2f220adfb0118b55e78aa3786871480eaded6bdb6500d2ece"} Oct 30 12:29:32.641407 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:32.641277 4908 scope.go:122] "RemoveContainer" containerID="b3332897c0fc28be62327823a761e65ea6d776c719ba72bdfc4b5a4975da7baa" Oct 30 12:29:32.655676 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:32.655649 4908 scope.go:122] "RemoveContainer" containerID="b3332897c0fc28be62327823a761e65ea6d776c719ba72bdfc4b5a4975da7baa" Oct 30 12:29:32.656101 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:29:32.656080 4908 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3332897c0fc28be62327823a761e65ea6d776c719ba72bdfc4b5a4975da7baa\": container with ID starting with b3332897c0fc28be62327823a761e65ea6d776c719ba72bdfc4b5a4975da7baa not found: ID does not exist" containerID="b3332897c0fc28be62327823a761e65ea6d776c719ba72bdfc4b5a4975da7baa" Oct 30 12:29:32.656200 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:32.656109 4908 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3332897c0fc28be62327823a761e65ea6d776c719ba72bdfc4b5a4975da7baa"} err="failed to get container status \"b3332897c0fc28be62327823a761e65ea6d776c719ba72bdfc4b5a4975da7baa\": rpc error: code = NotFound desc = could not find container \"b3332897c0fc28be62327823a761e65ea6d776c719ba72bdfc4b5a4975da7baa\": container with ID starting with b3332897c0fc28be62327823a761e65ea6d776c719ba72bdfc4b5a4975da7baa not found: ID does not exist" Oct 30 12:29:32.702666 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:32.702636 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zvsx7"] Oct 30 12:29:32.713978 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:32.713952 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zvsx7"] Oct 30 12:29:33.608058 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:33.608028 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06ac4d9d-7718-4f9d-a5b0-99047a324a3c" path="/var/lib/kubelet/pods/06ac4d9d-7718-4f9d-a5b0-99047a324a3c/volumes" Oct 30 12:29:34.489072 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:34.489036 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-rv92b"] Oct 30 12:29:34.489320 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:34.489281 4908 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-rv92b" podUID="4630b841-bf2e-4019-b450-12e85497afc1" containerName="istio-proxy" containerID="cri-o://8244474829cc517bf079988c2803e3cb5b3b5c8d9e715f7210f6e7a4a1620646" gracePeriod=30 Oct 30 12:29:38.268916 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:38.268885 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-8c5bs"] Oct 30 12:29:38.269440 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:38.269094 4908 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-8c5bs" podUID="07619fd1-c52a-45ce-851e-7436a3cb0723" containerName="istio-proxy" containerID="cri-o://5acbfb21e58944ad079746c35dad9949744f021119be9224397bc2c0cc111c32" gracePeriod=30 Oct 30 12:29:39.163584 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:39.163547 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-1e9675-openshift-default-7d7c758549-6hhbg"] Oct 30 12:29:39.164775 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:39.164745 4908 cpu_manager.go:414] "containerMap: removing container" podUID="06ac4d9d-7718-4f9d-a5b0-99047a324a3c" containerName="istio-proxy" Oct 30 12:29:39.164775 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:39.164775 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06ac4d9d-7718-4f9d-a5b0-99047a324a3c" containerName="istio-proxy" Oct 30 12:29:39.164985 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:39.164939 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="06ac4d9d-7718-4f9d-a5b0-99047a324a3c" containerName="istio-proxy" Oct 30 12:29:39.170371 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:39.170348 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-1e9675-openshift-default-7d7c758549-6hhbg" Oct 30 12:29:39.187544 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:39.187508 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62b94b35-a1bb-4eb2-9bc0-da2760557b3d-workload-socket\") pod \"e2e-praxis-b-1e9675-openshift-default-7d7c758549-6hhbg\" (UID: \"62b94b35-a1bb-4eb2-9bc0-da2760557b3d\") " pod="openshift-ingress/e2e-praxis-b-1e9675-openshift-default-7d7c758549-6hhbg" Oct 30 12:29:39.187671 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:39.187563 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62b94b35-a1bb-4eb2-9bc0-da2760557b3d-istio-token\") pod \"e2e-praxis-b-1e9675-openshift-default-7d7c758549-6hhbg\" (UID: \"62b94b35-a1bb-4eb2-9bc0-da2760557b3d\") " pod="openshift-ingress/e2e-praxis-b-1e9675-openshift-default-7d7c758549-6hhbg" Oct 30 12:29:39.187737 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:39.187676 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62b94b35-a1bb-4eb2-9bc0-da2760557b3d-istio-podinfo\") pod \"e2e-praxis-b-1e9675-openshift-default-7d7c758549-6hhbg\" (UID: \"62b94b35-a1bb-4eb2-9bc0-da2760557b3d\") " pod="openshift-ingress/e2e-praxis-b-1e9675-openshift-default-7d7c758549-6hhbg" Oct 30 12:29:39.187737 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:39.187707 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckrlf\" (UniqueName: \"kubernetes.io/projected/62b94b35-a1bb-4eb2-9bc0-da2760557b3d-kube-api-access-ckrlf\") pod \"e2e-praxis-b-1e9675-openshift-default-7d7c758549-6hhbg\" (UID: \"62b94b35-a1bb-4eb2-9bc0-da2760557b3d\") " pod="openshift-ingress/e2e-praxis-b-1e9675-openshift-default-7d7c758549-6hhbg" Oct 30 12:29:39.187865 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:39.187746 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62b94b35-a1bb-4eb2-9bc0-da2760557b3d-istio-data\") pod \"e2e-praxis-b-1e9675-openshift-default-7d7c758549-6hhbg\" (UID: \"62b94b35-a1bb-4eb2-9bc0-da2760557b3d\") " pod="openshift-ingress/e2e-praxis-b-1e9675-openshift-default-7d7c758549-6hhbg" Oct 30 12:29:39.187954 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:39.187928 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62b94b35-a1bb-4eb2-9bc0-da2760557b3d-workload-certs\") pod \"e2e-praxis-b-1e9675-openshift-default-7d7c758549-6hhbg\" (UID: \"62b94b35-a1bb-4eb2-9bc0-da2760557b3d\") " pod="openshift-ingress/e2e-praxis-b-1e9675-openshift-default-7d7c758549-6hhbg" Oct 30 12:29:39.188061 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:39.188044 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62b94b35-a1bb-4eb2-9bc0-da2760557b3d-credential-socket\") pod \"e2e-praxis-b-1e9675-openshift-default-7d7c758549-6hhbg\" (UID: \"62b94b35-a1bb-4eb2-9bc0-da2760557b3d\") " pod="openshift-ingress/e2e-praxis-b-1e9675-openshift-default-7d7c758549-6hhbg" Oct 30 12:29:39.188118 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:39.188091 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62b94b35-a1bb-4eb2-9bc0-da2760557b3d-istio-envoy\") pod \"e2e-praxis-b-1e9675-openshift-default-7d7c758549-6hhbg\" (UID: \"62b94b35-a1bb-4eb2-9bc0-da2760557b3d\") " pod="openshift-ingress/e2e-praxis-b-1e9675-openshift-default-7d7c758549-6hhbg" Oct 30 12:29:39.188179 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:39.188159 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62b94b35-a1bb-4eb2-9bc0-da2760557b3d-istiod-ca-cert\") pod \"e2e-praxis-b-1e9675-openshift-default-7d7c758549-6hhbg\" (UID: \"62b94b35-a1bb-4eb2-9bc0-da2760557b3d\") " pod="openshift-ingress/e2e-praxis-b-1e9675-openshift-default-7d7c758549-6hhbg" Oct 30 12:29:39.216438 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:39.216415 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-1e9675-openshift-default-dockercfg-8pfbt\"" Oct 30 12:29:39.231559 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:39.231524 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-1e9675-openshift-default-7d7c758549-6hhbg"] Oct 30 12:29:39.291504 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:39.289602 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62b94b35-a1bb-4eb2-9bc0-da2760557b3d-istio-data\") pod \"e2e-praxis-b-1e9675-openshift-default-7d7c758549-6hhbg\" (UID: \"62b94b35-a1bb-4eb2-9bc0-da2760557b3d\") " pod="openshift-ingress/e2e-praxis-b-1e9675-openshift-default-7d7c758549-6hhbg" Oct 30 12:29:39.291504 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:39.289715 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62b94b35-a1bb-4eb2-9bc0-da2760557b3d-workload-certs\") pod \"e2e-praxis-b-1e9675-openshift-default-7d7c758549-6hhbg\" (UID: \"62b94b35-a1bb-4eb2-9bc0-da2760557b3d\") " pod="openshift-ingress/e2e-praxis-b-1e9675-openshift-default-7d7c758549-6hhbg" Oct 30 12:29:39.291504 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:39.289799 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62b94b35-a1bb-4eb2-9bc0-da2760557b3d-credential-socket\") pod \"e2e-praxis-b-1e9675-openshift-default-7d7c758549-6hhbg\" (UID: \"62b94b35-a1bb-4eb2-9bc0-da2760557b3d\") " pod="openshift-ingress/e2e-praxis-b-1e9675-openshift-default-7d7c758549-6hhbg" Oct 30 12:29:39.291504 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:39.289863 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62b94b35-a1bb-4eb2-9bc0-da2760557b3d-istio-envoy\") pod \"e2e-praxis-b-1e9675-openshift-default-7d7c758549-6hhbg\" (UID: \"62b94b35-a1bb-4eb2-9bc0-da2760557b3d\") " pod="openshift-ingress/e2e-praxis-b-1e9675-openshift-default-7d7c758549-6hhbg" Oct 30 12:29:39.291504 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:39.289888 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62b94b35-a1bb-4eb2-9bc0-da2760557b3d-istiod-ca-cert\") pod \"e2e-praxis-b-1e9675-openshift-default-7d7c758549-6hhbg\" (UID: \"62b94b35-a1bb-4eb2-9bc0-da2760557b3d\") " pod="openshift-ingress/e2e-praxis-b-1e9675-openshift-default-7d7c758549-6hhbg" Oct 30 12:29:39.291504 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:39.289946 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62b94b35-a1bb-4eb2-9bc0-da2760557b3d-workload-socket\") pod \"e2e-praxis-b-1e9675-openshift-default-7d7c758549-6hhbg\" (UID: \"62b94b35-a1bb-4eb2-9bc0-da2760557b3d\") " pod="openshift-ingress/e2e-praxis-b-1e9675-openshift-default-7d7c758549-6hhbg" Oct 30 12:29:39.291504 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:39.289978 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62b94b35-a1bb-4eb2-9bc0-da2760557b3d-istio-token\") pod \"e2e-praxis-b-1e9675-openshift-default-7d7c758549-6hhbg\" (UID: \"62b94b35-a1bb-4eb2-9bc0-da2760557b3d\") " pod="openshift-ingress/e2e-praxis-b-1e9675-openshift-default-7d7c758549-6hhbg" Oct 30 12:29:39.291504 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:39.290069 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62b94b35-a1bb-4eb2-9bc0-da2760557b3d-istio-podinfo\") pod \"e2e-praxis-b-1e9675-openshift-default-7d7c758549-6hhbg\" (UID: \"62b94b35-a1bb-4eb2-9bc0-da2760557b3d\") " pod="openshift-ingress/e2e-praxis-b-1e9675-openshift-default-7d7c758549-6hhbg" Oct 30 12:29:39.291504 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:39.290096 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckrlf\" (UniqueName: \"kubernetes.io/projected/62b94b35-a1bb-4eb2-9bc0-da2760557b3d-kube-api-access-ckrlf\") pod \"e2e-praxis-b-1e9675-openshift-default-7d7c758549-6hhbg\" (UID: \"62b94b35-a1bb-4eb2-9bc0-da2760557b3d\") " pod="openshift-ingress/e2e-praxis-b-1e9675-openshift-default-7d7c758549-6hhbg" Oct 30 12:29:39.293701 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:39.293113 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62b94b35-a1bb-4eb2-9bc0-da2760557b3d-istio-data\") pod \"e2e-praxis-b-1e9675-openshift-default-7d7c758549-6hhbg\" (UID: \"62b94b35-a1bb-4eb2-9bc0-da2760557b3d\") " pod="openshift-ingress/e2e-praxis-b-1e9675-openshift-default-7d7c758549-6hhbg" Oct 30 12:29:39.293701 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:39.293374 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62b94b35-a1bb-4eb2-9bc0-da2760557b3d-workload-certs\") pod \"e2e-praxis-b-1e9675-openshift-default-7d7c758549-6hhbg\" (UID: \"62b94b35-a1bb-4eb2-9bc0-da2760557b3d\") " pod="openshift-ingress/e2e-praxis-b-1e9675-openshift-default-7d7c758549-6hhbg" Oct 30 12:29:39.294272 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:39.293916 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62b94b35-a1bb-4eb2-9bc0-da2760557b3d-credential-socket\") pod \"e2e-praxis-b-1e9675-openshift-default-7d7c758549-6hhbg\" (UID: \"62b94b35-a1bb-4eb2-9bc0-da2760557b3d\") " pod="openshift-ingress/e2e-praxis-b-1e9675-openshift-default-7d7c758549-6hhbg" Oct 30 12:29:39.294693 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:39.294670 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62b94b35-a1bb-4eb2-9bc0-da2760557b3d-workload-socket\") pod \"e2e-praxis-b-1e9675-openshift-default-7d7c758549-6hhbg\" (UID: \"62b94b35-a1bb-4eb2-9bc0-da2760557b3d\") " pod="openshift-ingress/e2e-praxis-b-1e9675-openshift-default-7d7c758549-6hhbg" Oct 30 12:29:39.296072 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:39.295677 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62b94b35-a1bb-4eb2-9bc0-da2760557b3d-istiod-ca-cert\") pod \"e2e-praxis-b-1e9675-openshift-default-7d7c758549-6hhbg\" (UID: \"62b94b35-a1bb-4eb2-9bc0-da2760557b3d\") " pod="openshift-ingress/e2e-praxis-b-1e9675-openshift-default-7d7c758549-6hhbg" Oct 30 12:29:39.338956 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:39.338924 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62b94b35-a1bb-4eb2-9bc0-da2760557b3d-istio-envoy\") pod \"e2e-praxis-b-1e9675-openshift-default-7d7c758549-6hhbg\" (UID: \"62b94b35-a1bb-4eb2-9bc0-da2760557b3d\") " pod="openshift-ingress/e2e-praxis-b-1e9675-openshift-default-7d7c758549-6hhbg" Oct 30 12:29:39.347851 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:39.347104 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckrlf\" (UniqueName: \"kubernetes.io/projected/62b94b35-a1bb-4eb2-9bc0-da2760557b3d-kube-api-access-ckrlf\") pod \"e2e-praxis-b-1e9675-openshift-default-7d7c758549-6hhbg\" (UID: \"62b94b35-a1bb-4eb2-9bc0-da2760557b3d\") " pod="openshift-ingress/e2e-praxis-b-1e9675-openshift-default-7d7c758549-6hhbg" Oct 30 12:29:39.389353 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:39.389318 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62b94b35-a1bb-4eb2-9bc0-da2760557b3d-istio-token\") pod \"e2e-praxis-b-1e9675-openshift-default-7d7c758549-6hhbg\" (UID: \"62b94b35-a1bb-4eb2-9bc0-da2760557b3d\") " pod="openshift-ingress/e2e-praxis-b-1e9675-openshift-default-7d7c758549-6hhbg" Oct 30 12:29:39.390155 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:39.390101 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62b94b35-a1bb-4eb2-9bc0-da2760557b3d-istio-podinfo\") pod \"e2e-praxis-b-1e9675-openshift-default-7d7c758549-6hhbg\" (UID: \"62b94b35-a1bb-4eb2-9bc0-da2760557b3d\") " pod="openshift-ingress/e2e-praxis-b-1e9675-openshift-default-7d7c758549-6hhbg" Oct 30 12:29:39.488877 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:39.488560 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-1e9675-openshift-default-7d7c758549-6hhbg" Oct 30 12:29:39.699254 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:39.699047 4908 generic.go:356] "Generic (PLEG): container finished" podID="4630b841-bf2e-4019-b450-12e85497afc1" containerID="8244474829cc517bf079988c2803e3cb5b3b5c8d9e715f7210f6e7a4a1620646" exitCode=0 Oct 30 12:29:39.699254 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:39.699186 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-rv92b" event={"ID":"4630b841-bf2e-4019-b450-12e85497afc1","Type":"ContainerDied","Data":"8244474829cc517bf079988c2803e3cb5b3b5c8d9e715f7210f6e7a4a1620646"} Oct 30 12:29:39.794048 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:39.791634 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-rv92b" Oct 30 12:29:39.894692 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:39.894668 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-1e9675-openshift-default-7d7c758549-6hhbg"] Oct 30 12:29:39.898059 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:29:39.898030 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62b94b35_a1bb_4eb2_9bc0_da2760557b3d.slice/crio-0ccf62b9cd4be90953ff1f3bab5a0bd3a72b82e563709c0f34d56ac0ce3d542a WatchSource:0}: Error finding container 0ccf62b9cd4be90953ff1f3bab5a0bd3a72b82e563709c0f34d56ac0ce3d542a: Status 404 returned error can't find the container with id 0ccf62b9cd4be90953ff1f3bab5a0bd3a72b82e563709c0f34d56ac0ce3d542a Oct 30 12:29:39.900715 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:39.900683 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:29:39.900798 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:39.900745 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:29:39.900798 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:39.900774 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:29:39.906331 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:39.906310 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4630b841-bf2e-4019-b450-12e85497afc1-kube-api-access-7md78\" (UniqueName: \"kubernetes.io/projected/4630b841-bf2e-4019-b450-12e85497afc1-kube-api-access-7md78\") pod \"4630b841-bf2e-4019-b450-12e85497afc1\" (UID: \"4630b841-bf2e-4019-b450-12e85497afc1\") " Oct 30 12:29:39.906432 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:39.906344 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4630b841-bf2e-4019-b450-12e85497afc1-istio-token\" (UniqueName: \"kubernetes.io/projected/4630b841-bf2e-4019-b450-12e85497afc1-istio-token\") pod \"4630b841-bf2e-4019-b450-12e85497afc1\" (UID: \"4630b841-bf2e-4019-b450-12e85497afc1\") " Oct 30 12:29:39.906432 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:39.906384 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4630b841-bf2e-4019-b450-12e85497afc1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4630b841-bf2e-4019-b450-12e85497afc1-istio-envoy\") pod \"4630b841-bf2e-4019-b450-12e85497afc1\" (UID: \"4630b841-bf2e-4019-b450-12e85497afc1\") " Oct 30 12:29:39.906432 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:39.906428 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4630b841-bf2e-4019-b450-12e85497afc1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4630b841-bf2e-4019-b450-12e85497afc1-workload-certs\") pod \"4630b841-bf2e-4019-b450-12e85497afc1\" (UID: \"4630b841-bf2e-4019-b450-12e85497afc1\") " Oct 30 12:29:39.906597 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:39.906457 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4630b841-bf2e-4019-b450-12e85497afc1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4630b841-bf2e-4019-b450-12e85497afc1-istio-data\") pod \"4630b841-bf2e-4019-b450-12e85497afc1\" (UID: \"4630b841-bf2e-4019-b450-12e85497afc1\") " Oct 30 12:29:39.906597 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:39.906553 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4630b841-bf2e-4019-b450-12e85497afc1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4630b841-bf2e-4019-b450-12e85497afc1-istio-podinfo\") pod \"4630b841-bf2e-4019-b450-12e85497afc1\" (UID: \"4630b841-bf2e-4019-b450-12e85497afc1\") " Oct 30 12:29:39.906701 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:39.906616 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4630b841-bf2e-4019-b450-12e85497afc1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4630b841-bf2e-4019-b450-12e85497afc1-istiod-ca-cert\") pod \"4630b841-bf2e-4019-b450-12e85497afc1\" (UID: \"4630b841-bf2e-4019-b450-12e85497afc1\") " Oct 30 12:29:39.906701 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:39.906640 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4630b841-bf2e-4019-b450-12e85497afc1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4630b841-bf2e-4019-b450-12e85497afc1-credential-socket\") pod \"4630b841-bf2e-4019-b450-12e85497afc1\" (UID: \"4630b841-bf2e-4019-b450-12e85497afc1\") " Oct 30 12:29:39.906701 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:39.906659 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4630b841-bf2e-4019-b450-12e85497afc1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4630b841-bf2e-4019-b450-12e85497afc1-workload-socket\") pod \"4630b841-bf2e-4019-b450-12e85497afc1\" (UID: \"4630b841-bf2e-4019-b450-12e85497afc1\") " Oct 30 12:29:39.907157 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:39.907133 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4630b841-bf2e-4019-b450-12e85497afc1-istiod-ca-cert" pod "4630b841-bf2e-4019-b450-12e85497afc1" (UID: "4630b841-bf2e-4019-b450-12e85497afc1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:29:39.907524 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:39.907286 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4630b841-bf2e-4019-b450-12e85497afc1-workload-socket" pod "4630b841-bf2e-4019-b450-12e85497afc1" (UID: "4630b841-bf2e-4019-b450-12e85497afc1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:29:39.907524 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:39.907472 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4630b841-bf2e-4019-b450-12e85497afc1-credential-socket" pod "4630b841-bf2e-4019-b450-12e85497afc1" (UID: "4630b841-bf2e-4019-b450-12e85497afc1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:29:39.907669 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:39.907510 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4630b841-bf2e-4019-b450-12e85497afc1-workload-certs" pod "4630b841-bf2e-4019-b450-12e85497afc1" (UID: "4630b841-bf2e-4019-b450-12e85497afc1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:29:39.907669 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:39.907534 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4630b841-bf2e-4019-b450-12e85497afc1-istio-data" pod "4630b841-bf2e-4019-b450-12e85497afc1" (UID: "4630b841-bf2e-4019-b450-12e85497afc1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:29:39.909528 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:39.909493 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4630b841-bf2e-4019-b450-12e85497afc1-istio-token" pod "4630b841-bf2e-4019-b450-12e85497afc1" (UID: "4630b841-bf2e-4019-b450-12e85497afc1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:29:39.909620 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:39.909563 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4630b841-bf2e-4019-b450-12e85497afc1-istio-podinfo" pod "4630b841-bf2e-4019-b450-12e85497afc1" (UID: "4630b841-bf2e-4019-b450-12e85497afc1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 12:29:39.909801 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:39.909780 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4630b841-bf2e-4019-b450-12e85497afc1-kube-api-access-7md78" pod "4630b841-bf2e-4019-b450-12e85497afc1" (UID: "4630b841-bf2e-4019-b450-12e85497afc1"). InnerVolumeSpecName "kube-api-access-7md78". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:29:39.911054 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:39.911032 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4630b841-bf2e-4019-b450-12e85497afc1-istio-envoy" pod "4630b841-bf2e-4019-b450-12e85497afc1" (UID: "4630b841-bf2e-4019-b450-12e85497afc1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:29:40.008426 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:40.008397 4908 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4630b841-bf2e-4019-b450-12e85497afc1-istio-podinfo\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:29:40.008426 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:40.008427 4908 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4630b841-bf2e-4019-b450-12e85497afc1-istiod-ca-cert\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:29:40.008568 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:40.008437 4908 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4630b841-bf2e-4019-b450-12e85497afc1-credential-socket\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:29:40.008568 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:40.008447 4908 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4630b841-bf2e-4019-b450-12e85497afc1-workload-socket\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:29:40.008568 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:40.008457 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7md78\" (UniqueName: \"kubernetes.io/projected/4630b841-bf2e-4019-b450-12e85497afc1-kube-api-access-7md78\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:29:40.008568 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:40.008466 4908 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4630b841-bf2e-4019-b450-12e85497afc1-istio-token\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:29:40.008568 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:40.008473 4908 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4630b841-bf2e-4019-b450-12e85497afc1-istio-envoy\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:29:40.008568 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:40.008485 4908 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4630b841-bf2e-4019-b450-12e85497afc1-workload-certs\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:29:40.008568 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:40.008496 4908 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4630b841-bf2e-4019-b450-12e85497afc1-istio-data\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:29:40.711084 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:40.710979 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-1e9675-openshift-default-7d7c758549-6hhbg" event={"ID":"62b94b35-a1bb-4eb2-9bc0-da2760557b3d","Type":"ContainerStarted","Data":"08fec614e1c0c235a1facb20245ba78167959eb61eb0a480c1b8f61483dcf3f5"} Oct 30 12:29:40.711084 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:40.711053 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-1e9675-openshift-default-7d7c758549-6hhbg" event={"ID":"62b94b35-a1bb-4eb2-9bc0-da2760557b3d","Type":"ContainerStarted","Data":"0ccf62b9cd4be90953ff1f3bab5a0bd3a72b82e563709c0f34d56ac0ce3d542a"} Oct 30 12:29:40.712526 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:40.712502 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-rv92b" Oct 30 12:29:40.712656 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:40.712524 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-rv92b" event={"ID":"4630b841-bf2e-4019-b450-12e85497afc1","Type":"ContainerDied","Data":"86b211fca371add4df61097c94ef65e489f7e8720ba3918129f1c2d26f02ff9c"} Oct 30 12:29:40.712656 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:40.712572 4908 scope.go:122] "RemoveContainer" containerID="8244474829cc517bf079988c2803e3cb5b3b5c8d9e715f7210f6e7a4a1620646" Oct 30 12:29:40.778499 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:40.778456 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-1e9675-openshift-default-7d7c758549-6hhbg" podStartSLOduration=1.778443685 podStartE2EDuration="1.778443685s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:29:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:29:40.764181074 +0000 UTC m=+2137.278609818" watchObservedRunningTime="2026-10-30 12:29:40.778443685 +0000 UTC m=+2137.292872429" Oct 30 12:29:40.799177 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:40.799150 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-rv92b"] Oct 30 12:29:40.809432 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:40.809404 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-rv92b"] Oct 30 12:29:41.489931 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:41.489888 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-1e9675-openshift-default-7d7c758549-6hhbg" Oct 30 12:29:41.497311 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:41.497284 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-1e9675-openshift-default-7d7c758549-6hhbg" Oct 30 12:29:41.616929 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:41.616864 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4630b841-bf2e-4019-b450-12e85497afc1" path="/var/lib/kubelet/pods/4630b841-bf2e-4019-b450-12e85497afc1/volumes" Oct 30 12:29:41.722086 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:41.722047 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-1e9675-openshift-default-7d7c758549-6hhbg" Oct 30 12:29:41.723159 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:41.723133 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-1e9675-openshift-default-7d7c758549-6hhbg" Oct 30 12:29:41.996652 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:41.996604 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-bnhn5"] Oct 30 12:29:41.997306 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:41.997286 4908 cpu_manager.go:414] "containerMap: removing container" podUID="4630b841-bf2e-4019-b450-12e85497afc1" containerName="istio-proxy" Oct 30 12:29:41.997306 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:41.997308 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4630b841-bf2e-4019-b450-12e85497afc1" containerName="istio-proxy" Oct 30 12:29:41.997460 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:41.997446 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="4630b841-bf2e-4019-b450-12e85497afc1" containerName="istio-proxy" Oct 30 12:29:42.002337 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:42.002309 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-bnhn5" Oct 30 12:29:42.021007 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:42.020978 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-41bdda-openshift-default-dockercfg-j9hg8\"" Oct 30 12:29:42.022231 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:42.022206 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-bnhn5"] Oct 30 12:29:42.055377 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:42.055351 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbf8595f-0d2c-471d-90dd-63425baa0510-workload-socket\") pod \"e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-bnhn5\" (UID: \"fbf8595f-0d2c-471d-90dd-63425baa0510\") " pod="openshift-ingress/e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-bnhn5" Oct 30 12:29:42.055547 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:42.055386 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbf8595f-0d2c-471d-90dd-63425baa0510-credential-socket\") pod \"e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-bnhn5\" (UID: \"fbf8595f-0d2c-471d-90dd-63425baa0510\") " pod="openshift-ingress/e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-bnhn5" Oct 30 12:29:42.055621 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:42.055547 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbf8595f-0d2c-471d-90dd-63425baa0510-istiod-ca-cert\") pod \"e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-bnhn5\" (UID: \"fbf8595f-0d2c-471d-90dd-63425baa0510\") " pod="openshift-ingress/e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-bnhn5" Oct 30 12:29:42.055678 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:42.055652 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbf8595f-0d2c-471d-90dd-63425baa0510-workload-certs\") pod \"e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-bnhn5\" (UID: \"fbf8595f-0d2c-471d-90dd-63425baa0510\") " pod="openshift-ingress/e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-bnhn5" Oct 30 12:29:42.055741 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:42.055699 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbf8595f-0d2c-471d-90dd-63425baa0510-istio-envoy\") pod \"e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-bnhn5\" (UID: \"fbf8595f-0d2c-471d-90dd-63425baa0510\") " pod="openshift-ingress/e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-bnhn5" Oct 30 12:29:42.055741 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:42.055730 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbf8595f-0d2c-471d-90dd-63425baa0510-istio-podinfo\") pod \"e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-bnhn5\" (UID: \"fbf8595f-0d2c-471d-90dd-63425baa0510\") " pod="openshift-ingress/e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-bnhn5" Oct 30 12:29:42.055869 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:42.055770 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbf8595f-0d2c-471d-90dd-63425baa0510-istio-token\") pod \"e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-bnhn5\" (UID: \"fbf8595f-0d2c-471d-90dd-63425baa0510\") " pod="openshift-ingress/e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-bnhn5" Oct 30 12:29:42.055951 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:42.055927 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvtww\" (UniqueName: \"kubernetes.io/projected/fbf8595f-0d2c-471d-90dd-63425baa0510-kube-api-access-vvtww\") pod \"e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-bnhn5\" (UID: \"fbf8595f-0d2c-471d-90dd-63425baa0510\") " pod="openshift-ingress/e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-bnhn5" Oct 30 12:29:42.056000 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:42.055986 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbf8595f-0d2c-471d-90dd-63425baa0510-istio-data\") pod \"e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-bnhn5\" (UID: \"fbf8595f-0d2c-471d-90dd-63425baa0510\") " pod="openshift-ingress/e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-bnhn5" Oct 30 12:29:42.156969 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:42.156933 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbf8595f-0d2c-471d-90dd-63425baa0510-workload-socket\") pod \"e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-bnhn5\" (UID: \"fbf8595f-0d2c-471d-90dd-63425baa0510\") " pod="openshift-ingress/e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-bnhn5" Oct 30 12:29:42.157159 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:42.156984 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbf8595f-0d2c-471d-90dd-63425baa0510-credential-socket\") pod \"e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-bnhn5\" (UID: \"fbf8595f-0d2c-471d-90dd-63425baa0510\") " pod="openshift-ingress/e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-bnhn5" Oct 30 12:29:42.157159 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:42.157034 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbf8595f-0d2c-471d-90dd-63425baa0510-istiod-ca-cert\") pod \"e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-bnhn5\" (UID: \"fbf8595f-0d2c-471d-90dd-63425baa0510\") " pod="openshift-ingress/e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-bnhn5" Oct 30 12:29:42.157159 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:42.157105 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbf8595f-0d2c-471d-90dd-63425baa0510-workload-certs\") pod \"e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-bnhn5\" (UID: \"fbf8595f-0d2c-471d-90dd-63425baa0510\") " pod="openshift-ingress/e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-bnhn5" Oct 30 12:29:42.157159 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:42.157143 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbf8595f-0d2c-471d-90dd-63425baa0510-istio-envoy\") pod \"e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-bnhn5\" (UID: \"fbf8595f-0d2c-471d-90dd-63425baa0510\") " pod="openshift-ingress/e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-bnhn5" Oct 30 12:29:42.157362 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:42.157174 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbf8595f-0d2c-471d-90dd-63425baa0510-istio-podinfo\") pod \"e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-bnhn5\" (UID: \"fbf8595f-0d2c-471d-90dd-63425baa0510\") " pod="openshift-ingress/e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-bnhn5" Oct 30 12:29:42.157362 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:42.157213 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbf8595f-0d2c-471d-90dd-63425baa0510-istio-token\") pod \"e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-bnhn5\" (UID: \"fbf8595f-0d2c-471d-90dd-63425baa0510\") " pod="openshift-ingress/e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-bnhn5" Oct 30 12:29:42.157362 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:42.157258 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbf8595f-0d2c-471d-90dd-63425baa0510-workload-socket\") pod \"e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-bnhn5\" (UID: \"fbf8595f-0d2c-471d-90dd-63425baa0510\") " pod="openshift-ingress/e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-bnhn5" Oct 30 12:29:42.157362 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:42.157268 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvtww\" (UniqueName: \"kubernetes.io/projected/fbf8595f-0d2c-471d-90dd-63425baa0510-kube-api-access-vvtww\") pod \"e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-bnhn5\" (UID: \"fbf8595f-0d2c-471d-90dd-63425baa0510\") " pod="openshift-ingress/e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-bnhn5" Oct 30 12:29:42.157362 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:42.157331 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbf8595f-0d2c-471d-90dd-63425baa0510-istio-data\") pod \"e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-bnhn5\" (UID: \"fbf8595f-0d2c-471d-90dd-63425baa0510\") " pod="openshift-ingress/e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-bnhn5" Oct 30 12:29:42.157362 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:42.157355 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbf8595f-0d2c-471d-90dd-63425baa0510-credential-socket\") pod \"e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-bnhn5\" (UID: \"fbf8595f-0d2c-471d-90dd-63425baa0510\") " pod="openshift-ingress/e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-bnhn5" Oct 30 12:29:42.157640 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:42.157514 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbf8595f-0d2c-471d-90dd-63425baa0510-workload-certs\") pod \"e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-bnhn5\" (UID: \"fbf8595f-0d2c-471d-90dd-63425baa0510\") " pod="openshift-ingress/e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-bnhn5" Oct 30 12:29:42.157751 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:42.157726 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbf8595f-0d2c-471d-90dd-63425baa0510-istio-data\") pod \"e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-bnhn5\" (UID: \"fbf8595f-0d2c-471d-90dd-63425baa0510\") " pod="openshift-ingress/e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-bnhn5" Oct 30 12:29:42.157933 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:42.157909 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbf8595f-0d2c-471d-90dd-63425baa0510-istiod-ca-cert\") pod \"e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-bnhn5\" (UID: \"fbf8595f-0d2c-471d-90dd-63425baa0510\") " pod="openshift-ingress/e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-bnhn5" Oct 30 12:29:42.159422 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:42.159401 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbf8595f-0d2c-471d-90dd-63425baa0510-istio-envoy\") pod \"e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-bnhn5\" (UID: \"fbf8595f-0d2c-471d-90dd-63425baa0510\") " pod="openshift-ingress/e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-bnhn5" Oct 30 12:29:42.160021 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:42.160002 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbf8595f-0d2c-471d-90dd-63425baa0510-istio-podinfo\") pod \"e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-bnhn5\" (UID: \"fbf8595f-0d2c-471d-90dd-63425baa0510\") " pod="openshift-ingress/e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-bnhn5" Oct 30 12:29:42.179200 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:42.179170 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbf8595f-0d2c-471d-90dd-63425baa0510-istio-token\") pod \"e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-bnhn5\" (UID: \"fbf8595f-0d2c-471d-90dd-63425baa0510\") " pod="openshift-ingress/e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-bnhn5" Oct 30 12:29:42.179300 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:42.179169 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvtww\" (UniqueName: \"kubernetes.io/projected/fbf8595f-0d2c-471d-90dd-63425baa0510-kube-api-access-vvtww\") pod \"e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-bnhn5\" (UID: \"fbf8595f-0d2c-471d-90dd-63425baa0510\") " pod="openshift-ingress/e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-bnhn5" Oct 30 12:29:42.321307 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:42.321229 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-bnhn5" Oct 30 12:29:42.647938 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:42.647909 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-bnhn5"] Oct 30 12:29:42.650590 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:29:42.650553 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbf8595f_0d2c_471d_90dd_63425baa0510.slice/crio-0c4d4770e9ac5f229ef8f28f1f03fc58bc44884101d163098d85c6e09c5b15bd WatchSource:0}: Error finding container 0c4d4770e9ac5f229ef8f28f1f03fc58bc44884101d163098d85c6e09c5b15bd: Status 404 returned error can't find the container with id 0c4d4770e9ac5f229ef8f28f1f03fc58bc44884101d163098d85c6e09c5b15bd Oct 30 12:29:42.653043 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:42.653012 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:29:42.653146 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:42.653074 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:29:42.653146 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:42.653103 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:29:42.732409 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:42.732378 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-bnhn5" event={"ID":"fbf8595f-0d2c-471d-90dd-63425baa0510","Type":"ContainerStarted","Data":"1f198fc51fb02ed8f0e2a3001b60164e221c8f8ec25eb1acbeace1647cc19b34"} Oct 30 12:29:42.732714 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:42.732435 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-bnhn5" event={"ID":"fbf8595f-0d2c-471d-90dd-63425baa0510","Type":"ContainerStarted","Data":"0c4d4770e9ac5f229ef8f28f1f03fc58bc44884101d163098d85c6e09c5b15bd"} Oct 30 12:29:42.779079 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:42.779019 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-bnhn5" podStartSLOduration=1.77900107 podStartE2EDuration="1.77900107s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:29:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:29:42.774418884 +0000 UTC m=+2139.288847630" watchObservedRunningTime="2026-10-30 12:29:42.77900107 +0000 UTC m=+2139.293429818" Oct 30 12:29:43.322566 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:43.322530 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-bnhn5" Oct 30 12:29:43.325320 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:43.325277 4908 patch_prober.go:29] interesting pod/e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-bnhn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Oct 30 12:29:43.325505 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:43.325355 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-bnhn5" podUID="fbf8595f-0d2c-471d-90dd-63425baa0510" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Oct 30 12:29:43.635802 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:43.635776 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-8c5bs" Oct 30 12:29:43.744299 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:43.744268 4908 generic.go:356] "Generic (PLEG): container finished" podID="07619fd1-c52a-45ce-851e-7436a3cb0723" containerID="5acbfb21e58944ad079746c35dad9949744f021119be9224397bc2c0cc111c32" exitCode=0 Oct 30 12:29:43.744772 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:43.744336 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-8c5bs" Oct 30 12:29:43.745134 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:43.744355 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-8c5bs" event={"ID":"07619fd1-c52a-45ce-851e-7436a3cb0723","Type":"ContainerDied","Data":"5acbfb21e58944ad079746c35dad9949744f021119be9224397bc2c0cc111c32"} Oct 30 12:29:43.745208 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:43.745168 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-8c5bs" event={"ID":"07619fd1-c52a-45ce-851e-7436a3cb0723","Type":"ContainerDied","Data":"31a9a3477177809d4aa10a5e0650ee9bdb24b0e064ac56ab040048e1a784fa4b"} Oct 30 12:29:43.745208 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:43.745202 4908 scope.go:122] "RemoveContainer" containerID="5acbfb21e58944ad079746c35dad9949744f021119be9224397bc2c0cc111c32" Oct 30 12:29:43.757350 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:43.757290 4908 scope.go:122] "RemoveContainer" containerID="5acbfb21e58944ad079746c35dad9949744f021119be9224397bc2c0cc111c32" Oct 30 12:29:43.757568 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:29:43.757551 4908 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5acbfb21e58944ad079746c35dad9949744f021119be9224397bc2c0cc111c32\": container with ID starting with 5acbfb21e58944ad079746c35dad9949744f021119be9224397bc2c0cc111c32 not found: ID does not exist" containerID="5acbfb21e58944ad079746c35dad9949744f021119be9224397bc2c0cc111c32" Oct 30 12:29:43.757655 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:43.757576 4908 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5acbfb21e58944ad079746c35dad9949744f021119be9224397bc2c0cc111c32"} err="failed to get container status \"5acbfb21e58944ad079746c35dad9949744f021119be9224397bc2c0cc111c32\": rpc error: code = NotFound desc = could not find container \"5acbfb21e58944ad079746c35dad9949744f021119be9224397bc2c0cc111c32\": container with ID starting with 5acbfb21e58944ad079746c35dad9949744f021119be9224397bc2c0cc111c32 not found: ID does not exist" Oct 30 12:29:43.794330 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:43.794298 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07619fd1-c52a-45ce-851e-7436a3cb0723-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07619fd1-c52a-45ce-851e-7436a3cb0723-istiod-ca-cert\") pod \"07619fd1-c52a-45ce-851e-7436a3cb0723\" (UID: \"07619fd1-c52a-45ce-851e-7436a3cb0723\") " Oct 30 12:29:43.794527 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:43.794517 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07619fd1-c52a-45ce-851e-7436a3cb0723-istio-token\" (UniqueName: \"kubernetes.io/projected/07619fd1-c52a-45ce-851e-7436a3cb0723-istio-token\") pod \"07619fd1-c52a-45ce-851e-7436a3cb0723\" (UID: \"07619fd1-c52a-45ce-851e-7436a3cb0723\") " Oct 30 12:29:43.794627 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:43.794618 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07619fd1-c52a-45ce-851e-7436a3cb0723-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07619fd1-c52a-45ce-851e-7436a3cb0723-istio-envoy\") pod \"07619fd1-c52a-45ce-851e-7436a3cb0723\" (UID: \"07619fd1-c52a-45ce-851e-7436a3cb0723\") " Oct 30 12:29:43.794714 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:43.794704 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07619fd1-c52a-45ce-851e-7436a3cb0723-kube-api-access-jjkfp\" (UniqueName: \"kubernetes.io/projected/07619fd1-c52a-45ce-851e-7436a3cb0723-kube-api-access-jjkfp\") pod \"07619fd1-c52a-45ce-851e-7436a3cb0723\" (UID: \"07619fd1-c52a-45ce-851e-7436a3cb0723\") " Oct 30 12:29:43.794796 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:43.794788 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/07619fd1-c52a-45ce-851e-7436a3cb0723-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07619fd1-c52a-45ce-851e-7436a3cb0723-istio-podinfo\") pod \"07619fd1-c52a-45ce-851e-7436a3cb0723\" (UID: \"07619fd1-c52a-45ce-851e-7436a3cb0723\") " Oct 30 12:29:43.794958 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:43.794942 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07619fd1-c52a-45ce-851e-7436a3cb0723-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07619fd1-c52a-45ce-851e-7436a3cb0723-credential-socket\") pod \"07619fd1-c52a-45ce-851e-7436a3cb0723\" (UID: \"07619fd1-c52a-45ce-851e-7436a3cb0723\") " Oct 30 12:29:43.795449 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:43.795425 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07619fd1-c52a-45ce-851e-7436a3cb0723-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07619fd1-c52a-45ce-851e-7436a3cb0723-istio-data\") pod \"07619fd1-c52a-45ce-851e-7436a3cb0723\" (UID: \"07619fd1-c52a-45ce-851e-7436a3cb0723\") " Oct 30 12:29:43.795673 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:43.795655 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07619fd1-c52a-45ce-851e-7436a3cb0723-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07619fd1-c52a-45ce-851e-7436a3cb0723-workload-certs\") pod \"07619fd1-c52a-45ce-851e-7436a3cb0723\" (UID: \"07619fd1-c52a-45ce-851e-7436a3cb0723\") " Oct 30 12:29:43.795909 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:43.795891 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07619fd1-c52a-45ce-851e-7436a3cb0723-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07619fd1-c52a-45ce-851e-7436a3cb0723-workload-socket\") pod \"07619fd1-c52a-45ce-851e-7436a3cb0723\" (UID: \"07619fd1-c52a-45ce-851e-7436a3cb0723\") " Oct 30 12:29:43.796985 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:43.795462 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07619fd1-c52a-45ce-851e-7436a3cb0723-istiod-ca-cert" pod "07619fd1-c52a-45ce-851e-7436a3cb0723" (UID: "07619fd1-c52a-45ce-851e-7436a3cb0723"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:29:43.797107 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:43.796107 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07619fd1-c52a-45ce-851e-7436a3cb0723-credential-socket" pod "07619fd1-c52a-45ce-851e-7436a3cb0723" (UID: "07619fd1-c52a-45ce-851e-7436a3cb0723"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:29:43.797189 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:43.796766 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07619fd1-c52a-45ce-851e-7436a3cb0723-workload-socket" pod "07619fd1-c52a-45ce-851e-7436a3cb0723" (UID: "07619fd1-c52a-45ce-851e-7436a3cb0723"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:29:43.797266 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:43.796964 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07619fd1-c52a-45ce-851e-7436a3cb0723-istio-data" pod "07619fd1-c52a-45ce-851e-7436a3cb0723" (UID: "07619fd1-c52a-45ce-851e-7436a3cb0723"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:29:43.797531 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:43.797513 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07619fd1-c52a-45ce-851e-7436a3cb0723-workload-certs" pod "07619fd1-c52a-45ce-851e-7436a3cb0723" (UID: "07619fd1-c52a-45ce-851e-7436a3cb0723"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:29:43.812256 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:43.812235 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07619fd1-c52a-45ce-851e-7436a3cb0723-kube-api-access-jjkfp" pod "07619fd1-c52a-45ce-851e-7436a3cb0723" (UID: "07619fd1-c52a-45ce-851e-7436a3cb0723"). InnerVolumeSpecName "kube-api-access-jjkfp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:29:43.818428 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:43.818397 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07619fd1-c52a-45ce-851e-7436a3cb0723-istio-token" pod "07619fd1-c52a-45ce-851e-7436a3cb0723" (UID: "07619fd1-c52a-45ce-851e-7436a3cb0723"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:29:43.819570 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:43.819545 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07619fd1-c52a-45ce-851e-7436a3cb0723-istio-envoy" pod "07619fd1-c52a-45ce-851e-7436a3cb0723" (UID: "07619fd1-c52a-45ce-851e-7436a3cb0723"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:29:43.820044 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:43.820020 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/07619fd1-c52a-45ce-851e-7436a3cb0723-istio-podinfo" pod "07619fd1-c52a-45ce-851e-7436a3cb0723" (UID: "07619fd1-c52a-45ce-851e-7436a3cb0723"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 12:29:43.899050 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:43.898964 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjkfp\" (UniqueName: \"kubernetes.io/projected/07619fd1-c52a-45ce-851e-7436a3cb0723-kube-api-access-jjkfp\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:29:43.899050 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:43.899005 4908 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07619fd1-c52a-45ce-851e-7436a3cb0723-istio-podinfo\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:29:43.899050 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:43.899020 4908 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07619fd1-c52a-45ce-851e-7436a3cb0723-credential-socket\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:29:43.899050 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:43.899034 4908 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07619fd1-c52a-45ce-851e-7436a3cb0723-istio-data\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:29:43.899050 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:43.899048 4908 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07619fd1-c52a-45ce-851e-7436a3cb0723-workload-certs\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:29:43.899332 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:43.899060 4908 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07619fd1-c52a-45ce-851e-7436a3cb0723-workload-socket\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:29:43.899332 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:43.899073 4908 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07619fd1-c52a-45ce-851e-7436a3cb0723-istiod-ca-cert\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:29:43.899332 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:43.899086 4908 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07619fd1-c52a-45ce-851e-7436a3cb0723-istio-token\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:29:43.899332 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:43.899099 4908 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07619fd1-c52a-45ce-851e-7436a3cb0723-istio-envoy\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:29:44.094167 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:44.094126 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-8c5bs"] Oct 30 12:29:44.105725 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:44.105691 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-8c5bs"] Oct 30 12:29:44.321776 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:44.321735 4908 patch_prober.go:29] interesting pod/e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-bnhn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Oct 30 12:29:44.321986 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:44.321792 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-bnhn5" podUID="fbf8595f-0d2c-471d-90dd-63425baa0510" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Oct 30 12:29:45.322014 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:45.321976 4908 patch_prober.go:29] interesting pod/e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-bnhn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Oct 30 12:29:45.322447 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:45.322028 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-bnhn5" podUID="fbf8595f-0d2c-471d-90dd-63425baa0510" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Oct 30 12:29:45.609016 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:45.608928 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07619fd1-c52a-45ce-851e-7436a3cb0723" path="/var/lib/kubelet/pods/07619fd1-c52a-45ce-851e-7436a3cb0723/volumes" Oct 30 12:29:46.327117 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:46.327077 4908 patch_prober.go:29] interesting pod/e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-bnhn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Oct 30 12:29:46.338860 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:46.327724 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-bnhn5" podUID="fbf8595f-0d2c-471d-90dd-63425baa0510" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Oct 30 12:29:47.341863 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:47.322268 4908 patch_prober.go:29] interesting pod/e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-bnhn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Oct 30 12:29:47.341863 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:47.322326 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-bnhn5" podUID="fbf8595f-0d2c-471d-90dd-63425baa0510" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Oct 30 12:29:48.322311 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:48.322272 4908 patch_prober.go:29] interesting pod/e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-bnhn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Oct 30 12:29:48.322509 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:48.322339 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-bnhn5" podUID="fbf8595f-0d2c-471d-90dd-63425baa0510" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Oct 30 12:29:49.326124 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:49.326091 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-bnhn5" Oct 30 12:29:49.326542 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:49.326519 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-bnhn5" Oct 30 12:29:49.327338 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:49.327317 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-bnhn5" Oct 30 12:29:49.683774 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:29:49.683678 4908 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vn6vv" containerName="extractor" Oct 30 12:29:54.829920 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:54.829890 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8e3112-4nrwf"] Oct 30 12:29:54.830431 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:54.830417 4908 cpu_manager.go:414] "containerMap: removing container" podUID="07619fd1-c52a-45ce-851e-7436a3cb0723" containerName="istio-proxy" Oct 30 12:29:54.830491 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:54.830431 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07619fd1-c52a-45ce-851e-7436a3cb0723" containerName="istio-proxy" Oct 30 12:29:54.830604 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:54.830593 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="07619fd1-c52a-45ce-851e-7436a3cb0723" containerName="istio-proxy" Oct 30 12:29:54.835893 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:54.835872 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8e3112-4nrwf" Oct 30 12:29:54.853515 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:54.853487 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8e3112-4nrwf"] Oct 30 12:29:54.860150 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:54.860085 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 12:29:54.860342 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:54.860148 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 12:29:54.860571 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:54.860552 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 12:29:54.860731 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:54.860663 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s75hq\"" Oct 30 12:29:54.875924 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:54.875892 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhfr7\" (UniqueName: \"kubernetes.io/projected/a81f4750-0da9-4032-a80e-1aab7a03ac82-kube-api-access-rhfr7\") pod \"maas-api-revoke-keys-e2e-ipp-a-8e3112-4nrwf\" (UID: \"a81f4750-0da9-4032-a80e-1aab7a03ac82\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8e3112-4nrwf" Oct 30 12:29:54.876048 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:54.876029 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a81f4750-0da9-4032-a80e-1aab7a03ac82-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-8e3112-4nrwf\" (UID: \"a81f4750-0da9-4032-a80e-1aab7a03ac82\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8e3112-4nrwf" Oct 30 12:29:54.876255 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:54.876236 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a81f4750-0da9-4032-a80e-1aab7a03ac82-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-8e3112-4nrwf\" (UID: \"a81f4750-0da9-4032-a80e-1aab7a03ac82\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8e3112-4nrwf" Oct 30 12:29:54.977336 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:54.977299 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a81f4750-0da9-4032-a80e-1aab7a03ac82-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-8e3112-4nrwf\" (UID: \"a81f4750-0da9-4032-a80e-1aab7a03ac82\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8e3112-4nrwf" Oct 30 12:29:54.977513 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:54.977497 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a81f4750-0da9-4032-a80e-1aab7a03ac82-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-8e3112-4nrwf\" (UID: \"a81f4750-0da9-4032-a80e-1aab7a03ac82\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8e3112-4nrwf" Oct 30 12:29:54.977577 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:54.977533 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhfr7\" (UniqueName: \"kubernetes.io/projected/a81f4750-0da9-4032-a80e-1aab7a03ac82-kube-api-access-rhfr7\") pod \"maas-api-revoke-keys-e2e-ipp-a-8e3112-4nrwf\" (UID: \"a81f4750-0da9-4032-a80e-1aab7a03ac82\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8e3112-4nrwf" Oct 30 12:29:54.978162 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:54.978138 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a81f4750-0da9-4032-a80e-1aab7a03ac82-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-8e3112-4nrwf\" (UID: \"a81f4750-0da9-4032-a80e-1aab7a03ac82\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8e3112-4nrwf" Oct 30 12:29:54.979777 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:54.979755 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a81f4750-0da9-4032-a80e-1aab7a03ac82-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-8e3112-4nrwf\" (UID: \"a81f4750-0da9-4032-a80e-1aab7a03ac82\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8e3112-4nrwf" Oct 30 12:29:55.000146 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:55.000117 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhfr7\" (UniqueName: \"kubernetes.io/projected/a81f4750-0da9-4032-a80e-1aab7a03ac82-kube-api-access-rhfr7\") pod \"maas-api-revoke-keys-e2e-ipp-a-8e3112-4nrwf\" (UID: \"a81f4750-0da9-4032-a80e-1aab7a03ac82\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8e3112-4nrwf" Oct 30 12:29:55.149162 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:55.149088 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8e3112-4nrwf" Oct 30 12:29:55.459795 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:55.459768 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8e3112-4nrwf"] Oct 30 12:29:55.461670 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:29:55.461638 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda81f4750_0da9_4032_a80e_1aab7a03ac82.slice/crio-39cc5eacf5cca850bd2e0003073027b82ba77e7028b3dacc00951a79acb6a503 WatchSource:0}: Error finding container 39cc5eacf5cca850bd2e0003073027b82ba77e7028b3dacc00951a79acb6a503: Status 404 returned error can't find the container with id 39cc5eacf5cca850bd2e0003073027b82ba77e7028b3dacc00951a79acb6a503 Oct 30 12:29:55.861727 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:55.861690 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8e3112-4nrwf" event={"ID":"a81f4750-0da9-4032-a80e-1aab7a03ac82","Type":"ContainerStarted","Data":"c27b1d681498e5c17ad86cf45797658d107f7cebb6ede02f548b51cf192a3387"} Oct 30 12:29:55.862150 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:55.861753 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8e3112-4nrwf" event={"ID":"a81f4750-0da9-4032-a80e-1aab7a03ac82","Type":"ContainerStarted","Data":"39cc5eacf5cca850bd2e0003073027b82ba77e7028b3dacc00951a79acb6a503"} Oct 30 12:29:55.900277 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:55.900227 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8e3112-4nrwf" podStartSLOduration=1.90021151 podStartE2EDuration="1.90021151s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:29:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:29:55.899408611 +0000 UTC m=+2152.413837357" watchObservedRunningTime="2026-10-30 12:29:55.90021151 +0000 UTC m=+2152.414640255" Oct 30 12:29:56.874048 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:56.874005 4908 generic.go:356] "Generic (PLEG): container finished" podID="a81f4750-0da9-4032-a80e-1aab7a03ac82" containerID="c27b1d681498e5c17ad86cf45797658d107f7cebb6ede02f548b51cf192a3387" exitCode=0 Oct 30 12:29:56.874515 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:56.874096 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8e3112-4nrwf" event={"ID":"a81f4750-0da9-4032-a80e-1aab7a03ac82","Type":"ContainerDied","Data":"c27b1d681498e5c17ad86cf45797658d107f7cebb6ede02f548b51cf192a3387"} Oct 30 12:29:58.048189 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:58.048165 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8e3112-4nrwf" Oct 30 12:29:58.116779 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:58.116742 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a81f4750-0da9-4032-a80e-1aab7a03ac82-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a81f4750-0da9-4032-a80e-1aab7a03ac82-maas-api-xks-ca\") pod \"a81f4750-0da9-4032-a80e-1aab7a03ac82\" (UID: \"a81f4750-0da9-4032-a80e-1aab7a03ac82\") " Oct 30 12:29:58.116960 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:58.116803 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a81f4750-0da9-4032-a80e-1aab7a03ac82-kube-api-access-rhfr7\" (UniqueName: \"kubernetes.io/projected/a81f4750-0da9-4032-a80e-1aab7a03ac82-kube-api-access-rhfr7\") pod \"a81f4750-0da9-4032-a80e-1aab7a03ac82\" (UID: \"a81f4750-0da9-4032-a80e-1aab7a03ac82\") " Oct 30 12:29:58.116960 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:58.116945 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a81f4750-0da9-4032-a80e-1aab7a03ac82-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a81f4750-0da9-4032-a80e-1aab7a03ac82-maas-api-openshift-ca\") pod \"a81f4750-0da9-4032-a80e-1aab7a03ac82\" (UID: \"a81f4750-0da9-4032-a80e-1aab7a03ac82\") " Oct 30 12:29:58.117374 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:58.117344 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a81f4750-0da9-4032-a80e-1aab7a03ac82-maas-api-openshift-ca" pod "a81f4750-0da9-4032-a80e-1aab7a03ac82" (UID: "a81f4750-0da9-4032-a80e-1aab7a03ac82"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:29:58.117558 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:58.117536 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a81f4750-0da9-4032-a80e-1aab7a03ac82-maas-api-openshift-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:29:58.119011 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:58.118982 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a81f4750-0da9-4032-a80e-1aab7a03ac82-maas-api-xks-ca" pod "a81f4750-0da9-4032-a80e-1aab7a03ac82" (UID: "a81f4750-0da9-4032-a80e-1aab7a03ac82"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:29:58.119365 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:58.119341 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a81f4750-0da9-4032-a80e-1aab7a03ac82-kube-api-access-rhfr7" pod "a81f4750-0da9-4032-a80e-1aab7a03ac82" (UID: "a81f4750-0da9-4032-a80e-1aab7a03ac82"). InnerVolumeSpecName "kube-api-access-rhfr7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:29:58.219003 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:58.218931 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a81f4750-0da9-4032-a80e-1aab7a03ac82-maas-api-xks-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:29:58.219003 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:58.218958 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhfr7\" (UniqueName: \"kubernetes.io/projected/a81f4750-0da9-4032-a80e-1aab7a03ac82-kube-api-access-rhfr7\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:29:58.893746 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:58.893701 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8e3112-4nrwf" event={"ID":"a81f4750-0da9-4032-a80e-1aab7a03ac82","Type":"ContainerDied","Data":"39cc5eacf5cca850bd2e0003073027b82ba77e7028b3dacc00951a79acb6a503"} Oct 30 12:29:58.893931 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:58.893717 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8e3112-4nrwf" Oct 30 12:29:58.893931 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:29:58.893770 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="39cc5eacf5cca850bd2e0003073027b82ba77e7028b3dacc00951a79acb6a503" Oct 30 12:30:00.161072 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:00.161032 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c1b1e8-29889390-hj6n9"] Oct 30 12:30:00.162082 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:00.162057 4908 cpu_manager.go:414] "containerMap: removing container" podUID="a81f4750-0da9-4032-a80e-1aab7a03ac82" containerName="revoke-keys" Oct 30 12:30:00.162082 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:00.162081 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a81f4750-0da9-4032-a80e-1aab7a03ac82" containerName="revoke-keys" Oct 30 12:30:00.162264 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:00.162253 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="a81f4750-0da9-4032-a80e-1aab7a03ac82" containerName="revoke-keys" Oct 30 12:30:00.166451 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:00.166424 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a7148-29889390-jpq2d"] Oct 30 12:30:00.166579 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:00.166557 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c1b1e8-29889390-hj6n9" Oct 30 12:30:00.177096 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:00.177063 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889390-pz259"] Oct 30 12:30:00.177215 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:00.177155 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a7148-29889390-jpq2d" Oct 30 12:30:00.186884 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:00.186819 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s75hq\"" Oct 30 12:30:00.187017 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:00.186980 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 12:30:00.187097 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:00.187082 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889390-pz259" Oct 30 12:30:00.189118 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:00.189093 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-c44eee-29889390-pjjzm"] Oct 30 12:30:00.189849 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:00.189803 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 12:30:00.315854 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:00.290693 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkwfh\" (UniqueName: \"kubernetes.io/projected/92ae9e02-1241-494f-8833-6081d874a34d-kube-api-access-kkwfh\") pod \"maas-api-key-cleanup-e2e-worker-w1-c1b1e8-29889390-hj6n9\" (UID: \"92ae9e02-1241-494f-8833-6081d874a34d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c1b1e8-29889390-hj6n9" Oct 30 12:30:00.315854 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:00.290786 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6npwr\" (UniqueName: \"kubernetes.io/projected/273ee4cd-dd11-4204-ba40-9073fde47d1d-kube-api-access-6npwr\") pod \"maas-api-key-cleanup-29889390-pz259\" (UID: \"273ee4cd-dd11-4204-ba40-9073fde47d1d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889390-pz259" Oct 30 12:30:00.315854 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:00.290866 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhtzc\" (UniqueName: \"kubernetes.io/projected/101bab62-abe6-4870-83e8-d30758312a0c-kube-api-access-hhtzc\") pod \"maas-api-key-cleanup-e2e-worker-w3-6a7148-29889390-jpq2d\" (UID: \"101bab62-abe6-4870-83e8-d30758312a0c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a7148-29889390-jpq2d" Oct 30 12:30:00.315854 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:00.290937 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/273ee4cd-dd11-4204-ba40-9073fde47d1d-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889390-pz259\" (UID: \"273ee4cd-dd11-4204-ba40-9073fde47d1d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889390-pz259" Oct 30 12:30:00.315854 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:00.290998 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92ae9e02-1241-494f-8833-6081d874a34d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-c1b1e8-29889390-hj6n9\" (UID: \"92ae9e02-1241-494f-8833-6081d874a34d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c1b1e8-29889390-hj6n9" Oct 30 12:30:00.315854 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:00.291040 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/101bab62-abe6-4870-83e8-d30758312a0c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-6a7148-29889390-jpq2d\" (UID: \"101bab62-abe6-4870-83e8-d30758312a0c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a7148-29889390-jpq2d" Oct 30 12:30:00.353700 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:00.352307 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f920f7-29889390-489k2"] Oct 30 12:30:00.358046 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:00.357036 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-c44eee-29889390-pjjzm" Oct 30 12:30:00.358046 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:00.357279 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c1b1e8-29889390-hj6n9"] Oct 30 12:30:00.358046 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:00.357326 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a7148-29889390-jpq2d"] Oct 30 12:30:00.358046 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:00.357365 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889390-pz259"] Oct 30 12:30:00.358046 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:00.357394 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-c44eee-29889390-pjjzm"] Oct 30 12:30:00.375859 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:00.374951 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-8e3112-29889390-sptjx"] Oct 30 12:30:00.393858 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:00.392125 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f920f7-29889390-489k2" Oct 30 12:30:00.428706 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:00.417585 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkwfh\" (UniqueName: \"kubernetes.io/projected/92ae9e02-1241-494f-8833-6081d874a34d-kube-api-access-kkwfh\") pod \"maas-api-key-cleanup-e2e-worker-w1-c1b1e8-29889390-hj6n9\" (UID: \"92ae9e02-1241-494f-8833-6081d874a34d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c1b1e8-29889390-hj6n9" Oct 30 12:30:00.428706 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:00.417656 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6npwr\" (UniqueName: \"kubernetes.io/projected/273ee4cd-dd11-4204-ba40-9073fde47d1d-kube-api-access-6npwr\") pod \"maas-api-key-cleanup-29889390-pz259\" (UID: \"273ee4cd-dd11-4204-ba40-9073fde47d1d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889390-pz259" Oct 30 12:30:00.428706 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:00.417706 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhtzc\" (UniqueName: \"kubernetes.io/projected/101bab62-abe6-4870-83e8-d30758312a0c-kube-api-access-hhtzc\") pod \"maas-api-key-cleanup-e2e-worker-w3-6a7148-29889390-jpq2d\" (UID: \"101bab62-abe6-4870-83e8-d30758312a0c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a7148-29889390-jpq2d" Oct 30 12:30:00.428706 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:00.417760 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/273ee4cd-dd11-4204-ba40-9073fde47d1d-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889390-pz259\" (UID: \"273ee4cd-dd11-4204-ba40-9073fde47d1d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889390-pz259" Oct 30 12:30:00.428706 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:00.417807 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92ae9e02-1241-494f-8833-6081d874a34d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-c1b1e8-29889390-hj6n9\" (UID: \"92ae9e02-1241-494f-8833-6081d874a34d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c1b1e8-29889390-hj6n9" Oct 30 12:30:00.428706 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:00.417859 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/101bab62-abe6-4870-83e8-d30758312a0c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-6a7148-29889390-jpq2d\" (UID: \"101bab62-abe6-4870-83e8-d30758312a0c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a7148-29889390-jpq2d" Oct 30 12:30:00.428706 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:00.418664 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/101bab62-abe6-4870-83e8-d30758312a0c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-6a7148-29889390-jpq2d\" (UID: \"101bab62-abe6-4870-83e8-d30758312a0c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a7148-29889390-jpq2d" Oct 30 12:30:00.428706 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:00.419889 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/273ee4cd-dd11-4204-ba40-9073fde47d1d-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889390-pz259\" (UID: \"273ee4cd-dd11-4204-ba40-9073fde47d1d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889390-pz259" Oct 30 12:30:00.428706 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:00.420477 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92ae9e02-1241-494f-8833-6081d874a34d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-c1b1e8-29889390-hj6n9\" (UID: \"92ae9e02-1241-494f-8833-6081d874a34d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c1b1e8-29889390-hj6n9" Oct 30 12:30:00.444141 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:00.443545 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6npwr\" (UniqueName: \"kubernetes.io/projected/273ee4cd-dd11-4204-ba40-9073fde47d1d-kube-api-access-6npwr\") pod \"maas-api-key-cleanup-29889390-pz259\" (UID: \"273ee4cd-dd11-4204-ba40-9073fde47d1d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889390-pz259" Oct 30 12:30:00.444455 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:00.444429 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkwfh\" (UniqueName: \"kubernetes.io/projected/92ae9e02-1241-494f-8833-6081d874a34d-kube-api-access-kkwfh\") pod \"maas-api-key-cleanup-e2e-worker-w1-c1b1e8-29889390-hj6n9\" (UID: \"92ae9e02-1241-494f-8833-6081d874a34d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c1b1e8-29889390-hj6n9" Oct 30 12:30:00.445063 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:00.445038 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-5c57af-29889390-hphdb"] Oct 30 12:30:00.445218 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:00.445201 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-8e3112-29889390-sptjx" Oct 30 12:30:00.449533 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:00.449511 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhtzc\" (UniqueName: \"kubernetes.io/projected/101bab62-abe6-4870-83e8-d30758312a0c-kube-api-access-hhtzc\") pod \"maas-api-key-cleanup-e2e-worker-w3-6a7148-29889390-jpq2d\" (UID: \"101bab62-abe6-4870-83e8-d30758312a0c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a7148-29889390-jpq2d" Oct 30 12:30:00.449763 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:00.449746 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-ab9381-29889390-lqgh7"] Oct 30 12:30:00.451289 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:00.451264 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-5c57af-29889390-hphdb" Oct 30 12:30:00.454302 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:00.454285 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f920f7-29889390-489k2"] Oct 30 12:30:00.454392 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:00.454329 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-5c57af-29889390-hphdb"] Oct 30 12:30:00.454392 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:00.454354 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-ab9381-29889390-lqgh7"] Oct 30 12:30:00.454392 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:00.454377 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-ec236e-29889390-2rvks"] Oct 30 12:30:00.454551 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:00.454391 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-ab9381-29889390-lqgh7" Oct 30 12:30:00.458861 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:00.458810 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-47fd99-29889390-jq8cp"] Oct 30 12:30:00.459173 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:00.459060 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-ec236e-29889390-2rvks" Oct 30 12:30:00.465552 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:00.464989 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-8e3112-29889390-sptjx"] Oct 30 12:30:00.465552 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:00.465029 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-ec236e-29889390-2rvks"] Oct 30 12:30:00.465552 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:00.465092 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9wwfc"] Oct 30 12:30:00.465552 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:00.465129 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-47fd99-29889390-jq8cp"] Oct 30 12:30:00.465921 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:00.465903 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-47fd99-29889390-jq8cp" Oct 30 12:30:00.512837 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:00.512794 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c1b1e8-29889390-hj6n9" Oct 30 12:30:00.519211 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:00.519185 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c7ac05a-ec43-4c05-9872-4694da6ec833-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-f920f7-29889390-489k2\" (UID: \"6c7ac05a-ec43-4c05-9872-4694da6ec833\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f920f7-29889390-489k2" Oct 30 12:30:00.519304 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:00.519267 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/76363248-837a-4227-974a-9613e1e598e0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-c44eee-29889390-pjjzm\" (UID: \"76363248-837a-4227-974a-9613e1e598e0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-c44eee-29889390-pjjzm" Oct 30 12:30:00.519430 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:00.519328 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjhvp\" (UniqueName: \"kubernetes.io/projected/6c7ac05a-ec43-4c05-9872-4694da6ec833-kube-api-access-hjhvp\") pod \"maas-api-key-cleanup-e2e-worker-w0-f920f7-29889390-489k2\" (UID: \"6c7ac05a-ec43-4c05-9872-4694da6ec833\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f920f7-29889390-489k2" Oct 30 12:30:00.522086 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:00.522057 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnr7v\" (UniqueName: \"kubernetes.io/projected/76363248-837a-4227-974a-9613e1e598e0-kube-api-access-nnr7v\") pod \"maas-api-key-cleanup-e2e-praxis-a-c44eee-29889390-pjjzm\" (UID: \"76363248-837a-4227-974a-9613e1e598e0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-c44eee-29889390-pjjzm" Oct 30 12:30:00.597270 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:00.597025 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a7148-29889390-jpq2d" Oct 30 12:30:00.624701 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:00.624602 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xj2h8\" (UniqueName: \"kubernetes.io/projected/a416c883-7c57-4862-baf0-5add62763ba9-kube-api-access-xj2h8\") pod \"maas-api-key-cleanup-e2e-worker-w2-47fd99-29889390-jq8cp\" (UID: \"a416c883-7c57-4862-baf0-5add62763ba9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-47fd99-29889390-jq8cp" Oct 30 12:30:00.624701 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:00.624665 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgqgz\" (UniqueName: \"kubernetes.io/projected/016efbc9-f46c-4dd9-8910-2dbec739d99a-kube-api-access-sgqgz\") pod \"maas-api-key-cleanup-e2e-ipp-b-5c57af-29889390-hphdb\" (UID: \"016efbc9-f46c-4dd9-8910-2dbec739d99a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-5c57af-29889390-hphdb" Oct 30 12:30:00.624919 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:00.624703 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnr7v\" (UniqueName: \"kubernetes.io/projected/76363248-837a-4227-974a-9613e1e598e0-kube-api-access-nnr7v\") pod \"maas-api-key-cleanup-e2e-praxis-a-c44eee-29889390-pjjzm\" (UID: \"76363248-837a-4227-974a-9613e1e598e0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-c44eee-29889390-pjjzm" Oct 30 12:30:00.624919 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:00.624780 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a582fb0-974b-41fe-bc01-0712b6df3b31-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-8e3112-29889390-sptjx\" (UID: \"0a582fb0-974b-41fe-bc01-0712b6df3b31\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-8e3112-29889390-sptjx" Oct 30 12:30:00.624919 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:00.624842 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dptqq\" (UniqueName: \"kubernetes.io/projected/cc32f2ae-3aa3-4caa-b338-18a8ce68b577-kube-api-access-dptqq\") pod \"maas-api-key-cleanup-e2e-praxis-a-ab9381-29889390-lqgh7\" (UID: \"cc32f2ae-3aa3-4caa-b338-18a8ce68b577\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-ab9381-29889390-lqgh7" Oct 30 12:30:00.625282 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:00.625248 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlfcm\" (UniqueName: \"kubernetes.io/projected/0a582fb0-974b-41fe-bc01-0712b6df3b31-kube-api-access-dlfcm\") pod \"maas-api-key-cleanup-e2e-ipp-a-8e3112-29889390-sptjx\" (UID: \"0a582fb0-974b-41fe-bc01-0712b6df3b31\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-8e3112-29889390-sptjx" Oct 30 12:30:00.625405 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:00.625316 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlppm\" (UniqueName: \"kubernetes.io/projected/1df31daa-0968-4f77-97a1-b9aa4b629cfc-kube-api-access-dlppm\") pod \"maas-api-key-cleanup-e2e-praxis-b-ec236e-29889390-2rvks\" (UID: \"1df31daa-0968-4f77-97a1-b9aa4b629cfc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-ec236e-29889390-2rvks" Oct 30 12:30:00.625405 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:00.625346 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/016efbc9-f46c-4dd9-8910-2dbec739d99a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-5c57af-29889390-hphdb\" (UID: \"016efbc9-f46c-4dd9-8910-2dbec739d99a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-5c57af-29889390-hphdb" Oct 30 12:30:00.625405 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:00.625390 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc32f2ae-3aa3-4caa-b338-18a8ce68b577-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-ab9381-29889390-lqgh7\" (UID: \"cc32f2ae-3aa3-4caa-b338-18a8ce68b577\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-ab9381-29889390-lqgh7" Oct 30 12:30:00.625574 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:00.625432 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a416c883-7c57-4862-baf0-5add62763ba9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-47fd99-29889390-jq8cp\" (UID: \"a416c883-7c57-4862-baf0-5add62763ba9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-47fd99-29889390-jq8cp" Oct 30 12:30:00.625574 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:00.625517 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1df31daa-0968-4f77-97a1-b9aa4b629cfc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-ec236e-29889390-2rvks\" (UID: \"1df31daa-0968-4f77-97a1-b9aa4b629cfc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-ec236e-29889390-2rvks" Oct 30 12:30:00.625574 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:00.625554 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c7ac05a-ec43-4c05-9872-4694da6ec833-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-f920f7-29889390-489k2\" (UID: \"6c7ac05a-ec43-4c05-9872-4694da6ec833\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f920f7-29889390-489k2" Oct 30 12:30:00.627072 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:00.625585 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/76363248-837a-4227-974a-9613e1e598e0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-c44eee-29889390-pjjzm\" (UID: \"76363248-837a-4227-974a-9613e1e598e0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-c44eee-29889390-pjjzm" Oct 30 12:30:00.627072 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:00.625611 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjhvp\" (UniqueName: \"kubernetes.io/projected/6c7ac05a-ec43-4c05-9872-4694da6ec833-kube-api-access-hjhvp\") pod \"maas-api-key-cleanup-e2e-worker-w0-f920f7-29889390-489k2\" (UID: \"6c7ac05a-ec43-4c05-9872-4694da6ec833\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f920f7-29889390-489k2" Oct 30 12:30:00.628007 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:00.627966 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c7ac05a-ec43-4c05-9872-4694da6ec833-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-f920f7-29889390-489k2\" (UID: \"6c7ac05a-ec43-4c05-9872-4694da6ec833\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f920f7-29889390-489k2" Oct 30 12:30:00.628667 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:00.628581 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/76363248-837a-4227-974a-9613e1e598e0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-c44eee-29889390-pjjzm\" (UID: \"76363248-837a-4227-974a-9613e1e598e0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-c44eee-29889390-pjjzm" Oct 30 12:30:00.647662 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:00.647635 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnr7v\" (UniqueName: \"kubernetes.io/projected/76363248-837a-4227-974a-9613e1e598e0-kube-api-access-nnr7v\") pod \"maas-api-key-cleanup-e2e-praxis-a-c44eee-29889390-pjjzm\" (UID: \"76363248-837a-4227-974a-9613e1e598e0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-c44eee-29889390-pjjzm" Oct 30 12:30:00.648356 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:00.648330 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjhvp\" (UniqueName: \"kubernetes.io/projected/6c7ac05a-ec43-4c05-9872-4694da6ec833-kube-api-access-hjhvp\") pod \"maas-api-key-cleanup-e2e-worker-w0-f920f7-29889390-489k2\" (UID: \"6c7ac05a-ec43-4c05-9872-4694da6ec833\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f920f7-29889390-489k2" Oct 30 12:30:00.705794 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:00.705762 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889390-pz259" Oct 30 12:30:00.731902 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:00.726595 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xj2h8\" (UniqueName: \"kubernetes.io/projected/a416c883-7c57-4862-baf0-5add62763ba9-kube-api-access-xj2h8\") pod \"maas-api-key-cleanup-e2e-worker-w2-47fd99-29889390-jq8cp\" (UID: \"a416c883-7c57-4862-baf0-5add62763ba9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-47fd99-29889390-jq8cp" Oct 30 12:30:00.731902 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:00.726655 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgqgz\" (UniqueName: \"kubernetes.io/projected/016efbc9-f46c-4dd9-8910-2dbec739d99a-kube-api-access-sgqgz\") pod \"maas-api-key-cleanup-e2e-ipp-b-5c57af-29889390-hphdb\" (UID: \"016efbc9-f46c-4dd9-8910-2dbec739d99a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-5c57af-29889390-hphdb" Oct 30 12:30:00.731902 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:00.726726 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a582fb0-974b-41fe-bc01-0712b6df3b31-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-8e3112-29889390-sptjx\" (UID: \"0a582fb0-974b-41fe-bc01-0712b6df3b31\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-8e3112-29889390-sptjx" Oct 30 12:30:00.731902 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:00.726773 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dptqq\" (UniqueName: \"kubernetes.io/projected/cc32f2ae-3aa3-4caa-b338-18a8ce68b577-kube-api-access-dptqq\") pod \"maas-api-key-cleanup-e2e-praxis-a-ab9381-29889390-lqgh7\" (UID: \"cc32f2ae-3aa3-4caa-b338-18a8ce68b577\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-ab9381-29889390-lqgh7" Oct 30 12:30:00.731902 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:00.726801 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlfcm\" (UniqueName: \"kubernetes.io/projected/0a582fb0-974b-41fe-bc01-0712b6df3b31-kube-api-access-dlfcm\") pod \"maas-api-key-cleanup-e2e-ipp-a-8e3112-29889390-sptjx\" (UID: \"0a582fb0-974b-41fe-bc01-0712b6df3b31\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-8e3112-29889390-sptjx" Oct 30 12:30:00.731902 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:00.726850 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlppm\" (UniqueName: \"kubernetes.io/projected/1df31daa-0968-4f77-97a1-b9aa4b629cfc-kube-api-access-dlppm\") pod \"maas-api-key-cleanup-e2e-praxis-b-ec236e-29889390-2rvks\" (UID: \"1df31daa-0968-4f77-97a1-b9aa4b629cfc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-ec236e-29889390-2rvks" Oct 30 12:30:00.731902 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:00.726879 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/016efbc9-f46c-4dd9-8910-2dbec739d99a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-5c57af-29889390-hphdb\" (UID: \"016efbc9-f46c-4dd9-8910-2dbec739d99a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-5c57af-29889390-hphdb" Oct 30 12:30:00.731902 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:00.726925 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc32f2ae-3aa3-4caa-b338-18a8ce68b577-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-ab9381-29889390-lqgh7\" (UID: \"cc32f2ae-3aa3-4caa-b338-18a8ce68b577\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-ab9381-29889390-lqgh7" Oct 30 12:30:00.731902 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:00.726968 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a416c883-7c57-4862-baf0-5add62763ba9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-47fd99-29889390-jq8cp\" (UID: \"a416c883-7c57-4862-baf0-5add62763ba9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-47fd99-29889390-jq8cp" Oct 30 12:30:00.731902 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:00.727031 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1df31daa-0968-4f77-97a1-b9aa4b629cfc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-ec236e-29889390-2rvks\" (UID: \"1df31daa-0968-4f77-97a1-b9aa4b629cfc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-ec236e-29889390-2rvks" Oct 30 12:30:00.731902 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:00.727772 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1df31daa-0968-4f77-97a1-b9aa4b629cfc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-ec236e-29889390-2rvks\" (UID: \"1df31daa-0968-4f77-97a1-b9aa4b629cfc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-ec236e-29889390-2rvks" Oct 30 12:30:00.731902 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:00.729584 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-c44eee-29889390-pjjzm" Oct 30 12:30:00.732735 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:00.732062 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc32f2ae-3aa3-4caa-b338-18a8ce68b577-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-ab9381-29889390-lqgh7\" (UID: \"cc32f2ae-3aa3-4caa-b338-18a8ce68b577\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-ab9381-29889390-lqgh7" Oct 30 12:30:00.732735 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:00.732624 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/016efbc9-f46c-4dd9-8910-2dbec739d99a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-5c57af-29889390-hphdb\" (UID: \"016efbc9-f46c-4dd9-8910-2dbec739d99a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-5c57af-29889390-hphdb" Oct 30 12:30:00.733264 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:00.733202 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a582fb0-974b-41fe-bc01-0712b6df3b31-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-8e3112-29889390-sptjx\" (UID: \"0a582fb0-974b-41fe-bc01-0712b6df3b31\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-8e3112-29889390-sptjx" Oct 30 12:30:00.733398 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:00.733345 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a416c883-7c57-4862-baf0-5add62763ba9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-47fd99-29889390-jq8cp\" (UID: \"a416c883-7c57-4862-baf0-5add62763ba9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-47fd99-29889390-jq8cp" Oct 30 12:30:00.743572 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:00.743554 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f920f7-29889390-489k2" Oct 30 12:30:00.758680 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:00.758655 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xj2h8\" (UniqueName: \"kubernetes.io/projected/a416c883-7c57-4862-baf0-5add62763ba9-kube-api-access-xj2h8\") pod \"maas-api-key-cleanup-e2e-worker-w2-47fd99-29889390-jq8cp\" (UID: \"a416c883-7c57-4862-baf0-5add62763ba9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-47fd99-29889390-jq8cp" Oct 30 12:30:00.766212 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:00.766111 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dptqq\" (UniqueName: \"kubernetes.io/projected/cc32f2ae-3aa3-4caa-b338-18a8ce68b577-kube-api-access-dptqq\") pod \"maas-api-key-cleanup-e2e-praxis-a-ab9381-29889390-lqgh7\" (UID: \"cc32f2ae-3aa3-4caa-b338-18a8ce68b577\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-ab9381-29889390-lqgh7" Oct 30 12:30:00.766212 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:00.766109 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgqgz\" (UniqueName: \"kubernetes.io/projected/016efbc9-f46c-4dd9-8910-2dbec739d99a-kube-api-access-sgqgz\") pod \"maas-api-key-cleanup-e2e-ipp-b-5c57af-29889390-hphdb\" (UID: \"016efbc9-f46c-4dd9-8910-2dbec739d99a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-5c57af-29889390-hphdb" Oct 30 12:30:00.767591 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:00.767560 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlppm\" (UniqueName: \"kubernetes.io/projected/1df31daa-0968-4f77-97a1-b9aa4b629cfc-kube-api-access-dlppm\") pod \"maas-api-key-cleanup-e2e-praxis-b-ec236e-29889390-2rvks\" (UID: \"1df31daa-0968-4f77-97a1-b9aa4b629cfc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-ec236e-29889390-2rvks" Oct 30 12:30:00.768031 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:00.768009 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlfcm\" (UniqueName: \"kubernetes.io/projected/0a582fb0-974b-41fe-bc01-0712b6df3b31-kube-api-access-dlfcm\") pod \"maas-api-key-cleanup-e2e-ipp-a-8e3112-29889390-sptjx\" (UID: \"0a582fb0-974b-41fe-bc01-0712b6df3b31\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-8e3112-29889390-sptjx" Oct 30 12:30:00.815851 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:00.810056 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-8e3112-29889390-sptjx" Oct 30 12:30:00.816086 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:00.816054 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-ab9381-29889390-lqgh7" Oct 30 12:30:00.816189 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:00.816177 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-5c57af-29889390-hphdb" Oct 30 12:30:00.829249 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:00.826526 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-ec236e-29889390-2rvks" Oct 30 12:30:00.857895 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:00.836899 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c1b1e8-29889390-hj6n9"] Oct 30 12:30:00.869209 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:00.868995 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-47fd99-29889390-jq8cp" Oct 30 12:30:01.057382 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:01.057293 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a7148-29889390-jpq2d"] Oct 30 12:30:01.065933 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:01.065905 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889390-pz259"] Oct 30 12:30:01.075584 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:01.075551 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c1b1e8-29889390-hj6n9" event={"ID":"92ae9e02-1241-494f-8833-6081d874a34d","Type":"ContainerStarted","Data":"b6c8631e030e169f85d21b379c7b8c5119bb31a69e4fba1d0ad6c7ea5ae5ba71"} Oct 30 12:30:01.084048 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:30:01.083976 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod273ee4cd_dd11_4204_ba40_9073fde47d1d.slice/crio-0c44e3dfea3392f0743fa0e40ed3bea90ad8e6afc87b2378bda58bcd6f2ab080 WatchSource:0}: Error finding container 0c44e3dfea3392f0743fa0e40ed3bea90ad8e6afc87b2378bda58bcd6f2ab080: Status 404 returned error can't find the container with id 0c44e3dfea3392f0743fa0e40ed3bea90ad8e6afc87b2378bda58bcd6f2ab080 Oct 30 12:30:01.095574 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:01.095548 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-c44eee-29889390-pjjzm"] Oct 30 12:30:01.310503 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:01.310456 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f920f7-29889390-489k2"] Oct 30 12:30:01.367332 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:30:01.365610 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c7ac05a_ec43_4c05_9872_4694da6ec833.slice/crio-ed4584f8de0552ab93ec4ee28db1108d47aa7ca55f3a381abb05c1bb76f7085a WatchSource:0}: Error finding container ed4584f8de0552ab93ec4ee28db1108d47aa7ca55f3a381abb05c1bb76f7085a: Status 404 returned error can't find the container with id ed4584f8de0552ab93ec4ee28db1108d47aa7ca55f3a381abb05c1bb76f7085a Oct 30 12:30:01.416403 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:01.416375 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-8e3112-29889390-sptjx"] Oct 30 12:30:01.439230 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:01.439127 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-ec236e-29889390-2rvks"] Oct 30 12:30:01.449416 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:01.449397 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-ab9381-29889390-lqgh7"] Oct 30 12:30:01.463014 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:30:01.462971 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a582fb0_974b_41fe_bc01_0712b6df3b31.slice/crio-f5bba0dd2803996150f3901b33459f3781d67f736dfa40efb54de1aa9c0ac789 WatchSource:0}: Error finding container f5bba0dd2803996150f3901b33459f3781d67f736dfa40efb54de1aa9c0ac789: Status 404 returned error can't find the container with id f5bba0dd2803996150f3901b33459f3781d67f736dfa40efb54de1aa9c0ac789 Oct 30 12:30:01.463876 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:30:01.463815 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1df31daa_0968_4f77_97a1_b9aa4b629cfc.slice/crio-0085b0b9394ca8e0455b736c4a318b5b4ad004bcc3f464c61bdff6888ba5806a WatchSource:0}: Error finding container 0085b0b9394ca8e0455b736c4a318b5b4ad004bcc3f464c61bdff6888ba5806a: Status 404 returned error can't find the container with id 0085b0b9394ca8e0455b736c4a318b5b4ad004bcc3f464c61bdff6888ba5806a Oct 30 12:30:01.465391 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:30:01.465365 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc32f2ae_3aa3_4caa_b338_18a8ce68b577.slice/crio-df5f4103ed16b48b877eccca9f9382232329974f6cc7732c97e55a684665987e WatchSource:0}: Error finding container df5f4103ed16b48b877eccca9f9382232329974f6cc7732c97e55a684665987e: Status 404 returned error can't find the container with id df5f4103ed16b48b877eccca9f9382232329974f6cc7732c97e55a684665987e Oct 30 12:30:01.479800 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:01.479775 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-5c57af-29889390-hphdb"] Oct 30 12:30:01.561702 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:01.561608 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-47fd99-29889390-jq8cp"] Oct 30 12:30:01.575521 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:30:01.574979 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod016efbc9_f46c_4dd9_8910_2dbec739d99a.slice/crio-c7d818974a7911112bd83c1661e543650899744b110dec527f39b1c3c9f1ae25 WatchSource:0}: Error finding container c7d818974a7911112bd83c1661e543650899744b110dec527f39b1c3c9f1ae25: Status 404 returned error can't find the container with id c7d818974a7911112bd83c1661e543650899744b110dec527f39b1c3c9f1ae25 Oct 30 12:30:02.093199 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:02.093166 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-47fd99-29889390-jq8cp" event={"ID":"a416c883-7c57-4862-baf0-5add62763ba9","Type":"ContainerStarted","Data":"1ba7cc4d6b9772dcf0dfe7ea78622d3c6a3d438b14dd8d82eb873a6add11f3c2"} Oct 30 12:30:02.093325 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:02.093228 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-47fd99-29889390-jq8cp" event={"ID":"a416c883-7c57-4862-baf0-5add62763ba9","Type":"ContainerStarted","Data":"d2be9cf2c676ea719dcce70e316481ffe1e529cb2608f3e5ac5713a73fa771e8"} Oct 30 12:30:02.095047 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:02.095017 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-ab9381-29889390-lqgh7" event={"ID":"cc32f2ae-3aa3-4caa-b338-18a8ce68b577","Type":"ContainerStarted","Data":"69af74926691597f7159bbf8b0c1a17ef3f50a6cb59101a644e9c00a4d501ee8"} Oct 30 12:30:02.095152 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:02.095070 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-ab9381-29889390-lqgh7" event={"ID":"cc32f2ae-3aa3-4caa-b338-18a8ce68b577","Type":"ContainerStarted","Data":"df5f4103ed16b48b877eccca9f9382232329974f6cc7732c97e55a684665987e"} Oct 30 12:30:02.097373 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:02.096859 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c1b1e8-29889390-hj6n9" event={"ID":"92ae9e02-1241-494f-8833-6081d874a34d","Type":"ContainerStarted","Data":"2655bc2d2acfcb3c73e00fcc1bef0825aeded36ed9db969fda8f7516a60f76b0"} Oct 30 12:30:02.098822 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:02.098799 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-5c57af-29889390-hphdb" event={"ID":"016efbc9-f46c-4dd9-8910-2dbec739d99a","Type":"ContainerStarted","Data":"4c9ea66489f4cccf1b73845374363a38f75c5b1166867e35dddc808249d27b5b"} Oct 30 12:30:02.098943 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:02.098876 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-5c57af-29889390-hphdb" event={"ID":"016efbc9-f46c-4dd9-8910-2dbec739d99a","Type":"ContainerStarted","Data":"c7d818974a7911112bd83c1661e543650899744b110dec527f39b1c3c9f1ae25"} Oct 30 12:30:02.101133 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:02.101110 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889390-pz259" event={"ID":"273ee4cd-dd11-4204-ba40-9073fde47d1d","Type":"ContainerStarted","Data":"a160b299bd77e69eb37aa665fe437d19bd27f8afc82f02791558afd8dfdae9cd"} Oct 30 12:30:02.101281 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:02.101263 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889390-pz259" event={"ID":"273ee4cd-dd11-4204-ba40-9073fde47d1d","Type":"ContainerStarted","Data":"0c44e3dfea3392f0743fa0e40ed3bea90ad8e6afc87b2378bda58bcd6f2ab080"} Oct 30 12:30:02.103595 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:02.103574 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-ec236e-29889390-2rvks" event={"ID":"1df31daa-0968-4f77-97a1-b9aa4b629cfc","Type":"ContainerStarted","Data":"f759c0acca86a93972e56b2b41961edfee48c4ec87473047ea092f8aec2511ca"} Oct 30 12:30:02.103694 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:02.103615 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-ec236e-29889390-2rvks" event={"ID":"1df31daa-0968-4f77-97a1-b9aa4b629cfc","Type":"ContainerStarted","Data":"0085b0b9394ca8e0455b736c4a318b5b4ad004bcc3f464c61bdff6888ba5806a"} Oct 30 12:30:02.105905 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:02.105873 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a7148-29889390-jpq2d" event={"ID":"101bab62-abe6-4870-83e8-d30758312a0c","Type":"ContainerStarted","Data":"09f2089d1bed306acdf0b96d894d2a432d887b9a3a6642df0da43b78c291d07d"} Oct 30 12:30:02.105990 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:02.105920 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a7148-29889390-jpq2d" event={"ID":"101bab62-abe6-4870-83e8-d30758312a0c","Type":"ContainerStarted","Data":"bbdaed24c1fc0bc24399b367f7f0d76e4437becc8c5167a5bc5b10e00e9e0d03"} Oct 30 12:30:02.107627 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:02.107606 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-c44eee-29889390-pjjzm" event={"ID":"76363248-837a-4227-974a-9613e1e598e0","Type":"ContainerStarted","Data":"9752de4ba5456c7162feebd680522836243ef985039e1da72489c2ef620a7804"} Oct 30 12:30:02.107709 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:02.107646 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-c44eee-29889390-pjjzm" event={"ID":"76363248-837a-4227-974a-9613e1e598e0","Type":"ContainerStarted","Data":"748e28003d52226b2950ac32c51275cbfe8826f050b11b767c30f5031a2c0de9"} Oct 30 12:30:02.108963 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:02.108937 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f920f7-29889390-489k2" event={"ID":"6c7ac05a-ec43-4c05-9872-4694da6ec833","Type":"ContainerStarted","Data":"ed4584f8de0552ab93ec4ee28db1108d47aa7ca55f3a381abb05c1bb76f7085a"} Oct 30 12:30:02.110228 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:02.110185 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-8e3112-29889390-sptjx" event={"ID":"0a582fb0-974b-41fe-bc01-0712b6df3b31","Type":"ContainerStarted","Data":"f5bba0dd2803996150f3901b33459f3781d67f736dfa40efb54de1aa9c0ac789"} Oct 30 12:30:02.132711 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:02.132624 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-47fd99-29889390-jq8cp" podStartSLOduration=2.132608948 podStartE2EDuration="2.132608948s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:30:02.130196239 +0000 UTC m=+2158.644624983" watchObservedRunningTime="2026-10-30 12:30:02.132608948 +0000 UTC m=+2158.647037695" Oct 30 12:30:02.169920 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:02.169874 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-c44eee-29889390-pjjzm" podStartSLOduration=2.169857541 podStartE2EDuration="2.169857541s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:30:02.166272553 +0000 UTC m=+2158.680701299" watchObservedRunningTime="2026-10-30 12:30:02.169857541 +0000 UTC m=+2158.684286303" Oct 30 12:30:02.205023 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:02.204949 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c1b1e8-29889390-hj6n9" podStartSLOduration=2.204930503 podStartE2EDuration="2.204930503s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:30:02.204040445 +0000 UTC m=+2158.718469189" watchObservedRunningTime="2026-10-30 12:30:02.204930503 +0000 UTC m=+2158.719359249" Oct 30 12:30:02.245267 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:02.245207 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-5c57af-29889390-hphdb" podStartSLOduration=2.245189934 podStartE2EDuration="2.245189934s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:30:02.241457972 +0000 UTC m=+2158.755886727" watchObservedRunningTime="2026-10-30 12:30:02.245189934 +0000 UTC m=+2158.759618678" Oct 30 12:30:02.282704 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:02.280329 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a7148-29889390-jpq2d" podStartSLOduration=2.280313304 podStartE2EDuration="2.280313304s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:30:02.279652043 +0000 UTC m=+2158.794080788" watchObservedRunningTime="2026-10-30 12:30:02.280313304 +0000 UTC m=+2158.794742048" Oct 30 12:30:02.318123 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:02.317454 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-ec236e-29889390-2rvks" podStartSLOduration=2.3174400139999998 podStartE2EDuration="2.317440014s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:30:02.316936805 +0000 UTC m=+2158.831365550" watchObservedRunningTime="2026-10-30 12:30:02.317440014 +0000 UTC m=+2158.831868772" Oct 30 12:30:02.374964 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:02.373130 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-ab9381-29889390-lqgh7" podStartSLOduration=2.373109024 podStartE2EDuration="2.373109024s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:30:02.353452882 +0000 UTC m=+2158.867881628" watchObservedRunningTime="2026-10-30 12:30:02.373109024 +0000 UTC m=+2158.887537774" Oct 30 12:30:03.124732 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:03.124691 4908 generic.go:356] "Generic (PLEG): container finished" podID="a416c883-7c57-4862-baf0-5add62763ba9" containerID="1ba7cc4d6b9772dcf0dfe7ea78622d3c6a3d438b14dd8d82eb873a6add11f3c2" exitCode=60 Oct 30 12:30:03.125034 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:03.124774 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-47fd99-29889390-jq8cp" event={"ID":"a416c883-7c57-4862-baf0-5add62763ba9","Type":"ContainerDied","Data":"1ba7cc4d6b9772dcf0dfe7ea78622d3c6a3d438b14dd8d82eb873a6add11f3c2"} Oct 30 12:30:03.125110 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:03.125047 4908 scope.go:122] "RemoveContainer" containerID="1ba7cc4d6b9772dcf0dfe7ea78622d3c6a3d438b14dd8d82eb873a6add11f3c2" Oct 30 12:30:03.126983 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:03.126963 4908 generic.go:356] "Generic (PLEG): container finished" podID="cc32f2ae-3aa3-4caa-b338-18a8ce68b577" containerID="69af74926691597f7159bbf8b0c1a17ef3f50a6cb59101a644e9c00a4d501ee8" exitCode=60 Oct 30 12:30:03.127103 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:03.127049 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-ab9381-29889390-lqgh7" event={"ID":"cc32f2ae-3aa3-4caa-b338-18a8ce68b577","Type":"ContainerDied","Data":"69af74926691597f7159bbf8b0c1a17ef3f50a6cb59101a644e9c00a4d501ee8"} Oct 30 12:30:03.127303 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:03.127279 4908 scope.go:122] "RemoveContainer" containerID="69af74926691597f7159bbf8b0c1a17ef3f50a6cb59101a644e9c00a4d501ee8" Oct 30 12:30:03.129286 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:03.128866 4908 generic.go:356] "Generic (PLEG): container finished" podID="92ae9e02-1241-494f-8833-6081d874a34d" containerID="2655bc2d2acfcb3c73e00fcc1bef0825aeded36ed9db969fda8f7516a60f76b0" exitCode=60 Oct 30 12:30:03.129286 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:03.128959 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c1b1e8-29889390-hj6n9" event={"ID":"92ae9e02-1241-494f-8833-6081d874a34d","Type":"ContainerDied","Data":"2655bc2d2acfcb3c73e00fcc1bef0825aeded36ed9db969fda8f7516a60f76b0"} Oct 30 12:30:03.129286 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:03.129146 4908 scope.go:122] "RemoveContainer" containerID="2655bc2d2acfcb3c73e00fcc1bef0825aeded36ed9db969fda8f7516a60f76b0" Oct 30 12:30:03.130793 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:03.130705 4908 generic.go:356] "Generic (PLEG): container finished" podID="016efbc9-f46c-4dd9-8910-2dbec739d99a" containerID="4c9ea66489f4cccf1b73845374363a38f75c5b1166867e35dddc808249d27b5b" exitCode=60 Oct 30 12:30:03.130793 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:03.130774 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-5c57af-29889390-hphdb" event={"ID":"016efbc9-f46c-4dd9-8910-2dbec739d99a","Type":"ContainerDied","Data":"4c9ea66489f4cccf1b73845374363a38f75c5b1166867e35dddc808249d27b5b"} Oct 30 12:30:03.131057 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:03.131036 4908 scope.go:122] "RemoveContainer" containerID="4c9ea66489f4cccf1b73845374363a38f75c5b1166867e35dddc808249d27b5b" Oct 30 12:30:03.132591 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:03.132570 4908 generic.go:356] "Generic (PLEG): container finished" podID="273ee4cd-dd11-4204-ba40-9073fde47d1d" containerID="a160b299bd77e69eb37aa665fe437d19bd27f8afc82f02791558afd8dfdae9cd" exitCode=60 Oct 30 12:30:03.132687 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:03.132658 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889390-pz259" event={"ID":"273ee4cd-dd11-4204-ba40-9073fde47d1d","Type":"ContainerDied","Data":"a160b299bd77e69eb37aa665fe437d19bd27f8afc82f02791558afd8dfdae9cd"} Oct 30 12:30:03.132821 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:03.132808 4908 scope.go:122] "RemoveContainer" containerID="a160b299bd77e69eb37aa665fe437d19bd27f8afc82f02791558afd8dfdae9cd" Oct 30 12:30:03.134240 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:03.134218 4908 generic.go:356] "Generic (PLEG): container finished" podID="1df31daa-0968-4f77-97a1-b9aa4b629cfc" containerID="f759c0acca86a93972e56b2b41961edfee48c4ec87473047ea092f8aec2511ca" exitCode=60 Oct 30 12:30:03.134329 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:03.134297 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-ec236e-29889390-2rvks" event={"ID":"1df31daa-0968-4f77-97a1-b9aa4b629cfc","Type":"ContainerDied","Data":"f759c0acca86a93972e56b2b41961edfee48c4ec87473047ea092f8aec2511ca"} Oct 30 12:30:03.134496 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:03.134481 4908 scope.go:122] "RemoveContainer" containerID="f759c0acca86a93972e56b2b41961edfee48c4ec87473047ea092f8aec2511ca" Oct 30 12:30:03.135844 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:03.135813 4908 generic.go:356] "Generic (PLEG): container finished" podID="101bab62-abe6-4870-83e8-d30758312a0c" containerID="09f2089d1bed306acdf0b96d894d2a432d887b9a3a6642df0da43b78c291d07d" exitCode=60 Oct 30 12:30:03.135923 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:03.135872 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a7148-29889390-jpq2d" event={"ID":"101bab62-abe6-4870-83e8-d30758312a0c","Type":"ContainerDied","Data":"09f2089d1bed306acdf0b96d894d2a432d887b9a3a6642df0da43b78c291d07d"} Oct 30 12:30:03.136954 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:03.136935 4908 scope.go:122] "RemoveContainer" containerID="09f2089d1bed306acdf0b96d894d2a432d887b9a3a6642df0da43b78c291d07d" Oct 30 12:30:03.139536 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:03.139509 4908 generic.go:356] "Generic (PLEG): container finished" podID="76363248-837a-4227-974a-9613e1e598e0" containerID="9752de4ba5456c7162feebd680522836243ef985039e1da72489c2ef620a7804" exitCode=60 Oct 30 12:30:03.139612 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:03.139535 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-c44eee-29889390-pjjzm" event={"ID":"76363248-837a-4227-974a-9613e1e598e0","Type":"ContainerDied","Data":"9752de4ba5456c7162feebd680522836243ef985039e1da72489c2ef620a7804"} Oct 30 12:30:03.139772 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:03.139756 4908 scope.go:122] "RemoveContainer" containerID="9752de4ba5456c7162feebd680522836243ef985039e1da72489c2ef620a7804" Oct 30 12:30:03.141140 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:03.141120 4908 generic.go:356] "Generic (PLEG): container finished" podID="6c7ac05a-ec43-4c05-9872-4694da6ec833" containerID="a3af282572dc2cdf07dfeed6802170215e3d772941eae5f5c26c81deb202599b" exitCode=60 Oct 30 12:30:03.141237 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:03.141202 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f920f7-29889390-489k2" event={"ID":"6c7ac05a-ec43-4c05-9872-4694da6ec833","Type":"ContainerDied","Data":"a3af282572dc2cdf07dfeed6802170215e3d772941eae5f5c26c81deb202599b"} Oct 30 12:30:03.141349 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:03.141336 4908 scope.go:122] "RemoveContainer" containerID="a3af282572dc2cdf07dfeed6802170215e3d772941eae5f5c26c81deb202599b" Oct 30 12:30:03.143121 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:03.143095 4908 generic.go:356] "Generic (PLEG): container finished" podID="0a582fb0-974b-41fe-bc01-0712b6df3b31" containerID="23a855a6555165ca48b51f4d42ddf832c514a33e44ddd6a1cc056d7f78a0a0e4" exitCode=60 Oct 30 12:30:03.143223 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:03.143129 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-8e3112-29889390-sptjx" event={"ID":"0a582fb0-974b-41fe-bc01-0712b6df3b31","Type":"ContainerDied","Data":"23a855a6555165ca48b51f4d42ddf832c514a33e44ddd6a1cc056d7f78a0a0e4"} Oct 30 12:30:03.143369 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:03.143351 4908 scope.go:122] "RemoveContainer" containerID="23a855a6555165ca48b51f4d42ddf832c514a33e44ddd6a1cc056d7f78a0a0e4" Oct 30 12:30:03.199910 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:03.177361 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889390-pz259" podStartSLOduration=3.177342542 podStartE2EDuration="3.177342542s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:30:02.407161729 +0000 UTC m=+2158.921590474" watchObservedRunningTime="2026-10-30 12:30:03.177342542 +0000 UTC m=+2159.691771288" Oct 30 12:30:04.201767 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:04.200756 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-5c57af-29889390-hphdb" event={"ID":"016efbc9-f46c-4dd9-8910-2dbec739d99a","Type":"ContainerStarted","Data":"0f3f74130158bb6ec65f0ebab62b5bb55689cbef9c963264cb90a0cf4c1fbf59"} Oct 30 12:30:04.206500 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:04.205627 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889390-pz259" event={"ID":"273ee4cd-dd11-4204-ba40-9073fde47d1d","Type":"ContainerStarted","Data":"def4c4063cd6ef4cb977f351627c3743cfbb70769b43b710908fc793e165a25e"} Oct 30 12:30:04.208233 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:04.208213 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-ec236e-29889390-2rvks" event={"ID":"1df31daa-0968-4f77-97a1-b9aa4b629cfc","Type":"ContainerStarted","Data":"f688c5b91104b1629223a6eced9cd8d92ed51f607d42f7a5fa866eb5d185b49a"} Oct 30 12:30:04.211334 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:04.211286 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a7148-29889390-jpq2d" event={"ID":"101bab62-abe6-4870-83e8-d30758312a0c","Type":"ContainerStarted","Data":"da8580ba591c8205250a6fbc281deb42224c17f9024cb427066110f34f821941"} Oct 30 12:30:04.213462 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:04.213300 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-c44eee-29889390-pjjzm" event={"ID":"76363248-837a-4227-974a-9613e1e598e0","Type":"ContainerStarted","Data":"c06e520993b53518031fdc47c583096aa97e4ceaef242e0826770ff912fd0c7e"} Oct 30 12:30:04.216103 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:04.215747 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f920f7-29889390-489k2" event={"ID":"6c7ac05a-ec43-4c05-9872-4694da6ec833","Type":"ContainerStarted","Data":"c132e3de070e6186dd4d418d926bf8c666b7da23315ecfa863cfe816b3212294"} Oct 30 12:30:04.218015 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:04.217998 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-8e3112-29889390-sptjx" event={"ID":"0a582fb0-974b-41fe-bc01-0712b6df3b31","Type":"ContainerStarted","Data":"55a715fc323ecab7724a0d24381d569b9b65a7ff935fee6a39e7b5504d131a04"} Oct 30 12:30:04.219742 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:04.219726 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-47fd99-29889390-jq8cp" event={"ID":"a416c883-7c57-4862-baf0-5add62763ba9","Type":"ContainerStarted","Data":"935540d8b74a57774d6e18e07c41c72f0b0d6dccc1e5ffce31f98246dd8de66c"} Oct 30 12:30:04.220113 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:04.220100 4908 scope.go:122] "RemoveContainer" containerID="935540d8b74a57774d6e18e07c41c72f0b0d6dccc1e5ffce31f98246dd8de66c" Oct 30 12:30:04.220385 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:30:04.220369 4908 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-47fd99-29889390-jq8cp_odh-ai-gateway-infra(a416c883-7c57-4862-baf0-5add62763ba9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-47fd99-29889390-jq8cp" podUID="a416c883-7c57-4862-baf0-5add62763ba9" Oct 30 12:30:04.222243 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:04.222227 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-ab9381-29889390-lqgh7" event={"ID":"cc32f2ae-3aa3-4caa-b338-18a8ce68b577","Type":"ContainerStarted","Data":"c703b92c71aab34fca4d697075be0e7e127ce8d701a4a6a4c1fe8a33a9d9f15e"} Oct 30 12:30:04.222565 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:04.222552 4908 scope.go:122] "RemoveContainer" containerID="c703b92c71aab34fca4d697075be0e7e127ce8d701a4a6a4c1fe8a33a9d9f15e" Oct 30 12:30:04.223127 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:30:04.223110 4908 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-ab9381-29889390-lqgh7_odh-ai-gateway-infra(cc32f2ae-3aa3-4caa-b338-18a8ce68b577)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-ab9381-29889390-lqgh7" podUID="cc32f2ae-3aa3-4caa-b338-18a8ce68b577" Oct 30 12:30:04.224913 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:04.224894 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c1b1e8-29889390-hj6n9" event={"ID":"92ae9e02-1241-494f-8833-6081d874a34d","Type":"ContainerStarted","Data":"8e7e31b1ee1ffc9ca8a460c0b571ec2e69a59323bbffe23681a04c8f31226b4f"} Oct 30 12:30:04.438717 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:04.438630 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-8e3112-29889390-sptjx" podStartSLOduration=4.438615923 podStartE2EDuration="4.438615923s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:30:04.435168236 +0000 UTC m=+2160.949597002" watchObservedRunningTime="2026-10-30 12:30:04.438615923 +0000 UTC m=+2160.953044667" Oct 30 12:30:04.516583 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:04.516528 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f920f7-29889390-489k2" podStartSLOduration=4.516513203 podStartE2EDuration="4.516513203s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:30:04.514303724 +0000 UTC m=+2161.028732470" watchObservedRunningTime="2026-10-30 12:30:04.516513203 +0000 UTC m=+2161.030941947" Oct 30 12:30:04.922927 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:04.922896 4908 scope.go:122] "RemoveContainer" containerID="186188cd6e795b7d461111050eaaf79462673726469dc83b0d99238b72fb5eae" Oct 30 12:30:04.933535 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:04.933509 4908 scope.go:122] "RemoveContainer" containerID="226c6a5e28afada6d8a15c17d8c23822c3ee8e3d2f738520caaeafc762a92c83" Oct 30 12:30:05.239382 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:05.238916 4908 generic.go:356] "Generic (PLEG): container finished" podID="1df31daa-0968-4f77-97a1-b9aa4b629cfc" containerID="f688c5b91104b1629223a6eced9cd8d92ed51f607d42f7a5fa866eb5d185b49a" exitCode=60 Oct 30 12:30:05.239382 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:05.239011 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-ec236e-29889390-2rvks" event={"ID":"1df31daa-0968-4f77-97a1-b9aa4b629cfc","Type":"ContainerDied","Data":"f688c5b91104b1629223a6eced9cd8d92ed51f607d42f7a5fa866eb5d185b49a"} Oct 30 12:30:05.239382 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:05.239082 4908 scope.go:122] "RemoveContainer" containerID="f759c0acca86a93972e56b2b41961edfee48c4ec87473047ea092f8aec2511ca" Oct 30 12:30:05.239382 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:05.239370 4908 scope.go:122] "RemoveContainer" containerID="f688c5b91104b1629223a6eced9cd8d92ed51f607d42f7a5fa866eb5d185b49a" Oct 30 12:30:05.239957 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:30:05.239718 4908 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-ec236e-29889390-2rvks_odh-ai-gateway-infra(1df31daa-0968-4f77-97a1-b9aa4b629cfc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-ec236e-29889390-2rvks" podUID="1df31daa-0968-4f77-97a1-b9aa4b629cfc" Oct 30 12:30:05.241382 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:05.241344 4908 generic.go:356] "Generic (PLEG): container finished" podID="101bab62-abe6-4870-83e8-d30758312a0c" containerID="da8580ba591c8205250a6fbc281deb42224c17f9024cb427066110f34f821941" exitCode=60 Oct 30 12:30:05.241477 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:05.241421 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a7148-29889390-jpq2d" event={"ID":"101bab62-abe6-4870-83e8-d30758312a0c","Type":"ContainerDied","Data":"da8580ba591c8205250a6fbc281deb42224c17f9024cb427066110f34f821941"} Oct 30 12:30:05.241614 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:05.241600 4908 scope.go:122] "RemoveContainer" containerID="da8580ba591c8205250a6fbc281deb42224c17f9024cb427066110f34f821941" Oct 30 12:30:05.241849 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:30:05.241816 4908 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-6a7148-29889390-jpq2d_odh-ai-gateway-infra(101bab62-abe6-4870-83e8-d30758312a0c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a7148-29889390-jpq2d" podUID="101bab62-abe6-4870-83e8-d30758312a0c" Oct 30 12:30:05.243423 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:05.243396 4908 generic.go:356] "Generic (PLEG): container finished" podID="76363248-837a-4227-974a-9613e1e598e0" containerID="c06e520993b53518031fdc47c583096aa97e4ceaef242e0826770ff912fd0c7e" exitCode=60 Oct 30 12:30:05.243533 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:05.243419 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-c44eee-29889390-pjjzm" event={"ID":"76363248-837a-4227-974a-9613e1e598e0","Type":"ContainerDied","Data":"c06e520993b53518031fdc47c583096aa97e4ceaef242e0826770ff912fd0c7e"} Oct 30 12:30:05.243701 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:05.243685 4908 scope.go:122] "RemoveContainer" containerID="c06e520993b53518031fdc47c583096aa97e4ceaef242e0826770ff912fd0c7e" Oct 30 12:30:05.243966 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:30:05.243942 4908 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-c44eee-29889390-pjjzm_odh-ai-gateway-infra(76363248-837a-4227-974a-9613e1e598e0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-c44eee-29889390-pjjzm" podUID="76363248-837a-4227-974a-9613e1e598e0" Oct 30 12:30:05.245805 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:05.245785 4908 generic.go:356] "Generic (PLEG): container finished" podID="6c7ac05a-ec43-4c05-9872-4694da6ec833" containerID="c132e3de070e6186dd4d418d926bf8c666b7da23315ecfa863cfe816b3212294" exitCode=60 Oct 30 12:30:05.245928 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:05.245899 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f920f7-29889390-489k2" event={"ID":"6c7ac05a-ec43-4c05-9872-4694da6ec833","Type":"ContainerDied","Data":"c132e3de070e6186dd4d418d926bf8c666b7da23315ecfa863cfe816b3212294"} Oct 30 12:30:05.246089 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:05.246074 4908 scope.go:122] "RemoveContainer" containerID="c132e3de070e6186dd4d418d926bf8c666b7da23315ecfa863cfe816b3212294" Oct 30 12:30:05.246307 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:30:05.246289 4908 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-f920f7-29889390-489k2_odh-ai-gateway-infra(6c7ac05a-ec43-4c05-9872-4694da6ec833)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f920f7-29889390-489k2" podUID="6c7ac05a-ec43-4c05-9872-4694da6ec833" Oct 30 12:30:05.247808 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:05.247790 4908 generic.go:356] "Generic (PLEG): container finished" podID="0a582fb0-974b-41fe-bc01-0712b6df3b31" containerID="55a715fc323ecab7724a0d24381d569b9b65a7ff935fee6a39e7b5504d131a04" exitCode=60 Oct 30 12:30:05.247932 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:05.247879 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-8e3112-29889390-sptjx" event={"ID":"0a582fb0-974b-41fe-bc01-0712b6df3b31","Type":"ContainerDied","Data":"55a715fc323ecab7724a0d24381d569b9b65a7ff935fee6a39e7b5504d131a04"} Oct 30 12:30:05.248083 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:05.248064 4908 scope.go:122] "RemoveContainer" containerID="55a715fc323ecab7724a0d24381d569b9b65a7ff935fee6a39e7b5504d131a04" Oct 30 12:30:05.248302 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:30:05.248275 4908 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-8e3112-29889390-sptjx_odh-ai-gateway-infra(0a582fb0-974b-41fe-bc01-0712b6df3b31)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-8e3112-29889390-sptjx" podUID="0a582fb0-974b-41fe-bc01-0712b6df3b31" Oct 30 12:30:05.250001 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:05.249982 4908 generic.go:356] "Generic (PLEG): container finished" podID="a416c883-7c57-4862-baf0-5add62763ba9" containerID="935540d8b74a57774d6e18e07c41c72f0b0d6dccc1e5ffce31f98246dd8de66c" exitCode=60 Oct 30 12:30:05.250085 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:05.250047 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-47fd99-29889390-jq8cp" event={"ID":"a416c883-7c57-4862-baf0-5add62763ba9","Type":"ContainerDied","Data":"935540d8b74a57774d6e18e07c41c72f0b0d6dccc1e5ffce31f98246dd8de66c"} Oct 30 12:30:05.250284 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:05.250267 4908 scope.go:122] "RemoveContainer" containerID="935540d8b74a57774d6e18e07c41c72f0b0d6dccc1e5ffce31f98246dd8de66c" Oct 30 12:30:05.250535 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:30:05.250510 4908 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-47fd99-29889390-jq8cp_odh-ai-gateway-infra(a416c883-7c57-4862-baf0-5add62763ba9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-47fd99-29889390-jq8cp" podUID="a416c883-7c57-4862-baf0-5add62763ba9" Oct 30 12:30:05.252155 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:05.252132 4908 generic.go:356] "Generic (PLEG): container finished" podID="cc32f2ae-3aa3-4caa-b338-18a8ce68b577" containerID="c703b92c71aab34fca4d697075be0e7e127ce8d701a4a6a4c1fe8a33a9d9f15e" exitCode=60 Oct 30 12:30:05.252260 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:05.252167 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-ab9381-29889390-lqgh7" event={"ID":"cc32f2ae-3aa3-4caa-b338-18a8ce68b577","Type":"ContainerDied","Data":"c703b92c71aab34fca4d697075be0e7e127ce8d701a4a6a4c1fe8a33a9d9f15e"} Oct 30 12:30:05.252414 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:05.252397 4908 scope.go:122] "RemoveContainer" containerID="c703b92c71aab34fca4d697075be0e7e127ce8d701a4a6a4c1fe8a33a9d9f15e" Oct 30 12:30:05.252608 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:30:05.252589 4908 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-ab9381-29889390-lqgh7_odh-ai-gateway-infra(cc32f2ae-3aa3-4caa-b338-18a8ce68b577)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-ab9381-29889390-lqgh7" podUID="cc32f2ae-3aa3-4caa-b338-18a8ce68b577" Oct 30 12:30:05.254023 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:05.254005 4908 generic.go:356] "Generic (PLEG): container finished" podID="92ae9e02-1241-494f-8833-6081d874a34d" containerID="8e7e31b1ee1ffc9ca8a460c0b571ec2e69a59323bbffe23681a04c8f31226b4f" exitCode=60 Oct 30 12:30:05.254102 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:05.254090 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c1b1e8-29889390-hj6n9" event={"ID":"92ae9e02-1241-494f-8833-6081d874a34d","Type":"ContainerDied","Data":"8e7e31b1ee1ffc9ca8a460c0b571ec2e69a59323bbffe23681a04c8f31226b4f"} Oct 30 12:30:05.254315 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:05.254303 4908 scope.go:122] "RemoveContainer" containerID="8e7e31b1ee1ffc9ca8a460c0b571ec2e69a59323bbffe23681a04c8f31226b4f" Oct 30 12:30:05.254464 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:30:05.254450 4908 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-c1b1e8-29889390-hj6n9_odh-ai-gateway-infra(92ae9e02-1241-494f-8833-6081d874a34d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c1b1e8-29889390-hj6n9" podUID="92ae9e02-1241-494f-8833-6081d874a34d" Oct 30 12:30:05.278849 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:05.274883 4908 scope.go:122] "RemoveContainer" containerID="09f2089d1bed306acdf0b96d894d2a432d887b9a3a6642df0da43b78c291d07d" Oct 30 12:30:05.293616 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:05.292520 4908 generic.go:356] "Generic (PLEG): container finished" podID="016efbc9-f46c-4dd9-8910-2dbec739d99a" containerID="0f3f74130158bb6ec65f0ebab62b5bb55689cbef9c963264cb90a0cf4c1fbf59" exitCode=60 Oct 30 12:30:05.293616 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:05.293144 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-5c57af-29889390-hphdb" event={"ID":"016efbc9-f46c-4dd9-8910-2dbec739d99a","Type":"ContainerDied","Data":"0f3f74130158bb6ec65f0ebab62b5bb55689cbef9c963264cb90a0cf4c1fbf59"} Oct 30 12:30:05.293616 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:05.293301 4908 scope.go:122] "RemoveContainer" containerID="0f3f74130158bb6ec65f0ebab62b5bb55689cbef9c963264cb90a0cf4c1fbf59" Oct 30 12:30:05.293616 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:30:05.293572 4908 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-5c57af-29889390-hphdb_odh-ai-gateway-infra(016efbc9-f46c-4dd9-8910-2dbec739d99a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-5c57af-29889390-hphdb" podUID="016efbc9-f46c-4dd9-8910-2dbec739d99a" Oct 30 12:30:05.296095 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:05.296071 4908 generic.go:356] "Generic (PLEG): container finished" podID="273ee4cd-dd11-4204-ba40-9073fde47d1d" containerID="def4c4063cd6ef4cb977f351627c3743cfbb70769b43b710908fc793e165a25e" exitCode=60 Oct 30 12:30:05.296199 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:05.296152 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889390-pz259" event={"ID":"273ee4cd-dd11-4204-ba40-9073fde47d1d","Type":"ContainerDied","Data":"def4c4063cd6ef4cb977f351627c3743cfbb70769b43b710908fc793e165a25e"} Oct 30 12:30:05.296442 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:05.296426 4908 scope.go:122] "RemoveContainer" containerID="def4c4063cd6ef4cb977f351627c3743cfbb70769b43b710908fc793e165a25e" Oct 30 12:30:05.296769 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:30:05.296748 4908 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889390-pz259_odh-ai-gateway-infra(273ee4cd-dd11-4204-ba40-9073fde47d1d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889390-pz259" podUID="273ee4cd-dd11-4204-ba40-9073fde47d1d" Oct 30 12:30:05.307477 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:05.307459 4908 scope.go:122] "RemoveContainer" containerID="9752de4ba5456c7162feebd680522836243ef985039e1da72489c2ef620a7804" Oct 30 12:30:05.321086 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:05.321060 4908 scope.go:122] "RemoveContainer" containerID="a3af282572dc2cdf07dfeed6802170215e3d772941eae5f5c26c81deb202599b" Oct 30 12:30:05.334361 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:05.334346 4908 scope.go:122] "RemoveContainer" containerID="23a855a6555165ca48b51f4d42ddf832c514a33e44ddd6a1cc056d7f78a0a0e4" Oct 30 12:30:05.349633 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:05.349610 4908 scope.go:122] "RemoveContainer" containerID="1ba7cc4d6b9772dcf0dfe7ea78622d3c6a3d438b14dd8d82eb873a6add11f3c2" Oct 30 12:30:05.388653 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:05.388610 4908 scope.go:122] "RemoveContainer" containerID="69af74926691597f7159bbf8b0c1a17ef3f50a6cb59101a644e9c00a4d501ee8" Oct 30 12:30:05.402874 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:05.402853 4908 scope.go:122] "RemoveContainer" containerID="2655bc2d2acfcb3c73e00fcc1bef0825aeded36ed9db969fda8f7516a60f76b0" Oct 30 12:30:05.416085 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:05.416067 4908 scope.go:122] "RemoveContainer" containerID="4c9ea66489f4cccf1b73845374363a38f75c5b1166867e35dddc808249d27b5b" Oct 30 12:30:05.439666 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:05.439643 4908 scope.go:122] "RemoveContainer" containerID="a160b299bd77e69eb37aa665fe437d19bd27f8afc82f02791558afd8dfdae9cd" Oct 30 12:30:06.305731 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:06.305703 4908 scope.go:122] "RemoveContainer" containerID="8e7e31b1ee1ffc9ca8a460c0b571ec2e69a59323bbffe23681a04c8f31226b4f" Oct 30 12:30:06.306219 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:30:06.306122 4908 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-c1b1e8-29889390-hj6n9_odh-ai-gateway-infra(92ae9e02-1241-494f-8833-6081d874a34d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c1b1e8-29889390-hj6n9" podUID="92ae9e02-1241-494f-8833-6081d874a34d" Oct 30 12:30:06.307576 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:06.307554 4908 scope.go:122] "RemoveContainer" containerID="0f3f74130158bb6ec65f0ebab62b5bb55689cbef9c963264cb90a0cf4c1fbf59" Oct 30 12:30:06.307911 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:30:06.307882 4908 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-5c57af-29889390-hphdb_odh-ai-gateway-infra(016efbc9-f46c-4dd9-8910-2dbec739d99a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-5c57af-29889390-hphdb" podUID="016efbc9-f46c-4dd9-8910-2dbec739d99a" Oct 30 12:30:06.309215 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:06.309199 4908 scope.go:122] "RemoveContainer" containerID="def4c4063cd6ef4cb977f351627c3743cfbb70769b43b710908fc793e165a25e" Oct 30 12:30:06.309413 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:30:06.309382 4908 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889390-pz259_odh-ai-gateway-infra(273ee4cd-dd11-4204-ba40-9073fde47d1d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889390-pz259" podUID="273ee4cd-dd11-4204-ba40-9073fde47d1d" Oct 30 12:30:06.311102 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:06.311086 4908 scope.go:122] "RemoveContainer" containerID="f688c5b91104b1629223a6eced9cd8d92ed51f607d42f7a5fa866eb5d185b49a" Oct 30 12:30:06.311302 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:30:06.311281 4908 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-ec236e-29889390-2rvks_odh-ai-gateway-infra(1df31daa-0968-4f77-97a1-b9aa4b629cfc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-ec236e-29889390-2rvks" podUID="1df31daa-0968-4f77-97a1-b9aa4b629cfc" Oct 30 12:30:06.312707 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:06.312690 4908 scope.go:122] "RemoveContainer" containerID="da8580ba591c8205250a6fbc281deb42224c17f9024cb427066110f34f821941" Oct 30 12:30:06.312961 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:30:06.312930 4908 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-6a7148-29889390-jpq2d_odh-ai-gateway-infra(101bab62-abe6-4870-83e8-d30758312a0c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a7148-29889390-jpq2d" podUID="101bab62-abe6-4870-83e8-d30758312a0c" Oct 30 12:30:06.314449 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:06.314430 4908 scope.go:122] "RemoveContainer" containerID="c06e520993b53518031fdc47c583096aa97e4ceaef242e0826770ff912fd0c7e" Oct 30 12:30:06.314667 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:30:06.314640 4908 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-c44eee-29889390-pjjzm_odh-ai-gateway-infra(76363248-837a-4227-974a-9613e1e598e0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-c44eee-29889390-pjjzm" podUID="76363248-837a-4227-974a-9613e1e598e0" Oct 30 12:30:06.315957 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:06.315939 4908 scope.go:122] "RemoveContainer" containerID="c132e3de070e6186dd4d418d926bf8c666b7da23315ecfa863cfe816b3212294" Oct 30 12:30:06.316145 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:30:06.316126 4908 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-f920f7-29889390-489k2_odh-ai-gateway-infra(6c7ac05a-ec43-4c05-9872-4694da6ec833)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f920f7-29889390-489k2" podUID="6c7ac05a-ec43-4c05-9872-4694da6ec833" Oct 30 12:30:06.317474 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:06.317454 4908 scope.go:122] "RemoveContainer" containerID="55a715fc323ecab7724a0d24381d569b9b65a7ff935fee6a39e7b5504d131a04" Oct 30 12:30:06.317657 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:30:06.317637 4908 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-8e3112-29889390-sptjx_odh-ai-gateway-infra(0a582fb0-974b-41fe-bc01-0712b6df3b31)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-8e3112-29889390-sptjx" podUID="0a582fb0-974b-41fe-bc01-0712b6df3b31" Oct 30 12:30:06.319157 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:06.319138 4908 scope.go:122] "RemoveContainer" containerID="935540d8b74a57774d6e18e07c41c72f0b0d6dccc1e5ffce31f98246dd8de66c" Oct 30 12:30:06.319359 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:30:06.319334 4908 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-47fd99-29889390-jq8cp_odh-ai-gateway-infra(a416c883-7c57-4862-baf0-5add62763ba9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-47fd99-29889390-jq8cp" podUID="a416c883-7c57-4862-baf0-5add62763ba9" Oct 30 12:30:06.320938 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:06.320922 4908 scope.go:122] "RemoveContainer" containerID="c703b92c71aab34fca4d697075be0e7e127ce8d701a4a6a4c1fe8a33a9d9f15e" Oct 30 12:30:06.321088 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:30:06.321069 4908 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-ab9381-29889390-lqgh7_odh-ai-gateway-infra(cc32f2ae-3aa3-4caa-b338-18a8ce68b577)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-ab9381-29889390-lqgh7" podUID="cc32f2ae-3aa3-4caa-b338-18a8ce68b577" Oct 30 12:30:11.364514 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:11.364485 4908 generic.go:356] "Generic (PLEG): container finished" podID="fcefbe36-2c1b-43bf-aec5-4e2def37ef97" containerID="a83eb4ce815d8daf3833499153def814577c34c6a9b1c3b022fa405292eda2c4" exitCode=0 Oct 30 12:30:11.364932 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:11.364565 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-c44eee-openshift-default-6758d9545-k5l44" event={"ID":"fcefbe36-2c1b-43bf-aec5-4e2def37ef97","Type":"ContainerDied","Data":"a83eb4ce815d8daf3833499153def814577c34c6a9b1c3b022fa405292eda2c4"} Oct 30 12:30:11.365073 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:11.365056 4908 scope.go:122] "RemoveContainer" containerID="a83eb4ce815d8daf3833499153def814577c34c6a9b1c3b022fa405292eda2c4" Oct 30 12:30:11.366889 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:11.366861 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:30:11.366967 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:11.366945 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:30:11.367004 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:11.366987 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:30:12.394333 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:12.394294 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-c44eee-openshift-default-6758d9545-k5l44" event={"ID":"fcefbe36-2c1b-43bf-aec5-4e2def37ef97","Type":"ContainerStarted","Data":"62e582122a2594963d6956e0f8456fce7f85096e0dd11a8432c4dde509a543b0"} Oct 30 12:30:12.723912 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:12.723809 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-6p2qs"] Oct 30 12:30:12.731895 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:12.724333 4908 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-6p2qs" podUID="1d8040e9-df95-42d1-8870-603107c66560" containerName="istio-proxy" containerID="cri-o://f0bc49434d657b8c2135dd865a0755c97d42fd1eff1fd8def32a3324fc0a45d0" gracePeriod=30 Oct 30 12:30:12.753857 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:12.751594 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-c44eee-openshift-default-6758d9545-k5l44" Oct 30 12:30:12.771081 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:12.770331 4908 patch_prober.go:29] interesting pod/e2e-praxis-a-c44eee-openshift-default-6758d9545-k5l44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body= Oct 30 12:30:12.771081 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:12.770399 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-c44eee-openshift-default-6758d9545-k5l44" podUID="fcefbe36-2c1b-43bf-aec5-4e2def37ef97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" Oct 30 12:30:13.749075 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:13.749038 4908 patch_prober.go:29] interesting pod/e2e-praxis-a-c44eee-openshift-default-6758d9545-k5l44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body= Oct 30 12:30:13.749470 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:13.749106 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-c44eee-openshift-default-6758d9545-k5l44" podUID="fcefbe36-2c1b-43bf-aec5-4e2def37ef97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" Oct 30 12:30:14.600857 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:14.600578 4908 scope.go:122] "RemoveContainer" containerID="c703b92c71aab34fca4d697075be0e7e127ce8d701a4a6a4c1fe8a33a9d9f15e" Oct 30 12:30:14.601854 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:14.601620 4908 scope.go:122] "RemoveContainer" containerID="935540d8b74a57774d6e18e07c41c72f0b0d6dccc1e5ffce31f98246dd8de66c" Oct 30 12:30:14.751339 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:14.751303 4908 patch_prober.go:29] interesting pod/e2e-praxis-a-c44eee-openshift-default-6758d9545-k5l44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body= Oct 30 12:30:14.751959 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:14.751906 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-c44eee-openshift-default-6758d9545-k5l44" podUID="fcefbe36-2c1b-43bf-aec5-4e2def37ef97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" Oct 30 12:30:15.458497 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:15.458460 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-47fd99-29889390-jq8cp" event={"ID":"a416c883-7c57-4862-baf0-5add62763ba9","Type":"ContainerStarted","Data":"2884454c6372ca74bfa419fd2f1b0c70398c751bc1a74f08241f55029e1343f6"} Oct 30 12:30:15.461970 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:15.461897 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-ab9381-29889390-lqgh7" event={"ID":"cc32f2ae-3aa3-4caa-b338-18a8ce68b577","Type":"ContainerStarted","Data":"f448abc6485733294bdf9f6960b126e7348df45f905c203989146bccd2e2d188"} Oct 30 12:30:15.613169 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:15.610984 4908 scope.go:122] "RemoveContainer" containerID="f688c5b91104b1629223a6eced9cd8d92ed51f607d42f7a5fa866eb5d185b49a" Oct 30 12:30:15.748982 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:15.748946 4908 patch_prober.go:29] interesting pod/e2e-praxis-a-c44eee-openshift-default-6758d9545-k5l44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body= Oct 30 12:30:15.749147 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:15.749021 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-c44eee-openshift-default-6758d9545-k5l44" podUID="fcefbe36-2c1b-43bf-aec5-4e2def37ef97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" Oct 30 12:30:16.417853 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:16.410940 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-c44eee-openshift-default-6758d9545-k5l44" Oct 30 12:30:16.718860 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:16.716041 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-47fd99-29889390-jq8cp"] Oct 30 12:30:16.718860 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:16.716648 4908 scope.go:122] "RemoveContainer" containerID="8e7e31b1ee1ffc9ca8a460c0b571ec2e69a59323bbffe23681a04c8f31226b4f" Oct 30 12:30:16.730309 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:16.727063 4908 scope.go:122] "RemoveContainer" containerID="0f3f74130158bb6ec65f0ebab62b5bb55689cbef9c963264cb90a0cf4c1fbf59" Oct 30 12:30:16.774730 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:16.774587 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-ab9381-29889390-lqgh7"] Oct 30 12:30:16.823024 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:16.822989 4908 patch_prober.go:29] interesting pod/e2e-praxis-a-c44eee-openshift-default-6758d9545-k5l44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body= Oct 30 12:30:16.823272 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:16.823245 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-c44eee-openshift-default-6758d9545-k5l44" podUID="fcefbe36-2c1b-43bf-aec5-4e2def37ef97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" Oct 30 12:30:16.825571 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:16.825542 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-ec236e-29889390-2rvks" event={"ID":"1df31daa-0968-4f77-97a1-b9aa4b629cfc","Type":"ContainerStarted","Data":"dcf35f7bd1c5c5ecfdbddae0aa85e4d437d16eeb704ddf25a17b1d6f2abadc32"} Oct 30 12:30:16.831062 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:16.830693 4908 generic.go:356] "Generic (PLEG): container finished" podID="a416c883-7c57-4862-baf0-5add62763ba9" containerID="2884454c6372ca74bfa419fd2f1b0c70398c751bc1a74f08241f55029e1343f6" exitCode=60 Oct 30 12:30:16.831062 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:16.830780 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-47fd99-29889390-jq8cp" event={"ID":"a416c883-7c57-4862-baf0-5add62763ba9","Type":"ContainerDied","Data":"2884454c6372ca74bfa419fd2f1b0c70398c751bc1a74f08241f55029e1343f6"} Oct 30 12:30:16.831062 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:16.830843 4908 scope.go:122] "RemoveContainer" containerID="935540d8b74a57774d6e18e07c41c72f0b0d6dccc1e5ffce31f98246dd8de66c" Oct 30 12:30:16.840849 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:16.839488 4908 generic.go:356] "Generic (PLEG): container finished" podID="cc32f2ae-3aa3-4caa-b338-18a8ce68b577" containerID="f448abc6485733294bdf9f6960b126e7348df45f905c203989146bccd2e2d188" exitCode=60 Oct 30 12:30:16.840849 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:16.839584 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-ab9381-29889390-lqgh7" event={"ID":"cc32f2ae-3aa3-4caa-b338-18a8ce68b577","Type":"ContainerDied","Data":"f448abc6485733294bdf9f6960b126e7348df45f905c203989146bccd2e2d188"} Oct 30 12:30:17.065689 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:17.065241 4908 scope.go:122] "RemoveContainer" containerID="c703b92c71aab34fca4d697075be0e7e127ce8d701a4a6a4c1fe8a33a9d9f15e" Oct 30 12:30:17.260937 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:17.257174 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ab9381-wqmpj"] Oct 30 12:30:17.336077 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:17.336016 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ab9381-wqmpj" Oct 30 12:30:17.344985 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:17.344950 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ab9381-wqmpj"] Oct 30 12:30:17.352338 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:17.352308 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 12:30:17.424620 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:17.412209 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a64aea77-808a-4b76-9e00-49faaba0314b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-ab9381-wqmpj\" (UID: \"a64aea77-808a-4b76-9e00-49faaba0314b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ab9381-wqmpj" Oct 30 12:30:17.424620 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:17.412253 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nmtt\" (UniqueName: \"kubernetes.io/projected/a64aea77-808a-4b76-9e00-49faaba0314b-kube-api-access-7nmtt\") pod \"maas-api-revoke-keys-e2e-praxis-a-ab9381-wqmpj\" (UID: \"a64aea77-808a-4b76-9e00-49faaba0314b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ab9381-wqmpj" Oct 30 12:30:17.424620 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:17.412479 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a64aea77-808a-4b76-9e00-49faaba0314b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-ab9381-wqmpj\" (UID: \"a64aea77-808a-4b76-9e00-49faaba0314b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ab9381-wqmpj" Oct 30 12:30:17.518906 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:17.513729 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a64aea77-808a-4b76-9e00-49faaba0314b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-ab9381-wqmpj\" (UID: \"a64aea77-808a-4b76-9e00-49faaba0314b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ab9381-wqmpj" Oct 30 12:30:17.518906 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:17.513877 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a64aea77-808a-4b76-9e00-49faaba0314b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-ab9381-wqmpj\" (UID: \"a64aea77-808a-4b76-9e00-49faaba0314b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ab9381-wqmpj" Oct 30 12:30:17.518906 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:17.513905 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nmtt\" (UniqueName: \"kubernetes.io/projected/a64aea77-808a-4b76-9e00-49faaba0314b-kube-api-access-7nmtt\") pod \"maas-api-revoke-keys-e2e-praxis-a-ab9381-wqmpj\" (UID: \"a64aea77-808a-4b76-9e00-49faaba0314b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ab9381-wqmpj" Oct 30 12:30:17.518906 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:17.515598 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a64aea77-808a-4b76-9e00-49faaba0314b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-ab9381-wqmpj\" (UID: \"a64aea77-808a-4b76-9e00-49faaba0314b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ab9381-wqmpj" Oct 30 12:30:17.528103 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:17.528074 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a64aea77-808a-4b76-9e00-49faaba0314b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-ab9381-wqmpj\" (UID: \"a64aea77-808a-4b76-9e00-49faaba0314b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ab9381-wqmpj" Oct 30 12:30:17.611854 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:17.611573 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nmtt\" (UniqueName: \"kubernetes.io/projected/a64aea77-808a-4b76-9e00-49faaba0314b-kube-api-access-7nmtt\") pod \"maas-api-revoke-keys-e2e-praxis-a-ab9381-wqmpj\" (UID: \"a64aea77-808a-4b76-9e00-49faaba0314b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ab9381-wqmpj" Oct 30 12:30:17.614867 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:17.613731 4908 scope.go:122] "RemoveContainer" containerID="55a715fc323ecab7724a0d24381d569b9b65a7ff935fee6a39e7b5504d131a04" Oct 30 12:30:17.615847 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:17.615806 4908 scope.go:122] "RemoveContainer" containerID="def4c4063cd6ef4cb977f351627c3743cfbb70769b43b710908fc793e165a25e" Oct 30 12:30:17.616496 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:17.616474 4908 scope.go:122] "RemoveContainer" containerID="c06e520993b53518031fdc47c583096aa97e4ceaef242e0826770ff912fd0c7e" Oct 30 12:30:17.618067 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:17.618049 4908 scope.go:122] "RemoveContainer" containerID="da8580ba591c8205250a6fbc281deb42224c17f9024cb427066110f34f821941" Oct 30 12:30:17.678862 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:17.674966 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-ab9381-29889390-lqgh7" Oct 30 12:30:17.745883 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:17.745855 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc32f2ae-3aa3-4caa-b338-18a8ce68b577-kube-api-access-dptqq\" (UniqueName: \"kubernetes.io/projected/cc32f2ae-3aa3-4caa-b338-18a8ce68b577-kube-api-access-dptqq\") pod \"cc32f2ae-3aa3-4caa-b338-18a8ce68b577\" (UID: \"cc32f2ae-3aa3-4caa-b338-18a8ce68b577\") " Oct 30 12:30:17.746191 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:17.746166 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc32f2ae-3aa3-4caa-b338-18a8ce68b577-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc32f2ae-3aa3-4caa-b338-18a8ce68b577-maas-api-service-ca\") pod \"cc32f2ae-3aa3-4caa-b338-18a8ce68b577\" (UID: \"cc32f2ae-3aa3-4caa-b338-18a8ce68b577\") " Oct 30 12:30:17.746650 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:17.746620 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc32f2ae-3aa3-4caa-b338-18a8ce68b577-maas-api-service-ca" pod "cc32f2ae-3aa3-4caa-b338-18a8ce68b577" (UID: "cc32f2ae-3aa3-4caa-b338-18a8ce68b577"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:30:17.747099 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:17.747080 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc32f2ae-3aa3-4caa-b338-18a8ce68b577-maas-api-service-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:30:17.749344 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:17.749321 4908 patch_prober.go:29] interesting pod/e2e-praxis-a-c44eee-openshift-default-6758d9545-k5l44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body= Oct 30 12:30:17.749571 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:17.749542 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-c44eee-openshift-default-6758d9545-k5l44" podUID="fcefbe36-2c1b-43bf-aec5-4e2def37ef97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" Oct 30 12:30:17.750206 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:17.750178 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc32f2ae-3aa3-4caa-b338-18a8ce68b577-kube-api-access-dptqq" pod "cc32f2ae-3aa3-4caa-b338-18a8ce68b577" (UID: "cc32f2ae-3aa3-4caa-b338-18a8ce68b577"). InnerVolumeSpecName "kube-api-access-dptqq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:30:17.783900 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:17.781707 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ab9381-wqmpj" Oct 30 12:30:17.805048 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:17.803735 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-47fd99-29889390-jq8cp" Oct 30 12:30:17.848365 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:17.848324 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a416c883-7c57-4862-baf0-5add62763ba9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a416c883-7c57-4862-baf0-5add62763ba9-maas-api-service-ca\") pod \"a416c883-7c57-4862-baf0-5add62763ba9\" (UID: \"a416c883-7c57-4862-baf0-5add62763ba9\") " Oct 30 12:30:17.867248 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:17.867219 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a416c883-7c57-4862-baf0-5add62763ba9-kube-api-access-xj2h8\" (UniqueName: \"kubernetes.io/projected/a416c883-7c57-4862-baf0-5add62763ba9-kube-api-access-xj2h8\") pod \"a416c883-7c57-4862-baf0-5add62763ba9\" (UID: \"a416c883-7c57-4862-baf0-5add62763ba9\") " Oct 30 12:30:17.879426 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:17.868378 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dptqq\" (UniqueName: \"kubernetes.io/projected/cc32f2ae-3aa3-4caa-b338-18a8ce68b577-kube-api-access-dptqq\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:30:17.922773 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:17.856952 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a416c883-7c57-4862-baf0-5add62763ba9-maas-api-service-ca" pod "a416c883-7c57-4862-baf0-5add62763ba9" (UID: "a416c883-7c57-4862-baf0-5add62763ba9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:30:17.922773 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:17.909563 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-47fd99-29889390-jq8cp" event={"ID":"a416c883-7c57-4862-baf0-5add62763ba9","Type":"ContainerDied","Data":"d2be9cf2c676ea719dcce70e316481ffe1e529cb2608f3e5ac5713a73fa771e8"} Oct 30 12:30:17.922773 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:17.909656 4908 scope.go:122] "RemoveContainer" containerID="2884454c6372ca74bfa419fd2f1b0c70398c751bc1a74f08241f55029e1343f6" Oct 30 12:30:17.922773 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:17.909799 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-47fd99-29889390-jq8cp" Oct 30 12:30:17.922773 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:17.914132 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-ab9381-29889390-lqgh7" event={"ID":"cc32f2ae-3aa3-4caa-b338-18a8ce68b577","Type":"ContainerDied","Data":"df5f4103ed16b48b877eccca9f9382232329974f6cc7732c97e55a684665987e"} Oct 30 12:30:17.922773 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:17.914317 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-ab9381-29889390-lqgh7" Oct 30 12:30:17.922773 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:17.916219 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a416c883-7c57-4862-baf0-5add62763ba9-kube-api-access-xj2h8" pod "a416c883-7c57-4862-baf0-5add62763ba9" (UID: "a416c883-7c57-4862-baf0-5add62763ba9"). InnerVolumeSpecName "kube-api-access-xj2h8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:30:17.922773 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:17.920379 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c1b1e8-29889390-hj6n9" event={"ID":"92ae9e02-1241-494f-8833-6081d874a34d","Type":"ContainerStarted","Data":"de1d5512889df12795779fd9afb89cc8afdb1a063ee1e0f330d6cd934fab12ca"} Oct 30 12:30:17.926274 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:17.925268 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-5c57af-29889390-hphdb" event={"ID":"016efbc9-f46c-4dd9-8910-2dbec739d99a","Type":"ContainerStarted","Data":"7235f44b9502c1d727ffb3a3f1fc077240eb5dea43d614424f7e45fab54e3ad6"} Oct 30 12:30:17.928594 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:17.928561 4908 generic.go:356] "Generic (PLEG): container finished" podID="1df31daa-0968-4f77-97a1-b9aa4b629cfc" containerID="dcf35f7bd1c5c5ecfdbddae0aa85e4d437d16eeb704ddf25a17b1d6f2abadc32" exitCode=60 Oct 30 12:30:17.928771 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:17.928753 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-ec236e-29889390-2rvks" event={"ID":"1df31daa-0968-4f77-97a1-b9aa4b629cfc","Type":"ContainerDied","Data":"dcf35f7bd1c5c5ecfdbddae0aa85e4d437d16eeb704ddf25a17b1d6f2abadc32"} Oct 30 12:30:17.929133 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:17.929116 4908 scope.go:122] "RemoveContainer" containerID="dcf35f7bd1c5c5ecfdbddae0aa85e4d437d16eeb704ddf25a17b1d6f2abadc32" Oct 30 12:30:17.929414 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:30:17.929389 4908 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-ec236e-29889390-2rvks_odh-ai-gateway-infra(1df31daa-0968-4f77-97a1-b9aa4b629cfc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-ec236e-29889390-2rvks" podUID="1df31daa-0968-4f77-97a1-b9aa4b629cfc" Oct 30 12:30:17.950512 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:17.950479 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-ec236e-29889390-2rvks"] Oct 30 12:30:17.954277 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:17.953390 4908 scope.go:122] "RemoveContainer" containerID="f448abc6485733294bdf9f6960b126e7348df45f905c203989146bccd2e2d188" Oct 30 12:30:18.007482 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:18.006963 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a416c883-7c57-4862-baf0-5add62763ba9-maas-api-service-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:30:18.007482 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:18.006994 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xj2h8\" (UniqueName: \"kubernetes.io/projected/a416c883-7c57-4862-baf0-5add62763ba9-kube-api-access-xj2h8\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:30:18.054706 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:18.054680 4908 scope.go:122] "RemoveContainer" containerID="f688c5b91104b1629223a6eced9cd8d92ed51f607d42f7a5fa866eb5d185b49a" Oct 30 12:30:18.208403 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:18.207534 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-ab9381-29889390-lqgh7"] Oct 30 12:30:18.214372 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:18.214338 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-ab9381-29889390-lqgh7"] Oct 30 12:30:18.257948 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:18.257920 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ab9381-wqmpj"] Oct 30 12:30:18.304743 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:18.304674 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-47fd99-29889390-jq8cp"] Oct 30 12:30:18.306340 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:18.306321 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-47fd99-29889390-jq8cp"] Oct 30 12:30:18.341869 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:30:18.341815 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda64aea77_808a_4b76_9e00_49faaba0314b.slice/crio-e5e15d32069083dc72b7c74e7bd2187e9d0b4034d074b82fd762c7d16addddee WatchSource:0}: Error finding container e5e15d32069083dc72b7c74e7bd2187e9d0b4034d074b82fd762c7d16addddee: Status 404 returned error can't find the container with id e5e15d32069083dc72b7c74e7bd2187e9d0b4034d074b82fd762c7d16addddee Oct 30 12:30:18.400948 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:18.400707 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-6p2qs" Oct 30 12:30:18.522388 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:18.521167 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d8040e9-df95-42d1-8870-603107c66560-istio-token\" (UniqueName: \"kubernetes.io/projected/1d8040e9-df95-42d1-8870-603107c66560-istio-token\") pod \"1d8040e9-df95-42d1-8870-603107c66560\" (UID: \"1d8040e9-df95-42d1-8870-603107c66560\") " Oct 30 12:30:18.522388 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:18.521225 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d8040e9-df95-42d1-8870-603107c66560-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d8040e9-df95-42d1-8870-603107c66560-istio-data\") pod \"1d8040e9-df95-42d1-8870-603107c66560\" (UID: \"1d8040e9-df95-42d1-8870-603107c66560\") " Oct 30 12:30:18.522388 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:18.521304 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d8040e9-df95-42d1-8870-603107c66560-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d8040e9-df95-42d1-8870-603107c66560-credential-socket\") pod \"1d8040e9-df95-42d1-8870-603107c66560\" (UID: \"1d8040e9-df95-42d1-8870-603107c66560\") " Oct 30 12:30:18.522388 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:18.521355 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d8040e9-df95-42d1-8870-603107c66560-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d8040e9-df95-42d1-8870-603107c66560-workload-certs\") pod \"1d8040e9-df95-42d1-8870-603107c66560\" (UID: \"1d8040e9-df95-42d1-8870-603107c66560\") " Oct 30 12:30:18.522388 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:18.521442 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d8040e9-df95-42d1-8870-603107c66560-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d8040e9-df95-42d1-8870-603107c66560-istio-envoy\") pod \"1d8040e9-df95-42d1-8870-603107c66560\" (UID: \"1d8040e9-df95-42d1-8870-603107c66560\") " Oct 30 12:30:18.522388 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:18.521478 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1d8040e9-df95-42d1-8870-603107c66560-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d8040e9-df95-42d1-8870-603107c66560-istio-podinfo\") pod \"1d8040e9-df95-42d1-8870-603107c66560\" (UID: \"1d8040e9-df95-42d1-8870-603107c66560\") " Oct 30 12:30:18.522388 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:18.521517 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d8040e9-df95-42d1-8870-603107c66560-kube-api-access-8pgsj\" (UniqueName: \"kubernetes.io/projected/1d8040e9-df95-42d1-8870-603107c66560-kube-api-access-8pgsj\") pod \"1d8040e9-df95-42d1-8870-603107c66560\" (UID: \"1d8040e9-df95-42d1-8870-603107c66560\") " Oct 30 12:30:18.522388 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:18.521608 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d8040e9-df95-42d1-8870-603107c66560-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d8040e9-df95-42d1-8870-603107c66560-istiod-ca-cert\") pod \"1d8040e9-df95-42d1-8870-603107c66560\" (UID: \"1d8040e9-df95-42d1-8870-603107c66560\") " Oct 30 12:30:18.522388 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:18.521708 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d8040e9-df95-42d1-8870-603107c66560-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d8040e9-df95-42d1-8870-603107c66560-workload-socket\") pod \"1d8040e9-df95-42d1-8870-603107c66560\" (UID: \"1d8040e9-df95-42d1-8870-603107c66560\") " Oct 30 12:30:18.523271 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:18.522570 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d8040e9-df95-42d1-8870-603107c66560-workload-socket" pod "1d8040e9-df95-42d1-8870-603107c66560" (UID: "1d8040e9-df95-42d1-8870-603107c66560"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:30:18.527907 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:18.523984 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d8040e9-df95-42d1-8870-603107c66560-credential-socket" pod "1d8040e9-df95-42d1-8870-603107c66560" (UID: "1d8040e9-df95-42d1-8870-603107c66560"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:30:18.527907 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:18.524160 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d8040e9-df95-42d1-8870-603107c66560-istio-data" pod "1d8040e9-df95-42d1-8870-603107c66560" (UID: "1d8040e9-df95-42d1-8870-603107c66560"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:30:18.531049 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:18.530373 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d8040e9-df95-42d1-8870-603107c66560-istiod-ca-cert" pod "1d8040e9-df95-42d1-8870-603107c66560" (UID: "1d8040e9-df95-42d1-8870-603107c66560"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:30:18.533653 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:18.531344 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d8040e9-df95-42d1-8870-603107c66560-workload-certs" pod "1d8040e9-df95-42d1-8870-603107c66560" (UID: "1d8040e9-df95-42d1-8870-603107c66560"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:30:18.570782 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:18.570739 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d8040e9-df95-42d1-8870-603107c66560-istio-token" pod "1d8040e9-df95-42d1-8870-603107c66560" (UID: "1d8040e9-df95-42d1-8870-603107c66560"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:30:18.571995 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:18.571957 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d8040e9-df95-42d1-8870-603107c66560-istio-envoy" pod "1d8040e9-df95-42d1-8870-603107c66560" (UID: "1d8040e9-df95-42d1-8870-603107c66560"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:30:18.572212 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:18.572183 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1d8040e9-df95-42d1-8870-603107c66560-istio-podinfo" pod "1d8040e9-df95-42d1-8870-603107c66560" (UID: "1d8040e9-df95-42d1-8870-603107c66560"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 12:30:18.572397 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:18.572371 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d8040e9-df95-42d1-8870-603107c66560-kube-api-access-8pgsj" pod "1d8040e9-df95-42d1-8870-603107c66560" (UID: "1d8040e9-df95-42d1-8870-603107c66560"). InnerVolumeSpecName "kube-api-access-8pgsj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:30:18.599860 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:18.598130 4908 scope.go:122] "RemoveContainer" containerID="c132e3de070e6186dd4d418d926bf8c666b7da23315ecfa863cfe816b3212294" Oct 30 12:30:18.623259 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:18.623229 4908 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d8040e9-df95-42d1-8870-603107c66560-istio-token\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:30:18.623387 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:18.623264 4908 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d8040e9-df95-42d1-8870-603107c66560-istio-data\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:30:18.623387 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:18.623278 4908 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d8040e9-df95-42d1-8870-603107c66560-credential-socket\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:30:18.623387 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:18.623291 4908 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d8040e9-df95-42d1-8870-603107c66560-workload-certs\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:30:18.623387 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:18.623304 4908 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d8040e9-df95-42d1-8870-603107c66560-istio-envoy\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:30:18.623387 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:18.623316 4908 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d8040e9-df95-42d1-8870-603107c66560-istio-podinfo\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:30:18.623387 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:18.623328 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8pgsj\" (UniqueName: \"kubernetes.io/projected/1d8040e9-df95-42d1-8870-603107c66560-kube-api-access-8pgsj\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:30:18.623387 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:18.623349 4908 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d8040e9-df95-42d1-8870-603107c66560-istiod-ca-cert\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:30:18.623387 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:18.623361 4908 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d8040e9-df95-42d1-8870-603107c66560-workload-socket\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:30:18.753431 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:18.753397 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-c44eee-openshift-default-6758d9545-k5l44" Oct 30 12:30:18.754613 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:18.754588 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-c44eee-openshift-default-6758d9545-k5l44" Oct 30 12:30:18.939186 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:18.939157 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-c44eee-29889390-pjjzm" event={"ID":"76363248-837a-4227-974a-9613e1e598e0","Type":"ContainerStarted","Data":"0f4aacb96ef666b3f90b52b5dd26070f443e7a27758adb5529a36ee6b2b864d4"} Oct 30 12:30:18.941954 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:18.941924 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-8e3112-29889390-sptjx" event={"ID":"0a582fb0-974b-41fe-bc01-0712b6df3b31","Type":"ContainerStarted","Data":"04567df8abf9fe74b6d0432d603fef568540df1e4052e08677a4b8ff3b45e8d7"} Oct 30 12:30:18.943818 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:18.943793 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ab9381-wqmpj" event={"ID":"a64aea77-808a-4b76-9e00-49faaba0314b","Type":"ContainerStarted","Data":"dc96eb27757db022136d47e820181a67052a243a8e9dec4714e8c5ad365a8b6d"} Oct 30 12:30:18.943979 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:18.943893 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ab9381-wqmpj" event={"ID":"a64aea77-808a-4b76-9e00-49faaba0314b","Type":"ContainerStarted","Data":"e5e15d32069083dc72b7c74e7bd2187e9d0b4034d074b82fd762c7d16addddee"} Oct 30 12:30:18.947469 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:18.947449 4908 generic.go:356] "Generic (PLEG): container finished" podID="92ae9e02-1241-494f-8833-6081d874a34d" containerID="de1d5512889df12795779fd9afb89cc8afdb1a063ee1e0f330d6cd934fab12ca" exitCode=60 Oct 30 12:30:18.947572 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:18.947524 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c1b1e8-29889390-hj6n9" event={"ID":"92ae9e02-1241-494f-8833-6081d874a34d","Type":"ContainerDied","Data":"de1d5512889df12795779fd9afb89cc8afdb1a063ee1e0f330d6cd934fab12ca"} Oct 30 12:30:18.947645 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:18.947585 4908 scope.go:122] "RemoveContainer" containerID="8e7e31b1ee1ffc9ca8a460c0b571ec2e69a59323bbffe23681a04c8f31226b4f" Oct 30 12:30:18.947905 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:18.947883 4908 scope.go:122] "RemoveContainer" containerID="de1d5512889df12795779fd9afb89cc8afdb1a063ee1e0f330d6cd934fab12ca" Oct 30 12:30:18.948174 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:30:18.948152 4908 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-c1b1e8-29889390-hj6n9_odh-ai-gateway-infra(92ae9e02-1241-494f-8833-6081d874a34d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c1b1e8-29889390-hj6n9" podUID="92ae9e02-1241-494f-8833-6081d874a34d" Oct 30 12:30:18.949078 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:18.949055 4908 generic.go:356] "Generic (PLEG): container finished" podID="1d8040e9-df95-42d1-8870-603107c66560" containerID="f0bc49434d657b8c2135dd865a0755c97d42fd1eff1fd8def32a3324fc0a45d0" exitCode=0 Oct 30 12:30:18.949181 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:18.949084 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-6p2qs" event={"ID":"1d8040e9-df95-42d1-8870-603107c66560","Type":"ContainerDied","Data":"f0bc49434d657b8c2135dd865a0755c97d42fd1eff1fd8def32a3324fc0a45d0"} Oct 30 12:30:18.949181 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:18.949119 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-6p2qs" Oct 30 12:30:18.949287 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:18.949120 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-6p2qs" event={"ID":"1d8040e9-df95-42d1-8870-603107c66560","Type":"ContainerDied","Data":"4c07cb9fcd515061bb4fd58156fe7aa0daabca6705a1b4212471617cd3aad768"} Oct 30 12:30:18.951020 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:18.950999 4908 generic.go:356] "Generic (PLEG): container finished" podID="016efbc9-f46c-4dd9-8910-2dbec739d99a" containerID="7235f44b9502c1d727ffb3a3f1fc077240eb5dea43d614424f7e45fab54e3ad6" exitCode=60 Oct 30 12:30:18.951130 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:18.951068 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-5c57af-29889390-hphdb" event={"ID":"016efbc9-f46c-4dd9-8910-2dbec739d99a","Type":"ContainerDied","Data":"7235f44b9502c1d727ffb3a3f1fc077240eb5dea43d614424f7e45fab54e3ad6"} Oct 30 12:30:18.951438 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:18.951417 4908 scope.go:122] "RemoveContainer" containerID="7235f44b9502c1d727ffb3a3f1fc077240eb5dea43d614424f7e45fab54e3ad6" Oct 30 12:30:18.951681 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:30:18.951657 4908 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-5c57af-29889390-hphdb_odh-ai-gateway-infra(016efbc9-f46c-4dd9-8910-2dbec739d99a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-5c57af-29889390-hphdb" podUID="016efbc9-f46c-4dd9-8910-2dbec739d99a" Oct 30 12:30:18.953696 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:18.953332 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889390-pz259" event={"ID":"273ee4cd-dd11-4204-ba40-9073fde47d1d","Type":"ContainerStarted","Data":"9c9ad3268c094a98445611726a4b228f67bbbae6a6636e7a22df933453cda8c9"} Oct 30 12:30:18.963211 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:18.962550 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a7148-29889390-jpq2d" event={"ID":"101bab62-abe6-4870-83e8-d30758312a0c","Type":"ContainerStarted","Data":"22e995d593f86f3d36f58819dfcd04054ea78dba21f77adea0fa394a8567b614"} Oct 30 12:30:18.968280 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:18.968244 4908 scope.go:122] "RemoveContainer" containerID="f0bc49434d657b8c2135dd865a0755c97d42fd1eff1fd8def32a3324fc0a45d0" Oct 30 12:30:18.990812 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:18.990783 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c1b1e8-29889390-hj6n9"] Oct 30 12:30:19.047428 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:19.047395 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-5c57af-29889390-hphdb"] Oct 30 12:30:19.167854 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:19.161711 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-6p2qs"] Oct 30 12:30:19.167854 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:19.167506 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-6p2qs"] Oct 30 12:30:19.319564 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:19.319519 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ab9381-wqmpj" podStartSLOduration=2.319505196 podStartE2EDuration="2.319505196s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:30:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:30:19.315766698 +0000 UTC m=+2175.830195441" watchObservedRunningTime="2026-10-30 12:30:19.319505196 +0000 UTC m=+2175.833933968" Oct 30 12:30:19.372543 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:19.372513 4908 scope.go:122] "RemoveContainer" containerID="f0bc49434d657b8c2135dd865a0755c97d42fd1eff1fd8def32a3324fc0a45d0" Oct 30 12:30:19.372953 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:30:19.372927 4908 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0bc49434d657b8c2135dd865a0755c97d42fd1eff1fd8def32a3324fc0a45d0\": container with ID starting with f0bc49434d657b8c2135dd865a0755c97d42fd1eff1fd8def32a3324fc0a45d0 not found: ID does not exist" containerID="f0bc49434d657b8c2135dd865a0755c97d42fd1eff1fd8def32a3324fc0a45d0" Oct 30 12:30:19.373067 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:19.372964 4908 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0bc49434d657b8c2135dd865a0755c97d42fd1eff1fd8def32a3324fc0a45d0"} err="failed to get container status \"f0bc49434d657b8c2135dd865a0755c97d42fd1eff1fd8def32a3324fc0a45d0\": rpc error: code = NotFound desc = could not find container \"f0bc49434d657b8c2135dd865a0755c97d42fd1eff1fd8def32a3324fc0a45d0\": container with ID starting with f0bc49434d657b8c2135dd865a0755c97d42fd1eff1fd8def32a3324fc0a45d0 not found: ID does not exist" Oct 30 12:30:19.373135 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:19.373088 4908 scope.go:122] "RemoveContainer" containerID="0f3f74130158bb6ec65f0ebab62b5bb55689cbef9c963264cb90a0cf4c1fbf59" Oct 30 12:30:19.379560 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:19.379535 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-ec236e-29889390-2rvks" Oct 30 12:30:19.538219 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:19.538181 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1df31daa-0968-4f77-97a1-b9aa4b629cfc-kube-api-access-dlppm\" (UniqueName: \"kubernetes.io/projected/1df31daa-0968-4f77-97a1-b9aa4b629cfc-kube-api-access-dlppm\") pod \"1df31daa-0968-4f77-97a1-b9aa4b629cfc\" (UID: \"1df31daa-0968-4f77-97a1-b9aa4b629cfc\") " Oct 30 12:30:19.538608 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:19.538227 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1df31daa-0968-4f77-97a1-b9aa4b629cfc-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1df31daa-0968-4f77-97a1-b9aa4b629cfc-maas-api-service-ca\") pod \"1df31daa-0968-4f77-97a1-b9aa4b629cfc\" (UID: \"1df31daa-0968-4f77-97a1-b9aa4b629cfc\") " Oct 30 12:30:19.539119 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:19.539095 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1df31daa-0968-4f77-97a1-b9aa4b629cfc-maas-api-service-ca" pod "1df31daa-0968-4f77-97a1-b9aa4b629cfc" (UID: "1df31daa-0968-4f77-97a1-b9aa4b629cfc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:30:19.540819 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:19.540733 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1df31daa-0968-4f77-97a1-b9aa4b629cfc-kube-api-access-dlppm" pod "1df31daa-0968-4f77-97a1-b9aa4b629cfc" (UID: "1df31daa-0968-4f77-97a1-b9aa4b629cfc"). InnerVolumeSpecName "kube-api-access-dlppm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:30:19.608000 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:19.607975 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d8040e9-df95-42d1-8870-603107c66560" path="/var/lib/kubelet/pods/1d8040e9-df95-42d1-8870-603107c66560/volumes" Oct 30 12:30:19.608406 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:19.608389 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a416c883-7c57-4862-baf0-5add62763ba9" path="/var/lib/kubelet/pods/a416c883-7c57-4862-baf0-5add62763ba9/volumes" Oct 30 12:30:19.608766 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:19.608751 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc32f2ae-3aa3-4caa-b338-18a8ce68b577" path="/var/lib/kubelet/pods/cc32f2ae-3aa3-4caa-b338-18a8ce68b577/volumes" Oct 30 12:30:19.639939 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:19.639886 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlppm\" (UniqueName: \"kubernetes.io/projected/1df31daa-0968-4f77-97a1-b9aa4b629cfc-kube-api-access-dlppm\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:30:19.639939 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:19.639910 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1df31daa-0968-4f77-97a1-b9aa4b629cfc-maas-api-service-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:30:19.683352 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:30:19.683322 4908 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vn6vv" containerName="extractor" Oct 30 12:30:19.971607 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:19.971529 4908 generic.go:356] "Generic (PLEG): container finished" podID="76363248-837a-4227-974a-9613e1e598e0" containerID="0f4aacb96ef666b3f90b52b5dd26070f443e7a27758adb5529a36ee6b2b864d4" exitCode=60 Oct 30 12:30:19.971759 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:19.971595 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-c44eee-29889390-pjjzm" event={"ID":"76363248-837a-4227-974a-9613e1e598e0","Type":"ContainerDied","Data":"0f4aacb96ef666b3f90b52b5dd26070f443e7a27758adb5529a36ee6b2b864d4"} Oct 30 12:30:19.971759 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:19.971662 4908 scope.go:122] "RemoveContainer" containerID="c06e520993b53518031fdc47c583096aa97e4ceaef242e0826770ff912fd0c7e" Oct 30 12:30:19.972121 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:19.972101 4908 scope.go:122] "RemoveContainer" containerID="0f4aacb96ef666b3f90b52b5dd26070f443e7a27758adb5529a36ee6b2b864d4" Oct 30 12:30:19.972600 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:30:19.972573 4908 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-c44eee-29889390-pjjzm_odh-ai-gateway-infra(76363248-837a-4227-974a-9613e1e598e0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-c44eee-29889390-pjjzm" podUID="76363248-837a-4227-974a-9613e1e598e0" Oct 30 12:30:19.973892 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:19.973869 4908 generic.go:356] "Generic (PLEG): container finished" podID="6c7ac05a-ec43-4c05-9872-4694da6ec833" containerID="ce9f770a62e5c011c2a1fcbd2f1ea7b7d46341465da41b06f71c6123c5b98237" exitCode=60 Oct 30 12:30:19.973998 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:19.973948 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f920f7-29889390-489k2" event={"ID":"6c7ac05a-ec43-4c05-9872-4694da6ec833","Type":"ContainerDied","Data":"ce9f770a62e5c011c2a1fcbd2f1ea7b7d46341465da41b06f71c6123c5b98237"} Oct 30 12:30:19.974314 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:19.974297 4908 scope.go:122] "RemoveContainer" containerID="ce9f770a62e5c011c2a1fcbd2f1ea7b7d46341465da41b06f71c6123c5b98237" Oct 30 12:30:19.974613 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:30:19.974590 4908 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-f920f7-29889390-489k2_odh-ai-gateway-infra(6c7ac05a-ec43-4c05-9872-4694da6ec833)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f920f7-29889390-489k2" podUID="6c7ac05a-ec43-4c05-9872-4694da6ec833" Oct 30 12:30:19.975879 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:19.975858 4908 generic.go:356] "Generic (PLEG): container finished" podID="0a582fb0-974b-41fe-bc01-0712b6df3b31" containerID="04567df8abf9fe74b6d0432d603fef568540df1e4052e08677a4b8ff3b45e8d7" exitCode=60 Oct 30 12:30:19.975984 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:19.975857 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-8e3112-29889390-sptjx" event={"ID":"0a582fb0-974b-41fe-bc01-0712b6df3b31","Type":"ContainerDied","Data":"04567df8abf9fe74b6d0432d603fef568540df1e4052e08677a4b8ff3b45e8d7"} Oct 30 12:30:19.976245 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:19.976227 4908 scope.go:122] "RemoveContainer" containerID="04567df8abf9fe74b6d0432d603fef568540df1e4052e08677a4b8ff3b45e8d7" Oct 30 12:30:19.976493 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:30:19.976465 4908 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-8e3112-29889390-sptjx_odh-ai-gateway-infra(0a582fb0-974b-41fe-bc01-0712b6df3b31)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-8e3112-29889390-sptjx" podUID="0a582fb0-974b-41fe-bc01-0712b6df3b31" Oct 30 12:30:19.977542 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:19.977517 4908 generic.go:356] "Generic (PLEG): container finished" podID="a64aea77-808a-4b76-9e00-49faaba0314b" containerID="dc96eb27757db022136d47e820181a67052a243a8e9dec4714e8c5ad365a8b6d" exitCode=0 Oct 30 12:30:19.977633 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:19.977546 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ab9381-wqmpj" event={"ID":"a64aea77-808a-4b76-9e00-49faaba0314b","Type":"ContainerDied","Data":"dc96eb27757db022136d47e820181a67052a243a8e9dec4714e8c5ad365a8b6d"} Oct 30 12:30:19.985401 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:19.985379 4908 generic.go:356] "Generic (PLEG): container finished" podID="273ee4cd-dd11-4204-ba40-9073fde47d1d" containerID="9c9ad3268c094a98445611726a4b228f67bbbae6a6636e7a22df933453cda8c9" exitCode=60 Oct 30 12:30:19.985580 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:19.985450 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889390-pz259" event={"ID":"273ee4cd-dd11-4204-ba40-9073fde47d1d","Type":"ContainerDied","Data":"9c9ad3268c094a98445611726a4b228f67bbbae6a6636e7a22df933453cda8c9"} Oct 30 12:30:19.985948 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:19.985932 4908 scope.go:122] "RemoveContainer" containerID="9c9ad3268c094a98445611726a4b228f67bbbae6a6636e7a22df933453cda8c9" Oct 30 12:30:19.986317 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:30:19.986285 4908 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889390-pz259_odh-ai-gateway-infra(273ee4cd-dd11-4204-ba40-9073fde47d1d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889390-pz259" podUID="273ee4cd-dd11-4204-ba40-9073fde47d1d" Oct 30 12:30:19.990247 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:19.990231 4908 scope.go:122] "RemoveContainer" containerID="c132e3de070e6186dd4d418d926bf8c666b7da23315ecfa863cfe816b3212294" Oct 30 12:30:19.992300 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:19.992269 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-ec236e-29889390-2rvks" Oct 30 12:30:19.992509 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:19.992274 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-ec236e-29889390-2rvks" event={"ID":"1df31daa-0968-4f77-97a1-b9aa4b629cfc","Type":"ContainerDied","Data":"0085b0b9394ca8e0455b736c4a318b5b4ad004bcc3f464c61bdff6888ba5806a"} Oct 30 12:30:19.992509 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:19.992440 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-c44eee-29889390-pjjzm"] Oct 30 12:30:19.999218 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:19.998438 4908 generic.go:356] "Generic (PLEG): container finished" podID="101bab62-abe6-4870-83e8-d30758312a0c" containerID="22e995d593f86f3d36f58819dfcd04054ea78dba21f77adea0fa394a8567b614" exitCode=60 Oct 30 12:30:19.999218 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:19.998560 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a7148-29889390-jpq2d" event={"ID":"101bab62-abe6-4870-83e8-d30758312a0c","Type":"ContainerDied","Data":"22e995d593f86f3d36f58819dfcd04054ea78dba21f77adea0fa394a8567b614"} Oct 30 12:30:19.999218 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:19.998973 4908 scope.go:122] "RemoveContainer" containerID="22e995d593f86f3d36f58819dfcd04054ea78dba21f77adea0fa394a8567b614" Oct 30 12:30:19.999218 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:30:19.999205 4908 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-6a7148-29889390-jpq2d_odh-ai-gateway-infra(101bab62-abe6-4870-83e8-d30758312a0c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a7148-29889390-jpq2d" podUID="101bab62-abe6-4870-83e8-d30758312a0c" Oct 30 12:30:20.011477 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:20.011453 4908 scope.go:122] "RemoveContainer" containerID="55a715fc323ecab7724a0d24381d569b9b65a7ff935fee6a39e7b5504d131a04" Oct 30 12:30:20.165282 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:20.163250 4908 scope.go:122] "RemoveContainer" containerID="def4c4063cd6ef4cb977f351627c3743cfbb70769b43b710908fc793e165a25e" Oct 30 12:30:20.188869 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:20.188406 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-8e3112-29889390-sptjx"] Oct 30 12:30:20.194896 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:20.194872 4908 scope.go:122] "RemoveContainer" containerID="dcf35f7bd1c5c5ecfdbddae0aa85e4d437d16eeb704ddf25a17b1d6f2abadc32" Oct 30 12:30:20.207388 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:20.207365 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a7148-29889390-jpq2d"] Oct 30 12:30:20.210448 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:20.210426 4908 scope.go:122] "RemoveContainer" containerID="da8580ba591c8205250a6fbc281deb42224c17f9024cb427066110f34f821941" Oct 30 12:30:20.243661 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:20.243626 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889390-pz259"] Oct 30 12:30:20.277235 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:20.276995 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c1b1e8-29889390-hj6n9" Oct 30 12:30:20.436652 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:20.433628 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92ae9e02-1241-494f-8833-6081d874a34d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92ae9e02-1241-494f-8833-6081d874a34d-maas-api-service-ca\") pod \"92ae9e02-1241-494f-8833-6081d874a34d\" (UID: \"92ae9e02-1241-494f-8833-6081d874a34d\") " Oct 30 12:30:20.436652 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:20.433943 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92ae9e02-1241-494f-8833-6081d874a34d-kube-api-access-kkwfh\" (UniqueName: \"kubernetes.io/projected/92ae9e02-1241-494f-8833-6081d874a34d-kube-api-access-kkwfh\") pod \"92ae9e02-1241-494f-8833-6081d874a34d\" (UID: \"92ae9e02-1241-494f-8833-6081d874a34d\") " Oct 30 12:30:20.441841 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:20.441795 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92ae9e02-1241-494f-8833-6081d874a34d-maas-api-service-ca" pod "92ae9e02-1241-494f-8833-6081d874a34d" (UID: "92ae9e02-1241-494f-8833-6081d874a34d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:30:20.447723 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:20.447679 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7f78d4-fb954f9d7-2b2qr"] Oct 30 12:30:20.449369 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:20.449342 4908 cpu_manager.go:414] "containerMap: removing container" podUID="cc32f2ae-3aa3-4caa-b338-18a8ce68b577" containerName="cleanup" Oct 30 12:30:20.451400 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:20.451364 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc32f2ae-3aa3-4caa-b338-18a8ce68b577" containerName="cleanup" Oct 30 12:30:20.454716 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:20.454697 4908 cpu_manager.go:414] "containerMap: removing container" podUID="1d8040e9-df95-42d1-8870-603107c66560" containerName="istio-proxy" Oct 30 12:30:20.456756 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:20.456729 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d8040e9-df95-42d1-8870-603107c66560" containerName="istio-proxy" Oct 30 12:30:20.457002 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:20.456965 4908 cpu_manager.go:414] "containerMap: removing container" podUID="1df31daa-0968-4f77-97a1-b9aa4b629cfc" containerName="cleanup" Oct 30 12:30:20.457125 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:20.457112 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1df31daa-0968-4f77-97a1-b9aa4b629cfc" containerName="cleanup" Oct 30 12:30:20.457248 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:20.457233 4908 cpu_manager.go:414] "containerMap: removing container" podUID="cc32f2ae-3aa3-4caa-b338-18a8ce68b577" containerName="cleanup" Oct 30 12:30:20.457383 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:20.457351 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc32f2ae-3aa3-4caa-b338-18a8ce68b577" containerName="cleanup" Oct 30 12:30:20.457500 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:20.457489 4908 cpu_manager.go:414] "containerMap: removing container" podUID="92ae9e02-1241-494f-8833-6081d874a34d" containerName="cleanup" Oct 30 12:30:20.457610 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:20.457584 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92ae9e02-1241-494f-8833-6081d874a34d" containerName="cleanup" Oct 30 12:30:20.457740 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:20.457729 4908 cpu_manager.go:414] "containerMap: removing container" podUID="a416c883-7c57-4862-baf0-5add62763ba9" containerName="cleanup" Oct 30 12:30:20.457847 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:20.457821 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a416c883-7c57-4862-baf0-5add62763ba9" containerName="cleanup" Oct 30 12:30:20.458062 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:20.458040 4908 cpu_manager.go:414] "containerMap: removing container" podUID="1df31daa-0968-4f77-97a1-b9aa4b629cfc" containerName="cleanup" Oct 30 12:30:20.458160 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:20.458141 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1df31daa-0968-4f77-97a1-b9aa4b629cfc" containerName="cleanup" Oct 30 12:30:20.458267 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:20.458250 4908 cpu_manager.go:414] "containerMap: removing container" podUID="a416c883-7c57-4862-baf0-5add62763ba9" containerName="cleanup" Oct 30 12:30:20.458343 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:20.458333 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a416c883-7c57-4862-baf0-5add62763ba9" containerName="cleanup" Oct 30 12:30:20.458435 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:20.458426 4908 cpu_manager.go:414] "containerMap: removing container" podUID="a416c883-7c57-4862-baf0-5add62763ba9" containerName="cleanup" Oct 30 12:30:20.458499 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:20.458490 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a416c883-7c57-4862-baf0-5add62763ba9" containerName="cleanup" Oct 30 12:30:20.458571 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:20.458563 4908 cpu_manager.go:414] "containerMap: removing container" podUID="92ae9e02-1241-494f-8833-6081d874a34d" containerName="cleanup" Oct 30 12:30:20.458633 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:20.458625 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92ae9e02-1241-494f-8833-6081d874a34d" containerName="cleanup" Oct 30 12:30:20.461013 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:20.460893 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="92ae9e02-1241-494f-8833-6081d874a34d" containerName="cleanup" Oct 30 12:30:20.461135 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:20.461120 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="1df31daa-0968-4f77-97a1-b9aa4b629cfc" containerName="cleanup" Oct 30 12:30:20.461209 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:20.461199 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="a416c883-7c57-4862-baf0-5add62763ba9" containerName="cleanup" Oct 30 12:30:20.461300 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:20.461290 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="a416c883-7c57-4862-baf0-5add62763ba9" containerName="cleanup" Oct 30 12:30:20.461373 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:20.461364 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="92ae9e02-1241-494f-8833-6081d874a34d" containerName="cleanup" Oct 30 12:30:20.461447 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:20.461438 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="a416c883-7c57-4862-baf0-5add62763ba9" containerName="cleanup" Oct 30 12:30:20.461575 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:20.461564 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc32f2ae-3aa3-4caa-b338-18a8ce68b577" containerName="cleanup" Oct 30 12:30:20.461687 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:20.461675 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d8040e9-df95-42d1-8870-603107c66560" containerName="istio-proxy" Oct 30 12:30:20.461800 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:20.461789 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="1df31daa-0968-4f77-97a1-b9aa4b629cfc" containerName="cleanup" Oct 30 12:30:20.461985 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:20.461971 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc32f2ae-3aa3-4caa-b338-18a8ce68b577" containerName="cleanup" Oct 30 12:30:20.466093 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:20.464994 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92ae9e02-1241-494f-8833-6081d874a34d-kube-api-access-kkwfh" pod "92ae9e02-1241-494f-8833-6081d874a34d" (UID: "92ae9e02-1241-494f-8833-6081d874a34d"). InnerVolumeSpecName "kube-api-access-kkwfh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:30:20.468727 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:20.468703 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7f78d4-fb954f9d7-2b2qr" Oct 30 12:30:20.482331 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:20.482280 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-5c57af-29889390-hphdb" Oct 30 12:30:20.501882 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:20.501847 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7f78d4-fb954f9d7-2b2qr"] Oct 30 12:30:20.506006 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:20.505983 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-7f78d4\"" Oct 30 12:30:20.520692 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:20.520671 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-7f78d4-dockercfg-5n628\"" Oct 30 12:30:20.534775 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:20.534750 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/016efbc9-f46c-4dd9-8910-2dbec739d99a-kube-api-access-sgqgz\" (UniqueName: \"kubernetes.io/projected/016efbc9-f46c-4dd9-8910-2dbec739d99a-kube-api-access-sgqgz\") pod \"016efbc9-f46c-4dd9-8910-2dbec739d99a\" (UID: \"016efbc9-f46c-4dd9-8910-2dbec739d99a\") " Oct 30 12:30:20.534944 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:20.534901 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/016efbc9-f46c-4dd9-8910-2dbec739d99a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/016efbc9-f46c-4dd9-8910-2dbec739d99a-maas-api-service-ca\") pod \"016efbc9-f46c-4dd9-8910-2dbec739d99a\" (UID: \"016efbc9-f46c-4dd9-8910-2dbec739d99a\") " Oct 30 12:30:20.535160 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:20.535140 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n64hf\" (UniqueName: \"kubernetes.io/projected/7b1f1ddf-f604-4b35-b246-9d40ff5fa84c-kube-api-access-n64hf\") pod \"payload-pre-processing-e2e-ipp-b-7f78d4-fb954f9d7-2b2qr\" (UID: \"7b1f1ddf-f604-4b35-b246-9d40ff5fa84c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7f78d4-fb954f9d7-2b2qr" Oct 30 12:30:20.535329 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:20.535305 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b1f1ddf-f604-4b35-b246-9d40ff5fa84c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-7f78d4-fb954f9d7-2b2qr\" (UID: \"7b1f1ddf-f604-4b35-b246-9d40ff5fa84c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7f78d4-fb954f9d7-2b2qr" Oct 30 12:30:20.535580 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:20.535560 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkwfh\" (UniqueName: \"kubernetes.io/projected/92ae9e02-1241-494f-8833-6081d874a34d-kube-api-access-kkwfh\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:30:20.535663 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:20.535584 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92ae9e02-1241-494f-8833-6081d874a34d-maas-api-service-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:30:20.535663 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:20.535234 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/016efbc9-f46c-4dd9-8910-2dbec739d99a-maas-api-service-ca" pod "016efbc9-f46c-4dd9-8910-2dbec739d99a" (UID: "016efbc9-f46c-4dd9-8910-2dbec739d99a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:30:20.567850 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:20.558245 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/016efbc9-f46c-4dd9-8910-2dbec739d99a-kube-api-access-sgqgz" pod "016efbc9-f46c-4dd9-8910-2dbec739d99a" (UID: "016efbc9-f46c-4dd9-8910-2dbec739d99a"). InnerVolumeSpecName "kube-api-access-sgqgz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:30:20.567850 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:20.564801 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7f78d4-c859fdc56-jzwrn"] Oct 30 12:30:20.567850 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:20.565869 4908 cpu_manager.go:414] "containerMap: removing container" podUID="1df31daa-0968-4f77-97a1-b9aa4b629cfc" containerName="cleanup" Oct 30 12:30:20.567850 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:20.565887 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1df31daa-0968-4f77-97a1-b9aa4b629cfc" containerName="cleanup" Oct 30 12:30:20.567850 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:20.565919 4908 cpu_manager.go:414] "containerMap: removing container" podUID="92ae9e02-1241-494f-8833-6081d874a34d" containerName="cleanup" Oct 30 12:30:20.567850 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:20.565926 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92ae9e02-1241-494f-8833-6081d874a34d" containerName="cleanup" Oct 30 12:30:20.567850 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:20.565949 4908 cpu_manager.go:414] "containerMap: removing container" podUID="cc32f2ae-3aa3-4caa-b338-18a8ce68b577" containerName="cleanup" Oct 30 12:30:20.567850 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:20.565956 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc32f2ae-3aa3-4caa-b338-18a8ce68b577" containerName="cleanup" Oct 30 12:30:20.567850 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:20.565968 4908 cpu_manager.go:414] "containerMap: removing container" podUID="016efbc9-f46c-4dd9-8910-2dbec739d99a" containerName="cleanup" Oct 30 12:30:20.567850 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:20.565975 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="016efbc9-f46c-4dd9-8910-2dbec739d99a" containerName="cleanup" Oct 30 12:30:20.567850 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:20.566024 4908 cpu_manager.go:414] "containerMap: removing container" podUID="016efbc9-f46c-4dd9-8910-2dbec739d99a" containerName="cleanup" Oct 30 12:30:20.567850 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:20.566030 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="016efbc9-f46c-4dd9-8910-2dbec739d99a" containerName="cleanup" Oct 30 12:30:20.567850 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:20.566140 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="016efbc9-f46c-4dd9-8910-2dbec739d99a" containerName="cleanup" Oct 30 12:30:20.567850 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:20.566154 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="1df31daa-0968-4f77-97a1-b9aa4b629cfc" containerName="cleanup" Oct 30 12:30:20.567850 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:20.566168 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="016efbc9-f46c-4dd9-8910-2dbec739d99a" containerName="cleanup" Oct 30 12:30:20.567850 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:20.566179 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="92ae9e02-1241-494f-8833-6081d874a34d" containerName="cleanup" Oct 30 12:30:20.567850 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:20.566191 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc32f2ae-3aa3-4caa-b338-18a8ce68b577" containerName="cleanup" Oct 30 12:30:20.583866 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:20.582968 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7f78d4-c859fdc56-jzwrn"] Oct 30 12:30:20.602809 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:20.602787 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-7f78d4-c859fdc56-jzwrn" Oct 30 12:30:20.638155 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:20.636752 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b1f1ddf-f604-4b35-b246-9d40ff5fa84c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-7f78d4-fb954f9d7-2b2qr\" (UID: \"7b1f1ddf-f604-4b35-b246-9d40ff5fa84c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7f78d4-fb954f9d7-2b2qr" Oct 30 12:30:20.638155 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:20.636800 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lz4hq\" (UniqueName: \"kubernetes.io/projected/a0fdb965-0b23-4560-8e89-b41e8c926cd8-kube-api-access-lz4hq\") pod \"payload-processing-e2e-ipp-b-7f78d4-c859fdc56-jzwrn\" (UID: \"a0fdb965-0b23-4560-8e89-b41e8c926cd8\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7f78d4-c859fdc56-jzwrn" Oct 30 12:30:20.638155 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:20.636961 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0fdb965-0b23-4560-8e89-b41e8c926cd8-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-7f78d4-c859fdc56-jzwrn\" (UID: \"a0fdb965-0b23-4560-8e89-b41e8c926cd8\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7f78d4-c859fdc56-jzwrn" Oct 30 12:30:20.638155 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:20.637107 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n64hf\" (UniqueName: \"kubernetes.io/projected/7b1f1ddf-f604-4b35-b246-9d40ff5fa84c-kube-api-access-n64hf\") pod \"payload-pre-processing-e2e-ipp-b-7f78d4-fb954f9d7-2b2qr\" (UID: \"7b1f1ddf-f604-4b35-b246-9d40ff5fa84c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7f78d4-fb954f9d7-2b2qr" Oct 30 12:30:20.638155 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:20.637185 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/016efbc9-f46c-4dd9-8910-2dbec739d99a-maas-api-service-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:30:20.638155 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:20.637200 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sgqgz\" (UniqueName: \"kubernetes.io/projected/016efbc9-f46c-4dd9-8910-2dbec739d99a-kube-api-access-sgqgz\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:30:20.638155 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:20.638108 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b1f1ddf-f604-4b35-b246-9d40ff5fa84c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-7f78d4-fb954f9d7-2b2qr\" (UID: \"7b1f1ddf-f604-4b35-b246-9d40ff5fa84c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7f78d4-fb954f9d7-2b2qr" Oct 30 12:30:20.642407 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:20.642344 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-ec236e-29889390-2rvks"] Oct 30 12:30:20.645810 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:20.645787 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-ec236e-29889390-2rvks"] Oct 30 12:30:20.659184 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:20.659163 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n64hf\" (UniqueName: \"kubernetes.io/projected/7b1f1ddf-f604-4b35-b246-9d40ff5fa84c-kube-api-access-n64hf\") pod \"payload-pre-processing-e2e-ipp-b-7f78d4-fb954f9d7-2b2qr\" (UID: \"7b1f1ddf-f604-4b35-b246-9d40ff5fa84c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7f78d4-fb954f9d7-2b2qr" Oct 30 12:30:20.740882 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:20.739453 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lz4hq\" (UniqueName: \"kubernetes.io/projected/a0fdb965-0b23-4560-8e89-b41e8c926cd8-kube-api-access-lz4hq\") pod \"payload-processing-e2e-ipp-b-7f78d4-c859fdc56-jzwrn\" (UID: \"a0fdb965-0b23-4560-8e89-b41e8c926cd8\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7f78d4-c859fdc56-jzwrn" Oct 30 12:30:20.740882 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:20.739590 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0fdb965-0b23-4560-8e89-b41e8c926cd8-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-7f78d4-c859fdc56-jzwrn\" (UID: \"a0fdb965-0b23-4560-8e89-b41e8c926cd8\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7f78d4-c859fdc56-jzwrn" Oct 30 12:30:20.740882 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:20.740523 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0fdb965-0b23-4560-8e89-b41e8c926cd8-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-7f78d4-c859fdc56-jzwrn\" (UID: \"a0fdb965-0b23-4560-8e89-b41e8c926cd8\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7f78d4-c859fdc56-jzwrn" Oct 30 12:30:20.763107 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:20.763076 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lz4hq\" (UniqueName: \"kubernetes.io/projected/a0fdb965-0b23-4560-8e89-b41e8c926cd8-kube-api-access-lz4hq\") pod \"payload-processing-e2e-ipp-b-7f78d4-c859fdc56-jzwrn\" (UID: \"a0fdb965-0b23-4560-8e89-b41e8c926cd8\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7f78d4-c859fdc56-jzwrn" Oct 30 12:30:20.798090 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:20.798054 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7f78d4-fb954f9d7-2b2qr" Oct 30 12:30:20.935140 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:20.935097 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-7f78d4-c859fdc56-jzwrn" Oct 30 12:30:21.022014 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:21.018749 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-5c57af-29889390-hphdb" event={"ID":"016efbc9-f46c-4dd9-8910-2dbec739d99a","Type":"ContainerDied","Data":"c7d818974a7911112bd83c1661e543650899744b110dec527f39b1c3c9f1ae25"} Oct 30 12:30:21.022014 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:21.018847 4908 scope.go:122] "RemoveContainer" containerID="7235f44b9502c1d727ffb3a3f1fc077240eb5dea43d614424f7e45fab54e3ad6" Oct 30 12:30:21.022014 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:21.018952 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-5c57af-29889390-hphdb" Oct 30 12:30:21.216529 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:21.209654 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-5c57af-29889390-hphdb"] Oct 30 12:30:21.218550 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:21.218522 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-5c57af-29889390-hphdb"] Oct 30 12:30:21.218708 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:21.218694 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7f78d4-fb954f9d7-2b2qr"] Oct 30 12:30:21.242617 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:21.242071 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c1b1e8-29889390-hj6n9" Oct 30 12:30:21.242791 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:21.242731 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c1b1e8-29889390-hj6n9" event={"ID":"92ae9e02-1241-494f-8833-6081d874a34d","Type":"ContainerDied","Data":"b6c8631e030e169f85d21b379c7b8c5119bb31a69e4fba1d0ad6c7ea5ae5ba71"} Oct 30 12:30:21.286551 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:21.276556 4908 scope.go:122] "RemoveContainer" containerID="de1d5512889df12795779fd9afb89cc8afdb1a063ee1e0f330d6cd934fab12ca" Oct 30 12:30:21.348807 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:21.348766 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7f78d4-c859fdc56-jzwrn"] Oct 30 12:30:21.352167 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:30:21.352119 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0fdb965_0b23_4560_8e89_b41e8c926cd8.slice/crio-ce355122a903b4320ad4d4012250941c602be0b30da48fb3efa06760941613db WatchSource:0}: Error finding container ce355122a903b4320ad4d4012250941c602be0b30da48fb3efa06760941613db: Status 404 returned error can't find the container with id ce355122a903b4320ad4d4012250941c602be0b30da48fb3efa06760941613db Oct 30 12:30:21.392115 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:21.390020 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c1b1e8-29889390-hj6n9"] Oct 30 12:30:21.399641 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:21.399613 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c1b1e8-29889390-hj6n9"] Oct 30 12:30:21.653880 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:21.652350 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="016efbc9-f46c-4dd9-8910-2dbec739d99a" path="/var/lib/kubelet/pods/016efbc9-f46c-4dd9-8910-2dbec739d99a/volumes" Oct 30 12:30:21.653880 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:21.653032 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1df31daa-0968-4f77-97a1-b9aa4b629cfc" path="/var/lib/kubelet/pods/1df31daa-0968-4f77-97a1-b9aa4b629cfc/volumes" Oct 30 12:30:21.653880 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:21.653471 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92ae9e02-1241-494f-8833-6081d874a34d" path="/var/lib/kubelet/pods/92ae9e02-1241-494f-8833-6081d874a34d/volumes" Oct 30 12:30:21.656129 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:21.656106 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f920f7-29889390-489k2"] Oct 30 12:30:21.657245 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:21.656269 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889390-pz259" Oct 30 12:30:21.669927 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:21.664187 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-c44eee-29889390-pjjzm" Oct 30 12:30:21.669927 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:21.668820 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a7148-29889390-jpq2d" Oct 30 12:30:21.681405 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:21.681389 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ab9381-wqmpj" Oct 30 12:30:21.683599 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:21.683585 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-8e3112-29889390-sptjx" Oct 30 12:30:21.795945 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:21.789568 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/101bab62-abe6-4870-83e8-d30758312a0c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/101bab62-abe6-4870-83e8-d30758312a0c-maas-api-service-ca\") pod \"101bab62-abe6-4870-83e8-d30758312a0c\" (UID: \"101bab62-abe6-4870-83e8-d30758312a0c\") " Oct 30 12:30:21.795945 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:21.789625 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a582fb0-974b-41fe-bc01-0712b6df3b31-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a582fb0-974b-41fe-bc01-0712b6df3b31-maas-api-service-ca\") pod \"0a582fb0-974b-41fe-bc01-0712b6df3b31\" (UID: \"0a582fb0-974b-41fe-bc01-0712b6df3b31\") " Oct 30 12:30:21.795945 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:21.789654 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/273ee4cd-dd11-4204-ba40-9073fde47d1d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/273ee4cd-dd11-4204-ba40-9073fde47d1d-maas-api-service-ca\") pod \"273ee4cd-dd11-4204-ba40-9073fde47d1d\" (UID: \"273ee4cd-dd11-4204-ba40-9073fde47d1d\") " Oct 30 12:30:21.795945 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:21.789697 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a582fb0-974b-41fe-bc01-0712b6df3b31-kube-api-access-dlfcm\" (UniqueName: \"kubernetes.io/projected/0a582fb0-974b-41fe-bc01-0712b6df3b31-kube-api-access-dlfcm\") pod \"0a582fb0-974b-41fe-bc01-0712b6df3b31\" (UID: \"0a582fb0-974b-41fe-bc01-0712b6df3b31\") " Oct 30 12:30:21.795945 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:21.789777 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/101bab62-abe6-4870-83e8-d30758312a0c-kube-api-access-hhtzc\" (UniqueName: \"kubernetes.io/projected/101bab62-abe6-4870-83e8-d30758312a0c-kube-api-access-hhtzc\") pod \"101bab62-abe6-4870-83e8-d30758312a0c\" (UID: \"101bab62-abe6-4870-83e8-d30758312a0c\") " Oct 30 12:30:21.795945 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:21.789807 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76363248-837a-4227-974a-9613e1e598e0-kube-api-access-nnr7v\" (UniqueName: \"kubernetes.io/projected/76363248-837a-4227-974a-9613e1e598e0-kube-api-access-nnr7v\") pod \"76363248-837a-4227-974a-9613e1e598e0\" (UID: \"76363248-837a-4227-974a-9613e1e598e0\") " Oct 30 12:30:21.795945 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:21.789854 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a64aea77-808a-4b76-9e00-49faaba0314b-kube-api-access-7nmtt\" (UniqueName: \"kubernetes.io/projected/a64aea77-808a-4b76-9e00-49faaba0314b-kube-api-access-7nmtt\") pod \"a64aea77-808a-4b76-9e00-49faaba0314b\" (UID: \"a64aea77-808a-4b76-9e00-49faaba0314b\") " Oct 30 12:30:21.795945 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:21.789932 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a64aea77-808a-4b76-9e00-49faaba0314b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a64aea77-808a-4b76-9e00-49faaba0314b-maas-api-openshift-ca\") pod \"a64aea77-808a-4b76-9e00-49faaba0314b\" (UID: \"a64aea77-808a-4b76-9e00-49faaba0314b\") " Oct 30 12:30:21.795945 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:21.789977 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a64aea77-808a-4b76-9e00-49faaba0314b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a64aea77-808a-4b76-9e00-49faaba0314b-maas-api-xks-ca\") pod \"a64aea77-808a-4b76-9e00-49faaba0314b\" (UID: \"a64aea77-808a-4b76-9e00-49faaba0314b\") " Oct 30 12:30:21.795945 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:21.790010 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76363248-837a-4227-974a-9613e1e598e0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/76363248-837a-4227-974a-9613e1e598e0-maas-api-service-ca\") pod \"76363248-837a-4227-974a-9613e1e598e0\" (UID: \"76363248-837a-4227-974a-9613e1e598e0\") " Oct 30 12:30:21.795945 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:21.790143 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/273ee4cd-dd11-4204-ba40-9073fde47d1d-kube-api-access-6npwr\" (UniqueName: \"kubernetes.io/projected/273ee4cd-dd11-4204-ba40-9073fde47d1d-kube-api-access-6npwr\") pod \"273ee4cd-dd11-4204-ba40-9073fde47d1d\" (UID: \"273ee4cd-dd11-4204-ba40-9073fde47d1d\") " Oct 30 12:30:21.798501 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:21.798462 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/101bab62-abe6-4870-83e8-d30758312a0c-maas-api-service-ca" pod "101bab62-abe6-4870-83e8-d30758312a0c" (UID: "101bab62-abe6-4870-83e8-d30758312a0c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:30:21.799409 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:21.799385 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a64aea77-808a-4b76-9e00-49faaba0314b-maas-api-openshift-ca" pod "a64aea77-808a-4b76-9e00-49faaba0314b" (UID: "a64aea77-808a-4b76-9e00-49faaba0314b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:30:21.799894 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:21.799874 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76363248-837a-4227-974a-9613e1e598e0-maas-api-service-ca" pod "76363248-837a-4227-974a-9613e1e598e0" (UID: "76363248-837a-4227-974a-9613e1e598e0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:30:21.800297 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:21.800280 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/273ee4cd-dd11-4204-ba40-9073fde47d1d-maas-api-service-ca" pod "273ee4cd-dd11-4204-ba40-9073fde47d1d" (UID: "273ee4cd-dd11-4204-ba40-9073fde47d1d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:30:21.800694 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:21.800677 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a582fb0-974b-41fe-bc01-0712b6df3b31-maas-api-service-ca" pod "0a582fb0-974b-41fe-bc01-0712b6df3b31" (UID: "0a582fb0-974b-41fe-bc01-0712b6df3b31"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:30:21.805455 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:21.805412 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a582fb0-974b-41fe-bc01-0712b6df3b31-kube-api-access-dlfcm" pod "0a582fb0-974b-41fe-bc01-0712b6df3b31" (UID: "0a582fb0-974b-41fe-bc01-0712b6df3b31"). InnerVolumeSpecName "kube-api-access-dlfcm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:30:21.808297 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:21.808236 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/101bab62-abe6-4870-83e8-d30758312a0c-kube-api-access-hhtzc" pod "101bab62-abe6-4870-83e8-d30758312a0c" (UID: "101bab62-abe6-4870-83e8-d30758312a0c"). InnerVolumeSpecName "kube-api-access-hhtzc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:30:21.809328 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:21.809293 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/273ee4cd-dd11-4204-ba40-9073fde47d1d-kube-api-access-6npwr" pod "273ee4cd-dd11-4204-ba40-9073fde47d1d" (UID: "273ee4cd-dd11-4204-ba40-9073fde47d1d"). InnerVolumeSpecName "kube-api-access-6npwr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:30:21.810867 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:21.810526 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76363248-837a-4227-974a-9613e1e598e0-kube-api-access-nnr7v" pod "76363248-837a-4227-974a-9613e1e598e0" (UID: "76363248-837a-4227-974a-9613e1e598e0"). InnerVolumeSpecName "kube-api-access-nnr7v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:30:21.811753 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:21.810966 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a64aea77-808a-4b76-9e00-49faaba0314b-kube-api-access-7nmtt" pod "a64aea77-808a-4b76-9e00-49faaba0314b" (UID: "a64aea77-808a-4b76-9e00-49faaba0314b"). InnerVolumeSpecName "kube-api-access-7nmtt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:30:21.815482 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:21.815452 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a64aea77-808a-4b76-9e00-49faaba0314b-maas-api-xks-ca" pod "a64aea77-808a-4b76-9e00-49faaba0314b" (UID: "a64aea77-808a-4b76-9e00-49faaba0314b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:30:21.892350 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:21.892317 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nnr7v\" (UniqueName: \"kubernetes.io/projected/76363248-837a-4227-974a-9613e1e598e0-kube-api-access-nnr7v\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:30:21.892350 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:21.892356 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7nmtt\" (UniqueName: \"kubernetes.io/projected/a64aea77-808a-4b76-9e00-49faaba0314b-kube-api-access-7nmtt\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:30:21.892541 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:21.892373 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a64aea77-808a-4b76-9e00-49faaba0314b-maas-api-openshift-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:30:21.892541 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:21.892387 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a64aea77-808a-4b76-9e00-49faaba0314b-maas-api-xks-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:30:21.892541 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:21.892399 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/76363248-837a-4227-974a-9613e1e598e0-maas-api-service-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:30:21.892541 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:21.892412 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6npwr\" (UniqueName: \"kubernetes.io/projected/273ee4cd-dd11-4204-ba40-9073fde47d1d-kube-api-access-6npwr\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:30:21.892541 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:21.892428 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/101bab62-abe6-4870-83e8-d30758312a0c-maas-api-service-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:30:21.892541 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:21.892441 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a582fb0-974b-41fe-bc01-0712b6df3b31-maas-api-service-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:30:21.892541 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:21.892456 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/273ee4cd-dd11-4204-ba40-9073fde47d1d-maas-api-service-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:30:21.892541 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:21.892489 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlfcm\" (UniqueName: \"kubernetes.io/projected/0a582fb0-974b-41fe-bc01-0712b6df3b31-kube-api-access-dlfcm\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:30:21.892541 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:21.892503 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhtzc\" (UniqueName: \"kubernetes.io/projected/101bab62-abe6-4870-83e8-d30758312a0c-kube-api-access-hhtzc\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:30:21.933252 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:21.933225 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f920f7-29889390-489k2" Oct 30 12:30:21.993693 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:21.993658 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c7ac05a-ec43-4c05-9872-4694da6ec833-kube-api-access-hjhvp\" (UniqueName: \"kubernetes.io/projected/6c7ac05a-ec43-4c05-9872-4694da6ec833-kube-api-access-hjhvp\") pod \"6c7ac05a-ec43-4c05-9872-4694da6ec833\" (UID: \"6c7ac05a-ec43-4c05-9872-4694da6ec833\") " Oct 30 12:30:21.993924 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:21.993904 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c7ac05a-ec43-4c05-9872-4694da6ec833-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c7ac05a-ec43-4c05-9872-4694da6ec833-maas-api-service-ca\") pod \"6c7ac05a-ec43-4c05-9872-4694da6ec833\" (UID: \"6c7ac05a-ec43-4c05-9872-4694da6ec833\") " Oct 30 12:30:21.994656 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:21.994632 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c7ac05a-ec43-4c05-9872-4694da6ec833-maas-api-service-ca" pod "6c7ac05a-ec43-4c05-9872-4694da6ec833" (UID: "6c7ac05a-ec43-4c05-9872-4694da6ec833"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:30:21.997103 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:21.997080 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c7ac05a-ec43-4c05-9872-4694da6ec833-kube-api-access-hjhvp" pod "6c7ac05a-ec43-4c05-9872-4694da6ec833" (UID: "6c7ac05a-ec43-4c05-9872-4694da6ec833"). InnerVolumeSpecName "kube-api-access-hjhvp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:30:22.095503 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:22.095382 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hjhvp\" (UniqueName: \"kubernetes.io/projected/6c7ac05a-ec43-4c05-9872-4694da6ec833-kube-api-access-hjhvp\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:30:22.095687 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:22.095671 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c7ac05a-ec43-4c05-9872-4694da6ec833-maas-api-service-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:30:22.253341 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:22.253010 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7f78d4-fb954f9d7-2b2qr" event={"ID":"7b1f1ddf-f604-4b35-b246-9d40ff5fa84c","Type":"ContainerStarted","Data":"b2215a41f390e74b4fa9da0c74491ed45a8ae4f2ea3e4cfe98d9e80b3635986f"} Oct 30 12:30:22.253575 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:22.253555 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7f78d4-fb954f9d7-2b2qr" Oct 30 12:30:22.253575 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:22.253586 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7f78d4-fb954f9d7-2b2qr" event={"ID":"7b1f1ddf-f604-4b35-b246-9d40ff5fa84c","Type":"ContainerStarted","Data":"575cf392461c2f4f803f1049080a3b3257abdcd1e9a319746d6373de4ebead30"} Oct 30 12:30:22.260392 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:22.260374 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889390-pz259" Oct 30 12:30:22.260611 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:22.260373 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889390-pz259" event={"ID":"273ee4cd-dd11-4204-ba40-9073fde47d1d","Type":"ContainerDied","Data":"0c44e3dfea3392f0743fa0e40ed3bea90ad8e6afc87b2378bda58bcd6f2ab080"} Oct 30 12:30:22.260682 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:22.260652 4908 scope.go:122] "RemoveContainer" containerID="9c9ad3268c094a98445611726a4b228f67bbbae6a6636e7a22df933453cda8c9" Oct 30 12:30:22.262253 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:22.262142 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a7148-29889390-jpq2d" Oct 30 12:30:22.262608 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:22.262585 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a7148-29889390-jpq2d" event={"ID":"101bab62-abe6-4870-83e8-d30758312a0c","Type":"ContainerDied","Data":"bbdaed24c1fc0bc24399b367f7f0d76e4437becc8c5167a5bc5b10e00e9e0d03"} Oct 30 12:30:22.264286 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:22.264264 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-7f78d4-c859fdc56-jzwrn" event={"ID":"a0fdb965-0b23-4560-8e89-b41e8c926cd8","Type":"ContainerStarted","Data":"b7e79e962b872168a1e3e9473e5565cfa4f5d97136e493da8ab5fc6549c719c2"} Oct 30 12:30:22.264381 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:22.264308 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-7f78d4-c859fdc56-jzwrn" event={"ID":"a0fdb965-0b23-4560-8e89-b41e8c926cd8","Type":"ContainerStarted","Data":"ce355122a903b4320ad4d4012250941c602be0b30da48fb3efa06760941613db"} Oct 30 12:30:22.264750 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:22.264733 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-7f78d4-c859fdc56-jzwrn" Oct 30 12:30:22.267232 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:22.267212 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-c44eee-29889390-pjjzm" event={"ID":"76363248-837a-4227-974a-9613e1e598e0","Type":"ContainerDied","Data":"748e28003d52226b2950ac32c51275cbfe8826f050b11b767c30f5031a2c0de9"} Oct 30 12:30:22.267401 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:22.267387 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-c44eee-29889390-pjjzm" Oct 30 12:30:22.276312 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:22.276291 4908 scope.go:122] "RemoveContainer" containerID="22e995d593f86f3d36f58819dfcd04054ea78dba21f77adea0fa394a8567b614" Oct 30 12:30:22.278928 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:22.278905 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f920f7-29889390-489k2" event={"ID":"6c7ac05a-ec43-4c05-9872-4694da6ec833","Type":"ContainerDied","Data":"ed4584f8de0552ab93ec4ee28db1108d47aa7ca55f3a381abb05c1bb76f7085a"} Oct 30 12:30:22.279025 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:22.278988 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f920f7-29889390-489k2" Oct 30 12:30:22.289338 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:22.289159 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-8e3112-29889390-sptjx" Oct 30 12:30:22.289338 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:22.289171 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-8e3112-29889390-sptjx" event={"ID":"0a582fb0-974b-41fe-bc01-0712b6df3b31","Type":"ContainerDied","Data":"f5bba0dd2803996150f3901b33459f3781d67f736dfa40efb54de1aa9c0ac789"} Oct 30 12:30:22.291911 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:22.291891 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ab9381-wqmpj" event={"ID":"a64aea77-808a-4b76-9e00-49faaba0314b","Type":"ContainerDied","Data":"e5e15d32069083dc72b7c74e7bd2187e9d0b4034d074b82fd762c7d16addddee"} Oct 30 12:30:22.292024 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:22.291923 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e5e15d32069083dc72b7c74e7bd2187e9d0b4034d074b82fd762c7d16addddee" Oct 30 12:30:22.292024 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:22.291963 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ab9381-wqmpj" Oct 30 12:30:22.295435 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:22.295419 4908 scope.go:122] "RemoveContainer" containerID="0f4aacb96ef666b3f90b52b5dd26070f443e7a27758adb5529a36ee6b2b864d4" Oct 30 12:30:22.303210 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:22.303162 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7f78d4-fb954f9d7-2b2qr" podStartSLOduration=2.303145885 podStartE2EDuration="2.303145885s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:30:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:30:22.297237735 +0000 UTC m=+2178.811666481" watchObservedRunningTime="2026-10-30 12:30:22.303145885 +0000 UTC m=+2178.817574637" Oct 30 12:30:22.310662 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:22.310643 4908 scope.go:122] "RemoveContainer" containerID="ce9f770a62e5c011c2a1fcbd2f1ea7b7d46341465da41b06f71c6123c5b98237" Oct 30 12:30:22.320352 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:22.320337 4908 scope.go:122] "RemoveContainer" containerID="04567df8abf9fe74b6d0432d603fef568540df1e4052e08677a4b8ff3b45e8d7" Oct 30 12:30:22.352450 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:22.352333 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-7f78d4-c859fdc56-jzwrn" podStartSLOduration=2.352314498 podStartE2EDuration="2.352314498s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:30:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:30:22.351117882 +0000 UTC m=+2178.865546653" watchObservedRunningTime="2026-10-30 12:30:22.352314498 +0000 UTC m=+2178.866743243" Oct 30 12:30:22.386119 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:22.386097 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889390-pz259"] Oct 30 12:30:22.389952 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:22.389932 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889390-pz259"] Oct 30 12:30:22.451846 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:22.451554 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a7148-29889390-jpq2d"] Oct 30 12:30:22.456545 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:22.456516 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a7148-29889390-jpq2d"] Oct 30 12:30:22.487642 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:22.487614 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-8e3112-29889390-sptjx"] Oct 30 12:30:22.492326 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:22.492296 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-8e3112-29889390-sptjx"] Oct 30 12:30:22.522940 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:22.522913 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f920f7-29889390-489k2"] Oct 30 12:30:22.527986 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:22.527959 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f920f7-29889390-489k2"] Oct 30 12:30:22.581926 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:22.581891 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-c44eee-29889390-pjjzm"] Oct 30 12:30:22.585692 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:22.585664 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-c44eee-29889390-pjjzm"] Oct 30 12:30:23.608042 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:23.607967 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a582fb0-974b-41fe-bc01-0712b6df3b31" path="/var/lib/kubelet/pods/0a582fb0-974b-41fe-bc01-0712b6df3b31/volumes" Oct 30 12:30:23.608596 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:23.608575 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="101bab62-abe6-4870-83e8-d30758312a0c" path="/var/lib/kubelet/pods/101bab62-abe6-4870-83e8-d30758312a0c/volumes" Oct 30 12:30:23.609110 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:23.609091 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="273ee4cd-dd11-4204-ba40-9073fde47d1d" path="/var/lib/kubelet/pods/273ee4cd-dd11-4204-ba40-9073fde47d1d/volumes" Oct 30 12:30:23.609752 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:23.609736 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c7ac05a-ec43-4c05-9872-4694da6ec833" path="/var/lib/kubelet/pods/6c7ac05a-ec43-4c05-9872-4694da6ec833/volumes" Oct 30 12:30:23.610156 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:23.610141 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76363248-837a-4227-974a-9613e1e598e0" path="/var/lib/kubelet/pods/76363248-837a-4227-974a-9613e1e598e0/volumes" Oct 30 12:30:24.323334 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:24.323304 4908 generic.go:356] "Generic (PLEG): container finished" podID="5ea7d109-957f-4d4f-9897-6eb86c402601" containerID="5129d25b769d5c0f940748bcb1555021168905a268bd4b3c31a2c6fba52703e1" exitCode=0 Oct 30 12:30:24.323524 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:24.323382 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-ec236e-openshift-default-f6575b7df-kwhwr" event={"ID":"5ea7d109-957f-4d4f-9897-6eb86c402601","Type":"ContainerDied","Data":"5129d25b769d5c0f940748bcb1555021168905a268bd4b3c31a2c6fba52703e1"} Oct 30 12:30:24.323904 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:24.323883 4908 scope.go:122] "RemoveContainer" containerID="5129d25b769d5c0f940748bcb1555021168905a268bd4b3c31a2c6fba52703e1" Oct 30 12:30:24.325289 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:24.325259 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:30:24.325406 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:24.325339 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:30:24.325406 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:24.325389 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:30:25.343707 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:25.343669 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-ec236e-openshift-default-f6575b7df-kwhwr" event={"ID":"5ea7d109-957f-4d4f-9897-6eb86c402601","Type":"ContainerStarted","Data":"a5d8ca0995f109f84aed0f9339343f70c6e299be9de38d2f25cacc6902008135"} Oct 30 12:30:25.675908 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:25.674883 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-ec236e-openshift-default-f6575b7df-kwhwr" Oct 30 12:30:25.679859 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:25.678910 4908 patch_prober.go:29] interesting pod/e2e-praxis-b-ec236e-openshift-default-f6575b7df-kwhwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body= Oct 30 12:30:25.679859 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:25.678962 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-ec236e-openshift-default-f6575b7df-kwhwr" podUID="5ea7d109-957f-4d4f-9897-6eb86c402601" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" Oct 30 12:30:26.673853 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:26.670091 4908 patch_prober.go:29] interesting pod/e2e-praxis-b-ec236e-openshift-default-f6575b7df-kwhwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body= Oct 30 12:30:26.673853 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:26.670150 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-ec236e-openshift-default-f6575b7df-kwhwr" podUID="5ea7d109-957f-4d4f-9897-6eb86c402601" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" Oct 30 12:30:27.680853 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:27.675144 4908 patch_prober.go:29] interesting pod/e2e-praxis-b-ec236e-openshift-default-f6575b7df-kwhwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body= Oct 30 12:30:27.680853 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:27.675205 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-ec236e-openshift-default-f6575b7df-kwhwr" podUID="5ea7d109-957f-4d4f-9897-6eb86c402601" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" Oct 30 12:30:28.675854 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:28.672107 4908 patch_prober.go:29] interesting pod/e2e-praxis-b-ec236e-openshift-default-f6575b7df-kwhwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body= Oct 30 12:30:28.675854 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:28.672164 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-ec236e-openshift-default-f6575b7df-kwhwr" podUID="5ea7d109-957f-4d4f-9897-6eb86c402601" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" Oct 30 12:30:29.588853 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:29.573052 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ab9381-wqmpj"] Oct 30 12:30:29.589349 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:29.589246 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ab9381-wqmpj"] Oct 30 12:30:29.611241 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:29.611210 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a64aea77-808a-4b76-9e00-49faaba0314b" path="/var/lib/kubelet/pods/a64aea77-808a-4b76-9e00-49faaba0314b/volumes" Oct 30 12:30:29.676356 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:29.675919 4908 patch_prober.go:29] interesting pod/e2e-praxis-b-ec236e-openshift-default-f6575b7df-kwhwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body= Oct 30 12:30:29.676356 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:29.675989 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-ec236e-openshift-default-f6575b7df-kwhwr" podUID="5ea7d109-957f-4d4f-9897-6eb86c402601" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" Oct 30 12:30:30.672784 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:30.672755 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-ec236e-openshift-default-f6575b7df-kwhwr" Oct 30 12:30:30.673253 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:30.673232 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-ec236e-openshift-default-f6575b7df-kwhwr" Oct 30 12:30:30.673954 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:30.673930 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-ec236e-openshift-default-f6575b7df-kwhwr" Oct 30 12:30:33.316205 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:33.316176 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7f78d4-fb954f9d7-2b2qr" Oct 30 12:30:33.652476 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:33.652384 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9wwfc"] Oct 30 12:30:34.332854 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:34.332454 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-7f78d4-c859fdc56-jzwrn" Oct 30 12:30:37.104923 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:37.104889 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7eee3b-dr2kd"] Oct 30 12:30:37.106346 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:37.106317 4908 cpu_manager.go:414] "containerMap: removing container" podUID="76363248-837a-4227-974a-9613e1e598e0" containerName="cleanup" Oct 30 12:30:37.106508 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:37.106494 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76363248-837a-4227-974a-9613e1e598e0" containerName="cleanup" Oct 30 12:30:37.106602 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:37.106591 4908 cpu_manager.go:414] "containerMap: removing container" podUID="101bab62-abe6-4870-83e8-d30758312a0c" containerName="cleanup" Oct 30 12:30:37.106678 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:37.106668 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="101bab62-abe6-4870-83e8-d30758312a0c" containerName="cleanup" Oct 30 12:30:37.106757 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:37.106748 4908 cpu_manager.go:414] "containerMap: removing container" podUID="6c7ac05a-ec43-4c05-9872-4694da6ec833" containerName="cleanup" Oct 30 12:30:37.106849 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:37.106820 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c7ac05a-ec43-4c05-9872-4694da6ec833" containerName="cleanup" Oct 30 12:30:37.107013 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:37.107000 4908 cpu_manager.go:414] "containerMap: removing container" podUID="0a582fb0-974b-41fe-bc01-0712b6df3b31" containerName="cleanup" Oct 30 12:30:37.107099 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:37.107087 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a582fb0-974b-41fe-bc01-0712b6df3b31" containerName="cleanup" Oct 30 12:30:37.107174 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:37.107166 4908 cpu_manager.go:414] "containerMap: removing container" podUID="6c7ac05a-ec43-4c05-9872-4694da6ec833" containerName="cleanup" Oct 30 12:30:37.107244 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:37.107234 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c7ac05a-ec43-4c05-9872-4694da6ec833" containerName="cleanup" Oct 30 12:30:37.107325 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:37.107314 4908 cpu_manager.go:414] "containerMap: removing container" podUID="273ee4cd-dd11-4204-ba40-9073fde47d1d" containerName="cleanup" Oct 30 12:30:37.107415 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:37.107403 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="273ee4cd-dd11-4204-ba40-9073fde47d1d" containerName="cleanup" Oct 30 12:30:37.107495 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:37.107486 4908 cpu_manager.go:414] "containerMap: removing container" podUID="101bab62-abe6-4870-83e8-d30758312a0c" containerName="cleanup" Oct 30 12:30:37.107571 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:37.107560 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="101bab62-abe6-4870-83e8-d30758312a0c" containerName="cleanup" Oct 30 12:30:37.107653 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:37.107644 4908 cpu_manager.go:414] "containerMap: removing container" podUID="a64aea77-808a-4b76-9e00-49faaba0314b" containerName="revoke-keys" Oct 30 12:30:37.107725 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:37.107716 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a64aea77-808a-4b76-9e00-49faaba0314b" containerName="revoke-keys" Oct 30 12:30:37.107802 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:37.107792 4908 cpu_manager.go:414] "containerMap: removing container" podUID="273ee4cd-dd11-4204-ba40-9073fde47d1d" containerName="cleanup" Oct 30 12:30:37.107888 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:37.107879 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="273ee4cd-dd11-4204-ba40-9073fde47d1d" containerName="cleanup" Oct 30 12:30:37.108026 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:37.108014 4908 cpu_manager.go:414] "containerMap: removing container" podUID="0a582fb0-974b-41fe-bc01-0712b6df3b31" containerName="cleanup" Oct 30 12:30:37.108111 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:37.108101 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a582fb0-974b-41fe-bc01-0712b6df3b31" containerName="cleanup" Oct 30 12:30:37.108194 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:37.108186 4908 cpu_manager.go:414] "containerMap: removing container" podUID="76363248-837a-4227-974a-9613e1e598e0" containerName="cleanup" Oct 30 12:30:37.108263 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:37.108254 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76363248-837a-4227-974a-9613e1e598e0" containerName="cleanup" Oct 30 12:30:37.108336 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:37.108326 4908 cpu_manager.go:414] "containerMap: removing container" podUID="016efbc9-f46c-4dd9-8910-2dbec739d99a" containerName="cleanup" Oct 30 12:30:37.108411 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:37.108399 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="016efbc9-f46c-4dd9-8910-2dbec739d99a" containerName="cleanup" Oct 30 12:30:37.108487 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:37.108479 4908 cpu_manager.go:414] "containerMap: removing container" podUID="6c7ac05a-ec43-4c05-9872-4694da6ec833" containerName="cleanup" Oct 30 12:30:37.108555 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:37.108547 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c7ac05a-ec43-4c05-9872-4694da6ec833" containerName="cleanup" Oct 30 12:30:37.108774 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:37.108762 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="76363248-837a-4227-974a-9613e1e598e0" containerName="cleanup" Oct 30 12:30:37.108888 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:37.108877 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="76363248-837a-4227-974a-9613e1e598e0" containerName="cleanup" Oct 30 12:30:37.108960 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:37.108951 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="101bab62-abe6-4870-83e8-d30758312a0c" containerName="cleanup" Oct 30 12:30:37.109027 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:37.109019 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c7ac05a-ec43-4c05-9872-4694da6ec833" containerName="cleanup" Oct 30 12:30:37.109095 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:37.109087 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a582fb0-974b-41fe-bc01-0712b6df3b31" containerName="cleanup" Oct 30 12:30:37.109164 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:37.109156 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c7ac05a-ec43-4c05-9872-4694da6ec833" containerName="cleanup" Oct 30 12:30:37.109234 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:37.109226 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="273ee4cd-dd11-4204-ba40-9073fde47d1d" containerName="cleanup" Oct 30 12:30:37.109302 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:37.109293 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="101bab62-abe6-4870-83e8-d30758312a0c" containerName="cleanup" Oct 30 12:30:37.109372 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:37.109363 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="a64aea77-808a-4b76-9e00-49faaba0314b" containerName="revoke-keys" Oct 30 12:30:37.109444 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:37.109436 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="273ee4cd-dd11-4204-ba40-9073fde47d1d" containerName="cleanup" Oct 30 12:30:37.109517 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:37.109509 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="016efbc9-f46c-4dd9-8910-2dbec739d99a" containerName="cleanup" Oct 30 12:30:37.109595 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:37.109586 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a582fb0-974b-41fe-bc01-0712b6df3b31" containerName="cleanup" Oct 30 12:30:37.116460 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:37.116442 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7eee3b-dr2kd"] Oct 30 12:30:37.116669 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:37.116656 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7eee3b-dr2kd" Oct 30 12:30:37.130767 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:37.130751 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 12:30:37.130975 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:37.130950 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 12:30:37.130975 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:37.130801 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 12:30:37.131089 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:37.130955 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s75hq\"" Oct 30 12:30:37.237476 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:37.237440 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z96gz\" (UniqueName: \"kubernetes.io/projected/24386a19-71a4-46f8-b129-145cae4b893b-kube-api-access-z96gz\") pod \"maas-api-revoke-keys-e2e-ipp-a-7eee3b-dr2kd\" (UID: \"24386a19-71a4-46f8-b129-145cae4b893b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7eee3b-dr2kd" Oct 30 12:30:37.237632 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:37.237481 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24386a19-71a4-46f8-b129-145cae4b893b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-7eee3b-dr2kd\" (UID: \"24386a19-71a4-46f8-b129-145cae4b893b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7eee3b-dr2kd" Oct 30 12:30:37.237632 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:37.237592 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24386a19-71a4-46f8-b129-145cae4b893b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-7eee3b-dr2kd\" (UID: \"24386a19-71a4-46f8-b129-145cae4b893b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7eee3b-dr2kd" Oct 30 12:30:37.338678 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:37.338634 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z96gz\" (UniqueName: \"kubernetes.io/projected/24386a19-71a4-46f8-b129-145cae4b893b-kube-api-access-z96gz\") pod \"maas-api-revoke-keys-e2e-ipp-a-7eee3b-dr2kd\" (UID: \"24386a19-71a4-46f8-b129-145cae4b893b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7eee3b-dr2kd" Oct 30 12:30:37.338678 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:37.338674 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24386a19-71a4-46f8-b129-145cae4b893b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-7eee3b-dr2kd\" (UID: \"24386a19-71a4-46f8-b129-145cae4b893b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7eee3b-dr2kd" Oct 30 12:30:37.338970 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:37.338702 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24386a19-71a4-46f8-b129-145cae4b893b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-7eee3b-dr2kd\" (UID: \"24386a19-71a4-46f8-b129-145cae4b893b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7eee3b-dr2kd" Oct 30 12:30:37.339473 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:37.339450 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24386a19-71a4-46f8-b129-145cae4b893b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-7eee3b-dr2kd\" (UID: \"24386a19-71a4-46f8-b129-145cae4b893b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7eee3b-dr2kd" Oct 30 12:30:37.341083 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:37.341060 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24386a19-71a4-46f8-b129-145cae4b893b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-7eee3b-dr2kd\" (UID: \"24386a19-71a4-46f8-b129-145cae4b893b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7eee3b-dr2kd" Oct 30 12:30:37.394542 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:37.394471 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z96gz\" (UniqueName: \"kubernetes.io/projected/24386a19-71a4-46f8-b129-145cae4b893b-kube-api-access-z96gz\") pod \"maas-api-revoke-keys-e2e-ipp-a-7eee3b-dr2kd\" (UID: \"24386a19-71a4-46f8-b129-145cae4b893b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7eee3b-dr2kd" Oct 30 12:30:37.426512 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:37.426468 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8e3112-4nrwf"] Oct 30 12:30:37.429765 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:37.429741 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8e3112-4nrwf"] Oct 30 12:30:37.435666 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:37.435644 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7eee3b-dr2kd" Oct 30 12:30:37.621362 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:37.619482 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a81f4750-0da9-4032-a80e-1aab7a03ac82" path="/var/lib/kubelet/pods/a81f4750-0da9-4032-a80e-1aab7a03ac82/volumes" Oct 30 12:30:37.811723 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:37.811693 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7eee3b-dr2kd"] Oct 30 12:30:37.812295 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:30:37.812269 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24386a19_71a4_46f8_b129_145cae4b893b.slice/crio-bd97bd1975f7193cad4e186c62c94546fda6c58d128f9de85d5df56f898fc0bf WatchSource:0}: Error finding container bd97bd1975f7193cad4e186c62c94546fda6c58d128f9de85d5df56f898fc0bf: Status 404 returned error can't find the container with id bd97bd1975f7193cad4e186c62c94546fda6c58d128f9de85d5df56f898fc0bf Oct 30 12:30:38.538084 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:38.538047 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7eee3b-dr2kd" event={"ID":"24386a19-71a4-46f8-b129-145cae4b893b","Type":"ContainerStarted","Data":"0ce70383eebd9ee67e2385e6e92bc8729117ac8644d52c1934fc407301016acf"} Oct 30 12:30:38.538470 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:38.538108 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7eee3b-dr2kd" event={"ID":"24386a19-71a4-46f8-b129-145cae4b893b","Type":"ContainerStarted","Data":"bd97bd1975f7193cad4e186c62c94546fda6c58d128f9de85d5df56f898fc0bf"} Oct 30 12:30:38.795457 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:38.795333 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7eee3b-dr2kd" podStartSLOduration=1.795312679 podStartE2EDuration="1.795312679s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:30:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:30:38.583055689 +0000 UTC m=+2195.097484434" watchObservedRunningTime="2026-10-30 12:30:38.795312679 +0000 UTC m=+2195.309741426" Oct 30 12:30:38.796345 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:38.796313 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ec236e-v8vlh"] Oct 30 12:30:38.797362 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:38.797339 4908 cpu_manager.go:414] "containerMap: removing container" podUID="76363248-837a-4227-974a-9613e1e598e0" containerName="cleanup" Oct 30 12:30:38.797460 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:38.797363 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76363248-837a-4227-974a-9613e1e598e0" containerName="cleanup" Oct 30 12:30:38.797460 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:38.797417 4908 cpu_manager.go:414] "containerMap: removing container" podUID="101bab62-abe6-4870-83e8-d30758312a0c" containerName="cleanup" Oct 30 12:30:38.797460 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:38.797427 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="101bab62-abe6-4870-83e8-d30758312a0c" containerName="cleanup" Oct 30 12:30:38.797460 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:38.797443 4908 cpu_manager.go:414] "containerMap: removing container" podUID="273ee4cd-dd11-4204-ba40-9073fde47d1d" containerName="cleanup" Oct 30 12:30:38.797460 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:38.797453 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="273ee4cd-dd11-4204-ba40-9073fde47d1d" containerName="cleanup" Oct 30 12:30:38.797713 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:38.797505 4908 cpu_manager.go:414] "containerMap: removing container" podUID="0a582fb0-974b-41fe-bc01-0712b6df3b31" containerName="cleanup" Oct 30 12:30:38.797713 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:38.797515 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a582fb0-974b-41fe-bc01-0712b6df3b31" containerName="cleanup" Oct 30 12:30:38.797809 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:38.797776 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="76363248-837a-4227-974a-9613e1e598e0" containerName="cleanup" Oct 30 12:30:38.797895 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:38.797809 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a582fb0-974b-41fe-bc01-0712b6df3b31" containerName="cleanup" Oct 30 12:30:38.797895 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:38.797818 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c7ac05a-ec43-4c05-9872-4694da6ec833" containerName="cleanup" Oct 30 12:30:38.797895 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:38.797844 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="101bab62-abe6-4870-83e8-d30758312a0c" containerName="cleanup" Oct 30 12:30:38.797895 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:38.797854 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="273ee4cd-dd11-4204-ba40-9073fde47d1d" containerName="cleanup" Oct 30 12:30:38.801351 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:38.801331 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ec236e-v8vlh"] Oct 30 12:30:38.801496 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:38.801473 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ec236e-v8vlh" Oct 30 12:30:38.903345 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:38.903292 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/111ec6e1-3267-4e49-a870-052bf799120d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-ec236e-v8vlh\" (UID: \"111ec6e1-3267-4e49-a870-052bf799120d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ec236e-v8vlh" Oct 30 12:30:38.903556 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:38.903536 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wkrb\" (UniqueName: \"kubernetes.io/projected/111ec6e1-3267-4e49-a870-052bf799120d-kube-api-access-2wkrb\") pod \"maas-api-revoke-keys-e2e-praxis-b-ec236e-v8vlh\" (UID: \"111ec6e1-3267-4e49-a870-052bf799120d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ec236e-v8vlh" Oct 30 12:30:38.903623 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:38.903586 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/111ec6e1-3267-4e49-a870-052bf799120d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-ec236e-v8vlh\" (UID: \"111ec6e1-3267-4e49-a870-052bf799120d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ec236e-v8vlh" Oct 30 12:30:39.005096 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:39.005065 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wkrb\" (UniqueName: \"kubernetes.io/projected/111ec6e1-3267-4e49-a870-052bf799120d-kube-api-access-2wkrb\") pod \"maas-api-revoke-keys-e2e-praxis-b-ec236e-v8vlh\" (UID: \"111ec6e1-3267-4e49-a870-052bf799120d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ec236e-v8vlh" Oct 30 12:30:39.005245 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:39.005134 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/111ec6e1-3267-4e49-a870-052bf799120d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-ec236e-v8vlh\" (UID: \"111ec6e1-3267-4e49-a870-052bf799120d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ec236e-v8vlh" Oct 30 12:30:39.005245 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:39.005198 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/111ec6e1-3267-4e49-a870-052bf799120d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-ec236e-v8vlh\" (UID: \"111ec6e1-3267-4e49-a870-052bf799120d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ec236e-v8vlh" Oct 30 12:30:39.005836 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:39.005799 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/111ec6e1-3267-4e49-a870-052bf799120d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-ec236e-v8vlh\" (UID: \"111ec6e1-3267-4e49-a870-052bf799120d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ec236e-v8vlh" Oct 30 12:30:39.008714 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:39.008684 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/111ec6e1-3267-4e49-a870-052bf799120d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-ec236e-v8vlh\" (UID: \"111ec6e1-3267-4e49-a870-052bf799120d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ec236e-v8vlh" Oct 30 12:30:39.027403 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:39.027356 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wkrb\" (UniqueName: \"kubernetes.io/projected/111ec6e1-3267-4e49-a870-052bf799120d-kube-api-access-2wkrb\") pod \"maas-api-revoke-keys-e2e-praxis-b-ec236e-v8vlh\" (UID: \"111ec6e1-3267-4e49-a870-052bf799120d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ec236e-v8vlh" Oct 30 12:30:39.125849 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:39.125759 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ec236e-v8vlh" Oct 30 12:30:39.500289 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:39.499937 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ec236e-v8vlh"] Oct 30 12:30:39.500289 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:30:39.500125 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod111ec6e1_3267_4e49_a870_052bf799120d.slice/crio-bce6f51fd8cc53026d491700a2691586e1986693b5227ca013236478f54920a8 WatchSource:0}: Error finding container bce6f51fd8cc53026d491700a2691586e1986693b5227ca013236478f54920a8: Status 404 returned error can't find the container with id bce6f51fd8cc53026d491700a2691586e1986693b5227ca013236478f54920a8 Oct 30 12:30:39.546062 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:39.546033 4908 generic.go:356] "Generic (PLEG): container finished" podID="24386a19-71a4-46f8-b129-145cae4b893b" containerID="0ce70383eebd9ee67e2385e6e92bc8729117ac8644d52c1934fc407301016acf" exitCode=0 Oct 30 12:30:39.546377 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:39.546103 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7eee3b-dr2kd" event={"ID":"24386a19-71a4-46f8-b129-145cae4b893b","Type":"ContainerDied","Data":"0ce70383eebd9ee67e2385e6e92bc8729117ac8644d52c1934fc407301016acf"} Oct 30 12:30:39.547558 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:39.547535 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ec236e-v8vlh" event={"ID":"111ec6e1-3267-4e49-a870-052bf799120d","Type":"ContainerStarted","Data":"bce6f51fd8cc53026d491700a2691586e1986693b5227ca013236478f54920a8"} Oct 30 12:30:40.556738 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:40.556704 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ec236e-v8vlh" event={"ID":"111ec6e1-3267-4e49-a870-052bf799120d","Type":"ContainerStarted","Data":"44db5df2fb480b1dada1a43f5a4c9c79487c30adb602551a4d4d0e81861e70a8"} Oct 30 12:30:40.598327 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:40.598243 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ec236e-v8vlh" podStartSLOduration=2.598212442 podStartE2EDuration="2.598212442s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:30:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:30:40.595697936 +0000 UTC m=+2197.110126681" watchObservedRunningTime="2026-10-30 12:30:40.598212442 +0000 UTC m=+2197.112641187" Oct 30 12:30:40.741877 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:40.741820 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7eee3b-dr2kd" Oct 30 12:30:40.852499 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:40.852467 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/24386a19-71a4-46f8-b129-145cae4b893b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24386a19-71a4-46f8-b129-145cae4b893b-maas-api-xks-ca\") pod \"24386a19-71a4-46f8-b129-145cae4b893b\" (UID: \"24386a19-71a4-46f8-b129-145cae4b893b\") " Oct 30 12:30:40.852682 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:40.852517 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24386a19-71a4-46f8-b129-145cae4b893b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24386a19-71a4-46f8-b129-145cae4b893b-maas-api-openshift-ca\") pod \"24386a19-71a4-46f8-b129-145cae4b893b\" (UID: \"24386a19-71a4-46f8-b129-145cae4b893b\") " Oct 30 12:30:40.852682 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:40.852552 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24386a19-71a4-46f8-b129-145cae4b893b-kube-api-access-z96gz\" (UniqueName: \"kubernetes.io/projected/24386a19-71a4-46f8-b129-145cae4b893b-kube-api-access-z96gz\") pod \"24386a19-71a4-46f8-b129-145cae4b893b\" (UID: \"24386a19-71a4-46f8-b129-145cae4b893b\") " Oct 30 12:30:40.852912 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:40.852888 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24386a19-71a4-46f8-b129-145cae4b893b-maas-api-openshift-ca" pod "24386a19-71a4-46f8-b129-145cae4b893b" (UID: "24386a19-71a4-46f8-b129-145cae4b893b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:30:40.853141 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:40.853126 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24386a19-71a4-46f8-b129-145cae4b893b-maas-api-openshift-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:30:40.854666 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:40.854640 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24386a19-71a4-46f8-b129-145cae4b893b-maas-api-xks-ca" pod "24386a19-71a4-46f8-b129-145cae4b893b" (UID: "24386a19-71a4-46f8-b129-145cae4b893b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:30:40.854666 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:40.854650 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24386a19-71a4-46f8-b129-145cae4b893b-kube-api-access-z96gz" pod "24386a19-71a4-46f8-b129-145cae4b893b" (UID: "24386a19-71a4-46f8-b129-145cae4b893b"). InnerVolumeSpecName "kube-api-access-z96gz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:30:40.954984 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:40.954954 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24386a19-71a4-46f8-b129-145cae4b893b-maas-api-xks-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:30:40.955117 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:40.954986 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z96gz\" (UniqueName: \"kubernetes.io/projected/24386a19-71a4-46f8-b129-145cae4b893b-kube-api-access-z96gz\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:30:41.571569 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:41.567541 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7eee3b-dr2kd" event={"ID":"24386a19-71a4-46f8-b129-145cae4b893b","Type":"ContainerDied","Data":"bd97bd1975f7193cad4e186c62c94546fda6c58d128f9de85d5df56f898fc0bf"} Oct 30 12:30:41.571569 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:41.567607 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bd97bd1975f7193cad4e186c62c94546fda6c58d128f9de85d5df56f898fc0bf" Oct 30 12:30:41.571569 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:41.567683 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7eee3b-dr2kd" Oct 30 12:30:41.572914 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:41.572883 4908 generic.go:356] "Generic (PLEG): container finished" podID="111ec6e1-3267-4e49-a870-052bf799120d" containerID="44db5df2fb480b1dada1a43f5a4c9c79487c30adb602551a4d4d0e81861e70a8" exitCode=0 Oct 30 12:30:41.573174 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:41.573152 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ec236e-v8vlh" event={"ID":"111ec6e1-3267-4e49-a870-052bf799120d","Type":"ContainerDied","Data":"44db5df2fb480b1dada1a43f5a4c9c79487c30adb602551a4d4d0e81861e70a8"} Oct 30 12:30:41.731866 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:41.731819 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-41bdda-7xgd2"] Oct 30 12:30:41.732388 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:41.732373 4908 cpu_manager.go:414] "containerMap: removing container" podUID="24386a19-71a4-46f8-b129-145cae4b893b" containerName="revoke-keys" Oct 30 12:30:41.732433 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:41.732389 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24386a19-71a4-46f8-b129-145cae4b893b" containerName="revoke-keys" Oct 30 12:30:41.732501 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:41.732492 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="24386a19-71a4-46f8-b129-145cae4b893b" containerName="revoke-keys" Oct 30 12:30:41.735923 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:41.735906 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-41bdda-7xgd2" Oct 30 12:30:41.754525 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:41.754497 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-41bdda-7xgd2"] Oct 30 12:30:41.764936 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:41.764904 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21bfd950-c0ea-46dd-b4f0-3dbb8167f696-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-41bdda-7xgd2\" (UID: \"21bfd950-c0ea-46dd-b4f0-3dbb8167f696\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-41bdda-7xgd2" Oct 30 12:30:41.765196 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:41.765178 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21bfd950-c0ea-46dd-b4f0-3dbb8167f696-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-41bdda-7xgd2\" (UID: \"21bfd950-c0ea-46dd-b4f0-3dbb8167f696\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-41bdda-7xgd2" Oct 30 12:30:41.765292 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:41.765233 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5cl5\" (UniqueName: \"kubernetes.io/projected/21bfd950-c0ea-46dd-b4f0-3dbb8167f696-kube-api-access-v5cl5\") pod \"maas-api-revoke-keys-e2e-praxis-a-41bdda-7xgd2\" (UID: \"21bfd950-c0ea-46dd-b4f0-3dbb8167f696\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-41bdda-7xgd2" Oct 30 12:30:41.866868 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:41.866770 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21bfd950-c0ea-46dd-b4f0-3dbb8167f696-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-41bdda-7xgd2\" (UID: \"21bfd950-c0ea-46dd-b4f0-3dbb8167f696\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-41bdda-7xgd2" Oct 30 12:30:41.866868 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:41.866840 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5cl5\" (UniqueName: \"kubernetes.io/projected/21bfd950-c0ea-46dd-b4f0-3dbb8167f696-kube-api-access-v5cl5\") pod \"maas-api-revoke-keys-e2e-praxis-a-41bdda-7xgd2\" (UID: \"21bfd950-c0ea-46dd-b4f0-3dbb8167f696\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-41bdda-7xgd2" Oct 30 12:30:41.867227 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:41.866894 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21bfd950-c0ea-46dd-b4f0-3dbb8167f696-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-41bdda-7xgd2\" (UID: \"21bfd950-c0ea-46dd-b4f0-3dbb8167f696\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-41bdda-7xgd2" Oct 30 12:30:41.867689 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:41.867661 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21bfd950-c0ea-46dd-b4f0-3dbb8167f696-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-41bdda-7xgd2\" (UID: \"21bfd950-c0ea-46dd-b4f0-3dbb8167f696\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-41bdda-7xgd2" Oct 30 12:30:41.869171 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:41.869150 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21bfd950-c0ea-46dd-b4f0-3dbb8167f696-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-41bdda-7xgd2\" (UID: \"21bfd950-c0ea-46dd-b4f0-3dbb8167f696\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-41bdda-7xgd2" Oct 30 12:30:41.889008 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:41.888979 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5cl5\" (UniqueName: \"kubernetes.io/projected/21bfd950-c0ea-46dd-b4f0-3dbb8167f696-kube-api-access-v5cl5\") pod \"maas-api-revoke-keys-e2e-praxis-a-41bdda-7xgd2\" (UID: \"21bfd950-c0ea-46dd-b4f0-3dbb8167f696\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-41bdda-7xgd2" Oct 30 12:30:42.049298 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:42.049260 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-41bdda-7xgd2" Oct 30 12:30:42.367554 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:42.367518 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-41bdda-7xgd2"] Oct 30 12:30:42.371353 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:30:42.371318 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21bfd950_c0ea_46dd_b4f0_3dbb8167f696.slice/crio-28cdff9e7e06102c955defd62f86d11ab528c459258cad37bdc7006e29818e53 WatchSource:0}: Error finding container 28cdff9e7e06102c955defd62f86d11ab528c459258cad37bdc7006e29818e53: Status 404 returned error can't find the container with id 28cdff9e7e06102c955defd62f86d11ab528c459258cad37bdc7006e29818e53 Oct 30 12:30:42.585105 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:42.585072 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-41bdda-7xgd2" event={"ID":"21bfd950-c0ea-46dd-b4f0-3dbb8167f696","Type":"ContainerStarted","Data":"28cdff9e7e06102c955defd62f86d11ab528c459258cad37bdc7006e29818e53"} Oct 30 12:30:42.726822 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:42.726798 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ec236e-v8vlh" Oct 30 12:30:42.764885 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:42.764472 4908 patch_prober.go:29] interesting pod/e2e-praxis-b-1e9675-openshift-default-7d7c758549-6hhbg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 12:30:42.764885 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:42.764538 4908 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-1e9675-openshift-default-7d7c758549-6hhbg" podUID="62b94b35-a1bb-4eb2-9bc0-da2760557b3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 12:30:42.778958 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:42.778651 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/111ec6e1-3267-4e49-a870-052bf799120d-kube-api-access-2wkrb\" (UniqueName: \"kubernetes.io/projected/111ec6e1-3267-4e49-a870-052bf799120d-kube-api-access-2wkrb\") pod \"111ec6e1-3267-4e49-a870-052bf799120d\" (UID: \"111ec6e1-3267-4e49-a870-052bf799120d\") " Oct 30 12:30:42.778958 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:42.778712 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/111ec6e1-3267-4e49-a870-052bf799120d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/111ec6e1-3267-4e49-a870-052bf799120d-maas-api-xks-ca\") pod \"111ec6e1-3267-4e49-a870-052bf799120d\" (UID: \"111ec6e1-3267-4e49-a870-052bf799120d\") " Oct 30 12:30:42.778958 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:42.778863 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/111ec6e1-3267-4e49-a870-052bf799120d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/111ec6e1-3267-4e49-a870-052bf799120d-maas-api-openshift-ca\") pod \"111ec6e1-3267-4e49-a870-052bf799120d\" (UID: \"111ec6e1-3267-4e49-a870-052bf799120d\") " Oct 30 12:30:42.779343 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:42.779183 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/111ec6e1-3267-4e49-a870-052bf799120d-maas-api-openshift-ca" pod "111ec6e1-3267-4e49-a870-052bf799120d" (UID: "111ec6e1-3267-4e49-a870-052bf799120d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:30:42.779902 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:42.779876 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/111ec6e1-3267-4e49-a870-052bf799120d-maas-api-openshift-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:30:42.781860 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:42.781813 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/111ec6e1-3267-4e49-a870-052bf799120d-kube-api-access-2wkrb" pod "111ec6e1-3267-4e49-a870-052bf799120d" (UID: "111ec6e1-3267-4e49-a870-052bf799120d"). InnerVolumeSpecName "kube-api-access-2wkrb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:30:42.782026 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:42.782003 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/111ec6e1-3267-4e49-a870-052bf799120d-maas-api-xks-ca" pod "111ec6e1-3267-4e49-a870-052bf799120d" (UID: "111ec6e1-3267-4e49-a870-052bf799120d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:30:42.881289 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:42.881256 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2wkrb\" (UniqueName: \"kubernetes.io/projected/111ec6e1-3267-4e49-a870-052bf799120d-kube-api-access-2wkrb\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:30:42.881413 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:42.881295 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/111ec6e1-3267-4e49-a870-052bf799120d-maas-api-xks-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:30:43.595694 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:43.595605 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ec236e-v8vlh" Oct 30 12:30:43.595694 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:43.595634 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ec236e-v8vlh" event={"ID":"111ec6e1-3267-4e49-a870-052bf799120d","Type":"ContainerDied","Data":"bce6f51fd8cc53026d491700a2691586e1986693b5227ca013236478f54920a8"} Oct 30 12:30:43.596265 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:43.595741 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bce6f51fd8cc53026d491700a2691586e1986693b5227ca013236478f54920a8" Oct 30 12:30:43.597905 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:43.597874 4908 generic.go:356] "Generic (PLEG): container finished" podID="21bfd950-c0ea-46dd-b4f0-3dbb8167f696" containerID="de5ed5804305f8cc691e17d69815d55fced91baf12befade01b898e662ecae53" exitCode=0 Oct 30 12:30:43.608455 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:43.608426 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-41bdda-7xgd2" event={"ID":"21bfd950-c0ea-46dd-b4f0-3dbb8167f696","Type":"ContainerDied","Data":"de5ed5804305f8cc691e17d69815d55fced91baf12befade01b898e662ecae53"} Oct 30 12:30:44.609403 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:44.609370 4908 generic.go:356] "Generic (PLEG): container finished" podID="62b94b35-a1bb-4eb2-9bc0-da2760557b3d" containerID="08fec614e1c0c235a1facb20245ba78167959eb61eb0a480c1b8f61483dcf3f5" exitCode=0 Oct 30 12:30:44.609974 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:44.609443 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-1e9675-openshift-default-7d7c758549-6hhbg" event={"ID":"62b94b35-a1bb-4eb2-9bc0-da2760557b3d","Type":"ContainerDied","Data":"08fec614e1c0c235a1facb20245ba78167959eb61eb0a480c1b8f61483dcf3f5"} Oct 30 12:30:44.609974 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:44.609902 4908 scope.go:122] "RemoveContainer" containerID="08fec614e1c0c235a1facb20245ba78167959eb61eb0a480c1b8f61483dcf3f5" Oct 30 12:30:44.612307 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:44.612276 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:30:44.612441 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:44.612360 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:30:44.612441 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:44.612404 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:30:44.816022 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:44.815995 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-41bdda-7xgd2" Oct 30 12:30:44.915880 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:44.915789 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21bfd950-c0ea-46dd-b4f0-3dbb8167f696-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21bfd950-c0ea-46dd-b4f0-3dbb8167f696-maas-api-xks-ca\") pod \"21bfd950-c0ea-46dd-b4f0-3dbb8167f696\" (UID: \"21bfd950-c0ea-46dd-b4f0-3dbb8167f696\") " Oct 30 12:30:44.916023 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:44.915916 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21bfd950-c0ea-46dd-b4f0-3dbb8167f696-kube-api-access-v5cl5\" (UniqueName: \"kubernetes.io/projected/21bfd950-c0ea-46dd-b4f0-3dbb8167f696-kube-api-access-v5cl5\") pod \"21bfd950-c0ea-46dd-b4f0-3dbb8167f696\" (UID: \"21bfd950-c0ea-46dd-b4f0-3dbb8167f696\") " Oct 30 12:30:44.916405 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:44.916340 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21bfd950-c0ea-46dd-b4f0-3dbb8167f696-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21bfd950-c0ea-46dd-b4f0-3dbb8167f696-maas-api-openshift-ca\") pod \"21bfd950-c0ea-46dd-b4f0-3dbb8167f696\" (UID: \"21bfd950-c0ea-46dd-b4f0-3dbb8167f696\") " Oct 30 12:30:44.916941 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:44.916913 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21bfd950-c0ea-46dd-b4f0-3dbb8167f696-maas-api-openshift-ca" pod "21bfd950-c0ea-46dd-b4f0-3dbb8167f696" (UID: "21bfd950-c0ea-46dd-b4f0-3dbb8167f696"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:30:44.917107 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:44.917090 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21bfd950-c0ea-46dd-b4f0-3dbb8167f696-maas-api-openshift-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:30:44.918390 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:44.918366 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21bfd950-c0ea-46dd-b4f0-3dbb8167f696-maas-api-xks-ca" pod "21bfd950-c0ea-46dd-b4f0-3dbb8167f696" (UID: "21bfd950-c0ea-46dd-b4f0-3dbb8167f696"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:30:44.918510 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:44.918393 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21bfd950-c0ea-46dd-b4f0-3dbb8167f696-kube-api-access-v5cl5" pod "21bfd950-c0ea-46dd-b4f0-3dbb8167f696" (UID: "21bfd950-c0ea-46dd-b4f0-3dbb8167f696"). InnerVolumeSpecName "kube-api-access-v5cl5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:30:45.018958 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:45.018925 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21bfd950-c0ea-46dd-b4f0-3dbb8167f696-maas-api-xks-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:30:45.019113 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:45.018963 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v5cl5\" (UniqueName: \"kubernetes.io/projected/21bfd950-c0ea-46dd-b4f0-3dbb8167f696-kube-api-access-v5cl5\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:30:45.623356 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:45.623320 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-1e9675-openshift-default-7d7c758549-6hhbg" event={"ID":"62b94b35-a1bb-4eb2-9bc0-da2760557b3d","Type":"ContainerStarted","Data":"a45855efd1f7e4fd82b582d43c7deefe3da174c5f5a3aca8912ebbbcfc01f517"} Oct 30 12:30:45.625273 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:45.625246 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-41bdda-7xgd2" event={"ID":"21bfd950-c0ea-46dd-b4f0-3dbb8167f696","Type":"ContainerDied","Data":"28cdff9e7e06102c955defd62f86d11ab528c459258cad37bdc7006e29818e53"} Oct 30 12:30:45.625397 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:45.625285 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="28cdff9e7e06102c955defd62f86d11ab528c459258cad37bdc7006e29818e53" Oct 30 12:30:45.625397 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:45.625258 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-41bdda-7xgd2" Oct 30 12:30:46.489742 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:46.489707 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-1e9675-openshift-default-7d7c758549-6hhbg" Oct 30 12:30:46.490500 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:46.490476 4908 patch_prober.go:29] interesting pod/e2e-praxis-b-1e9675-openshift-default-7d7c758549-6hhbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body= Oct 30 12:30:46.490611 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:46.490523 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-1e9675-openshift-default-7d7c758549-6hhbg" podUID="62b94b35-a1bb-4eb2-9bc0-da2760557b3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" Oct 30 12:30:47.490139 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:47.490107 4908 patch_prober.go:29] interesting pod/e2e-praxis-b-1e9675-openshift-default-7d7c758549-6hhbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body= Oct 30 12:30:47.490646 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:47.490327 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-1e9675-openshift-default-7d7c758549-6hhbg" podUID="62b94b35-a1bb-4eb2-9bc0-da2760557b3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" Oct 30 12:30:48.548563 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:48.548525 4908 patch_prober.go:29] interesting pod/e2e-praxis-b-1e9675-openshift-default-7d7c758549-6hhbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body= Oct 30 12:30:48.565940 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:48.565896 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-1e9675-openshift-default-7d7c758549-6hhbg" podUID="62b94b35-a1bb-4eb2-9bc0-da2760557b3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" Oct 30 12:30:48.754266 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:48.754233 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7eee3b-dr2kd"] Oct 30 12:30:48.769366 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:48.769324 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7eee3b-dr2kd"] Oct 30 12:30:49.490307 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:49.490257 4908 patch_prober.go:29] interesting pod/e2e-praxis-b-1e9675-openshift-default-7d7c758549-6hhbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body= Oct 30 12:30:49.490650 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:49.490617 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-1e9675-openshift-default-7d7c758549-6hhbg" podUID="62b94b35-a1bb-4eb2-9bc0-da2760557b3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" Oct 30 12:30:49.608564 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:49.608527 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24386a19-71a4-46f8-b129-145cae4b893b" path="/var/lib/kubelet/pods/24386a19-71a4-46f8-b129-145cae4b893b/volumes" Oct 30 12:30:49.712320 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:30:49.712285 4908 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vn6vv" containerName="extractor" Oct 30 12:30:50.490395 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:50.490357 4908 patch_prober.go:29] interesting pod/e2e-praxis-b-1e9675-openshift-default-7d7c758549-6hhbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body= Oct 30 12:30:50.490604 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:50.490430 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-1e9675-openshift-default-7d7c758549-6hhbg" podUID="62b94b35-a1bb-4eb2-9bc0-da2760557b3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" Oct 30 12:30:51.493968 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:51.493938 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-1e9675-openshift-default-7d7c758549-6hhbg" Oct 30 12:30:51.494520 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:51.494497 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-1e9675-openshift-default-7d7c758549-6hhbg" Oct 30 12:30:51.495119 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:30:51.495097 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-1e9675-openshift-default-7d7c758549-6hhbg" Oct 30 12:31:01.929611 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:01.929575 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ec236e-v8vlh"] Oct 30 12:31:01.941415 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:01.941385 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ec236e-v8vlh"] Oct 30 12:31:03.606897 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:03.606865 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="111ec6e1-3267-4e49-a870-052bf799120d" path="/var/lib/kubelet/pods/111ec6e1-3267-4e49-a870-052bf799120d/volumes" Oct 30 12:31:05.008800 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:05.008763 4908 scope.go:122] "RemoveContainer" containerID="43088f232fe3bfc5ec3ae83f893d8ef64ccb12cc2e6689ae4466e7b9cf67d996" Oct 30 12:31:06.941578 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:06.941548 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7f78d4-ddkzg"] Oct 30 12:31:06.942152 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:06.942135 4908 cpu_manager.go:414] "containerMap: removing container" podUID="21bfd950-c0ea-46dd-b4f0-3dbb8167f696" containerName="revoke-keys" Oct 30 12:31:06.942195 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:06.942153 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21bfd950-c0ea-46dd-b4f0-3dbb8167f696" containerName="revoke-keys" Oct 30 12:31:06.942195 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:06.942180 4908 cpu_manager.go:414] "containerMap: removing container" podUID="111ec6e1-3267-4e49-a870-052bf799120d" containerName="revoke-keys" Oct 30 12:31:06.942195 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:06.942185 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="111ec6e1-3267-4e49-a870-052bf799120d" containerName="revoke-keys" Oct 30 12:31:06.942286 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:06.942261 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="21bfd950-c0ea-46dd-b4f0-3dbb8167f696" containerName="revoke-keys" Oct 30 12:31:06.942286 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:06.942276 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="111ec6e1-3267-4e49-a870-052bf799120d" containerName="revoke-keys" Oct 30 12:31:06.946415 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:06.946399 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7f78d4-ddkzg" Oct 30 12:31:06.962916 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:06.962889 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 12:31:06.973766 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:06.973743 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 12:31:06.973935 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:06.973821 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s75hq\"" Oct 30 12:31:06.973935 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:06.973820 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 12:31:06.979352 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:06.979327 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7f78d4-ddkzg"] Oct 30 12:31:07.068576 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:07.068543 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45a55357-916a-402b-9c5a-cee4b3403b55-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-7f78d4-ddkzg\" (UID: \"45a55357-916a-402b-9c5a-cee4b3403b55\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7f78d4-ddkzg" Oct 30 12:31:07.068710 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:07.068610 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45a55357-916a-402b-9c5a-cee4b3403b55-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-7f78d4-ddkzg\" (UID: \"45a55357-916a-402b-9c5a-cee4b3403b55\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7f78d4-ddkzg" Oct 30 12:31:07.068755 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:07.068706 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmmg7\" (UniqueName: \"kubernetes.io/projected/45a55357-916a-402b-9c5a-cee4b3403b55-kube-api-access-dmmg7\") pod \"maas-api-revoke-keys-e2e-ipp-b-7f78d4-ddkzg\" (UID: \"45a55357-916a-402b-9c5a-cee4b3403b55\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7f78d4-ddkzg" Oct 30 12:31:07.170288 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:07.170257 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45a55357-916a-402b-9c5a-cee4b3403b55-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-7f78d4-ddkzg\" (UID: \"45a55357-916a-402b-9c5a-cee4b3403b55\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7f78d4-ddkzg" Oct 30 12:31:07.170476 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:07.170303 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45a55357-916a-402b-9c5a-cee4b3403b55-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-7f78d4-ddkzg\" (UID: \"45a55357-916a-402b-9c5a-cee4b3403b55\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7f78d4-ddkzg" Oct 30 12:31:07.170476 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:07.170331 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmmg7\" (UniqueName: \"kubernetes.io/projected/45a55357-916a-402b-9c5a-cee4b3403b55-kube-api-access-dmmg7\") pod \"maas-api-revoke-keys-e2e-ipp-b-7f78d4-ddkzg\" (UID: \"45a55357-916a-402b-9c5a-cee4b3403b55\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7f78d4-ddkzg" Oct 30 12:31:07.171066 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:07.171037 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45a55357-916a-402b-9c5a-cee4b3403b55-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-7f78d4-ddkzg\" (UID: \"45a55357-916a-402b-9c5a-cee4b3403b55\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7f78d4-ddkzg" Oct 30 12:31:07.172803 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:07.172774 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45a55357-916a-402b-9c5a-cee4b3403b55-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-7f78d4-ddkzg\" (UID: \"45a55357-916a-402b-9c5a-cee4b3403b55\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7f78d4-ddkzg" Oct 30 12:31:07.192541 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:07.192487 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmmg7\" (UniqueName: \"kubernetes.io/projected/45a55357-916a-402b-9c5a-cee4b3403b55-kube-api-access-dmmg7\") pod \"maas-api-revoke-keys-e2e-ipp-b-7f78d4-ddkzg\" (UID: \"45a55357-916a-402b-9c5a-cee4b3403b55\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7f78d4-ddkzg" Oct 30 12:31:07.259584 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:07.259561 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7f78d4-ddkzg" Oct 30 12:31:07.567017 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:07.565191 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7f78d4-ddkzg"] Oct 30 12:31:08.007728 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:08.007689 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7f78d4-ddkzg" event={"ID":"45a55357-916a-402b-9c5a-cee4b3403b55","Type":"ContainerStarted","Data":"6bb846d7a23e9b48f6372682f8fd6188f3c99eb48a76bae6a24d88dbd609cec0"} Oct 30 12:31:08.008234 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:08.007748 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7f78d4-ddkzg" event={"ID":"45a55357-916a-402b-9c5a-cee4b3403b55","Type":"ContainerStarted","Data":"b9ec6df0509d4e3811e76d93f7e73b0521e9f559d9fb2fb985a720d469ef4a68"} Oct 30 12:31:08.045475 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:08.045431 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7f78d4-ddkzg" podStartSLOduration=2.045418714 podStartE2EDuration="2.045418714s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:31:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:31:08.044898555 +0000 UTC m=+2224.559327323" watchObservedRunningTime="2026-10-30 12:31:08.045418714 +0000 UTC m=+2224.559847505" Oct 30 12:31:09.023849 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:09.022921 4908 generic.go:356] "Generic (PLEG): container finished" podID="45a55357-916a-402b-9c5a-cee4b3403b55" containerID="6bb846d7a23e9b48f6372682f8fd6188f3c99eb48a76bae6a24d88dbd609cec0" exitCode=0 Oct 30 12:31:09.023849 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:09.023003 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7f78d4-ddkzg" event={"ID":"45a55357-916a-402b-9c5a-cee4b3403b55","Type":"ContainerDied","Data":"6bb846d7a23e9b48f6372682f8fd6188f3c99eb48a76bae6a24d88dbd609cec0"} Oct 30 12:31:09.340702 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:09.340287 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-ec236e-openshift-default-f6575b7df-kwhwr"] Oct 30 12:31:09.340702 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:09.340559 4908 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-ec236e-openshift-default-f6575b7df-kwhwr" podUID="5ea7d109-957f-4d4f-9897-6eb86c402601" containerName="istio-proxy" containerID="cri-o://a5d8ca0995f109f84aed0f9339343f70c6e299be9de38d2f25cacc6902008135" gracePeriod=30 Oct 30 12:31:10.230561 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:10.230537 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7f78d4-ddkzg" Oct 30 12:31:10.324083 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:10.324009 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/45a55357-916a-402b-9c5a-cee4b3403b55-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45a55357-916a-402b-9c5a-cee4b3403b55-maas-api-xks-ca\") pod \"45a55357-916a-402b-9c5a-cee4b3403b55\" (UID: \"45a55357-916a-402b-9c5a-cee4b3403b55\") " Oct 30 12:31:10.324083 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:10.324074 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45a55357-916a-402b-9c5a-cee4b3403b55-kube-api-access-dmmg7\" (UniqueName: \"kubernetes.io/projected/45a55357-916a-402b-9c5a-cee4b3403b55-kube-api-access-dmmg7\") pod \"45a55357-916a-402b-9c5a-cee4b3403b55\" (UID: \"45a55357-916a-402b-9c5a-cee4b3403b55\") " Oct 30 12:31:10.324256 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:10.324199 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45a55357-916a-402b-9c5a-cee4b3403b55-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45a55357-916a-402b-9c5a-cee4b3403b55-maas-api-openshift-ca\") pod \"45a55357-916a-402b-9c5a-cee4b3403b55\" (UID: \"45a55357-916a-402b-9c5a-cee4b3403b55\") " Oct 30 12:31:10.324574 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:10.324544 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45a55357-916a-402b-9c5a-cee4b3403b55-maas-api-openshift-ca" pod "45a55357-916a-402b-9c5a-cee4b3403b55" (UID: "45a55357-916a-402b-9c5a-cee4b3403b55"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:31:10.326379 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:10.326357 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45a55357-916a-402b-9c5a-cee4b3403b55-maas-api-xks-ca" pod "45a55357-916a-402b-9c5a-cee4b3403b55" (UID: "45a55357-916a-402b-9c5a-cee4b3403b55"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:31:10.326596 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:10.326572 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45a55357-916a-402b-9c5a-cee4b3403b55-kube-api-access-dmmg7" pod "45a55357-916a-402b-9c5a-cee4b3403b55" (UID: "45a55357-916a-402b-9c5a-cee4b3403b55"). InnerVolumeSpecName "kube-api-access-dmmg7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:31:10.426153 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:10.426127 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45a55357-916a-402b-9c5a-cee4b3403b55-maas-api-openshift-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:31:10.426153 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:10.426153 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45a55357-916a-402b-9c5a-cee4b3403b55-maas-api-xks-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:31:10.426303 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:10.426164 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dmmg7\" (UniqueName: \"kubernetes.io/projected/45a55357-916a-402b-9c5a-cee4b3403b55-kube-api-access-dmmg7\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:31:11.037842 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:11.037812 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7f78d4-ddkzg" Oct 30 12:31:11.037842 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:11.037809 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7f78d4-ddkzg" event={"ID":"45a55357-916a-402b-9c5a-cee4b3403b55","Type":"ContainerDied","Data":"b9ec6df0509d4e3811e76d93f7e73b0521e9f559d9fb2fb985a720d469ef4a68"} Oct 30 12:31:11.038055 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:11.037881 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b9ec6df0509d4e3811e76d93f7e73b0521e9f559d9fb2fb985a720d469ef4a68" Oct 30 12:31:11.929222 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:11.929180 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9wwfc"] Oct 30 12:31:14.142553 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:14.142517 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-41bdda-7xgd2"] Oct 30 12:31:14.154454 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:14.154425 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-41bdda-7xgd2"] Oct 30 12:31:14.626986 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:14.626960 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-ec236e-openshift-default-f6575b7df-kwhwr" Oct 30 12:31:14.704188 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:14.704154 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ea7d109-957f-4d4f-9897-6eb86c402601-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ea7d109-957f-4d4f-9897-6eb86c402601-workload-socket\") pod \"5ea7d109-957f-4d4f-9897-6eb86c402601\" (UID: \"5ea7d109-957f-4d4f-9897-6eb86c402601\") " Oct 30 12:31:14.704368 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:14.704204 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ea7d109-957f-4d4f-9897-6eb86c402601-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ea7d109-957f-4d4f-9897-6eb86c402601-istiod-ca-cert\") pod \"5ea7d109-957f-4d4f-9897-6eb86c402601\" (UID: \"5ea7d109-957f-4d4f-9897-6eb86c402601\") " Oct 30 12:31:14.704368 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:14.704294 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ea7d109-957f-4d4f-9897-6eb86c402601-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ea7d109-957f-4d4f-9897-6eb86c402601-istio-envoy\") pod \"5ea7d109-957f-4d4f-9897-6eb86c402601\" (UID: \"5ea7d109-957f-4d4f-9897-6eb86c402601\") " Oct 30 12:31:14.704368 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:14.704356 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ea7d109-957f-4d4f-9897-6eb86c402601-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ea7d109-957f-4d4f-9897-6eb86c402601-istio-data\") pod \"5ea7d109-957f-4d4f-9897-6eb86c402601\" (UID: \"5ea7d109-957f-4d4f-9897-6eb86c402601\") " Oct 30 12:31:14.704569 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:14.704471 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ea7d109-957f-4d4f-9897-6eb86c402601-workload-socket" pod "5ea7d109-957f-4d4f-9897-6eb86c402601" (UID: "5ea7d109-957f-4d4f-9897-6eb86c402601"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:31:14.704673 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:14.704652 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ea7d109-957f-4d4f-9897-6eb86c402601-istio-data" pod "5ea7d109-957f-4d4f-9897-6eb86c402601" (UID: "5ea7d109-957f-4d4f-9897-6eb86c402601"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:31:14.704747 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:14.704670 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ea7d109-957f-4d4f-9897-6eb86c402601-istiod-ca-cert" pod "5ea7d109-957f-4d4f-9897-6eb86c402601" (UID: "5ea7d109-957f-4d4f-9897-6eb86c402601"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:31:14.704747 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:14.704715 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5ea7d109-957f-4d4f-9897-6eb86c402601-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ea7d109-957f-4d4f-9897-6eb86c402601-istio-podinfo\") pod \"5ea7d109-957f-4d4f-9897-6eb86c402601\" (UID: \"5ea7d109-957f-4d4f-9897-6eb86c402601\") " Oct 30 12:31:14.704889 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:14.704758 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ea7d109-957f-4d4f-9897-6eb86c402601-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ea7d109-957f-4d4f-9897-6eb86c402601-credential-socket\") pod \"5ea7d109-957f-4d4f-9897-6eb86c402601\" (UID: \"5ea7d109-957f-4d4f-9897-6eb86c402601\") " Oct 30 12:31:14.704889 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:14.704822 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ea7d109-957f-4d4f-9897-6eb86c402601-istio-token\" (UniqueName: \"kubernetes.io/projected/5ea7d109-957f-4d4f-9897-6eb86c402601-istio-token\") pod \"5ea7d109-957f-4d4f-9897-6eb86c402601\" (UID: \"5ea7d109-957f-4d4f-9897-6eb86c402601\") " Oct 30 12:31:14.704889 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:14.704879 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ea7d109-957f-4d4f-9897-6eb86c402601-kube-api-access-ptdcf\" (UniqueName: \"kubernetes.io/projected/5ea7d109-957f-4d4f-9897-6eb86c402601-kube-api-access-ptdcf\") pod \"5ea7d109-957f-4d4f-9897-6eb86c402601\" (UID: \"5ea7d109-957f-4d4f-9897-6eb86c402601\") " Oct 30 12:31:14.705059 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:14.705009 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ea7d109-957f-4d4f-9897-6eb86c402601-credential-socket" pod "5ea7d109-957f-4d4f-9897-6eb86c402601" (UID: "5ea7d109-957f-4d4f-9897-6eb86c402601"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:31:14.705487 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:14.705465 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ea7d109-957f-4d4f-9897-6eb86c402601-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ea7d109-957f-4d4f-9897-6eb86c402601-workload-certs\") pod \"5ea7d109-957f-4d4f-9897-6eb86c402601\" (UID: \"5ea7d109-957f-4d4f-9897-6eb86c402601\") " Oct 30 12:31:14.705876 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:14.705702 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ea7d109-957f-4d4f-9897-6eb86c402601-workload-certs" pod "5ea7d109-957f-4d4f-9897-6eb86c402601" (UID: "5ea7d109-957f-4d4f-9897-6eb86c402601"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:31:14.706558 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:14.706535 4908 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ea7d109-957f-4d4f-9897-6eb86c402601-workload-socket\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:31:14.706683 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:14.706669 4908 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ea7d109-957f-4d4f-9897-6eb86c402601-istiod-ca-cert\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:31:14.706774 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:14.706763 4908 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ea7d109-957f-4d4f-9897-6eb86c402601-istio-data\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:31:14.706906 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:14.706892 4908 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ea7d109-957f-4d4f-9897-6eb86c402601-credential-socket\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:31:14.707009 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:14.706995 4908 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ea7d109-957f-4d4f-9897-6eb86c402601-workload-certs\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:31:14.707093 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:14.707054 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ea7d109-957f-4d4f-9897-6eb86c402601-istio-envoy" pod "5ea7d109-957f-4d4f-9897-6eb86c402601" (UID: "5ea7d109-957f-4d4f-9897-6eb86c402601"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:31:14.707204 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:14.707186 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5ea7d109-957f-4d4f-9897-6eb86c402601-istio-podinfo" pod "5ea7d109-957f-4d4f-9897-6eb86c402601" (UID: "5ea7d109-957f-4d4f-9897-6eb86c402601"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 12:31:14.708183 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:14.708157 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ea7d109-957f-4d4f-9897-6eb86c402601-istio-token" pod "5ea7d109-957f-4d4f-9897-6eb86c402601" (UID: "5ea7d109-957f-4d4f-9897-6eb86c402601"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:31:14.708355 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:14.708338 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ea7d109-957f-4d4f-9897-6eb86c402601-kube-api-access-ptdcf" pod "5ea7d109-957f-4d4f-9897-6eb86c402601" (UID: "5ea7d109-957f-4d4f-9897-6eb86c402601"). InnerVolumeSpecName "kube-api-access-ptdcf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:31:14.743029 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:14.743003 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-34a210df-s2hkm"] Oct 30 12:31:14.743588 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:14.743573 4908 cpu_manager.go:414] "containerMap: removing container" podUID="45a55357-916a-402b-9c5a-cee4b3403b55" containerName="revoke-keys" Oct 30 12:31:14.743639 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:14.743588 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45a55357-916a-402b-9c5a-cee4b3403b55" containerName="revoke-keys" Oct 30 12:31:14.743639 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:14.743624 4908 cpu_manager.go:414] "containerMap: removing container" podUID="5ea7d109-957f-4d4f-9897-6eb86c402601" containerName="istio-proxy" Oct 30 12:31:14.743639 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:14.743630 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ea7d109-957f-4d4f-9897-6eb86c402601" containerName="istio-proxy" Oct 30 12:31:14.743734 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:14.743715 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ea7d109-957f-4d4f-9897-6eb86c402601" containerName="istio-proxy" Oct 30 12:31:14.743734 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:14.743726 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ea7d109-957f-4d4f-9897-6eb86c402601" containerName="istio-proxy" Oct 30 12:31:14.743794 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:14.743736 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="45a55357-916a-402b-9c5a-cee4b3403b55" containerName="revoke-keys" Oct 30 12:31:14.751497 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:14.751473 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34a210df-s2hkm" Oct 30 12:31:14.765539 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:14.765518 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 12:31:14.765642 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:14.765592 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 12:31:14.774246 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:14.774189 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-34a210df-s2hkm"] Oct 30 12:31:14.808795 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:14.808756 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8898760f-3f4d-4a4d-bb30-69744a87e78f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-34a210df-s2hkm\" (UID: \"8898760f-3f4d-4a4d-bb30-69744a87e78f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34a210df-s2hkm" Oct 30 12:31:14.808966 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:14.808804 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8898760f-3f4d-4a4d-bb30-69744a87e78f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-34a210df-s2hkm\" (UID: \"8898760f-3f4d-4a4d-bb30-69744a87e78f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34a210df-s2hkm" Oct 30 12:31:14.808966 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:14.808851 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8ndr\" (UniqueName: \"kubernetes.io/projected/8898760f-3f4d-4a4d-bb30-69744a87e78f-kube-api-access-f8ndr\") pod \"maas-api-revoke-sub-34a210df-s2hkm\" (UID: \"8898760f-3f4d-4a4d-bb30-69744a87e78f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34a210df-s2hkm" Oct 30 12:31:14.809087 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:14.809044 4908 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ea7d109-957f-4d4f-9897-6eb86c402601-istio-envoy\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:31:14.809087 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:14.809061 4908 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ea7d109-957f-4d4f-9897-6eb86c402601-istio-podinfo\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:31:14.809087 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:14.809075 4908 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ea7d109-957f-4d4f-9897-6eb86c402601-istio-token\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:31:14.809238 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:14.809088 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptdcf\" (UniqueName: \"kubernetes.io/projected/5ea7d109-957f-4d4f-9897-6eb86c402601-kube-api-access-ptdcf\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:31:14.820005 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:14.819979 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 12:31:14.820114 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:14.820059 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s75hq\"" Oct 30 12:31:14.909841 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:14.909800 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8898760f-3f4d-4a4d-bb30-69744a87e78f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-34a210df-s2hkm\" (UID: \"8898760f-3f4d-4a4d-bb30-69744a87e78f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34a210df-s2hkm" Oct 30 12:31:14.910022 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:14.909874 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8898760f-3f4d-4a4d-bb30-69744a87e78f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-34a210df-s2hkm\" (UID: \"8898760f-3f4d-4a4d-bb30-69744a87e78f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34a210df-s2hkm" Oct 30 12:31:14.910022 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:14.909903 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8ndr\" (UniqueName: \"kubernetes.io/projected/8898760f-3f4d-4a4d-bb30-69744a87e78f-kube-api-access-f8ndr\") pod \"maas-api-revoke-sub-34a210df-s2hkm\" (UID: \"8898760f-3f4d-4a4d-bb30-69744a87e78f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34a210df-s2hkm" Oct 30 12:31:14.910740 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:14.910717 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8898760f-3f4d-4a4d-bb30-69744a87e78f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-34a210df-s2hkm\" (UID: \"8898760f-3f4d-4a4d-bb30-69744a87e78f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34a210df-s2hkm" Oct 30 12:31:14.912230 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:14.912213 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8898760f-3f4d-4a4d-bb30-69744a87e78f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-34a210df-s2hkm\" (UID: \"8898760f-3f4d-4a4d-bb30-69744a87e78f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34a210df-s2hkm" Oct 30 12:31:14.931881 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:14.931860 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8ndr\" (UniqueName: \"kubernetes.io/projected/8898760f-3f4d-4a4d-bb30-69744a87e78f-kube-api-access-f8ndr\") pod \"maas-api-revoke-sub-34a210df-s2hkm\" (UID: \"8898760f-3f4d-4a4d-bb30-69744a87e78f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34a210df-s2hkm" Oct 30 12:31:15.066714 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:15.066677 4908 generic.go:356] "Generic (PLEG): container finished" podID="5ea7d109-957f-4d4f-9897-6eb86c402601" containerID="a5d8ca0995f109f84aed0f9339343f70c6e299be9de38d2f25cacc6902008135" exitCode=0 Oct 30 12:31:15.066931 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:15.066732 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-ec236e-openshift-default-f6575b7df-kwhwr" Oct 30 12:31:15.066931 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:15.066751 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-ec236e-openshift-default-f6575b7df-kwhwr" event={"ID":"5ea7d109-957f-4d4f-9897-6eb86c402601","Type":"ContainerDied","Data":"a5d8ca0995f109f84aed0f9339343f70c6e299be9de38d2f25cacc6902008135"} Oct 30 12:31:15.066931 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:15.066839 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-ec236e-openshift-default-f6575b7df-kwhwr" event={"ID":"5ea7d109-957f-4d4f-9897-6eb86c402601","Type":"ContainerDied","Data":"ff5a52763ef94c689d6ef44284e3c86b24f90a5b83cfa8919d54b76ccf793ca4"} Oct 30 12:31:15.066931 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:15.066872 4908 scope.go:122] "RemoveContainer" containerID="a5d8ca0995f109f84aed0f9339343f70c6e299be9de38d2f25cacc6902008135" Oct 30 12:31:15.067748 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:15.067726 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34a210df-s2hkm" Oct 30 12:31:15.085777 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:15.085711 4908 scope.go:122] "RemoveContainer" containerID="5129d25b769d5c0f940748bcb1555021168905a268bd4b3c31a2c6fba52703e1" Oct 30 12:31:15.096073 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:15.096054 4908 scope.go:122] "RemoveContainer" containerID="a5d8ca0995f109f84aed0f9339343f70c6e299be9de38d2f25cacc6902008135" Oct 30 12:31:15.096407 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:31:15.096372 4908 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5d8ca0995f109f84aed0f9339343f70c6e299be9de38d2f25cacc6902008135\": container with ID starting with a5d8ca0995f109f84aed0f9339343f70c6e299be9de38d2f25cacc6902008135 not found: ID does not exist" containerID="a5d8ca0995f109f84aed0f9339343f70c6e299be9de38d2f25cacc6902008135" Oct 30 12:31:15.096515 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:15.096416 4908 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5d8ca0995f109f84aed0f9339343f70c6e299be9de38d2f25cacc6902008135"} err="failed to get container status \"a5d8ca0995f109f84aed0f9339343f70c6e299be9de38d2f25cacc6902008135\": rpc error: code = NotFound desc = could not find container \"a5d8ca0995f109f84aed0f9339343f70c6e299be9de38d2f25cacc6902008135\": container with ID starting with a5d8ca0995f109f84aed0f9339343f70c6e299be9de38d2f25cacc6902008135 not found: ID does not exist" Oct 30 12:31:15.096515 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:15.096441 4908 scope.go:122] "RemoveContainer" containerID="5129d25b769d5c0f940748bcb1555021168905a268bd4b3c31a2c6fba52703e1" Oct 30 12:31:15.096751 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:31:15.096730 4908 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5129d25b769d5c0f940748bcb1555021168905a268bd4b3c31a2c6fba52703e1\": container with ID starting with 5129d25b769d5c0f940748bcb1555021168905a268bd4b3c31a2c6fba52703e1 not found: ID does not exist" containerID="5129d25b769d5c0f940748bcb1555021168905a268bd4b3c31a2c6fba52703e1" Oct 30 12:31:15.096843 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:15.096760 4908 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5129d25b769d5c0f940748bcb1555021168905a268bd4b3c31a2c6fba52703e1"} err="failed to get container status \"5129d25b769d5c0f940748bcb1555021168905a268bd4b3c31a2c6fba52703e1\": rpc error: code = NotFound desc = could not find container \"5129d25b769d5c0f940748bcb1555021168905a268bd4b3c31a2c6fba52703e1\": container with ID starting with 5129d25b769d5c0f940748bcb1555021168905a268bd4b3c31a2c6fba52703e1 not found: ID does not exist" Oct 30 12:31:15.119729 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:15.119702 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-ec236e-openshift-default-f6575b7df-kwhwr"] Oct 30 12:31:15.131465 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:15.131295 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-ec236e-openshift-default-f6575b7df-kwhwr"] Oct 30 12:31:15.388101 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:15.388073 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-34a210df-s2hkm"] Oct 30 12:31:15.390787 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:31:15.390756 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8898760f_3f4d_4a4d_bb30_69744a87e78f.slice/crio-64bf658f5a0632fe3b11a7d7f46ccf966e944ebb7189d7d163a12975318075a8 WatchSource:0}: Error finding container 64bf658f5a0632fe3b11a7d7f46ccf966e944ebb7189d7d163a12975318075a8: Status 404 returned error can't find the container with id 64bf658f5a0632fe3b11a7d7f46ccf966e944ebb7189d7d163a12975318075a8 Oct 30 12:31:15.616352 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:15.616322 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21bfd950-c0ea-46dd-b4f0-3dbb8167f696" path="/var/lib/kubelet/pods/21bfd950-c0ea-46dd-b4f0-3dbb8167f696/volumes" Oct 30 12:31:15.616780 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:15.616767 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ea7d109-957f-4d4f-9897-6eb86c402601" path="/var/lib/kubelet/pods/5ea7d109-957f-4d4f-9897-6eb86c402601/volumes" Oct 30 12:31:16.078524 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:16.078477 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34a210df-s2hkm" event={"ID":"8898760f-3f4d-4a4d-bb30-69744a87e78f","Type":"ContainerStarted","Data":"88ca2dd55da2cd140f2f3ddee213b4a8a72bc7df64ef4c716d3f96827795b0f7"} Oct 30 12:31:16.078724 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:16.078545 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34a210df-s2hkm" event={"ID":"8898760f-3f4d-4a4d-bb30-69744a87e78f","Type":"ContainerStarted","Data":"64bf658f5a0632fe3b11a7d7f46ccf966e944ebb7189d7d163a12975318075a8"} Oct 30 12:31:16.118848 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:16.118786 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34a210df-s2hkm" podStartSLOduration=2.118773235 podStartE2EDuration="2.118773235s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:31:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:31:16.117642722 +0000 UTC m=+2232.632071467" watchObservedRunningTime="2026-10-30 12:31:16.118773235 +0000 UTC m=+2232.633201979" Oct 30 12:31:17.089582 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:17.089541 4908 generic.go:356] "Generic (PLEG): container finished" podID="8898760f-3f4d-4a4d-bb30-69744a87e78f" containerID="88ca2dd55da2cd140f2f3ddee213b4a8a72bc7df64ef4c716d3f96827795b0f7" exitCode=0 Oct 30 12:31:17.090017 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:17.089699 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34a210df-s2hkm" event={"ID":"8898760f-3f4d-4a4d-bb30-69744a87e78f","Type":"ContainerDied","Data":"88ca2dd55da2cd140f2f3ddee213b4a8a72bc7df64ef4c716d3f96827795b0f7"} Oct 30 12:31:18.279183 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:18.279156 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34a210df-s2hkm" Oct 30 12:31:18.351020 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:18.350948 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8898760f-3f4d-4a4d-bb30-69744a87e78f-kube-api-access-f8ndr\" (UniqueName: \"kubernetes.io/projected/8898760f-3f4d-4a4d-bb30-69744a87e78f-kube-api-access-f8ndr\") pod \"8898760f-3f4d-4a4d-bb30-69744a87e78f\" (UID: \"8898760f-3f4d-4a4d-bb30-69744a87e78f\") " Oct 30 12:31:18.351020 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:18.351006 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8898760f-3f4d-4a4d-bb30-69744a87e78f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8898760f-3f4d-4a4d-bb30-69744a87e78f-maas-api-openshift-ca\") pod \"8898760f-3f4d-4a4d-bb30-69744a87e78f\" (UID: \"8898760f-3f4d-4a4d-bb30-69744a87e78f\") " Oct 30 12:31:18.351228 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:18.351084 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8898760f-3f4d-4a4d-bb30-69744a87e78f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8898760f-3f4d-4a4d-bb30-69744a87e78f-maas-api-xks-ca\") pod \"8898760f-3f4d-4a4d-bb30-69744a87e78f\" (UID: \"8898760f-3f4d-4a4d-bb30-69744a87e78f\") " Oct 30 12:31:18.351450 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:18.351418 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8898760f-3f4d-4a4d-bb30-69744a87e78f-maas-api-openshift-ca" pod "8898760f-3f4d-4a4d-bb30-69744a87e78f" (UID: "8898760f-3f4d-4a4d-bb30-69744a87e78f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:31:18.351747 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:18.351728 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8898760f-3f4d-4a4d-bb30-69744a87e78f-maas-api-openshift-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:31:18.353278 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:18.353243 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8898760f-3f4d-4a4d-bb30-69744a87e78f-maas-api-xks-ca" pod "8898760f-3f4d-4a4d-bb30-69744a87e78f" (UID: "8898760f-3f4d-4a4d-bb30-69744a87e78f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:31:18.353515 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:18.353494 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8898760f-3f4d-4a4d-bb30-69744a87e78f-kube-api-access-f8ndr" pod "8898760f-3f4d-4a4d-bb30-69744a87e78f" (UID: "8898760f-3f4d-4a4d-bb30-69744a87e78f"). InnerVolumeSpecName "kube-api-access-f8ndr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:31:18.453260 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:18.453236 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8ndr\" (UniqueName: \"kubernetes.io/projected/8898760f-3f4d-4a4d-bb30-69744a87e78f-kube-api-access-f8ndr\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:31:18.453260 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:18.453258 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8898760f-3f4d-4a4d-bb30-69744a87e78f-maas-api-xks-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:31:19.122225 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:19.122195 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34a210df-s2hkm" Oct 30 12:31:19.122225 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:19.122218 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34a210df-s2hkm" event={"ID":"8898760f-3f4d-4a4d-bb30-69744a87e78f","Type":"ContainerDied","Data":"64bf658f5a0632fe3b11a7d7f46ccf966e944ebb7189d7d163a12975318075a8"} Oct 30 12:31:19.122443 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:19.122269 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="64bf658f5a0632fe3b11a7d7f46ccf966e944ebb7189d7d163a12975318075a8" Oct 30 12:31:19.713585 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:31:19.713544 4908 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vn6vv" containerName="extractor" Oct 30 12:31:21.895267 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:21.895225 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-bnhn5"] Oct 30 12:31:21.896602 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:21.896569 4908 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-bnhn5" podUID="fbf8595f-0d2c-471d-90dd-63425baa0510" containerName="istio-proxy" containerID="cri-o://1f198fc51fb02ed8f0e2a3001b60164e221c8f8ec25eb1acbeace1647cc19b34" gracePeriod=30 Oct 30 12:31:22.548285 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:22.548253 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-1e9675-6q54x"] Oct 30 12:31:22.548818 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:22.548802 4908 cpu_manager.go:414] "containerMap: removing container" podUID="5ea7d109-957f-4d4f-9897-6eb86c402601" containerName="istio-proxy" Oct 30 12:31:22.548941 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:22.548819 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ea7d109-957f-4d4f-9897-6eb86c402601" containerName="istio-proxy" Oct 30 12:31:22.548941 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:22.548875 4908 cpu_manager.go:414] "containerMap: removing container" podUID="8898760f-3f4d-4a4d-bb30-69744a87e78f" containerName="revoke-keys" Oct 30 12:31:22.548941 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:22.548882 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8898760f-3f4d-4a4d-bb30-69744a87e78f" containerName="revoke-keys" Oct 30 12:31:22.549083 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:22.548974 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="8898760f-3f4d-4a4d-bb30-69744a87e78f" containerName="revoke-keys" Oct 30 12:31:22.554583 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:22.554563 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-1e9675-6q54x" Oct 30 12:31:22.568176 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:22.568156 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 12:31:22.572415 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:22.572393 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-1e9675-6q54x"] Oct 30 12:31:22.579434 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:22.579208 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 12:31:22.579434 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:22.579219 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 12:31:22.579434 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:22.579296 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s75hq\"" Oct 30 12:31:22.631810 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:22.631779 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62b98818-9688-4fee-9742-529d1a6927d0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-1e9675-6q54x\" (UID: \"62b98818-9688-4fee-9742-529d1a6927d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-1e9675-6q54x" Oct 30 12:31:22.631956 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:22.631840 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4dgd\" (UniqueName: \"kubernetes.io/projected/62b98818-9688-4fee-9742-529d1a6927d0-kube-api-access-f4dgd\") pod \"maas-api-revoke-keys-e2e-praxis-b-1e9675-6q54x\" (UID: \"62b98818-9688-4fee-9742-529d1a6927d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-1e9675-6q54x" Oct 30 12:31:22.632170 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:22.632144 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62b98818-9688-4fee-9742-529d1a6927d0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-1e9675-6q54x\" (UID: \"62b98818-9688-4fee-9742-529d1a6927d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-1e9675-6q54x" Oct 30 12:31:22.733491 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:22.733461 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4dgd\" (UniqueName: \"kubernetes.io/projected/62b98818-9688-4fee-9742-529d1a6927d0-kube-api-access-f4dgd\") pod \"maas-api-revoke-keys-e2e-praxis-b-1e9675-6q54x\" (UID: \"62b98818-9688-4fee-9742-529d1a6927d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-1e9675-6q54x" Oct 30 12:31:22.733638 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:22.733598 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62b98818-9688-4fee-9742-529d1a6927d0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-1e9675-6q54x\" (UID: \"62b98818-9688-4fee-9742-529d1a6927d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-1e9675-6q54x" Oct 30 12:31:22.733720 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:22.733661 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62b98818-9688-4fee-9742-529d1a6927d0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-1e9675-6q54x\" (UID: \"62b98818-9688-4fee-9742-529d1a6927d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-1e9675-6q54x" Oct 30 12:31:22.734357 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:22.734334 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62b98818-9688-4fee-9742-529d1a6927d0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-1e9675-6q54x\" (UID: \"62b98818-9688-4fee-9742-529d1a6927d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-1e9675-6q54x" Oct 30 12:31:22.735978 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:22.735953 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62b98818-9688-4fee-9742-529d1a6927d0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-1e9675-6q54x\" (UID: \"62b98818-9688-4fee-9742-529d1a6927d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-1e9675-6q54x" Oct 30 12:31:22.755207 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:22.755181 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4dgd\" (UniqueName: \"kubernetes.io/projected/62b98818-9688-4fee-9742-529d1a6927d0-kube-api-access-f4dgd\") pod \"maas-api-revoke-keys-e2e-praxis-b-1e9675-6q54x\" (UID: \"62b98818-9688-4fee-9742-529d1a6927d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-1e9675-6q54x" Oct 30 12:31:22.867733 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:22.867675 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-1e9675-6q54x" Oct 30 12:31:23.167427 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:23.167403 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-1e9675-6q54x"] Oct 30 12:31:23.170455 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:31:23.170422 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62b98818_9688_4fee_9742_529d1a6927d0.slice/crio-d1c8beb0aa2116e1511aac30cc1ee0800665d098ac8b6c9970c2b36937a57bf3 WatchSource:0}: Error finding container d1c8beb0aa2116e1511aac30cc1ee0800665d098ac8b6c9970c2b36937a57bf3: Status 404 returned error can't find the container with id d1c8beb0aa2116e1511aac30cc1ee0800665d098ac8b6c9970c2b36937a57bf3 Oct 30 12:31:24.159653 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:24.159615 4908 generic.go:356] "Generic (PLEG): container finished" podID="62b98818-9688-4fee-9742-529d1a6927d0" containerID="882e5912e768ef799ce7a13998ce3fb9156654dd0c6ffe07f5a718d624cccc88" exitCode=0 Oct 30 12:31:24.159847 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:24.159708 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-1e9675-6q54x" event={"ID":"62b98818-9688-4fee-9742-529d1a6927d0","Type":"ContainerDied","Data":"882e5912e768ef799ce7a13998ce3fb9156654dd0c6ffe07f5a718d624cccc88"} Oct 30 12:31:24.159847 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:24.159759 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-1e9675-6q54x" event={"ID":"62b98818-9688-4fee-9742-529d1a6927d0","Type":"ContainerStarted","Data":"d1c8beb0aa2116e1511aac30cc1ee0800665d098ac8b6c9970c2b36937a57bf3"} Oct 30 12:31:25.345279 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:25.345249 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-1e9675-6q54x" Oct 30 12:31:25.473925 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:25.473810 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62b98818-9688-4fee-9742-529d1a6927d0-kube-api-access-f4dgd\" (UniqueName: \"kubernetes.io/projected/62b98818-9688-4fee-9742-529d1a6927d0-kube-api-access-f4dgd\") pod \"62b98818-9688-4fee-9742-529d1a6927d0\" (UID: \"62b98818-9688-4fee-9742-529d1a6927d0\") " Oct 30 12:31:25.474091 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:25.473927 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/62b98818-9688-4fee-9742-529d1a6927d0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62b98818-9688-4fee-9742-529d1a6927d0-maas-api-xks-ca\") pod \"62b98818-9688-4fee-9742-529d1a6927d0\" (UID: \"62b98818-9688-4fee-9742-529d1a6927d0\") " Oct 30 12:31:25.474091 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:25.474059 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62b98818-9688-4fee-9742-529d1a6927d0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62b98818-9688-4fee-9742-529d1a6927d0-maas-api-openshift-ca\") pod \"62b98818-9688-4fee-9742-529d1a6927d0\" (UID: \"62b98818-9688-4fee-9742-529d1a6927d0\") " Oct 30 12:31:25.474424 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:25.474390 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62b98818-9688-4fee-9742-529d1a6927d0-maas-api-openshift-ca" pod "62b98818-9688-4fee-9742-529d1a6927d0" (UID: "62b98818-9688-4fee-9742-529d1a6927d0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:31:25.475958 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:25.475933 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62b98818-9688-4fee-9742-529d1a6927d0-maas-api-xks-ca" pod "62b98818-9688-4fee-9742-529d1a6927d0" (UID: "62b98818-9688-4fee-9742-529d1a6927d0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:31:25.476118 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:25.475977 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62b98818-9688-4fee-9742-529d1a6927d0-kube-api-access-f4dgd" pod "62b98818-9688-4fee-9742-529d1a6927d0" (UID: "62b98818-9688-4fee-9742-529d1a6927d0"). InnerVolumeSpecName "kube-api-access-f4dgd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:31:25.575432 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:25.575403 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62b98818-9688-4fee-9742-529d1a6927d0-maas-api-openshift-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:31:25.575432 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:25.575431 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4dgd\" (UniqueName: \"kubernetes.io/projected/62b98818-9688-4fee-9742-529d1a6927d0-kube-api-access-f4dgd\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:31:25.575432 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:25.575441 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62b98818-9688-4fee-9742-529d1a6927d0-maas-api-xks-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:31:26.177076 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:26.177049 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-1e9675-6q54x" Oct 30 12:31:26.177242 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:26.177093 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-1e9675-6q54x" event={"ID":"62b98818-9688-4fee-9742-529d1a6927d0","Type":"ContainerDied","Data":"d1c8beb0aa2116e1511aac30cc1ee0800665d098ac8b6c9970c2b36937a57bf3"} Oct 30 12:31:26.177242 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:26.177143 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d1c8beb0aa2116e1511aac30cc1ee0800665d098ac8b6c9970c2b36937a57bf3" Oct 30 12:31:27.177134 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:27.177071 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-bnhn5" Oct 30 12:31:27.186196 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:27.186167 4908 generic.go:356] "Generic (PLEG): container finished" podID="fbf8595f-0d2c-471d-90dd-63425baa0510" containerID="1f198fc51fb02ed8f0e2a3001b60164e221c8f8ec25eb1acbeace1647cc19b34" exitCode=0 Oct 30 12:31:27.186328 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:27.186213 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-bnhn5" Oct 30 12:31:27.186328 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:27.186254 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-bnhn5" event={"ID":"fbf8595f-0d2c-471d-90dd-63425baa0510","Type":"ContainerDied","Data":"1f198fc51fb02ed8f0e2a3001b60164e221c8f8ec25eb1acbeace1647cc19b34"} Oct 30 12:31:27.186328 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:27.186303 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-bnhn5" event={"ID":"fbf8595f-0d2c-471d-90dd-63425baa0510","Type":"ContainerDied","Data":"0c4d4770e9ac5f229ef8f28f1f03fc58bc44884101d163098d85c6e09c5b15bd"} Oct 30 12:31:27.186328 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:27.186326 4908 scope.go:122] "RemoveContainer" containerID="1f198fc51fb02ed8f0e2a3001b60164e221c8f8ec25eb1acbeace1647cc19b34" Oct 30 12:31:27.211375 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:27.211258 4908 scope.go:122] "RemoveContainer" containerID="1f198fc51fb02ed8f0e2a3001b60164e221c8f8ec25eb1acbeace1647cc19b34" Oct 30 12:31:27.214680 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:31:27.214645 4908 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f198fc51fb02ed8f0e2a3001b60164e221c8f8ec25eb1acbeace1647cc19b34\": container with ID starting with 1f198fc51fb02ed8f0e2a3001b60164e221c8f8ec25eb1acbeace1647cc19b34 not found: ID does not exist" containerID="1f198fc51fb02ed8f0e2a3001b60164e221c8f8ec25eb1acbeace1647cc19b34" Oct 30 12:31:27.214907 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:27.214695 4908 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f198fc51fb02ed8f0e2a3001b60164e221c8f8ec25eb1acbeace1647cc19b34"} err="failed to get container status \"1f198fc51fb02ed8f0e2a3001b60164e221c8f8ec25eb1acbeace1647cc19b34\": rpc error: code = NotFound desc = could not find container \"1f198fc51fb02ed8f0e2a3001b60164e221c8f8ec25eb1acbeace1647cc19b34\": container with ID starting with 1f198fc51fb02ed8f0e2a3001b60164e221c8f8ec25eb1acbeace1647cc19b34 not found: ID does not exist" Oct 30 12:31:27.346824 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:27.346794 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbf8595f-0d2c-471d-90dd-63425baa0510-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbf8595f-0d2c-471d-90dd-63425baa0510-credential-socket\") pod \"fbf8595f-0d2c-471d-90dd-63425baa0510\" (UID: \"fbf8595f-0d2c-471d-90dd-63425baa0510\") " Oct 30 12:31:27.347044 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:27.346897 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbf8595f-0d2c-471d-90dd-63425baa0510-kube-api-access-vvtww\" (UniqueName: \"kubernetes.io/projected/fbf8595f-0d2c-471d-90dd-63425baa0510-kube-api-access-vvtww\") pod \"fbf8595f-0d2c-471d-90dd-63425baa0510\" (UID: \"fbf8595f-0d2c-471d-90dd-63425baa0510\") " Oct 30 12:31:27.347044 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:27.346966 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbf8595f-0d2c-471d-90dd-63425baa0510-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbf8595f-0d2c-471d-90dd-63425baa0510-workload-socket\") pod \"fbf8595f-0d2c-471d-90dd-63425baa0510\" (UID: \"fbf8595f-0d2c-471d-90dd-63425baa0510\") " Oct 30 12:31:27.347044 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:27.347008 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbf8595f-0d2c-471d-90dd-63425baa0510-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbf8595f-0d2c-471d-90dd-63425baa0510-workload-certs\") pod \"fbf8595f-0d2c-471d-90dd-63425baa0510\" (UID: \"fbf8595f-0d2c-471d-90dd-63425baa0510\") " Oct 30 12:31:27.347213 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:27.347058 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbf8595f-0d2c-471d-90dd-63425baa0510-credential-socket" pod "fbf8595f-0d2c-471d-90dd-63425baa0510" (UID: "fbf8595f-0d2c-471d-90dd-63425baa0510"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:31:27.347213 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:27.347063 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbf8595f-0d2c-471d-90dd-63425baa0510-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbf8595f-0d2c-471d-90dd-63425baa0510-istio-data\") pod \"fbf8595f-0d2c-471d-90dd-63425baa0510\" (UID: \"fbf8595f-0d2c-471d-90dd-63425baa0510\") " Oct 30 12:31:27.347213 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:27.347117 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbf8595f-0d2c-471d-90dd-63425baa0510-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbf8595f-0d2c-471d-90dd-63425baa0510-istiod-ca-cert\") pod \"fbf8595f-0d2c-471d-90dd-63425baa0510\" (UID: \"fbf8595f-0d2c-471d-90dd-63425baa0510\") " Oct 30 12:31:27.347213 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:27.347157 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbf8595f-0d2c-471d-90dd-63425baa0510-istio-token\" (UniqueName: \"kubernetes.io/projected/fbf8595f-0d2c-471d-90dd-63425baa0510-istio-token\") pod \"fbf8595f-0d2c-471d-90dd-63425baa0510\" (UID: \"fbf8595f-0d2c-471d-90dd-63425baa0510\") " Oct 30 12:31:27.347418 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:27.347253 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fbf8595f-0d2c-471d-90dd-63425baa0510-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbf8595f-0d2c-471d-90dd-63425baa0510-istio-podinfo\") pod \"fbf8595f-0d2c-471d-90dd-63425baa0510\" (UID: \"fbf8595f-0d2c-471d-90dd-63425baa0510\") " Oct 30 12:31:27.347418 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:27.347285 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbf8595f-0d2c-471d-90dd-63425baa0510-istio-data" pod "fbf8595f-0d2c-471d-90dd-63425baa0510" (UID: "fbf8595f-0d2c-471d-90dd-63425baa0510"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:31:27.347418 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:27.347296 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbf8595f-0d2c-471d-90dd-63425baa0510-workload-socket" pod "fbf8595f-0d2c-471d-90dd-63425baa0510" (UID: "fbf8595f-0d2c-471d-90dd-63425baa0510"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:31:27.347418 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:27.347315 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbf8595f-0d2c-471d-90dd-63425baa0510-workload-certs" pod "fbf8595f-0d2c-471d-90dd-63425baa0510" (UID: "fbf8595f-0d2c-471d-90dd-63425baa0510"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:31:27.347418 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:27.347351 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbf8595f-0d2c-471d-90dd-63425baa0510-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbf8595f-0d2c-471d-90dd-63425baa0510-istio-envoy\") pod \"fbf8595f-0d2c-471d-90dd-63425baa0510\" (UID: \"fbf8595f-0d2c-471d-90dd-63425baa0510\") " Oct 30 12:31:27.347690 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:27.347545 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbf8595f-0d2c-471d-90dd-63425baa0510-istiod-ca-cert" pod "fbf8595f-0d2c-471d-90dd-63425baa0510" (UID: "fbf8595f-0d2c-471d-90dd-63425baa0510"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:31:27.347951 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:27.347929 4908 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbf8595f-0d2c-471d-90dd-63425baa0510-credential-socket\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:31:27.348051 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:27.347956 4908 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbf8595f-0d2c-471d-90dd-63425baa0510-workload-socket\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:31:27.348051 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:27.347973 4908 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbf8595f-0d2c-471d-90dd-63425baa0510-workload-certs\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:31:27.348051 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:27.347990 4908 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbf8595f-0d2c-471d-90dd-63425baa0510-istio-data\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:31:27.348051 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:27.348002 4908 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbf8595f-0d2c-471d-90dd-63425baa0510-istiod-ca-cert\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:31:27.349295 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:27.349264 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbf8595f-0d2c-471d-90dd-63425baa0510-kube-api-access-vvtww" pod "fbf8595f-0d2c-471d-90dd-63425baa0510" (UID: "fbf8595f-0d2c-471d-90dd-63425baa0510"). InnerVolumeSpecName "kube-api-access-vvtww". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:31:27.349415 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:27.349353 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbf8595f-0d2c-471d-90dd-63425baa0510-istio-envoy" pod "fbf8595f-0d2c-471d-90dd-63425baa0510" (UID: "fbf8595f-0d2c-471d-90dd-63425baa0510"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:31:27.349474 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:27.349426 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbf8595f-0d2c-471d-90dd-63425baa0510-istio-token" pod "fbf8595f-0d2c-471d-90dd-63425baa0510" (UID: "fbf8595f-0d2c-471d-90dd-63425baa0510"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:31:27.349513 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:27.349477 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fbf8595f-0d2c-471d-90dd-63425baa0510-istio-podinfo" pod "fbf8595f-0d2c-471d-90dd-63425baa0510" (UID: "fbf8595f-0d2c-471d-90dd-63425baa0510"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 12:31:27.449208 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:27.449156 4908 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbf8595f-0d2c-471d-90dd-63425baa0510-istio-podinfo\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:31:27.449208 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:27.449179 4908 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbf8595f-0d2c-471d-90dd-63425baa0510-istio-envoy\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:31:27.449208 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:27.449189 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvtww\" (UniqueName: \"kubernetes.io/projected/fbf8595f-0d2c-471d-90dd-63425baa0510-kube-api-access-vvtww\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:31:27.449208 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:27.449200 4908 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbf8595f-0d2c-471d-90dd-63425baa0510-istio-token\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:31:27.533721 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:27.533690 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-bnhn5"] Oct 30 12:31:27.545441 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:27.545413 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-bnhn5"] Oct 30 12:31:27.608269 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:27.608234 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbf8595f-0d2c-471d-90dd-63425baa0510" path="/var/lib/kubelet/pods/fbf8595f-0d2c-471d-90dd-63425baa0510/volumes" Oct 30 12:31:30.402943 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:30.402801 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7f78d4-c859fdc56-jzwrn"] Oct 30 12:31:30.403595 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:30.403113 4908 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-7f78d4-c859fdc56-jzwrn" podUID="a0fdb965-0b23-4560-8e89-b41e8c926cd8" containerName="payload-processing" containerID="cri-o://b7e79e962b872168a1e3e9473e5565cfa4f5d97136e493da8ab5fc6549c719c2" gracePeriod=30 Oct 30 12:31:30.422572 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:30.422537 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7f78d4-fb954f9d7-2b2qr"] Oct 30 12:31:30.423074 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:30.422993 4908 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7f78d4-fb954f9d7-2b2qr" podUID="7b1f1ddf-f604-4b35-b246-9d40ff5fa84c" containerName="payload-pre-processing" containerID="cri-o://b2215a41f390e74b4fa9da0c74491ed45a8ae4f2ea3e4cfe98d9e80b3635986f" gracePeriod=30 Oct 30 12:31:30.888480 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:30.888449 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-7f78d4-c859fdc56-jzwrn" Oct 30 12:31:30.920288 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:30.920158 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0fdb965-0b23-4560-8e89-b41e8c926cd8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0fdb965-0b23-4560-8e89-b41e8c926cd8-plugins-config-volume\") pod \"a0fdb965-0b23-4560-8e89-b41e8c926cd8\" (UID: \"a0fdb965-0b23-4560-8e89-b41e8c926cd8\") " Oct 30 12:31:30.920288 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:30.920216 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0fdb965-0b23-4560-8e89-b41e8c926cd8-kube-api-access-lz4hq\" (UniqueName: \"kubernetes.io/projected/a0fdb965-0b23-4560-8e89-b41e8c926cd8-kube-api-access-lz4hq\") pod \"a0fdb965-0b23-4560-8e89-b41e8c926cd8\" (UID: \"a0fdb965-0b23-4560-8e89-b41e8c926cd8\") " Oct 30 12:31:30.920555 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:30.920529 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0fdb965-0b23-4560-8e89-b41e8c926cd8-plugins-config-volume" pod "a0fdb965-0b23-4560-8e89-b41e8c926cd8" (UID: "a0fdb965-0b23-4560-8e89-b41e8c926cd8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:31:30.920773 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:30.920756 4908 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0fdb965-0b23-4560-8e89-b41e8c926cd8-plugins-config-volume\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:31:30.922100 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:30.922074 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0fdb965-0b23-4560-8e89-b41e8c926cd8-kube-api-access-lz4hq" pod "a0fdb965-0b23-4560-8e89-b41e8c926cd8" (UID: "a0fdb965-0b23-4560-8e89-b41e8c926cd8"). InnerVolumeSpecName "kube-api-access-lz4hq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:31:30.928033 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:30.928013 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7f78d4-fb954f9d7-2b2qr" Oct 30 12:31:31.021967 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:31.021935 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b1f1ddf-f604-4b35-b246-9d40ff5fa84c-kube-api-access-n64hf\" (UniqueName: \"kubernetes.io/projected/7b1f1ddf-f604-4b35-b246-9d40ff5fa84c-kube-api-access-n64hf\") pod \"7b1f1ddf-f604-4b35-b246-9d40ff5fa84c\" (UID: \"7b1f1ddf-f604-4b35-b246-9d40ff5fa84c\") " Oct 30 12:31:31.022167 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:31.022067 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b1f1ddf-f604-4b35-b246-9d40ff5fa84c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b1f1ddf-f604-4b35-b246-9d40ff5fa84c-plugins-config-volume\") pod \"7b1f1ddf-f604-4b35-b246-9d40ff5fa84c\" (UID: \"7b1f1ddf-f604-4b35-b246-9d40ff5fa84c\") " Oct 30 12:31:31.022772 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:31.022746 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lz4hq\" (UniqueName: \"kubernetes.io/projected/a0fdb965-0b23-4560-8e89-b41e8c926cd8-kube-api-access-lz4hq\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:31:31.023136 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:31.023105 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b1f1ddf-f604-4b35-b246-9d40ff5fa84c-plugins-config-volume" pod "7b1f1ddf-f604-4b35-b246-9d40ff5fa84c" (UID: "7b1f1ddf-f604-4b35-b246-9d40ff5fa84c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:31:31.025153 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:31.025127 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b1f1ddf-f604-4b35-b246-9d40ff5fa84c-kube-api-access-n64hf" pod "7b1f1ddf-f604-4b35-b246-9d40ff5fa84c" (UID: "7b1f1ddf-f604-4b35-b246-9d40ff5fa84c"). InnerVolumeSpecName "kube-api-access-n64hf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:31:31.098871 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:31.098819 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7f78d4-ddkzg"] Oct 30 12:31:31.111267 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:31.111240 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7f78d4-ddkzg"] Oct 30 12:31:31.123575 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:31.123550 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n64hf\" (UniqueName: \"kubernetes.io/projected/7b1f1ddf-f604-4b35-b246-9d40ff5fa84c-kube-api-access-n64hf\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:31:31.123661 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:31.123585 4908 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b1f1ddf-f604-4b35-b246-9d40ff5fa84c-plugins-config-volume\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:31:31.221607 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:31.221579 4908 generic.go:356] "Generic (PLEG): container finished" podID="a0fdb965-0b23-4560-8e89-b41e8c926cd8" containerID="b7e79e962b872168a1e3e9473e5565cfa4f5d97136e493da8ab5fc6549c719c2" exitCode=0 Oct 30 12:31:31.221789 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:31.221626 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-7f78d4-c859fdc56-jzwrn" Oct 30 12:31:31.221789 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:31.221660 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-7f78d4-c859fdc56-jzwrn" event={"ID":"a0fdb965-0b23-4560-8e89-b41e8c926cd8","Type":"ContainerDied","Data":"b7e79e962b872168a1e3e9473e5565cfa4f5d97136e493da8ab5fc6549c719c2"} Oct 30 12:31:31.221789 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:31.221708 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-7f78d4-c859fdc56-jzwrn" event={"ID":"a0fdb965-0b23-4560-8e89-b41e8c926cd8","Type":"ContainerDied","Data":"ce355122a903b4320ad4d4012250941c602be0b30da48fb3efa06760941613db"} Oct 30 12:31:31.221789 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:31.221736 4908 scope.go:122] "RemoveContainer" containerID="b7e79e962b872168a1e3e9473e5565cfa4f5d97136e493da8ab5fc6549c719c2" Oct 30 12:31:31.223223 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:31.223198 4908 generic.go:356] "Generic (PLEG): container finished" podID="7b1f1ddf-f604-4b35-b246-9d40ff5fa84c" containerID="b2215a41f390e74b4fa9da0c74491ed45a8ae4f2ea3e4cfe98d9e80b3635986f" exitCode=0 Oct 30 12:31:31.223333 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:31.223235 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7f78d4-fb954f9d7-2b2qr" Oct 30 12:31:31.223333 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:31.223280 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7f78d4-fb954f9d7-2b2qr" event={"ID":"7b1f1ddf-f604-4b35-b246-9d40ff5fa84c","Type":"ContainerDied","Data":"b2215a41f390e74b4fa9da0c74491ed45a8ae4f2ea3e4cfe98d9e80b3635986f"} Oct 30 12:31:31.223333 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:31.223330 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7f78d4-fb954f9d7-2b2qr" event={"ID":"7b1f1ddf-f604-4b35-b246-9d40ff5fa84c","Type":"ContainerDied","Data":"575cf392461c2f4f803f1049080a3b3257abdcd1e9a319746d6373de4ebead30"} Oct 30 12:31:31.232560 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:31.232539 4908 scope.go:122] "RemoveContainer" containerID="b7e79e962b872168a1e3e9473e5565cfa4f5d97136e493da8ab5fc6549c719c2" Oct 30 12:31:31.232872 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:31:31.232816 4908 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7e79e962b872168a1e3e9473e5565cfa4f5d97136e493da8ab5fc6549c719c2\": container with ID starting with b7e79e962b872168a1e3e9473e5565cfa4f5d97136e493da8ab5fc6549c719c2 not found: ID does not exist" containerID="b7e79e962b872168a1e3e9473e5565cfa4f5d97136e493da8ab5fc6549c719c2" Oct 30 12:31:31.232872 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:31.232860 4908 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7e79e962b872168a1e3e9473e5565cfa4f5d97136e493da8ab5fc6549c719c2"} err="failed to get container status \"b7e79e962b872168a1e3e9473e5565cfa4f5d97136e493da8ab5fc6549c719c2\": rpc error: code = NotFound desc = could not find container \"b7e79e962b872168a1e3e9473e5565cfa4f5d97136e493da8ab5fc6549c719c2\": container with ID starting with b7e79e962b872168a1e3e9473e5565cfa4f5d97136e493da8ab5fc6549c719c2 not found: ID does not exist" Oct 30 12:31:31.233000 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:31.232878 4908 scope.go:122] "RemoveContainer" containerID="b2215a41f390e74b4fa9da0c74491ed45a8ae4f2ea3e4cfe98d9e80b3635986f" Oct 30 12:31:31.242115 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:31.242096 4908 scope.go:122] "RemoveContainer" containerID="b2215a41f390e74b4fa9da0c74491ed45a8ae4f2ea3e4cfe98d9e80b3635986f" Oct 30 12:31:31.242350 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:31:31.242331 4908 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2215a41f390e74b4fa9da0c74491ed45a8ae4f2ea3e4cfe98d9e80b3635986f\": container with ID starting with b2215a41f390e74b4fa9da0c74491ed45a8ae4f2ea3e4cfe98d9e80b3635986f not found: ID does not exist" containerID="b2215a41f390e74b4fa9da0c74491ed45a8ae4f2ea3e4cfe98d9e80b3635986f" Oct 30 12:31:31.242395 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:31.242356 4908 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2215a41f390e74b4fa9da0c74491ed45a8ae4f2ea3e4cfe98d9e80b3635986f"} err="failed to get container status \"b2215a41f390e74b4fa9da0c74491ed45a8ae4f2ea3e4cfe98d9e80b3635986f\": rpc error: code = NotFound desc = could not find container \"b2215a41f390e74b4fa9da0c74491ed45a8ae4f2ea3e4cfe98d9e80b3635986f\": container with ID starting with b2215a41f390e74b4fa9da0c74491ed45a8ae4f2ea3e4cfe98d9e80b3635986f not found: ID does not exist" Oct 30 12:31:31.273252 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:31.273194 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7f78d4-fb954f9d7-2b2qr"] Oct 30 12:31:31.281790 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:31.281765 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7f78d4-fb954f9d7-2b2qr"] Oct 30 12:31:31.313958 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:31.313938 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7f78d4-c859fdc56-jzwrn"] Oct 30 12:31:31.319576 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:31.319557 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7f78d4-c859fdc56-jzwrn"] Oct 30 12:31:31.605919 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:31.605847 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45a55357-916a-402b-9c5a-cee4b3403b55" path="/var/lib/kubelet/pods/45a55357-916a-402b-9c5a-cee4b3403b55/volumes" Oct 30 12:31:31.606307 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:31.606246 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b1f1ddf-f604-4b35-b246-9d40ff5fa84c" path="/var/lib/kubelet/pods/7b1f1ddf-f604-4b35-b246-9d40ff5fa84c/volumes" Oct 30 12:31:31.606596 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:31.606582 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0fdb965-0b23-4560-8e89-b41e8c926cd8" path="/var/lib/kubelet/pods/a0fdb965-0b23-4560-8e89-b41e8c926cd8/volumes" Oct 30 12:31:31.814939 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:31.814904 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-264ed6ef-jvzzz"] Oct 30 12:31:31.815495 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:31.815468 4908 cpu_manager.go:414] "containerMap: removing container" podUID="a0fdb965-0b23-4560-8e89-b41e8c926cd8" containerName="payload-processing" Oct 30 12:31:31.815608 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:31.815496 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0fdb965-0b23-4560-8e89-b41e8c926cd8" containerName="payload-processing" Oct 30 12:31:31.815608 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:31.815544 4908 cpu_manager.go:414] "containerMap: removing container" podUID="62b98818-9688-4fee-9742-529d1a6927d0" containerName="revoke-keys" Oct 30 12:31:31.815608 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:31.815553 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62b98818-9688-4fee-9742-529d1a6927d0" containerName="revoke-keys" Oct 30 12:31:31.815608 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:31.815565 4908 cpu_manager.go:414] "containerMap: removing container" podUID="7b1f1ddf-f604-4b35-b246-9d40ff5fa84c" containerName="payload-pre-processing" Oct 30 12:31:31.815608 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:31.815572 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b1f1ddf-f604-4b35-b246-9d40ff5fa84c" containerName="payload-pre-processing" Oct 30 12:31:31.815608 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:31.815587 4908 cpu_manager.go:414] "containerMap: removing container" podUID="fbf8595f-0d2c-471d-90dd-63425baa0510" containerName="istio-proxy" Oct 30 12:31:31.815608 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:31.815594 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbf8595f-0d2c-471d-90dd-63425baa0510" containerName="istio-proxy" Oct 30 12:31:31.815993 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:31.815719 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="62b98818-9688-4fee-9742-529d1a6927d0" containerName="revoke-keys" Oct 30 12:31:31.815993 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:31.815735 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b1f1ddf-f604-4b35-b246-9d40ff5fa84c" containerName="payload-pre-processing" Oct 30 12:31:31.815993 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:31.815748 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbf8595f-0d2c-471d-90dd-63425baa0510" containerName="istio-proxy" Oct 30 12:31:31.815993 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:31.815798 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0fdb965-0b23-4560-8e89-b41e8c926cd8" containerName="payload-processing" Oct 30 12:31:31.820406 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:31.820386 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-264ed6ef-jvzzz" Oct 30 12:31:31.838863 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:31.838841 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-264ed6ef-jvzzz"] Oct 30 12:31:31.845723 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:31.845703 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 12:31:31.845723 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:31.845715 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 12:31:31.845989 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:31.845972 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s75hq\"" Oct 30 12:31:31.846048 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:31.845991 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 12:31:31.931555 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:31.931479 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7tbd\" (UniqueName: \"kubernetes.io/projected/6fe995aa-83ea-4d67-b827-780415870c80-kube-api-access-m7tbd\") pod \"maas-api-revoke-sub-264ed6ef-jvzzz\" (UID: \"6fe995aa-83ea-4d67-b827-780415870c80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-264ed6ef-jvzzz" Oct 30 12:31:31.931673 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:31.931601 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6fe995aa-83ea-4d67-b827-780415870c80-maas-api-xks-ca\") pod \"maas-api-revoke-sub-264ed6ef-jvzzz\" (UID: \"6fe995aa-83ea-4d67-b827-780415870c80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-264ed6ef-jvzzz" Oct 30 12:31:31.931673 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:31.931639 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6fe995aa-83ea-4d67-b827-780415870c80-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-264ed6ef-jvzzz\" (UID: \"6fe995aa-83ea-4d67-b827-780415870c80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-264ed6ef-jvzzz" Oct 30 12:31:32.032420 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:32.032392 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7tbd\" (UniqueName: \"kubernetes.io/projected/6fe995aa-83ea-4d67-b827-780415870c80-kube-api-access-m7tbd\") pod \"maas-api-revoke-sub-264ed6ef-jvzzz\" (UID: \"6fe995aa-83ea-4d67-b827-780415870c80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-264ed6ef-jvzzz" Oct 30 12:31:32.032578 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:32.032454 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6fe995aa-83ea-4d67-b827-780415870c80-maas-api-xks-ca\") pod \"maas-api-revoke-sub-264ed6ef-jvzzz\" (UID: \"6fe995aa-83ea-4d67-b827-780415870c80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-264ed6ef-jvzzz" Oct 30 12:31:32.032578 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:32.032476 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6fe995aa-83ea-4d67-b827-780415870c80-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-264ed6ef-jvzzz\" (UID: \"6fe995aa-83ea-4d67-b827-780415870c80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-264ed6ef-jvzzz" Oct 30 12:31:32.033117 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:32.033097 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6fe995aa-83ea-4d67-b827-780415870c80-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-264ed6ef-jvzzz\" (UID: \"6fe995aa-83ea-4d67-b827-780415870c80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-264ed6ef-jvzzz" Oct 30 12:31:32.034712 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:32.034694 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6fe995aa-83ea-4d67-b827-780415870c80-maas-api-xks-ca\") pod \"maas-api-revoke-sub-264ed6ef-jvzzz\" (UID: \"6fe995aa-83ea-4d67-b827-780415870c80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-264ed6ef-jvzzz" Oct 30 12:31:32.054706 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:32.054680 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7tbd\" (UniqueName: \"kubernetes.io/projected/6fe995aa-83ea-4d67-b827-780415870c80-kube-api-access-m7tbd\") pod \"maas-api-revoke-sub-264ed6ef-jvzzz\" (UID: \"6fe995aa-83ea-4d67-b827-780415870c80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-264ed6ef-jvzzz" Oct 30 12:31:32.133637 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:32.133610 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-264ed6ef-jvzzz" Oct 30 12:31:32.492039 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:32.490701 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-264ed6ef-jvzzz"] Oct 30 12:31:32.504900 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:31:32.504868 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fe995aa_83ea_4d67_b827_780415870c80.slice/crio-54b7229aed145eac648a3770d6252fb9b691912385099435bfad3b5c1c096216 WatchSource:0}: Error finding container 54b7229aed145eac648a3770d6252fb9b691912385099435bfad3b5c1c096216: Status 404 returned error can't find the container with id 54b7229aed145eac648a3770d6252fb9b691912385099435bfad3b5c1c096216 Oct 30 12:31:33.258616 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:33.258575 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-264ed6ef-jvzzz" event={"ID":"6fe995aa-83ea-4d67-b827-780415870c80","Type":"ContainerStarted","Data":"8558774ba3c8b13ceaca17d6c3a13abd2a48c16c9f2cc8de7c2b2c4fa9a37324"} Oct 30 12:31:33.259061 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:33.258634 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-264ed6ef-jvzzz" event={"ID":"6fe995aa-83ea-4d67-b827-780415870c80","Type":"ContainerStarted","Data":"54b7229aed145eac648a3770d6252fb9b691912385099435bfad3b5c1c096216"} Oct 30 12:31:33.308140 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:33.308076 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-264ed6ef-jvzzz" podStartSLOduration=2.308063928 podStartE2EDuration="2.308063928s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:31:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:31:33.296069693 +0000 UTC m=+2249.810498464" watchObservedRunningTime="2026-10-30 12:31:33.308063928 +0000 UTC m=+2249.822492671" Oct 30 12:31:34.268353 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:34.268314 4908 generic.go:356] "Generic (PLEG): container finished" podID="6fe995aa-83ea-4d67-b827-780415870c80" containerID="8558774ba3c8b13ceaca17d6c3a13abd2a48c16c9f2cc8de7c2b2c4fa9a37324" exitCode=0 Oct 30 12:31:34.268789 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:34.268401 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-264ed6ef-jvzzz" event={"ID":"6fe995aa-83ea-4d67-b827-780415870c80","Type":"ContainerDied","Data":"8558774ba3c8b13ceaca17d6c3a13abd2a48c16c9f2cc8de7c2b2c4fa9a37324"} Oct 30 12:31:35.422783 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:35.422758 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-264ed6ef-jvzzz" Oct 30 12:31:35.472494 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:35.472469 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6fe995aa-83ea-4d67-b827-780415870c80-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6fe995aa-83ea-4d67-b827-780415870c80-maas-api-openshift-ca\") pod \"6fe995aa-83ea-4d67-b827-780415870c80\" (UID: \"6fe995aa-83ea-4d67-b827-780415870c80\") " Oct 30 12:31:35.472648 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:35.472521 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6fe995aa-83ea-4d67-b827-780415870c80-kube-api-access-m7tbd\" (UniqueName: \"kubernetes.io/projected/6fe995aa-83ea-4d67-b827-780415870c80-kube-api-access-m7tbd\") pod \"6fe995aa-83ea-4d67-b827-780415870c80\" (UID: \"6fe995aa-83ea-4d67-b827-780415870c80\") " Oct 30 12:31:35.472648 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:35.472552 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6fe995aa-83ea-4d67-b827-780415870c80-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6fe995aa-83ea-4d67-b827-780415870c80-maas-api-xks-ca\") pod \"6fe995aa-83ea-4d67-b827-780415870c80\" (UID: \"6fe995aa-83ea-4d67-b827-780415870c80\") " Oct 30 12:31:35.472920 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:35.472892 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fe995aa-83ea-4d67-b827-780415870c80-maas-api-openshift-ca" pod "6fe995aa-83ea-4d67-b827-780415870c80" (UID: "6fe995aa-83ea-4d67-b827-780415870c80"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:31:35.473208 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:35.473187 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6fe995aa-83ea-4d67-b827-780415870c80-maas-api-openshift-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:31:35.475226 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:35.475191 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6fe995aa-83ea-4d67-b827-780415870c80-maas-api-xks-ca" pod "6fe995aa-83ea-4d67-b827-780415870c80" (UID: "6fe995aa-83ea-4d67-b827-780415870c80"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:31:35.475354 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:35.475330 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fe995aa-83ea-4d67-b827-780415870c80-kube-api-access-m7tbd" pod "6fe995aa-83ea-4d67-b827-780415870c80" (UID: "6fe995aa-83ea-4d67-b827-780415870c80"). InnerVolumeSpecName "kube-api-access-m7tbd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:31:35.574400 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:35.574331 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7tbd\" (UniqueName: \"kubernetes.io/projected/6fe995aa-83ea-4d67-b827-780415870c80-kube-api-access-m7tbd\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:31:35.574400 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:35.574356 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6fe995aa-83ea-4d67-b827-780415870c80-maas-api-xks-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:31:36.285460 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:36.285432 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-264ed6ef-jvzzz" Oct 30 12:31:36.285628 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:36.285475 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-264ed6ef-jvzzz" event={"ID":"6fe995aa-83ea-4d67-b827-780415870c80","Type":"ContainerDied","Data":"54b7229aed145eac648a3770d6252fb9b691912385099435bfad3b5c1c096216"} Oct 30 12:31:36.285628 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:36.285535 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="54b7229aed145eac648a3770d6252fb9b691912385099435bfad3b5c1c096216" Oct 30 12:31:36.471431 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:36.471399 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9wwfc"] Oct 30 12:31:40.994634 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:40.994600 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-1e9675-6q54x"] Oct 30 12:31:40.999597 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:40.999567 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-1e9675-6q54x"] Oct 30 12:31:41.609070 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:41.609033 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62b98818-9688-4fee-9742-529d1a6927d0" path="/var/lib/kubelet/pods/62b98818-9688-4fee-9742-529d1a6927d0/volumes" Oct 30 12:31:45.087199 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:45.087162 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5c57af-h96hh"] Oct 30 12:31:45.087766 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:45.087747 4908 cpu_manager.go:414] "containerMap: removing container" podUID="6fe995aa-83ea-4d67-b827-780415870c80" containerName="revoke-keys" Oct 30 12:31:45.087809 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:45.087767 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6fe995aa-83ea-4d67-b827-780415870c80" containerName="revoke-keys" Oct 30 12:31:45.087904 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:45.087893 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="6fe995aa-83ea-4d67-b827-780415870c80" containerName="revoke-keys" Oct 30 12:31:45.092543 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:45.092522 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5c57af-h96hh" Oct 30 12:31:45.110207 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:45.110182 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5c57af-h96hh"] Oct 30 12:31:45.117100 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:45.117077 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 12:31:45.117271 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:45.117251 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 12:31:45.192907 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:45.192885 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 12:31:45.193193 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:45.193176 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s75hq\"" Oct 30 12:31:45.242945 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:45.242921 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkhvs\" (UniqueName: \"kubernetes.io/projected/e545bc36-bf6e-49c3-ad8f-9e1d57e62c6f-kube-api-access-jkhvs\") pod \"maas-api-revoke-keys-e2e-ipp-b-5c57af-h96hh\" (UID: \"e545bc36-bf6e-49c3-ad8f-9e1d57e62c6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5c57af-h96hh" Oct 30 12:31:45.243043 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:45.243018 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e545bc36-bf6e-49c3-ad8f-9e1d57e62c6f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-5c57af-h96hh\" (UID: \"e545bc36-bf6e-49c3-ad8f-9e1d57e62c6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5c57af-h96hh" Oct 30 12:31:45.243094 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:45.243049 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e545bc36-bf6e-49c3-ad8f-9e1d57e62c6f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-5c57af-h96hh\" (UID: \"e545bc36-bf6e-49c3-ad8f-9e1d57e62c6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5c57af-h96hh" Oct 30 12:31:45.262328 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:45.262306 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f4d3590-7lc6p"] Oct 30 12:31:45.266240 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:45.266223 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f4d3590-7lc6p" Oct 30 12:31:45.284763 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:45.284740 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f4d3590-7lc6p"] Oct 30 12:31:45.344235 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:45.344161 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkhvs\" (UniqueName: \"kubernetes.io/projected/e545bc36-bf6e-49c3-ad8f-9e1d57e62c6f-kube-api-access-jkhvs\") pod \"maas-api-revoke-keys-e2e-ipp-b-5c57af-h96hh\" (UID: \"e545bc36-bf6e-49c3-ad8f-9e1d57e62c6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5c57af-h96hh" Oct 30 12:31:45.344235 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:45.344214 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e545bc36-bf6e-49c3-ad8f-9e1d57e62c6f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-5c57af-h96hh\" (UID: \"e545bc36-bf6e-49c3-ad8f-9e1d57e62c6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5c57af-h96hh" Oct 30 12:31:45.344430 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:45.344243 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e545bc36-bf6e-49c3-ad8f-9e1d57e62c6f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-5c57af-h96hh\" (UID: \"e545bc36-bf6e-49c3-ad8f-9e1d57e62c6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5c57af-h96hh" Oct 30 12:31:45.344430 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:45.344262 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ace46019-d978-4707-ab77-e1b4dc45dc32-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2f4d3590-7lc6p\" (UID: \"ace46019-d978-4707-ab77-e1b4dc45dc32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f4d3590-7lc6p" Oct 30 12:31:45.344430 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:45.344326 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbtmr\" (UniqueName: \"kubernetes.io/projected/ace46019-d978-4707-ab77-e1b4dc45dc32-kube-api-access-pbtmr\") pod \"maas-api-revoke-sub-2f4d3590-7lc6p\" (UID: \"ace46019-d978-4707-ab77-e1b4dc45dc32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f4d3590-7lc6p" Oct 30 12:31:45.344585 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:45.344514 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ace46019-d978-4707-ab77-e1b4dc45dc32-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2f4d3590-7lc6p\" (UID: \"ace46019-d978-4707-ab77-e1b4dc45dc32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f4d3590-7lc6p" Oct 30 12:31:45.345042 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:45.345018 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e545bc36-bf6e-49c3-ad8f-9e1d57e62c6f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-5c57af-h96hh\" (UID: \"e545bc36-bf6e-49c3-ad8f-9e1d57e62c6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5c57af-h96hh" Oct 30 12:31:45.346526 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:45.346506 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e545bc36-bf6e-49c3-ad8f-9e1d57e62c6f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-5c57af-h96hh\" (UID: \"e545bc36-bf6e-49c3-ad8f-9e1d57e62c6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5c57af-h96hh" Oct 30 12:31:45.366412 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:45.366393 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkhvs\" (UniqueName: \"kubernetes.io/projected/e545bc36-bf6e-49c3-ad8f-9e1d57e62c6f-kube-api-access-jkhvs\") pod \"maas-api-revoke-keys-e2e-ipp-b-5c57af-h96hh\" (UID: \"e545bc36-bf6e-49c3-ad8f-9e1d57e62c6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5c57af-h96hh" Oct 30 12:31:45.405857 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:45.405810 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5c57af-h96hh" Oct 30 12:31:45.446096 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:45.446067 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ace46019-d978-4707-ab77-e1b4dc45dc32-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2f4d3590-7lc6p\" (UID: \"ace46019-d978-4707-ab77-e1b4dc45dc32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f4d3590-7lc6p" Oct 30 12:31:45.446230 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:45.446127 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbtmr\" (UniqueName: \"kubernetes.io/projected/ace46019-d978-4707-ab77-e1b4dc45dc32-kube-api-access-pbtmr\") pod \"maas-api-revoke-sub-2f4d3590-7lc6p\" (UID: \"ace46019-d978-4707-ab77-e1b4dc45dc32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f4d3590-7lc6p" Oct 30 12:31:45.446230 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:45.446187 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ace46019-d978-4707-ab77-e1b4dc45dc32-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2f4d3590-7lc6p\" (UID: \"ace46019-d978-4707-ab77-e1b4dc45dc32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f4d3590-7lc6p" Oct 30 12:31:45.447262 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:45.447222 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ace46019-d978-4707-ab77-e1b4dc45dc32-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2f4d3590-7lc6p\" (UID: \"ace46019-d978-4707-ab77-e1b4dc45dc32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f4d3590-7lc6p" Oct 30 12:31:45.460981 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:45.460903 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ace46019-d978-4707-ab77-e1b4dc45dc32-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2f4d3590-7lc6p\" (UID: \"ace46019-d978-4707-ab77-e1b4dc45dc32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f4d3590-7lc6p" Oct 30 12:31:45.470970 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:45.470891 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbtmr\" (UniqueName: \"kubernetes.io/projected/ace46019-d978-4707-ab77-e1b4dc45dc32-kube-api-access-pbtmr\") pod \"maas-api-revoke-sub-2f4d3590-7lc6p\" (UID: \"ace46019-d978-4707-ab77-e1b4dc45dc32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f4d3590-7lc6p" Oct 30 12:31:45.579225 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:45.579176 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f4d3590-7lc6p" Oct 30 12:31:45.724943 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:45.722529 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5c57af-h96hh"] Oct 30 12:31:45.726063 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:31:45.726033 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode545bc36_bf6e_49c3_ad8f_9e1d57e62c6f.slice/crio-e7d4eec4b42883d516505dbbba2a7fa6bb59bb70a2ee748225aa2d17aa67bee0 WatchSource:0}: Error finding container e7d4eec4b42883d516505dbbba2a7fa6bb59bb70a2ee748225aa2d17aa67bee0: Status 404 returned error can't find the container with id e7d4eec4b42883d516505dbbba2a7fa6bb59bb70a2ee748225aa2d17aa67bee0 Oct 30 12:31:45.891879 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:45.891799 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f4d3590-7lc6p"] Oct 30 12:31:45.895279 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:31:45.895249 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podace46019_d978_4707_ab77_e1b4dc45dc32.slice/crio-f8b9b6776bd158368971fd8db40bf28bcca2cbba31005304f9d0ed86fbb039bc WatchSource:0}: Error finding container f8b9b6776bd158368971fd8db40bf28bcca2cbba31005304f9d0ed86fbb039bc: Status 404 returned error can't find the container with id f8b9b6776bd158368971fd8db40bf28bcca2cbba31005304f9d0ed86fbb039bc Oct 30 12:31:46.369758 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:46.369721 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f4d3590-7lc6p" event={"ID":"ace46019-d978-4707-ab77-e1b4dc45dc32","Type":"ContainerStarted","Data":"82417fef55ee8287db5d8f937e8fd624bd887a864a64b262e54cdecaa56d4bb9"} Oct 30 12:31:46.370245 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:46.369778 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f4d3590-7lc6p" event={"ID":"ace46019-d978-4707-ab77-e1b4dc45dc32","Type":"ContainerStarted","Data":"f8b9b6776bd158368971fd8db40bf28bcca2cbba31005304f9d0ed86fbb039bc"} Oct 30 12:31:46.371558 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:46.371526 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5c57af-h96hh" event={"ID":"e545bc36-bf6e-49c3-ad8f-9e1d57e62c6f","Type":"ContainerStarted","Data":"7dc80f4dccb74baa4941bcb7daa376d5406bdc1aef0c1d1c035bc8e138445f1d"} Oct 30 12:31:46.371689 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:46.371578 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5c57af-h96hh" event={"ID":"e545bc36-bf6e-49c3-ad8f-9e1d57e62c6f","Type":"ContainerStarted","Data":"e7d4eec4b42883d516505dbbba2a7fa6bb59bb70a2ee748225aa2d17aa67bee0"} Oct 30 12:31:46.411154 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:46.411086 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f4d3590-7lc6p" podStartSLOduration=1.4110690049999999 podStartE2EDuration="1.411069005s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:31:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:31:46.407703054 +0000 UTC m=+2262.922131800" watchObservedRunningTime="2026-10-30 12:31:46.411069005 +0000 UTC m=+2262.925497752" Oct 30 12:31:46.447319 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:46.447269 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5c57af-h96hh" podStartSLOduration=1.447256314 podStartE2EDuration="1.447256314s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:31:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:31:46.44508414 +0000 UTC m=+2262.959512889" watchObservedRunningTime="2026-10-30 12:31:46.447256314 +0000 UTC m=+2262.961685060" Oct 30 12:31:47.381089 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:47.381051 4908 generic.go:356] "Generic (PLEG): container finished" podID="ace46019-d978-4707-ab77-e1b4dc45dc32" containerID="82417fef55ee8287db5d8f937e8fd624bd887a864a64b262e54cdecaa56d4bb9" exitCode=0 Oct 30 12:31:47.381514 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:47.381141 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f4d3590-7lc6p" event={"ID":"ace46019-d978-4707-ab77-e1b4dc45dc32","Type":"ContainerDied","Data":"82417fef55ee8287db5d8f937e8fd624bd887a864a64b262e54cdecaa56d4bb9"} Oct 30 12:31:47.382992 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:47.382967 4908 generic.go:356] "Generic (PLEG): container finished" podID="e545bc36-bf6e-49c3-ad8f-9e1d57e62c6f" containerID="7dc80f4dccb74baa4941bcb7daa376d5406bdc1aef0c1d1c035bc8e138445f1d" exitCode=0 Oct 30 12:31:47.383134 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:47.383048 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5c57af-h96hh" event={"ID":"e545bc36-bf6e-49c3-ad8f-9e1d57e62c6f","Type":"ContainerDied","Data":"7dc80f4dccb74baa4941bcb7daa376d5406bdc1aef0c1d1c035bc8e138445f1d"} Oct 30 12:31:48.401953 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:48.401919 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-1e9675-openshift-default-7d7c758549-6hhbg"] Oct 30 12:31:48.402394 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:48.402151 4908 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-1e9675-openshift-default-7d7c758549-6hhbg" podUID="62b94b35-a1bb-4eb2-9bc0-da2760557b3d" containerName="istio-proxy" containerID="cri-o://a45855efd1f7e4fd82b582d43c7deefe3da174c5f5a3aca8912ebbbcfc01f517" gracePeriod=30 Oct 30 12:31:48.683147 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:48.683120 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f4d3590-7lc6p" Oct 30 12:31:48.725619 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:48.725592 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ace46019-d978-4707-ab77-e1b4dc45dc32-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ace46019-d978-4707-ab77-e1b4dc45dc32-maas-api-openshift-ca\") pod \"ace46019-d978-4707-ab77-e1b4dc45dc32\" (UID: \"ace46019-d978-4707-ab77-e1b4dc45dc32\") " Oct 30 12:31:48.725771 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:48.725649 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ace46019-d978-4707-ab77-e1b4dc45dc32-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ace46019-d978-4707-ab77-e1b4dc45dc32-maas-api-xks-ca\") pod \"ace46019-d978-4707-ab77-e1b4dc45dc32\" (UID: \"ace46019-d978-4707-ab77-e1b4dc45dc32\") " Oct 30 12:31:48.725771 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:48.725723 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ace46019-d978-4707-ab77-e1b4dc45dc32-kube-api-access-pbtmr\" (UniqueName: \"kubernetes.io/projected/ace46019-d978-4707-ab77-e1b4dc45dc32-kube-api-access-pbtmr\") pod \"ace46019-d978-4707-ab77-e1b4dc45dc32\" (UID: \"ace46019-d978-4707-ab77-e1b4dc45dc32\") " Oct 30 12:31:48.726476 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:48.726309 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ace46019-d978-4707-ab77-e1b4dc45dc32-maas-api-openshift-ca" pod "ace46019-d978-4707-ab77-e1b4dc45dc32" (UID: "ace46019-d978-4707-ab77-e1b4dc45dc32"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:31:48.729205 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:48.729181 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ace46019-d978-4707-ab77-e1b4dc45dc32-kube-api-access-pbtmr" pod "ace46019-d978-4707-ab77-e1b4dc45dc32" (UID: "ace46019-d978-4707-ab77-e1b4dc45dc32"). InnerVolumeSpecName "kube-api-access-pbtmr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:31:48.730444 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:48.730420 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ace46019-d978-4707-ab77-e1b4dc45dc32-maas-api-xks-ca" pod "ace46019-d978-4707-ab77-e1b4dc45dc32" (UID: "ace46019-d978-4707-ab77-e1b4dc45dc32"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:31:48.753248 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:48.753223 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5c57af-h96hh" Oct 30 12:31:48.844222 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:48.844186 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e545bc36-bf6e-49c3-ad8f-9e1d57e62c6f-kube-api-access-jkhvs\" (UniqueName: \"kubernetes.io/projected/e545bc36-bf6e-49c3-ad8f-9e1d57e62c6f-kube-api-access-jkhvs\") pod \"e545bc36-bf6e-49c3-ad8f-9e1d57e62c6f\" (UID: \"e545bc36-bf6e-49c3-ad8f-9e1d57e62c6f\") " Oct 30 12:31:48.844222 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:48.844271 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e545bc36-bf6e-49c3-ad8f-9e1d57e62c6f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e545bc36-bf6e-49c3-ad8f-9e1d57e62c6f-maas-api-openshift-ca\") pod \"e545bc36-bf6e-49c3-ad8f-9e1d57e62c6f\" (UID: \"e545bc36-bf6e-49c3-ad8f-9e1d57e62c6f\") " Oct 30 12:31:48.846247 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:48.845814 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e545bc36-bf6e-49c3-ad8f-9e1d57e62c6f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e545bc36-bf6e-49c3-ad8f-9e1d57e62c6f-maas-api-xks-ca\") pod \"e545bc36-bf6e-49c3-ad8f-9e1d57e62c6f\" (UID: \"e545bc36-bf6e-49c3-ad8f-9e1d57e62c6f\") " Oct 30 12:31:48.846558 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:48.846537 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ace46019-d978-4707-ab77-e1b4dc45dc32-maas-api-openshift-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:31:48.846650 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:48.846564 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ace46019-d978-4707-ab77-e1b4dc45dc32-maas-api-xks-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:31:48.846650 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:48.846580 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbtmr\" (UniqueName: \"kubernetes.io/projected/ace46019-d978-4707-ab77-e1b4dc45dc32-kube-api-access-pbtmr\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:31:48.847844 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:48.847779 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e545bc36-bf6e-49c3-ad8f-9e1d57e62c6f-maas-api-openshift-ca" pod "e545bc36-bf6e-49c3-ad8f-9e1d57e62c6f" (UID: "e545bc36-bf6e-49c3-ad8f-9e1d57e62c6f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:31:48.861507 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:48.861464 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e545bc36-bf6e-49c3-ad8f-9e1d57e62c6f-kube-api-access-jkhvs" pod "e545bc36-bf6e-49c3-ad8f-9e1d57e62c6f" (UID: "e545bc36-bf6e-49c3-ad8f-9e1d57e62c6f"). InnerVolumeSpecName "kube-api-access-jkhvs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:31:48.863475 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:48.863447 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e545bc36-bf6e-49c3-ad8f-9e1d57e62c6f-maas-api-xks-ca" pod "e545bc36-bf6e-49c3-ad8f-9e1d57e62c6f" (UID: "e545bc36-bf6e-49c3-ad8f-9e1d57e62c6f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:31:48.947864 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:48.947771 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkhvs\" (UniqueName: \"kubernetes.io/projected/e545bc36-bf6e-49c3-ad8f-9e1d57e62c6f-kube-api-access-jkhvs\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:31:48.947864 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:48.947798 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e545bc36-bf6e-49c3-ad8f-9e1d57e62c6f-maas-api-openshift-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:31:48.947864 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:48.947809 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e545bc36-bf6e-49c3-ad8f-9e1d57e62c6f-maas-api-xks-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:31:49.413839 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:49.413798 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f4d3590-7lc6p" Oct 30 12:31:49.414401 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:49.413799 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f4d3590-7lc6p" event={"ID":"ace46019-d978-4707-ab77-e1b4dc45dc32","Type":"ContainerDied","Data":"f8b9b6776bd158368971fd8db40bf28bcca2cbba31005304f9d0ed86fbb039bc"} Oct 30 12:31:49.414546 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:49.414469 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f8b9b6776bd158368971fd8db40bf28bcca2cbba31005304f9d0ed86fbb039bc" Oct 30 12:31:49.416986 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:49.416962 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5c57af-h96hh" Oct 30 12:31:49.417072 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:49.416960 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5c57af-h96hh" event={"ID":"e545bc36-bf6e-49c3-ad8f-9e1d57e62c6f","Type":"ContainerDied","Data":"e7d4eec4b42883d516505dbbba2a7fa6bb59bb70a2ee748225aa2d17aa67bee0"} Oct 30 12:31:49.417111 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:49.417090 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e7d4eec4b42883d516505dbbba2a7fa6bb59bb70a2ee748225aa2d17aa67bee0" Oct 30 12:31:49.685679 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:31:49.685601 4908 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vn6vv" containerName="extractor" Oct 30 12:31:51.257404 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:51.257372 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9933f47-s6tcr"] Oct 30 12:31:51.257891 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:51.257876 4908 cpu_manager.go:414] "containerMap: removing container" podUID="ace46019-d978-4707-ab77-e1b4dc45dc32" containerName="revoke-keys" Oct 30 12:31:51.257936 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:51.257891 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ace46019-d978-4707-ab77-e1b4dc45dc32" containerName="revoke-keys" Oct 30 12:31:51.257936 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:51.257921 4908 cpu_manager.go:414] "containerMap: removing container" podUID="e545bc36-bf6e-49c3-ad8f-9e1d57e62c6f" containerName="revoke-keys" Oct 30 12:31:51.257936 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:51.257927 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e545bc36-bf6e-49c3-ad8f-9e1d57e62c6f" containerName="revoke-keys" Oct 30 12:31:51.258036 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:51.258026 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="ace46019-d978-4707-ab77-e1b4dc45dc32" containerName="revoke-keys" Oct 30 12:31:51.258068 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:51.258042 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="e545bc36-bf6e-49c3-ad8f-9e1d57e62c6f" containerName="revoke-keys" Oct 30 12:31:51.261749 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:51.261729 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9933f47-s6tcr" Oct 30 12:31:51.276416 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:51.276386 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 12:31:51.276515 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:51.276507 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 12:31:51.276592 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:51.276544 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 12:31:51.287855 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:51.287694 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s75hq\"" Oct 30 12:31:51.291644 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:51.291620 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9933f47-s6tcr"] Oct 30 12:31:51.373783 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:51.373755 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/452ea436-3165-416d-93c1-6563af3cd263-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b9933f47-s6tcr\" (UID: \"452ea436-3165-416d-93c1-6563af3cd263\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9933f47-s6tcr" Oct 30 12:31:51.373990 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:51.373971 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/452ea436-3165-416d-93c1-6563af3cd263-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b9933f47-s6tcr\" (UID: \"452ea436-3165-416d-93c1-6563af3cd263\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9933f47-s6tcr" Oct 30 12:31:51.374034 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:51.374016 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckj6w\" (UniqueName: \"kubernetes.io/projected/452ea436-3165-416d-93c1-6563af3cd263-kube-api-access-ckj6w\") pod \"maas-api-revoke-sub-b9933f47-s6tcr\" (UID: \"452ea436-3165-416d-93c1-6563af3cd263\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9933f47-s6tcr" Oct 30 12:31:51.475521 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:51.475492 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/452ea436-3165-416d-93c1-6563af3cd263-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b9933f47-s6tcr\" (UID: \"452ea436-3165-416d-93c1-6563af3cd263\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9933f47-s6tcr" Oct 30 12:31:51.475638 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:51.475544 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckj6w\" (UniqueName: \"kubernetes.io/projected/452ea436-3165-416d-93c1-6563af3cd263-kube-api-access-ckj6w\") pod \"maas-api-revoke-sub-b9933f47-s6tcr\" (UID: \"452ea436-3165-416d-93c1-6563af3cd263\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9933f47-s6tcr" Oct 30 12:31:51.475638 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:51.475613 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/452ea436-3165-416d-93c1-6563af3cd263-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b9933f47-s6tcr\" (UID: \"452ea436-3165-416d-93c1-6563af3cd263\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9933f47-s6tcr" Oct 30 12:31:51.476437 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:51.476405 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/452ea436-3165-416d-93c1-6563af3cd263-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b9933f47-s6tcr\" (UID: \"452ea436-3165-416d-93c1-6563af3cd263\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9933f47-s6tcr" Oct 30 12:31:51.477928 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:51.477909 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/452ea436-3165-416d-93c1-6563af3cd263-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b9933f47-s6tcr\" (UID: \"452ea436-3165-416d-93c1-6563af3cd263\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9933f47-s6tcr" Oct 30 12:31:51.497077 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:51.497051 4908 patch_prober.go:29] interesting pod/e2e-praxis-b-1e9675-openshift-default-7d7c758549-6hhbg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 12:31:51.497177 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:51.497098 4908 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-1e9675-openshift-default-7d7c758549-6hhbg" podUID="62b94b35-a1bb-4eb2-9bc0-da2760557b3d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 12:31:51.498049 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:51.498030 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckj6w\" (UniqueName: \"kubernetes.io/projected/452ea436-3165-416d-93c1-6563af3cd263-kube-api-access-ckj6w\") pod \"maas-api-revoke-sub-b9933f47-s6tcr\" (UID: \"452ea436-3165-416d-93c1-6563af3cd263\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9933f47-s6tcr" Oct 30 12:31:51.575602 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:51.575541 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9933f47-s6tcr" Oct 30 12:31:51.886451 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:51.886230 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9933f47-s6tcr"] Oct 30 12:31:51.889940 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:31:51.889912 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod452ea436_3165_416d_93c1_6563af3cd263.slice/crio-20cf6be6649ebf671854d24e5da1a6e0b87d93423fd595bdb67a29859a9f54bc WatchSource:0}: Error finding container 20cf6be6649ebf671854d24e5da1a6e0b87d93423fd595bdb67a29859a9f54bc: Status 404 returned error can't find the container with id 20cf6be6649ebf671854d24e5da1a6e0b87d93423fd595bdb67a29859a9f54bc Oct 30 12:31:52.441940 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:52.441822 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9933f47-s6tcr" event={"ID":"452ea436-3165-416d-93c1-6563af3cd263","Type":"ContainerStarted","Data":"c816c843e168b8606377a972505948870e2aa37e0dd5e0f40f111834123f0ae7"} Oct 30 12:31:52.441940 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:52.441919 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9933f47-s6tcr" event={"ID":"452ea436-3165-416d-93c1-6563af3cd263","Type":"ContainerStarted","Data":"20cf6be6649ebf671854d24e5da1a6e0b87d93423fd595bdb67a29859a9f54bc"} Oct 30 12:31:52.484157 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:52.484104 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9933f47-s6tcr" podStartSLOduration=1.484086649 podStartE2EDuration="1.484086649s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:31:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:31:52.478366828 +0000 UTC m=+2268.992795575" watchObservedRunningTime="2026-10-30 12:31:52.484086649 +0000 UTC m=+2268.998515398" Oct 30 12:31:53.451718 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:53.451682 4908 generic.go:356] "Generic (PLEG): container finished" podID="452ea436-3165-416d-93c1-6563af3cd263" containerID="c816c843e168b8606377a972505948870e2aa37e0dd5e0f40f111834123f0ae7" exitCode=0 Oct 30 12:31:53.452085 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:53.451767 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9933f47-s6tcr" event={"ID":"452ea436-3165-416d-93c1-6563af3cd263","Type":"ContainerDied","Data":"c816c843e168b8606377a972505948870e2aa37e0dd5e0f40f111834123f0ae7"} Oct 30 12:31:53.693095 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:53.693065 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-1e9675-openshift-default-7d7c758549-6hhbg" Oct 30 12:31:53.810220 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:53.810185 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62b94b35-a1bb-4eb2-9bc0-da2760557b3d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62b94b35-a1bb-4eb2-9bc0-da2760557b3d-workload-socket\") pod \"62b94b35-a1bb-4eb2-9bc0-da2760557b3d\" (UID: \"62b94b35-a1bb-4eb2-9bc0-da2760557b3d\") " Oct 30 12:31:53.810220 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:53.810218 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62b94b35-a1bb-4eb2-9bc0-da2760557b3d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62b94b35-a1bb-4eb2-9bc0-da2760557b3d-workload-certs\") pod \"62b94b35-a1bb-4eb2-9bc0-da2760557b3d\" (UID: \"62b94b35-a1bb-4eb2-9bc0-da2760557b3d\") " Oct 30 12:31:53.810452 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:53.810261 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/62b94b35-a1bb-4eb2-9bc0-da2760557b3d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62b94b35-a1bb-4eb2-9bc0-da2760557b3d-istio-podinfo\") pod \"62b94b35-a1bb-4eb2-9bc0-da2760557b3d\" (UID: \"62b94b35-a1bb-4eb2-9bc0-da2760557b3d\") " Oct 30 12:31:53.810452 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:53.810279 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62b94b35-a1bb-4eb2-9bc0-da2760557b3d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62b94b35-a1bb-4eb2-9bc0-da2760557b3d-credential-socket\") pod \"62b94b35-a1bb-4eb2-9bc0-da2760557b3d\" (UID: \"62b94b35-a1bb-4eb2-9bc0-da2760557b3d\") " Oct 30 12:31:53.810452 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:53.810296 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62b94b35-a1bb-4eb2-9bc0-da2760557b3d-istio-token\" (UniqueName: \"kubernetes.io/projected/62b94b35-a1bb-4eb2-9bc0-da2760557b3d-istio-token\") pod \"62b94b35-a1bb-4eb2-9bc0-da2760557b3d\" (UID: \"62b94b35-a1bb-4eb2-9bc0-da2760557b3d\") " Oct 30 12:31:53.810452 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:53.810326 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62b94b35-a1bb-4eb2-9bc0-da2760557b3d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62b94b35-a1bb-4eb2-9bc0-da2760557b3d-istiod-ca-cert\") pod \"62b94b35-a1bb-4eb2-9bc0-da2760557b3d\" (UID: \"62b94b35-a1bb-4eb2-9bc0-da2760557b3d\") " Oct 30 12:31:53.810452 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:53.810384 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62b94b35-a1bb-4eb2-9bc0-da2760557b3d-kube-api-access-ckrlf\" (UniqueName: \"kubernetes.io/projected/62b94b35-a1bb-4eb2-9bc0-da2760557b3d-kube-api-access-ckrlf\") pod \"62b94b35-a1bb-4eb2-9bc0-da2760557b3d\" (UID: \"62b94b35-a1bb-4eb2-9bc0-da2760557b3d\") " Oct 30 12:31:53.810452 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:53.810420 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62b94b35-a1bb-4eb2-9bc0-da2760557b3d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62b94b35-a1bb-4eb2-9bc0-da2760557b3d-istio-envoy\") pod \"62b94b35-a1bb-4eb2-9bc0-da2760557b3d\" (UID: \"62b94b35-a1bb-4eb2-9bc0-da2760557b3d\") " Oct 30 12:31:53.810753 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:53.810483 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62b94b35-a1bb-4eb2-9bc0-da2760557b3d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62b94b35-a1bb-4eb2-9bc0-da2760557b3d-istio-data\") pod \"62b94b35-a1bb-4eb2-9bc0-da2760557b3d\" (UID: \"62b94b35-a1bb-4eb2-9bc0-da2760557b3d\") " Oct 30 12:31:53.810753 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:53.810513 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62b94b35-a1bb-4eb2-9bc0-da2760557b3d-workload-socket" pod "62b94b35-a1bb-4eb2-9bc0-da2760557b3d" (UID: "62b94b35-a1bb-4eb2-9bc0-da2760557b3d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:31:53.810753 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:53.810632 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62b94b35-a1bb-4eb2-9bc0-da2760557b3d-credential-socket" pod "62b94b35-a1bb-4eb2-9bc0-da2760557b3d" (UID: "62b94b35-a1bb-4eb2-9bc0-da2760557b3d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:31:53.810753 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:53.810727 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62b94b35-a1bb-4eb2-9bc0-da2760557b3d-workload-certs" pod "62b94b35-a1bb-4eb2-9bc0-da2760557b3d" (UID: "62b94b35-a1bb-4eb2-9bc0-da2760557b3d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:31:53.810987 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:53.810750 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62b94b35-a1bb-4eb2-9bc0-da2760557b3d-istio-data" pod "62b94b35-a1bb-4eb2-9bc0-da2760557b3d" (UID: "62b94b35-a1bb-4eb2-9bc0-da2760557b3d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:31:53.811105 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:53.811083 4908 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62b94b35-a1bb-4eb2-9bc0-da2760557b3d-istio-data\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:31:53.811105 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:53.811107 4908 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62b94b35-a1bb-4eb2-9bc0-da2760557b3d-workload-socket\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:31:53.811268 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:53.811120 4908 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62b94b35-a1bb-4eb2-9bc0-da2760557b3d-workload-certs\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:31:53.811268 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:53.811133 4908 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62b94b35-a1bb-4eb2-9bc0-da2760557b3d-credential-socket\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:31:53.811268 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:53.811166 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62b94b35-a1bb-4eb2-9bc0-da2760557b3d-istiod-ca-cert" pod "62b94b35-a1bb-4eb2-9bc0-da2760557b3d" (UID: "62b94b35-a1bb-4eb2-9bc0-da2760557b3d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:31:53.812635 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:53.812608 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/62b94b35-a1bb-4eb2-9bc0-da2760557b3d-istio-podinfo" pod "62b94b35-a1bb-4eb2-9bc0-da2760557b3d" (UID: "62b94b35-a1bb-4eb2-9bc0-da2760557b3d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 12:31:53.812635 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:53.812623 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62b94b35-a1bb-4eb2-9bc0-da2760557b3d-istio-token" pod "62b94b35-a1bb-4eb2-9bc0-da2760557b3d" (UID: "62b94b35-a1bb-4eb2-9bc0-da2760557b3d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:31:53.813158 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:53.813136 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62b94b35-a1bb-4eb2-9bc0-da2760557b3d-kube-api-access-ckrlf" pod "62b94b35-a1bb-4eb2-9bc0-da2760557b3d" (UID: "62b94b35-a1bb-4eb2-9bc0-da2760557b3d"). InnerVolumeSpecName "kube-api-access-ckrlf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:31:53.813248 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:53.813233 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62b94b35-a1bb-4eb2-9bc0-da2760557b3d-istio-envoy" pod "62b94b35-a1bb-4eb2-9bc0-da2760557b3d" (UID: "62b94b35-a1bb-4eb2-9bc0-da2760557b3d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:31:53.911988 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:53.911961 4908 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62b94b35-a1bb-4eb2-9bc0-da2760557b3d-istio-podinfo\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:31:53.911988 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:53.911985 4908 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62b94b35-a1bb-4eb2-9bc0-da2760557b3d-istio-token\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:31:53.912150 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:53.911996 4908 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62b94b35-a1bb-4eb2-9bc0-da2760557b3d-istiod-ca-cert\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:31:53.912150 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:53.912004 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ckrlf\" (UniqueName: \"kubernetes.io/projected/62b94b35-a1bb-4eb2-9bc0-da2760557b3d-kube-api-access-ckrlf\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:31:53.912150 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:53.912013 4908 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62b94b35-a1bb-4eb2-9bc0-da2760557b3d-istio-envoy\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:31:54.461137 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:54.461100 4908 generic.go:356] "Generic (PLEG): container finished" podID="62b94b35-a1bb-4eb2-9bc0-da2760557b3d" containerID="a45855efd1f7e4fd82b582d43c7deefe3da174c5f5a3aca8912ebbbcfc01f517" exitCode=0 Oct 30 12:31:54.461619 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:54.461175 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-1e9675-openshift-default-7d7c758549-6hhbg" Oct 30 12:31:54.461619 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:54.461179 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-1e9675-openshift-default-7d7c758549-6hhbg" event={"ID":"62b94b35-a1bb-4eb2-9bc0-da2760557b3d","Type":"ContainerDied","Data":"a45855efd1f7e4fd82b582d43c7deefe3da174c5f5a3aca8912ebbbcfc01f517"} Oct 30 12:31:54.461619 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:54.461284 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-1e9675-openshift-default-7d7c758549-6hhbg" event={"ID":"62b94b35-a1bb-4eb2-9bc0-da2760557b3d","Type":"ContainerDied","Data":"0ccf62b9cd4be90953ff1f3bab5a0bd3a72b82e563709c0f34d56ac0ce3d542a"} Oct 30 12:31:54.461619 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:54.461318 4908 scope.go:122] "RemoveContainer" containerID="a45855efd1f7e4fd82b582d43c7deefe3da174c5f5a3aca8912ebbbcfc01f517" Oct 30 12:31:54.478638 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:54.478601 4908 scope.go:122] "RemoveContainer" containerID="08fec614e1c0c235a1facb20245ba78167959eb61eb0a480c1b8f61483dcf3f5" Oct 30 12:31:54.493260 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:54.493238 4908 scope.go:122] "RemoveContainer" containerID="a45855efd1f7e4fd82b582d43c7deefe3da174c5f5a3aca8912ebbbcfc01f517" Oct 30 12:31:54.493599 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:31:54.493572 4908 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a45855efd1f7e4fd82b582d43c7deefe3da174c5f5a3aca8912ebbbcfc01f517\": container with ID starting with a45855efd1f7e4fd82b582d43c7deefe3da174c5f5a3aca8912ebbbcfc01f517 not found: ID does not exist" containerID="a45855efd1f7e4fd82b582d43c7deefe3da174c5f5a3aca8912ebbbcfc01f517" Oct 30 12:31:54.493673 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:54.493615 4908 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a45855efd1f7e4fd82b582d43c7deefe3da174c5f5a3aca8912ebbbcfc01f517"} err="failed to get container status \"a45855efd1f7e4fd82b582d43c7deefe3da174c5f5a3aca8912ebbbcfc01f517\": rpc error: code = NotFound desc = could not find container \"a45855efd1f7e4fd82b582d43c7deefe3da174c5f5a3aca8912ebbbcfc01f517\": container with ID starting with a45855efd1f7e4fd82b582d43c7deefe3da174c5f5a3aca8912ebbbcfc01f517 not found: ID does not exist" Oct 30 12:31:54.493673 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:54.493641 4908 scope.go:122] "RemoveContainer" containerID="08fec614e1c0c235a1facb20245ba78167959eb61eb0a480c1b8f61483dcf3f5" Oct 30 12:31:54.494052 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:31:54.494019 4908 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08fec614e1c0c235a1facb20245ba78167959eb61eb0a480c1b8f61483dcf3f5\": container with ID starting with 08fec614e1c0c235a1facb20245ba78167959eb61eb0a480c1b8f61483dcf3f5 not found: ID does not exist" containerID="08fec614e1c0c235a1facb20245ba78167959eb61eb0a480c1b8f61483dcf3f5" Oct 30 12:31:54.494140 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:54.494058 4908 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08fec614e1c0c235a1facb20245ba78167959eb61eb0a480c1b8f61483dcf3f5"} err="failed to get container status \"08fec614e1c0c235a1facb20245ba78167959eb61eb0a480c1b8f61483dcf3f5\": rpc error: code = NotFound desc = could not find container \"08fec614e1c0c235a1facb20245ba78167959eb61eb0a480c1b8f61483dcf3f5\": container with ID starting with 08fec614e1c0c235a1facb20245ba78167959eb61eb0a480c1b8f61483dcf3f5 not found: ID does not exist" Oct 30 12:31:54.512176 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:54.512150 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-1e9675-openshift-default-7d7c758549-6hhbg"] Oct 30 12:31:54.522287 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:54.522265 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-1e9675-openshift-default-7d7c758549-6hhbg"] Oct 30 12:31:54.618085 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:54.618061 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9933f47-s6tcr" Oct 30 12:31:54.720245 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:54.720178 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/452ea436-3165-416d-93c1-6563af3cd263-kube-api-access-ckj6w\" (UniqueName: \"kubernetes.io/projected/452ea436-3165-416d-93c1-6563af3cd263-kube-api-access-ckj6w\") pod \"452ea436-3165-416d-93c1-6563af3cd263\" (UID: \"452ea436-3165-416d-93c1-6563af3cd263\") " Oct 30 12:31:54.720245 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:54.720215 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/452ea436-3165-416d-93c1-6563af3cd263-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/452ea436-3165-416d-93c1-6563af3cd263-maas-api-xks-ca\") pod \"452ea436-3165-416d-93c1-6563af3cd263\" (UID: \"452ea436-3165-416d-93c1-6563af3cd263\") " Oct 30 12:31:54.720465 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:54.720265 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/452ea436-3165-416d-93c1-6563af3cd263-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/452ea436-3165-416d-93c1-6563af3cd263-maas-api-openshift-ca\") pod \"452ea436-3165-416d-93c1-6563af3cd263\" (UID: \"452ea436-3165-416d-93c1-6563af3cd263\") " Oct 30 12:31:54.720634 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:54.720606 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/452ea436-3165-416d-93c1-6563af3cd263-maas-api-openshift-ca" pod "452ea436-3165-416d-93c1-6563af3cd263" (UID: "452ea436-3165-416d-93c1-6563af3cd263"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:31:54.720778 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:54.720760 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/452ea436-3165-416d-93c1-6563af3cd263-maas-api-openshift-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:31:54.722397 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:54.722371 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/452ea436-3165-416d-93c1-6563af3cd263-kube-api-access-ckj6w" pod "452ea436-3165-416d-93c1-6563af3cd263" (UID: "452ea436-3165-416d-93c1-6563af3cd263"). InnerVolumeSpecName "kube-api-access-ckj6w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:31:54.722488 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:54.722380 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/452ea436-3165-416d-93c1-6563af3cd263-maas-api-xks-ca" pod "452ea436-3165-416d-93c1-6563af3cd263" (UID: "452ea436-3165-416d-93c1-6563af3cd263"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:31:54.821494 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:54.821467 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ckj6w\" (UniqueName: \"kubernetes.io/projected/452ea436-3165-416d-93c1-6563af3cd263-kube-api-access-ckj6w\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:31:54.821494 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:54.821496 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/452ea436-3165-416d-93c1-6563af3cd263-maas-api-xks-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:31:55.475376 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:55.475349 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9933f47-s6tcr" Oct 30 12:31:55.475919 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:55.475343 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9933f47-s6tcr" event={"ID":"452ea436-3165-416d-93c1-6563af3cd263","Type":"ContainerDied","Data":"20cf6be6649ebf671854d24e5da1a6e0b87d93423fd595bdb67a29859a9f54bc"} Oct 30 12:31:55.475919 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:55.475468 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="20cf6be6649ebf671854d24e5da1a6e0b87d93423fd595bdb67a29859a9f54bc" Oct 30 12:31:55.610784 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:55.610746 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62b94b35-a1bb-4eb2-9bc0-da2760557b3d" path="/var/lib/kubelet/pods/62b94b35-a1bb-4eb2-9bc0-da2760557b3d/volumes" Oct 30 12:31:58.286453 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:58.286419 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a32072a-9gttn"] Oct 30 12:31:58.287150 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:58.287127 4908 cpu_manager.go:414] "containerMap: removing container" podUID="452ea436-3165-416d-93c1-6563af3cd263" containerName="revoke-keys" Oct 30 12:31:58.287257 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:58.287151 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="452ea436-3165-416d-93c1-6563af3cd263" containerName="revoke-keys" Oct 30 12:31:58.287257 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:58.287215 4908 cpu_manager.go:414] "containerMap: removing container" podUID="62b94b35-a1bb-4eb2-9bc0-da2760557b3d" containerName="istio-proxy" Oct 30 12:31:58.287257 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:58.287224 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62b94b35-a1bb-4eb2-9bc0-da2760557b3d" containerName="istio-proxy" Oct 30 12:31:58.287257 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:58.287247 4908 cpu_manager.go:414] "containerMap: removing container" podUID="62b94b35-a1bb-4eb2-9bc0-da2760557b3d" containerName="istio-proxy" Oct 30 12:31:58.287257 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:58.287255 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62b94b35-a1bb-4eb2-9bc0-da2760557b3d" containerName="istio-proxy" Oct 30 12:31:58.287522 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:58.287384 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="62b94b35-a1bb-4eb2-9bc0-da2760557b3d" containerName="istio-proxy" Oct 30 12:31:58.287522 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:58.287403 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="452ea436-3165-416d-93c1-6563af3cd263" containerName="revoke-keys" Oct 30 12:31:58.292214 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:58.292185 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a32072a-9gttn" Oct 30 12:31:58.319674 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:58.319648 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 12:31:58.319915 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:58.319889 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 12:31:58.320066 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:58.319989 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s75hq\"" Oct 30 12:31:58.320194 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:58.320175 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 12:31:58.324059 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:58.324034 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a32072a-9gttn"] Oct 30 12:31:58.364462 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:58.364431 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f16f29d-fe4d-4fbd-91d7-c07a188222b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8a32072a-9gttn\" (UID: \"8f16f29d-fe4d-4fbd-91d7-c07a188222b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a32072a-9gttn" Oct 30 12:31:58.364618 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:58.364475 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f16f29d-fe4d-4fbd-91d7-c07a188222b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8a32072a-9gttn\" (UID: \"8f16f29d-fe4d-4fbd-91d7-c07a188222b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a32072a-9gttn" Oct 30 12:31:58.364784 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:58.364758 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bhnk\" (UniqueName: \"kubernetes.io/projected/8f16f29d-fe4d-4fbd-91d7-c07a188222b7-kube-api-access-5bhnk\") pod \"maas-api-revoke-sub-8a32072a-9gttn\" (UID: \"8f16f29d-fe4d-4fbd-91d7-c07a188222b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a32072a-9gttn" Oct 30 12:31:58.466102 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:58.466060 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bhnk\" (UniqueName: \"kubernetes.io/projected/8f16f29d-fe4d-4fbd-91d7-c07a188222b7-kube-api-access-5bhnk\") pod \"maas-api-revoke-sub-8a32072a-9gttn\" (UID: \"8f16f29d-fe4d-4fbd-91d7-c07a188222b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a32072a-9gttn" Oct 30 12:31:58.466298 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:58.466177 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f16f29d-fe4d-4fbd-91d7-c07a188222b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8a32072a-9gttn\" (UID: \"8f16f29d-fe4d-4fbd-91d7-c07a188222b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a32072a-9gttn" Oct 30 12:31:58.466298 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:58.466221 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f16f29d-fe4d-4fbd-91d7-c07a188222b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8a32072a-9gttn\" (UID: \"8f16f29d-fe4d-4fbd-91d7-c07a188222b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a32072a-9gttn" Oct 30 12:31:58.467195 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:58.467162 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f16f29d-fe4d-4fbd-91d7-c07a188222b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8a32072a-9gttn\" (UID: \"8f16f29d-fe4d-4fbd-91d7-c07a188222b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a32072a-9gttn" Oct 30 12:31:58.470493 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:58.470465 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f16f29d-fe4d-4fbd-91d7-c07a188222b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8a32072a-9gttn\" (UID: \"8f16f29d-fe4d-4fbd-91d7-c07a188222b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a32072a-9gttn" Oct 30 12:31:58.489489 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:58.489461 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bhnk\" (UniqueName: \"kubernetes.io/projected/8f16f29d-fe4d-4fbd-91d7-c07a188222b7-kube-api-access-5bhnk\") pod \"maas-api-revoke-sub-8a32072a-9gttn\" (UID: \"8f16f29d-fe4d-4fbd-91d7-c07a188222b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a32072a-9gttn" Oct 30 12:31:58.606918 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:58.606802 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a32072a-9gttn" Oct 30 12:31:58.919778 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:58.919750 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a32072a-9gttn"] Oct 30 12:31:58.922150 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:31:58.922120 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f16f29d_fe4d_4fbd_91d7_c07a188222b7.slice/crio-c4a026860b8da71de1d2d14885677281c8ddfafa9613aa448f16cdc65f1e18dc WatchSource:0}: Error finding container c4a026860b8da71de1d2d14885677281c8ddfafa9613aa448f16cdc65f1e18dc: Status 404 returned error can't find the container with id c4a026860b8da71de1d2d14885677281c8ddfafa9613aa448f16cdc65f1e18dc Oct 30 12:31:59.516535 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:59.516464 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a32072a-9gttn" event={"ID":"8f16f29d-fe4d-4fbd-91d7-c07a188222b7","Type":"ContainerStarted","Data":"c2891b6d1b422baf500e782d89e0dcf22fbee954bff97c950104fa74bc0093c7"} Oct 30 12:31:59.516535 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:59.516536 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a32072a-9gttn" event={"ID":"8f16f29d-fe4d-4fbd-91d7-c07a188222b7","Type":"ContainerStarted","Data":"c4a026860b8da71de1d2d14885677281c8ddfafa9613aa448f16cdc65f1e18dc"} Oct 30 12:31:59.559044 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:31:59.558994 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a32072a-9gttn" podStartSLOduration=1.5589810910000002 podStartE2EDuration="1.558981091s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:31:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:31:59.555477018 +0000 UTC m=+2276.069905801" watchObservedRunningTime="2026-10-30 12:31:59.558981091 +0000 UTC m=+2276.073409836" Oct 30 12:32:00.527408 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:00.527365 4908 generic.go:356] "Generic (PLEG): container finished" podID="8f16f29d-fe4d-4fbd-91d7-c07a188222b7" containerID="c2891b6d1b422baf500e782d89e0dcf22fbee954bff97c950104fa74bc0093c7" exitCode=0 Oct 30 12:32:00.527871 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:00.527507 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a32072a-9gttn" event={"ID":"8f16f29d-fe4d-4fbd-91d7-c07a188222b7","Type":"ContainerDied","Data":"c2891b6d1b422baf500e782d89e0dcf22fbee954bff97c950104fa74bc0093c7"} Oct 30 12:32:01.702152 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:01.702092 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-539475a0-b9vp6"] Oct 30 12:32:01.703781 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:01.703737 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="62b94b35-a1bb-4eb2-9bc0-da2760557b3d" containerName="istio-proxy" Oct 30 12:32:01.709816 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:01.709785 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-539475a0-b9vp6" Oct 30 12:32:01.745637 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:01.745605 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-539475a0-b9vp6"] Oct 30 12:32:01.819740 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:01.819697 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68c31061-4f8a-434e-863f-e1199f5b7c1b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-539475a0-b9vp6\" (UID: \"68c31061-4f8a-434e-863f-e1199f5b7c1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-539475a0-b9vp6" Oct 30 12:32:01.820648 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:01.820624 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nj76b\" (UniqueName: \"kubernetes.io/projected/68c31061-4f8a-434e-863f-e1199f5b7c1b-kube-api-access-nj76b\") pod \"maas-api-revoke-sub-539475a0-b9vp6\" (UID: \"68c31061-4f8a-434e-863f-e1199f5b7c1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-539475a0-b9vp6" Oct 30 12:32:01.820898 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:01.820878 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68c31061-4f8a-434e-863f-e1199f5b7c1b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-539475a0-b9vp6\" (UID: \"68c31061-4f8a-434e-863f-e1199f5b7c1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-539475a0-b9vp6" Oct 30 12:32:01.903312 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:01.903235 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a32072a-9gttn" Oct 30 12:32:01.921675 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:01.921645 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f16f29d-fe4d-4fbd-91d7-c07a188222b7-kube-api-access-5bhnk\" (UniqueName: \"kubernetes.io/projected/8f16f29d-fe4d-4fbd-91d7-c07a188222b7-kube-api-access-5bhnk\") pod \"8f16f29d-fe4d-4fbd-91d7-c07a188222b7\" (UID: \"8f16f29d-fe4d-4fbd-91d7-c07a188222b7\") " Oct 30 12:32:01.921947 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:01.921927 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f16f29d-fe4d-4fbd-91d7-c07a188222b7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f16f29d-fe4d-4fbd-91d7-c07a188222b7-maas-api-openshift-ca\") pod \"8f16f29d-fe4d-4fbd-91d7-c07a188222b7\" (UID: \"8f16f29d-fe4d-4fbd-91d7-c07a188222b7\") " Oct 30 12:32:01.922595 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:01.922573 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f16f29d-fe4d-4fbd-91d7-c07a188222b7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f16f29d-fe4d-4fbd-91d7-c07a188222b7-maas-api-xks-ca\") pod \"8f16f29d-fe4d-4fbd-91d7-c07a188222b7\" (UID: \"8f16f29d-fe4d-4fbd-91d7-c07a188222b7\") " Oct 30 12:32:01.922937 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:01.922915 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68c31061-4f8a-434e-863f-e1199f5b7c1b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-539475a0-b9vp6\" (UID: \"68c31061-4f8a-434e-863f-e1199f5b7c1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-539475a0-b9vp6" Oct 30 12:32:01.924013 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:01.923992 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nj76b\" (UniqueName: \"kubernetes.io/projected/68c31061-4f8a-434e-863f-e1199f5b7c1b-kube-api-access-nj76b\") pod \"maas-api-revoke-sub-539475a0-b9vp6\" (UID: \"68c31061-4f8a-434e-863f-e1199f5b7c1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-539475a0-b9vp6" Oct 30 12:32:01.924486 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:01.924469 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68c31061-4f8a-434e-863f-e1199f5b7c1b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-539475a0-b9vp6\" (UID: \"68c31061-4f8a-434e-863f-e1199f5b7c1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-539475a0-b9vp6" Oct 30 12:32:01.924857 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:01.923856 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68c31061-4f8a-434e-863f-e1199f5b7c1b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-539475a0-b9vp6\" (UID: \"68c31061-4f8a-434e-863f-e1199f5b7c1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-539475a0-b9vp6" Oct 30 12:32:01.924993 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:01.922410 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f16f29d-fe4d-4fbd-91d7-c07a188222b7-maas-api-openshift-ca" pod "8f16f29d-fe4d-4fbd-91d7-c07a188222b7" (UID: "8f16f29d-fe4d-4fbd-91d7-c07a188222b7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:32:01.925081 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:01.924489 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f16f29d-fe4d-4fbd-91d7-c07a188222b7-kube-api-access-5bhnk" pod "8f16f29d-fe4d-4fbd-91d7-c07a188222b7" (UID: "8f16f29d-fe4d-4fbd-91d7-c07a188222b7"). InnerVolumeSpecName "kube-api-access-5bhnk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:32:01.926595 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:01.926273 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f16f29d-fe4d-4fbd-91d7-c07a188222b7-maas-api-xks-ca" pod "8f16f29d-fe4d-4fbd-91d7-c07a188222b7" (UID: "8f16f29d-fe4d-4fbd-91d7-c07a188222b7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:32:01.927118 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:01.927098 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68c31061-4f8a-434e-863f-e1199f5b7c1b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-539475a0-b9vp6\" (UID: \"68c31061-4f8a-434e-863f-e1199f5b7c1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-539475a0-b9vp6" Oct 30 12:32:01.948930 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:01.948896 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nj76b\" (UniqueName: \"kubernetes.io/projected/68c31061-4f8a-434e-863f-e1199f5b7c1b-kube-api-access-nj76b\") pod \"maas-api-revoke-sub-539475a0-b9vp6\" (UID: \"68c31061-4f8a-434e-863f-e1199f5b7c1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-539475a0-b9vp6" Oct 30 12:32:02.027511 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:02.027460 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f16f29d-fe4d-4fbd-91d7-c07a188222b7-maas-api-xks-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:32:02.027511 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:02.027507 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5bhnk\" (UniqueName: \"kubernetes.io/projected/8f16f29d-fe4d-4fbd-91d7-c07a188222b7-kube-api-access-5bhnk\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:32:02.027807 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:02.027524 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f16f29d-fe4d-4fbd-91d7-c07a188222b7-maas-api-openshift-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:32:02.055772 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:02.055739 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-539475a0-b9vp6" Oct 30 12:32:02.404853 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:02.402995 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-539475a0-b9vp6"] Oct 30 12:32:02.625340 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:02.625311 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a32072a-9gttn" Oct 30 12:32:02.625340 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:02.625332 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a32072a-9gttn" event={"ID":"8f16f29d-fe4d-4fbd-91d7-c07a188222b7","Type":"ContainerDied","Data":"c4a026860b8da71de1d2d14885677281c8ddfafa9613aa448f16cdc65f1e18dc"} Oct 30 12:32:02.625558 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:02.625382 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c4a026860b8da71de1d2d14885677281c8ddfafa9613aa448f16cdc65f1e18dc" Oct 30 12:32:02.626695 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:02.626671 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-539475a0-b9vp6" event={"ID":"68c31061-4f8a-434e-863f-e1199f5b7c1b","Type":"ContainerStarted","Data":"e8d913d8609e43cd50a5ce95643a75361d7e9d26e274171b6749d8669ffad50e"} Oct 30 12:32:03.635503 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:03.635464 4908 generic.go:356] "Generic (PLEG): container finished" podID="68c31061-4f8a-434e-863f-e1199f5b7c1b" containerID="1892285d1c5dd75fc039fb0e92de34f4befa7d7e4069137992706d256f3df3da" exitCode=0 Oct 30 12:32:03.635924 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:03.635517 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-539475a0-b9vp6" event={"ID":"68c31061-4f8a-434e-863f-e1199f5b7c1b","Type":"ContainerDied","Data":"1892285d1c5dd75fc039fb0e92de34f4befa7d7e4069137992706d256f3df3da"} Oct 30 12:32:04.842718 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:04.842693 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-539475a0-b9vp6" Oct 30 12:32:04.858328 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:04.858305 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68c31061-4f8a-434e-863f-e1199f5b7c1b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68c31061-4f8a-434e-863f-e1199f5b7c1b-maas-api-openshift-ca\") pod \"68c31061-4f8a-434e-863f-e1199f5b7c1b\" (UID: \"68c31061-4f8a-434e-863f-e1199f5b7c1b\") " Oct 30 12:32:04.858452 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:04.858418 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/68c31061-4f8a-434e-863f-e1199f5b7c1b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68c31061-4f8a-434e-863f-e1199f5b7c1b-maas-api-xks-ca\") pod \"68c31061-4f8a-434e-863f-e1199f5b7c1b\" (UID: \"68c31061-4f8a-434e-863f-e1199f5b7c1b\") " Oct 30 12:32:04.858452 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:04.858445 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68c31061-4f8a-434e-863f-e1199f5b7c1b-kube-api-access-nj76b\" (UniqueName: \"kubernetes.io/projected/68c31061-4f8a-434e-863f-e1199f5b7c1b-kube-api-access-nj76b\") pod \"68c31061-4f8a-434e-863f-e1199f5b7c1b\" (UID: \"68c31061-4f8a-434e-863f-e1199f5b7c1b\") " Oct 30 12:32:04.858632 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:04.858609 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68c31061-4f8a-434e-863f-e1199f5b7c1b-maas-api-openshift-ca" pod "68c31061-4f8a-434e-863f-e1199f5b7c1b" (UID: "68c31061-4f8a-434e-863f-e1199f5b7c1b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:32:04.858801 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:04.858786 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68c31061-4f8a-434e-863f-e1199f5b7c1b-maas-api-openshift-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:32:04.860498 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:04.860469 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68c31061-4f8a-434e-863f-e1199f5b7c1b-maas-api-xks-ca" pod "68c31061-4f8a-434e-863f-e1199f5b7c1b" (UID: "68c31061-4f8a-434e-863f-e1199f5b7c1b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:32:04.860714 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:04.860691 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68c31061-4f8a-434e-863f-e1199f5b7c1b-kube-api-access-nj76b" pod "68c31061-4f8a-434e-863f-e1199f5b7c1b" (UID: "68c31061-4f8a-434e-863f-e1199f5b7c1b"). InnerVolumeSpecName "kube-api-access-nj76b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:32:04.959932 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:04.959814 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68c31061-4f8a-434e-863f-e1199f5b7c1b-maas-api-xks-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:32:04.959932 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:04.959880 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nj76b\" (UniqueName: \"kubernetes.io/projected/68c31061-4f8a-434e-863f-e1199f5b7c1b-kube-api-access-nj76b\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:32:05.095776 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:05.095745 4908 scope.go:122] "RemoveContainer" containerID="25c1a2c0ba09560ee93bbd92687c64163cfd6faada412c983043583bcd9e221e" Oct 30 12:32:05.650436 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:05.650407 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-539475a0-b9vp6" Oct 30 12:32:05.650595 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:05.650435 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-539475a0-b9vp6" event={"ID":"68c31061-4f8a-434e-863f-e1199f5b7c1b","Type":"ContainerDied","Data":"e8d913d8609e43cd50a5ce95643a75361d7e9d26e274171b6749d8669ffad50e"} Oct 30 12:32:05.650595 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:05.650480 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e8d913d8609e43cd50a5ce95643a75361d7e9d26e274171b6749d8669ffad50e" Oct 30 12:32:07.885542 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:07.885500 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5c57af-6d8484d895-78hm2"] Oct 30 12:32:07.886129 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:07.885745 4908 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5c57af-6d8484d895-78hm2" podUID="3834e8b4-8ff0-43cf-8aef-65de733483e6" containerName="payload-pre-processing" containerID="cri-o://f728842bc6344419396dbfd3e985df1140741aa79af14bc68134404da6b00f1d" gracePeriod=30 Oct 30 12:32:08.271928 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:08.271549 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5c57af-6d8484d895-78hm2" Oct 30 12:32:08.352113 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:08.347041 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3834e8b4-8ff0-43cf-8aef-65de733483e6-kube-api-access-9xqn9\" (UniqueName: \"kubernetes.io/projected/3834e8b4-8ff0-43cf-8aef-65de733483e6-kube-api-access-9xqn9\") pod \"3834e8b4-8ff0-43cf-8aef-65de733483e6\" (UID: \"3834e8b4-8ff0-43cf-8aef-65de733483e6\") " Oct 30 12:32:08.352113 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:08.347135 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3834e8b4-8ff0-43cf-8aef-65de733483e6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3834e8b4-8ff0-43cf-8aef-65de733483e6-plugins-config-volume\") pod \"3834e8b4-8ff0-43cf-8aef-65de733483e6\" (UID: \"3834e8b4-8ff0-43cf-8aef-65de733483e6\") " Oct 30 12:32:08.352113 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:08.348156 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3834e8b4-8ff0-43cf-8aef-65de733483e6-plugins-config-volume" pod "3834e8b4-8ff0-43cf-8aef-65de733483e6" (UID: "3834e8b4-8ff0-43cf-8aef-65de733483e6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:32:08.352113 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:08.348771 4908 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3834e8b4-8ff0-43cf-8aef-65de733483e6-plugins-config-volume\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:32:08.352113 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:08.351661 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3834e8b4-8ff0-43cf-8aef-65de733483e6-kube-api-access-9xqn9" pod "3834e8b4-8ff0-43cf-8aef-65de733483e6" (UID: "3834e8b4-8ff0-43cf-8aef-65de733483e6"). InnerVolumeSpecName "kube-api-access-9xqn9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:32:08.449596 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:08.449513 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9xqn9\" (UniqueName: \"kubernetes.io/projected/3834e8b4-8ff0-43cf-8aef-65de733483e6-kube-api-access-9xqn9\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:32:08.470333 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:08.470293 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5c57af-h96hh"] Oct 30 12:32:08.474628 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:08.474599 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5c57af-h96hh"] Oct 30 12:32:08.676879 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:08.676842 4908 generic.go:356] "Generic (PLEG): container finished" podID="3834e8b4-8ff0-43cf-8aef-65de733483e6" containerID="f728842bc6344419396dbfd3e985df1140741aa79af14bc68134404da6b00f1d" exitCode=0 Oct 30 12:32:08.677081 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:08.676884 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5c57af-6d8484d895-78hm2" Oct 30 12:32:08.677081 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:08.676923 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5c57af-6d8484d895-78hm2" event={"ID":"3834e8b4-8ff0-43cf-8aef-65de733483e6","Type":"ContainerDied","Data":"f728842bc6344419396dbfd3e985df1140741aa79af14bc68134404da6b00f1d"} Oct 30 12:32:08.677081 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:08.676982 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5c57af-6d8484d895-78hm2" event={"ID":"3834e8b4-8ff0-43cf-8aef-65de733483e6","Type":"ContainerDied","Data":"d22da190f9ae04b1da96e1beda0b522e173515fb11b854721d4fd1d3773cf592"} Oct 30 12:32:08.677081 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:08.677012 4908 scope.go:122] "RemoveContainer" containerID="f728842bc6344419396dbfd3e985df1140741aa79af14bc68134404da6b00f1d" Oct 30 12:32:08.688577 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:08.688548 4908 scope.go:122] "RemoveContainer" containerID="f728842bc6344419396dbfd3e985df1140741aa79af14bc68134404da6b00f1d" Oct 30 12:32:08.688938 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:32:08.688898 4908 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f728842bc6344419396dbfd3e985df1140741aa79af14bc68134404da6b00f1d\": container with ID starting with f728842bc6344419396dbfd3e985df1140741aa79af14bc68134404da6b00f1d not found: ID does not exist" containerID="f728842bc6344419396dbfd3e985df1140741aa79af14bc68134404da6b00f1d" Oct 30 12:32:08.689028 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:08.688953 4908 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f728842bc6344419396dbfd3e985df1140741aa79af14bc68134404da6b00f1d"} err="failed to get container status \"f728842bc6344419396dbfd3e985df1140741aa79af14bc68134404da6b00f1d\": rpc error: code = NotFound desc = could not find container \"f728842bc6344419396dbfd3e985df1140741aa79af14bc68134404da6b00f1d\": container with ID starting with f728842bc6344419396dbfd3e985df1140741aa79af14bc68134404da6b00f1d not found: ID does not exist" Oct 30 12:32:08.728698 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:08.728669 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5c57af-6d8484d895-78hm2"] Oct 30 12:32:08.731777 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:08.731756 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5c57af-6d8484d895-78hm2"] Oct 30 12:32:08.805625 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:08.805586 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f894569-vs6m2"] Oct 30 12:32:08.806135 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:08.806120 4908 cpu_manager.go:414] "containerMap: removing container" podUID="8f16f29d-fe4d-4fbd-91d7-c07a188222b7" containerName="revoke-keys" Oct 30 12:32:08.806188 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:08.806136 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f16f29d-fe4d-4fbd-91d7-c07a188222b7" containerName="revoke-keys" Oct 30 12:32:08.806188 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:08.806152 4908 cpu_manager.go:414] "containerMap: removing container" podUID="3834e8b4-8ff0-43cf-8aef-65de733483e6" containerName="payload-pre-processing" Oct 30 12:32:08.806188 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:08.806158 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3834e8b4-8ff0-43cf-8aef-65de733483e6" containerName="payload-pre-processing" Oct 30 12:32:08.806188 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:08.806172 4908 cpu_manager.go:414] "containerMap: removing container" podUID="68c31061-4f8a-434e-863f-e1199f5b7c1b" containerName="revoke-keys" Oct 30 12:32:08.806188 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:08.806177 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68c31061-4f8a-434e-863f-e1199f5b7c1b" containerName="revoke-keys" Oct 30 12:32:08.806344 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:08.806261 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="68c31061-4f8a-434e-863f-e1199f5b7c1b" containerName="revoke-keys" Oct 30 12:32:08.806344 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:08.806279 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f16f29d-fe4d-4fbd-91d7-c07a188222b7" containerName="revoke-keys" Oct 30 12:32:08.806344 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:08.806285 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="3834e8b4-8ff0-43cf-8aef-65de733483e6" containerName="payload-pre-processing" Oct 30 12:32:08.811891 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:08.811871 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f894569-vs6m2" Oct 30 12:32:08.826197 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:08.826174 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 12:32:08.826336 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:08.826288 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s75hq\"" Oct 30 12:32:08.826666 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:08.826575 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 12:32:08.826666 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:08.826635 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 12:32:08.828118 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:08.828095 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f894569-vs6m2"] Oct 30 12:32:08.854283 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:08.854242 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggrh2\" (UniqueName: \"kubernetes.io/projected/c090356f-b5ff-4678-aab5-db6f893b2cb1-kube-api-access-ggrh2\") pod \"maas-api-revoke-sub-6f894569-vs6m2\" (UID: \"c090356f-b5ff-4678-aab5-db6f893b2cb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f894569-vs6m2" Oct 30 12:32:08.854421 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:08.854376 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c090356f-b5ff-4678-aab5-db6f893b2cb1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6f894569-vs6m2\" (UID: \"c090356f-b5ff-4678-aab5-db6f893b2cb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f894569-vs6m2" Oct 30 12:32:08.854544 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:08.854522 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c090356f-b5ff-4678-aab5-db6f893b2cb1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6f894569-vs6m2\" (UID: \"c090356f-b5ff-4678-aab5-db6f893b2cb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f894569-vs6m2" Oct 30 12:32:08.955740 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:08.955709 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggrh2\" (UniqueName: \"kubernetes.io/projected/c090356f-b5ff-4678-aab5-db6f893b2cb1-kube-api-access-ggrh2\") pod \"maas-api-revoke-sub-6f894569-vs6m2\" (UID: \"c090356f-b5ff-4678-aab5-db6f893b2cb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f894569-vs6m2" Oct 30 12:32:08.956220 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:08.955761 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c090356f-b5ff-4678-aab5-db6f893b2cb1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6f894569-vs6m2\" (UID: \"c090356f-b5ff-4678-aab5-db6f893b2cb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f894569-vs6m2" Oct 30 12:32:08.956220 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:08.955856 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c090356f-b5ff-4678-aab5-db6f893b2cb1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6f894569-vs6m2\" (UID: \"c090356f-b5ff-4678-aab5-db6f893b2cb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f894569-vs6m2" Oct 30 12:32:08.956546 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:08.956527 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c090356f-b5ff-4678-aab5-db6f893b2cb1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6f894569-vs6m2\" (UID: \"c090356f-b5ff-4678-aab5-db6f893b2cb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f894569-vs6m2" Oct 30 12:32:08.958166 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:08.958148 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c090356f-b5ff-4678-aab5-db6f893b2cb1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6f894569-vs6m2\" (UID: \"c090356f-b5ff-4678-aab5-db6f893b2cb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f894569-vs6m2" Oct 30 12:32:08.978182 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:08.978152 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggrh2\" (UniqueName: \"kubernetes.io/projected/c090356f-b5ff-4678-aab5-db6f893b2cb1-kube-api-access-ggrh2\") pod \"maas-api-revoke-sub-6f894569-vs6m2\" (UID: \"c090356f-b5ff-4678-aab5-db6f893b2cb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f894569-vs6m2" Oct 30 12:32:09.126457 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:09.126365 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f894569-vs6m2" Oct 30 12:32:09.434706 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:09.434675 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f894569-vs6m2"] Oct 30 12:32:09.438321 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:32:09.438287 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc090356f_b5ff_4678_aab5_db6f893b2cb1.slice/crio-e3f725a776e198b67d04682605487ac862fa0c0bfe147e57b03247c8a321d236 WatchSource:0}: Error finding container e3f725a776e198b67d04682605487ac862fa0c0bfe147e57b03247c8a321d236: Status 404 returned error can't find the container with id e3f725a776e198b67d04682605487ac862fa0c0bfe147e57b03247c8a321d236 Oct 30 12:32:09.606053 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:09.606022 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3834e8b4-8ff0-43cf-8aef-65de733483e6" path="/var/lib/kubelet/pods/3834e8b4-8ff0-43cf-8aef-65de733483e6/volumes" Oct 30 12:32:09.606511 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:09.606492 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e545bc36-bf6e-49c3-ad8f-9e1d57e62c6f" path="/var/lib/kubelet/pods/e545bc36-bf6e-49c3-ad8f-9e1d57e62c6f/volumes" Oct 30 12:32:09.688391 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:09.688312 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f894569-vs6m2" event={"ID":"c090356f-b5ff-4678-aab5-db6f893b2cb1","Type":"ContainerStarted","Data":"e3f725a776e198b67d04682605487ac862fa0c0bfe147e57b03247c8a321d236"} Oct 30 12:32:10.697805 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:10.697716 4908 generic.go:356] "Generic (PLEG): container finished" podID="c090356f-b5ff-4678-aab5-db6f893b2cb1" containerID="cdaf4c8a90744ae4aff020e781769e163ecda1e4eed3152cee854d2351b9b0c7" exitCode=0 Oct 30 12:32:10.698228 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:10.697801 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f894569-vs6m2" event={"ID":"c090356f-b5ff-4678-aab5-db6f893b2cb1","Type":"ContainerDied","Data":"cdaf4c8a90744ae4aff020e781769e163ecda1e4eed3152cee854d2351b9b0c7"} Oct 30 12:32:11.896391 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:11.896360 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f894569-vs6m2" Oct 30 12:32:11.990996 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:11.990956 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c090356f-b5ff-4678-aab5-db6f893b2cb1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c090356f-b5ff-4678-aab5-db6f893b2cb1-maas-api-openshift-ca\") pod \"c090356f-b5ff-4678-aab5-db6f893b2cb1\" (UID: \"c090356f-b5ff-4678-aab5-db6f893b2cb1\") " Oct 30 12:32:11.991208 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:11.991068 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c090356f-b5ff-4678-aab5-db6f893b2cb1-kube-api-access-ggrh2\" (UniqueName: \"kubernetes.io/projected/c090356f-b5ff-4678-aab5-db6f893b2cb1-kube-api-access-ggrh2\") pod \"c090356f-b5ff-4678-aab5-db6f893b2cb1\" (UID: \"c090356f-b5ff-4678-aab5-db6f893b2cb1\") " Oct 30 12:32:11.991208 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:11.991142 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c090356f-b5ff-4678-aab5-db6f893b2cb1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c090356f-b5ff-4678-aab5-db6f893b2cb1-maas-api-xks-ca\") pod \"c090356f-b5ff-4678-aab5-db6f893b2cb1\" (UID: \"c090356f-b5ff-4678-aab5-db6f893b2cb1\") " Oct 30 12:32:11.991431 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:11.991403 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c090356f-b5ff-4678-aab5-db6f893b2cb1-maas-api-openshift-ca" pod "c090356f-b5ff-4678-aab5-db6f893b2cb1" (UID: "c090356f-b5ff-4678-aab5-db6f893b2cb1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:32:11.991585 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:11.991566 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c090356f-b5ff-4678-aab5-db6f893b2cb1-maas-api-openshift-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:32:11.993291 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:11.993264 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c090356f-b5ff-4678-aab5-db6f893b2cb1-maas-api-xks-ca" pod "c090356f-b5ff-4678-aab5-db6f893b2cb1" (UID: "c090356f-b5ff-4678-aab5-db6f893b2cb1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:32:11.993679 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:11.993660 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c090356f-b5ff-4678-aab5-db6f893b2cb1-kube-api-access-ggrh2" pod "c090356f-b5ff-4678-aab5-db6f893b2cb1" (UID: "c090356f-b5ff-4678-aab5-db6f893b2cb1"). InnerVolumeSpecName "kube-api-access-ggrh2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:32:12.093252 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:12.093212 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ggrh2\" (UniqueName: \"kubernetes.io/projected/c090356f-b5ff-4678-aab5-db6f893b2cb1-kube-api-access-ggrh2\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:32:12.093252 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:12.093248 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c090356f-b5ff-4678-aab5-db6f893b2cb1-maas-api-xks-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:32:12.716030 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:12.715987 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f894569-vs6m2" event={"ID":"c090356f-b5ff-4678-aab5-db6f893b2cb1","Type":"ContainerDied","Data":"e3f725a776e198b67d04682605487ac862fa0c0bfe147e57b03247c8a321d236"} Oct 30 12:32:12.716030 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:12.716008 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f894569-vs6m2" Oct 30 12:32:12.716030 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:12.716037 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e3f725a776e198b67d04682605487ac862fa0c0bfe147e57b03247c8a321d236" Oct 30 12:32:14.696147 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:14.696114 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00d131b1-sv5v8"] Oct 30 12:32:14.696691 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:14.696675 4908 cpu_manager.go:414] "containerMap: removing container" podUID="c090356f-b5ff-4678-aab5-db6f893b2cb1" containerName="revoke-keys" Oct 30 12:32:14.696691 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:14.696691 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c090356f-b5ff-4678-aab5-db6f893b2cb1" containerName="revoke-keys" Oct 30 12:32:14.696854 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:14.696818 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="c090356f-b5ff-4678-aab5-db6f893b2cb1" containerName="revoke-keys" Oct 30 12:32:14.700223 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:14.700205 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00d131b1-sv5v8" Oct 30 12:32:14.717925 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:14.717893 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00d131b1-sv5v8"] Oct 30 12:32:14.726974 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:14.726865 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 12:32:14.726974 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:14.726885 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 12:32:14.726974 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:14.726950 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 12:32:14.727213 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:14.727000 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s75hq\"" Oct 30 12:32:14.825258 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:14.825228 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/856d3aa7-ff0f-4df5-ab38-369fb6a8757d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-00d131b1-sv5v8\" (UID: \"856d3aa7-ff0f-4df5-ab38-369fb6a8757d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00d131b1-sv5v8" Oct 30 12:32:14.825391 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:14.825271 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6rrd\" (UniqueName: \"kubernetes.io/projected/856d3aa7-ff0f-4df5-ab38-369fb6a8757d-kube-api-access-w6rrd\") pod \"maas-api-revoke-sub-00d131b1-sv5v8\" (UID: \"856d3aa7-ff0f-4df5-ab38-369fb6a8757d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00d131b1-sv5v8" Oct 30 12:32:14.825493 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:14.825472 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/856d3aa7-ff0f-4df5-ab38-369fb6a8757d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-00d131b1-sv5v8\" (UID: \"856d3aa7-ff0f-4df5-ab38-369fb6a8757d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00d131b1-sv5v8" Oct 30 12:32:14.926227 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:14.926199 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/856d3aa7-ff0f-4df5-ab38-369fb6a8757d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-00d131b1-sv5v8\" (UID: \"856d3aa7-ff0f-4df5-ab38-369fb6a8757d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00d131b1-sv5v8" Oct 30 12:32:14.926357 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:14.926252 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/856d3aa7-ff0f-4df5-ab38-369fb6a8757d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-00d131b1-sv5v8\" (UID: \"856d3aa7-ff0f-4df5-ab38-369fb6a8757d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00d131b1-sv5v8" Oct 30 12:32:14.926357 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:14.926283 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6rrd\" (UniqueName: \"kubernetes.io/projected/856d3aa7-ff0f-4df5-ab38-369fb6a8757d-kube-api-access-w6rrd\") pod \"maas-api-revoke-sub-00d131b1-sv5v8\" (UID: \"856d3aa7-ff0f-4df5-ab38-369fb6a8757d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00d131b1-sv5v8" Oct 30 12:32:14.926994 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:14.926968 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/856d3aa7-ff0f-4df5-ab38-369fb6a8757d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-00d131b1-sv5v8\" (UID: \"856d3aa7-ff0f-4df5-ab38-369fb6a8757d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00d131b1-sv5v8" Oct 30 12:32:14.928577 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:14.928559 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/856d3aa7-ff0f-4df5-ab38-369fb6a8757d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-00d131b1-sv5v8\" (UID: \"856d3aa7-ff0f-4df5-ab38-369fb6a8757d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00d131b1-sv5v8" Oct 30 12:32:14.948791 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:14.948736 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6rrd\" (UniqueName: \"kubernetes.io/projected/856d3aa7-ff0f-4df5-ab38-369fb6a8757d-kube-api-access-w6rrd\") pod \"maas-api-revoke-sub-00d131b1-sv5v8\" (UID: \"856d3aa7-ff0f-4df5-ab38-369fb6a8757d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00d131b1-sv5v8" Oct 30 12:32:15.016226 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:15.016193 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00d131b1-sv5v8" Oct 30 12:32:15.327699 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:15.327667 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00d131b1-sv5v8"] Oct 30 12:32:15.330976 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:32:15.330948 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod856d3aa7_ff0f_4df5_ab38_369fb6a8757d.slice/crio-ae2039ea04e167ba648b786fc322de6a025ed4635d8932186ead8c3754feaa9d WatchSource:0}: Error finding container ae2039ea04e167ba648b786fc322de6a025ed4635d8932186ead8c3754feaa9d: Status 404 returned error can't find the container with id ae2039ea04e167ba648b786fc322de6a025ed4635d8932186ead8c3754feaa9d Oct 30 12:32:15.785882 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:15.785416 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00d131b1-sv5v8" event={"ID":"856d3aa7-ff0f-4df5-ab38-369fb6a8757d","Type":"ContainerStarted","Data":"ae2039ea04e167ba648b786fc322de6a025ed4635d8932186ead8c3754feaa9d"} Oct 30 12:32:16.797011 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:16.796972 4908 generic.go:356] "Generic (PLEG): container finished" podID="856d3aa7-ff0f-4df5-ab38-369fb6a8757d" containerID="1218b01a178df40f6763d11e286abe5be656f4b567567f834e5f624e6a547ba6" exitCode=0 Oct 30 12:32:16.797430 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:16.797019 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00d131b1-sv5v8" event={"ID":"856d3aa7-ff0f-4df5-ab38-369fb6a8757d","Type":"ContainerDied","Data":"1218b01a178df40f6763d11e286abe5be656f4b567567f834e5f624e6a547ba6"} Oct 30 12:32:17.998850 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:17.982561 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9wwfc"] Oct 30 12:32:18.137201 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:18.137165 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00d131b1-sv5v8" Oct 30 12:32:18.238848 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:18.235306 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/856d3aa7-ff0f-4df5-ab38-369fb6a8757d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/856d3aa7-ff0f-4df5-ab38-369fb6a8757d-maas-api-openshift-ca\") pod \"856d3aa7-ff0f-4df5-ab38-369fb6a8757d\" (UID: \"856d3aa7-ff0f-4df5-ab38-369fb6a8757d\") " Oct 30 12:32:18.238848 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:18.235451 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/856d3aa7-ff0f-4df5-ab38-369fb6a8757d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/856d3aa7-ff0f-4df5-ab38-369fb6a8757d-maas-api-xks-ca\") pod \"856d3aa7-ff0f-4df5-ab38-369fb6a8757d\" (UID: \"856d3aa7-ff0f-4df5-ab38-369fb6a8757d\") " Oct 30 12:32:18.238848 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:18.235498 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/856d3aa7-ff0f-4df5-ab38-369fb6a8757d-kube-api-access-w6rrd\" (UniqueName: \"kubernetes.io/projected/856d3aa7-ff0f-4df5-ab38-369fb6a8757d-kube-api-access-w6rrd\") pod \"856d3aa7-ff0f-4df5-ab38-369fb6a8757d\" (UID: \"856d3aa7-ff0f-4df5-ab38-369fb6a8757d\") " Oct 30 12:32:18.255472 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:18.255432 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/856d3aa7-ff0f-4df5-ab38-369fb6a8757d-maas-api-openshift-ca" pod "856d3aa7-ff0f-4df5-ab38-369fb6a8757d" (UID: "856d3aa7-ff0f-4df5-ab38-369fb6a8757d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:32:18.267891 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:18.263762 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/856d3aa7-ff0f-4df5-ab38-369fb6a8757d-maas-api-xks-ca" pod "856d3aa7-ff0f-4df5-ab38-369fb6a8757d" (UID: "856d3aa7-ff0f-4df5-ab38-369fb6a8757d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:32:18.268096 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:18.264816 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/856d3aa7-ff0f-4df5-ab38-369fb6a8757d-kube-api-access-w6rrd" pod "856d3aa7-ff0f-4df5-ab38-369fb6a8757d" (UID: "856d3aa7-ff0f-4df5-ab38-369fb6a8757d"). InnerVolumeSpecName "kube-api-access-w6rrd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:32:18.337205 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:18.337168 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/856d3aa7-ff0f-4df5-ab38-369fb6a8757d-maas-api-openshift-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:32:18.337205 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:18.337205 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/856d3aa7-ff0f-4df5-ab38-369fb6a8757d-maas-api-xks-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:32:18.337405 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:18.337222 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6rrd\" (UniqueName: \"kubernetes.io/projected/856d3aa7-ff0f-4df5-ab38-369fb6a8757d-kube-api-access-w6rrd\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:32:18.816903 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:18.816862 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00d131b1-sv5v8" event={"ID":"856d3aa7-ff0f-4df5-ab38-369fb6a8757d","Type":"ContainerDied","Data":"ae2039ea04e167ba648b786fc322de6a025ed4635d8932186ead8c3754feaa9d"} Oct 30 12:32:18.817141 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:18.816927 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ae2039ea04e167ba648b786fc322de6a025ed4635d8932186ead8c3754feaa9d" Oct 30 12:32:18.817141 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:18.816882 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00d131b1-sv5v8" Oct 30 12:32:19.702040 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:32:19.701999 4908 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vn6vv" containerName="extractor" Oct 30 12:32:21.447934 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:21.447902 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-mcrfm"] Oct 30 12:32:21.448606 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:21.448582 4908 cpu_manager.go:414] "containerMap: removing container" podUID="856d3aa7-ff0f-4df5-ab38-369fb6a8757d" containerName="revoke-keys" Oct 30 12:32:21.448709 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:21.448606 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="856d3aa7-ff0f-4df5-ab38-369fb6a8757d" containerName="revoke-keys" Oct 30 12:32:21.448793 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:21.448780 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="856d3aa7-ff0f-4df5-ab38-369fb6a8757d" containerName="revoke-keys" Oct 30 12:32:21.453545 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:21.453525 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-mcrfm" Oct 30 12:32:21.476099 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:21.476070 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-mcrfm"] Oct 30 12:32:21.534424 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:21.534389 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcd6e5fa-dded-4c97-b694-a2a58766816e-credential-socket\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-mcrfm\" (UID: \"bcd6e5fa-dded-4c97-b694-a2a58766816e\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-mcrfm" Oct 30 12:32:21.534589 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:21.534434 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bcd6e5fa-dded-4c97-b694-a2a58766816e-istio-envoy\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-mcrfm\" (UID: \"bcd6e5fa-dded-4c97-b694-a2a58766816e\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-mcrfm" Oct 30 12:32:21.534646 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:21.534586 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bcd6e5fa-dded-4c97-b694-a2a58766816e-istiod-ca-cert\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-mcrfm\" (UID: \"bcd6e5fa-dded-4c97-b694-a2a58766816e\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-mcrfm" Oct 30 12:32:21.534704 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:21.534650 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bcd6e5fa-dded-4c97-b694-a2a58766816e-istio-token\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-mcrfm\" (UID: \"bcd6e5fa-dded-4c97-b694-a2a58766816e\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-mcrfm" Oct 30 12:32:21.534765 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:21.534747 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bcd6e5fa-dded-4c97-b694-a2a58766816e-istio-data\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-mcrfm\" (UID: \"bcd6e5fa-dded-4c97-b694-a2a58766816e\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-mcrfm" Oct 30 12:32:21.534840 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:21.534786 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmbwq\" (UniqueName: \"kubernetes.io/projected/bcd6e5fa-dded-4c97-b694-a2a58766816e-kube-api-access-zmbwq\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-mcrfm\" (UID: \"bcd6e5fa-dded-4c97-b694-a2a58766816e\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-mcrfm" Oct 30 12:32:21.534925 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:21.534909 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bcd6e5fa-dded-4c97-b694-a2a58766816e-workload-certs\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-mcrfm\" (UID: \"bcd6e5fa-dded-4c97-b694-a2a58766816e\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-mcrfm" Oct 30 12:32:21.534978 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:21.534957 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bcd6e5fa-dded-4c97-b694-a2a58766816e-istio-podinfo\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-mcrfm\" (UID: \"bcd6e5fa-dded-4c97-b694-a2a58766816e\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-mcrfm" Oct 30 12:32:21.535079 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:21.535064 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcd6e5fa-dded-4c97-b694-a2a58766816e-workload-socket\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-mcrfm\" (UID: \"bcd6e5fa-dded-4c97-b694-a2a58766816e\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-mcrfm" Oct 30 12:32:21.636238 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:21.636199 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bcd6e5fa-dded-4c97-b694-a2a58766816e-istio-token\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-mcrfm\" (UID: \"bcd6e5fa-dded-4c97-b694-a2a58766816e\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-mcrfm" Oct 30 12:32:21.636406 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:21.636251 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bcd6e5fa-dded-4c97-b694-a2a58766816e-istio-data\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-mcrfm\" (UID: \"bcd6e5fa-dded-4c97-b694-a2a58766816e\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-mcrfm" Oct 30 12:32:21.636406 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:21.636278 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmbwq\" (UniqueName: \"kubernetes.io/projected/bcd6e5fa-dded-4c97-b694-a2a58766816e-kube-api-access-zmbwq\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-mcrfm\" (UID: \"bcd6e5fa-dded-4c97-b694-a2a58766816e\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-mcrfm" Oct 30 12:32:21.636406 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:21.636339 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bcd6e5fa-dded-4c97-b694-a2a58766816e-workload-certs\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-mcrfm\" (UID: \"bcd6e5fa-dded-4c97-b694-a2a58766816e\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-mcrfm" Oct 30 12:32:21.636406 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:21.636380 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bcd6e5fa-dded-4c97-b694-a2a58766816e-istio-podinfo\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-mcrfm\" (UID: \"bcd6e5fa-dded-4c97-b694-a2a58766816e\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-mcrfm" Oct 30 12:32:21.636676 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:21.636449 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcd6e5fa-dded-4c97-b694-a2a58766816e-workload-socket\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-mcrfm\" (UID: \"bcd6e5fa-dded-4c97-b694-a2a58766816e\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-mcrfm" Oct 30 12:32:21.636676 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:21.636484 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcd6e5fa-dded-4c97-b694-a2a58766816e-credential-socket\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-mcrfm\" (UID: \"bcd6e5fa-dded-4c97-b694-a2a58766816e\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-mcrfm" Oct 30 12:32:21.636676 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:21.636523 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bcd6e5fa-dded-4c97-b694-a2a58766816e-istio-envoy\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-mcrfm\" (UID: \"bcd6e5fa-dded-4c97-b694-a2a58766816e\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-mcrfm" Oct 30 12:32:21.636676 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:21.636619 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bcd6e5fa-dded-4c97-b694-a2a58766816e-istiod-ca-cert\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-mcrfm\" (UID: \"bcd6e5fa-dded-4c97-b694-a2a58766816e\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-mcrfm" Oct 30 12:32:21.636947 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:21.636710 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bcd6e5fa-dded-4c97-b694-a2a58766816e-istio-data\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-mcrfm\" (UID: \"bcd6e5fa-dded-4c97-b694-a2a58766816e\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-mcrfm" Oct 30 12:32:21.636947 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:21.636779 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bcd6e5fa-dded-4c97-b694-a2a58766816e-workload-certs\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-mcrfm\" (UID: \"bcd6e5fa-dded-4c97-b694-a2a58766816e\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-mcrfm" Oct 30 12:32:21.636947 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:21.636864 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcd6e5fa-dded-4c97-b694-a2a58766816e-workload-socket\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-mcrfm\" (UID: \"bcd6e5fa-dded-4c97-b694-a2a58766816e\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-mcrfm" Oct 30 12:32:21.637109 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:21.637046 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcd6e5fa-dded-4c97-b694-a2a58766816e-credential-socket\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-mcrfm\" (UID: \"bcd6e5fa-dded-4c97-b694-a2a58766816e\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-mcrfm" Oct 30 12:32:21.637394 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:21.637367 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bcd6e5fa-dded-4c97-b694-a2a58766816e-istiod-ca-cert\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-mcrfm\" (UID: \"bcd6e5fa-dded-4c97-b694-a2a58766816e\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-mcrfm" Oct 30 12:32:21.638966 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:21.638943 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bcd6e5fa-dded-4c97-b694-a2a58766816e-istio-envoy\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-mcrfm\" (UID: \"bcd6e5fa-dded-4c97-b694-a2a58766816e\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-mcrfm" Oct 30 12:32:21.639385 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:21.639354 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bcd6e5fa-dded-4c97-b694-a2a58766816e-istio-podinfo\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-mcrfm\" (UID: \"bcd6e5fa-dded-4c97-b694-a2a58766816e\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-mcrfm" Oct 30 12:32:21.667222 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:21.667185 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmbwq\" (UniqueName: \"kubernetes.io/projected/bcd6e5fa-dded-4c97-b694-a2a58766816e-kube-api-access-zmbwq\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-mcrfm\" (UID: \"bcd6e5fa-dded-4c97-b694-a2a58766816e\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-mcrfm" Oct 30 12:32:21.669524 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:21.669497 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bcd6e5fa-dded-4c97-b694-a2a58766816e-istio-token\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-mcrfm\" (UID: \"bcd6e5fa-dded-4c97-b694-a2a58766816e\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-mcrfm" Oct 30 12:32:21.769649 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:21.769600 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-mcrfm" Oct 30 12:32:22.126594 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:22.126505 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-460afc00-5s8cg"] Oct 30 12:32:22.131432 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:22.131407 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-460afc00-5s8cg" Oct 30 12:32:22.145138 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:22.145114 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 12:32:22.151500 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:22.151223 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-460afc00-5s8cg"] Oct 30 12:32:22.157858 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:22.155858 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 12:32:22.157858 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:22.155993 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 12:32:22.157858 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:22.156043 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s75hq\"" Oct 30 12:32:22.245736 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:22.245710 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-mcrfm"] Oct 30 12:32:22.247147 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:32:22.247122 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcd6e5fa_dded_4c97_b694_a2a58766816e.slice/crio-563e52b1a5ef1fe18006962e75ced2b6fb35f0715c6bc39e10389262628aa3d8 WatchSource:0}: Error finding container 563e52b1a5ef1fe18006962e75ced2b6fb35f0715c6bc39e10389262628aa3d8: Status 404 returned error can't find the container with id 563e52b1a5ef1fe18006962e75ced2b6fb35f0715c6bc39e10389262628aa3d8 Oct 30 12:32:22.250200 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:22.250131 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:32:22.250392 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:22.250252 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:32:22.250392 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:22.250291 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:32:22.252900 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:22.252875 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwftd\" (UniqueName: \"kubernetes.io/projected/1f008d28-b03a-4c7c-8a89-c3f9a03a4a5f-kube-api-access-dwftd\") pod \"maas-api-revoke-sub-460afc00-5s8cg\" (UID: \"1f008d28-b03a-4c7c-8a89-c3f9a03a4a5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-460afc00-5s8cg" Oct 30 12:32:22.253042 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:22.253018 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f008d28-b03a-4c7c-8a89-c3f9a03a4a5f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-460afc00-5s8cg\" (UID: \"1f008d28-b03a-4c7c-8a89-c3f9a03a4a5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-460afc00-5s8cg" Oct 30 12:32:22.253106 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:22.253088 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f008d28-b03a-4c7c-8a89-c3f9a03a4a5f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-460afc00-5s8cg\" (UID: \"1f008d28-b03a-4c7c-8a89-c3f9a03a4a5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-460afc00-5s8cg" Oct 30 12:32:22.354687 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:22.354555 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwftd\" (UniqueName: \"kubernetes.io/projected/1f008d28-b03a-4c7c-8a89-c3f9a03a4a5f-kube-api-access-dwftd\") pod \"maas-api-revoke-sub-460afc00-5s8cg\" (UID: \"1f008d28-b03a-4c7c-8a89-c3f9a03a4a5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-460afc00-5s8cg" Oct 30 12:32:22.354854 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:22.354726 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f008d28-b03a-4c7c-8a89-c3f9a03a4a5f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-460afc00-5s8cg\" (UID: \"1f008d28-b03a-4c7c-8a89-c3f9a03a4a5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-460afc00-5s8cg" Oct 30 12:32:22.354854 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:22.354792 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f008d28-b03a-4c7c-8a89-c3f9a03a4a5f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-460afc00-5s8cg\" (UID: \"1f008d28-b03a-4c7c-8a89-c3f9a03a4a5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-460afc00-5s8cg" Oct 30 12:32:22.355781 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:22.355705 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f008d28-b03a-4c7c-8a89-c3f9a03a4a5f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-460afc00-5s8cg\" (UID: \"1f008d28-b03a-4c7c-8a89-c3f9a03a4a5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-460afc00-5s8cg" Oct 30 12:32:22.357793 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:22.357766 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f008d28-b03a-4c7c-8a89-c3f9a03a4a5f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-460afc00-5s8cg\" (UID: \"1f008d28-b03a-4c7c-8a89-c3f9a03a4a5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-460afc00-5s8cg" Oct 30 12:32:22.382978 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:22.382908 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwftd\" (UniqueName: \"kubernetes.io/projected/1f008d28-b03a-4c7c-8a89-c3f9a03a4a5f-kube-api-access-dwftd\") pod \"maas-api-revoke-sub-460afc00-5s8cg\" (UID: \"1f008d28-b03a-4c7c-8a89-c3f9a03a4a5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-460afc00-5s8cg" Oct 30 12:32:22.452411 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:22.452376 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-460afc00-5s8cg" Oct 30 12:32:22.800859 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:22.800812 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-460afc00-5s8cg"] Oct 30 12:32:22.906046 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:22.905949 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-460afc00-5s8cg" event={"ID":"1f008d28-b03a-4c7c-8a89-c3f9a03a4a5f","Type":"ContainerStarted","Data":"ac8fc76aa5f9ca15e9d7356b5634bee1f0fa634b6698cceb37a4e2f2212de73d"} Oct 30 12:32:22.907868 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:22.907801 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-mcrfm" event={"ID":"bcd6e5fa-dded-4c97-b694-a2a58766816e","Type":"ContainerStarted","Data":"811ec441fec961d76ef653554df13ecc16d64e0d186014187caf53628a2714a7"} Oct 30 12:32:22.908004 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:22.907878 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-mcrfm" event={"ID":"bcd6e5fa-dded-4c97-b694-a2a58766816e","Type":"ContainerStarted","Data":"563e52b1a5ef1fe18006962e75ced2b6fb35f0715c6bc39e10389262628aa3d8"} Oct 30 12:32:22.954310 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:22.954259 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-mcrfm" podStartSLOduration=1.954244496 podStartE2EDuration="1.954244496s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:32:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:32:22.951436318 +0000 UTC m=+2299.465865073" watchObservedRunningTime="2026-10-30 12:32:22.954244496 +0000 UTC m=+2299.468673241" Oct 30 12:32:23.783855 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:23.779189 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-mcrfm" Oct 30 12:32:23.783855 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:23.781048 4908 patch_prober.go:29] interesting pod/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-mcrfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body= Oct 30 12:32:23.783855 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:23.781117 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-mcrfm" podUID="bcd6e5fa-dded-4c97-b694-a2a58766816e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" Oct 30 12:32:23.924491 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:23.920373 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-460afc00-5s8cg" event={"ID":"1f008d28-b03a-4c7c-8a89-c3f9a03a4a5f","Type":"ContainerStarted","Data":"9762fe3fa5c075c77601e330914209d4ac01b159c733d31a9437767c07fa41ca"} Oct 30 12:32:23.970854 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:23.959590 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-460afc00-5s8cg" podStartSLOduration=1.959571779 podStartE2EDuration="1.959571779s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:32:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:32:23.95780692 +0000 UTC m=+2300.472235668" watchObservedRunningTime="2026-10-30 12:32:23.959571779 +0000 UTC m=+2300.474000551" Oct 30 12:32:24.771852 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:24.771181 4908 patch_prober.go:29] interesting pod/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-mcrfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body= Oct 30 12:32:24.771852 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:24.771243 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-mcrfm" podUID="bcd6e5fa-dded-4c97-b694-a2a58766816e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" Oct 30 12:32:24.928784 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:24.928745 4908 generic.go:356] "Generic (PLEG): container finished" podID="1f008d28-b03a-4c7c-8a89-c3f9a03a4a5f" containerID="9762fe3fa5c075c77601e330914209d4ac01b159c733d31a9437767c07fa41ca" exitCode=0 Oct 30 12:32:24.929197 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:24.928805 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-460afc00-5s8cg" event={"ID":"1f008d28-b03a-4c7c-8a89-c3f9a03a4a5f","Type":"ContainerDied","Data":"9762fe3fa5c075c77601e330914209d4ac01b159c733d31a9437767c07fa41ca"} Oct 30 12:32:25.772861 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:25.772139 4908 patch_prober.go:29] interesting pod/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-mcrfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body= Oct 30 12:32:25.772861 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:25.772209 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-mcrfm" podUID="bcd6e5fa-dded-4c97-b694-a2a58766816e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" Oct 30 12:32:26.134872 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:26.134849 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-460afc00-5s8cg" Oct 30 12:32:26.236402 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:26.236370 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f008d28-b03a-4c7c-8a89-c3f9a03a4a5f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f008d28-b03a-4c7c-8a89-c3f9a03a4a5f-maas-api-openshift-ca\") pod \"1f008d28-b03a-4c7c-8a89-c3f9a03a4a5f\" (UID: \"1f008d28-b03a-4c7c-8a89-c3f9a03a4a5f\") " Oct 30 12:32:26.236608 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:26.236469 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f008d28-b03a-4c7c-8a89-c3f9a03a4a5f-kube-api-access-dwftd\" (UniqueName: \"kubernetes.io/projected/1f008d28-b03a-4c7c-8a89-c3f9a03a4a5f-kube-api-access-dwftd\") pod \"1f008d28-b03a-4c7c-8a89-c3f9a03a4a5f\" (UID: \"1f008d28-b03a-4c7c-8a89-c3f9a03a4a5f\") " Oct 30 12:32:26.236608 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:26.236564 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f008d28-b03a-4c7c-8a89-c3f9a03a4a5f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f008d28-b03a-4c7c-8a89-c3f9a03a4a5f-maas-api-xks-ca\") pod \"1f008d28-b03a-4c7c-8a89-c3f9a03a4a5f\" (UID: \"1f008d28-b03a-4c7c-8a89-c3f9a03a4a5f\") " Oct 30 12:32:26.238004 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:26.237969 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f008d28-b03a-4c7c-8a89-c3f9a03a4a5f-maas-api-openshift-ca" pod "1f008d28-b03a-4c7c-8a89-c3f9a03a4a5f" (UID: "1f008d28-b03a-4c7c-8a89-c3f9a03a4a5f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:32:26.239626 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:26.239590 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f008d28-b03a-4c7c-8a89-c3f9a03a4a5f-maas-api-xks-ca" pod "1f008d28-b03a-4c7c-8a89-c3f9a03a4a5f" (UID: "1f008d28-b03a-4c7c-8a89-c3f9a03a4a5f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:32:26.243371 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:26.243341 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f008d28-b03a-4c7c-8a89-c3f9a03a4a5f-kube-api-access-dwftd" pod "1f008d28-b03a-4c7c-8a89-c3f9a03a4a5f" (UID: "1f008d28-b03a-4c7c-8a89-c3f9a03a4a5f"). InnerVolumeSpecName "kube-api-access-dwftd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:32:26.340766 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:26.340688 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f008d28-b03a-4c7c-8a89-c3f9a03a4a5f-maas-api-openshift-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:32:26.340766 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:26.340725 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwftd\" (UniqueName: \"kubernetes.io/projected/1f008d28-b03a-4c7c-8a89-c3f9a03a4a5f-kube-api-access-dwftd\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:32:26.340766 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:26.340741 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f008d28-b03a-4c7c-8a89-c3f9a03a4a5f-maas-api-xks-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:32:26.794610 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:26.794574 4908 patch_prober.go:29] interesting pod/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-mcrfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body= Oct 30 12:32:26.794996 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:26.794964 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-mcrfm" podUID="bcd6e5fa-dded-4c97-b694-a2a58766816e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" Oct 30 12:32:26.949764 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:26.949730 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-460afc00-5s8cg" event={"ID":"1f008d28-b03a-4c7c-8a89-c3f9a03a4a5f","Type":"ContainerDied","Data":"ac8fc76aa5f9ca15e9d7356b5634bee1f0fa634b6698cceb37a4e2f2212de73d"} Oct 30 12:32:26.949939 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:26.949769 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-460afc00-5s8cg" Oct 30 12:32:26.949939 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:26.949777 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ac8fc76aa5f9ca15e9d7356b5634bee1f0fa634b6698cceb37a4e2f2212de73d" Oct 30 12:32:27.773853 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:27.771188 4908 patch_prober.go:29] interesting pod/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-mcrfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body= Oct 30 12:32:27.773853 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:27.771249 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-mcrfm" podUID="bcd6e5fa-dded-4c97-b694-a2a58766816e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" Oct 30 12:32:28.773282 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:28.772778 4908 patch_prober.go:29] interesting pod/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-mcrfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body= Oct 30 12:32:28.773282 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:28.772898 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-mcrfm" podUID="bcd6e5fa-dded-4c97-b694-a2a58766816e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" Oct 30 12:32:29.879014 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:29.878978 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-mcrfm" Oct 30 12:32:29.879749 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:29.879651 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-mcrfm" Oct 30 12:32:29.880440 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:29.880418 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-mcrfm" Oct 30 12:32:34.186690 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:34.186649 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ac1c19b-4prjc"] Oct 30 12:32:34.187471 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:34.187447 4908 cpu_manager.go:414] "containerMap: removing container" podUID="1f008d28-b03a-4c7c-8a89-c3f9a03a4a5f" containerName="revoke-keys" Oct 30 12:32:34.187592 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:34.187473 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f008d28-b03a-4c7c-8a89-c3f9a03a4a5f" containerName="revoke-keys" Oct 30 12:32:34.187651 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:34.187614 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f008d28-b03a-4c7c-8a89-c3f9a03a4a5f" containerName="revoke-keys" Oct 30 12:32:34.191431 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:34.191407 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ac1c19b-4prjc" Oct 30 12:32:34.208975 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:34.208950 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ac1c19b-4prjc"] Oct 30 12:32:34.215484 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:34.215458 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 12:32:34.215631 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:34.215530 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 12:32:34.215712 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:34.215638 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 12:32:34.215887 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:34.215872 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s75hq\"" Oct 30 12:32:34.257129 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:34.257099 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7qkw\" (UniqueName: \"kubernetes.io/projected/a314719d-98ed-41dd-bb49-b0813194fbec-kube-api-access-z7qkw\") pod \"maas-api-revoke-sub-0ac1c19b-4prjc\" (UID: \"a314719d-98ed-41dd-bb49-b0813194fbec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ac1c19b-4prjc" Oct 30 12:32:34.257256 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:34.257158 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a314719d-98ed-41dd-bb49-b0813194fbec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0ac1c19b-4prjc\" (UID: \"a314719d-98ed-41dd-bb49-b0813194fbec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ac1c19b-4prjc" Oct 30 12:32:34.257256 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:34.257181 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a314719d-98ed-41dd-bb49-b0813194fbec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0ac1c19b-4prjc\" (UID: \"a314719d-98ed-41dd-bb49-b0813194fbec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ac1c19b-4prjc" Oct 30 12:32:34.358262 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:34.358234 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7qkw\" (UniqueName: \"kubernetes.io/projected/a314719d-98ed-41dd-bb49-b0813194fbec-kube-api-access-z7qkw\") pod \"maas-api-revoke-sub-0ac1c19b-4prjc\" (UID: \"a314719d-98ed-41dd-bb49-b0813194fbec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ac1c19b-4prjc" Oct 30 12:32:34.358421 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:34.358284 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a314719d-98ed-41dd-bb49-b0813194fbec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0ac1c19b-4prjc\" (UID: \"a314719d-98ed-41dd-bb49-b0813194fbec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ac1c19b-4prjc" Oct 30 12:32:34.358421 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:34.358304 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a314719d-98ed-41dd-bb49-b0813194fbec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0ac1c19b-4prjc\" (UID: \"a314719d-98ed-41dd-bb49-b0813194fbec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ac1c19b-4prjc" Oct 30 12:32:34.358928 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:34.358903 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a314719d-98ed-41dd-bb49-b0813194fbec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0ac1c19b-4prjc\" (UID: \"a314719d-98ed-41dd-bb49-b0813194fbec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ac1c19b-4prjc" Oct 30 12:32:34.360877 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:34.360848 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a314719d-98ed-41dd-bb49-b0813194fbec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0ac1c19b-4prjc\" (UID: \"a314719d-98ed-41dd-bb49-b0813194fbec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ac1c19b-4prjc" Oct 30 12:32:34.380249 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:34.380227 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7qkw\" (UniqueName: \"kubernetes.io/projected/a314719d-98ed-41dd-bb49-b0813194fbec-kube-api-access-z7qkw\") pod \"maas-api-revoke-sub-0ac1c19b-4prjc\" (UID: \"a314719d-98ed-41dd-bb49-b0813194fbec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ac1c19b-4prjc" Oct 30 12:32:34.506468 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:34.506397 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ac1c19b-4prjc" Oct 30 12:32:34.822552 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:34.822512 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ac1c19b-4prjc"] Oct 30 12:32:35.069865 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:35.062428 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ac1c19b-4prjc" event={"ID":"a314719d-98ed-41dd-bb49-b0813194fbec","Type":"ContainerStarted","Data":"fa41b4da4c7a2653a4f16b4e6db007e4d5be7469d664e2eec0ac50d7ce388110"} Oct 30 12:32:36.099069 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:36.093818 4908 generic.go:356] "Generic (PLEG): container finished" podID="a314719d-98ed-41dd-bb49-b0813194fbec" containerID="151968bffad60e1c41cb6f0e393e6db13ac05c29e581582d4625309495a4f7ea" exitCode=0 Oct 30 12:32:36.099069 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:36.094089 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ac1c19b-4prjc" event={"ID":"a314719d-98ed-41dd-bb49-b0813194fbec","Type":"ContainerDied","Data":"151968bffad60e1c41cb6f0e393e6db13ac05c29e581582d4625309495a4f7ea"} Oct 30 12:32:37.268743 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:37.268714 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ac1c19b-4prjc" Oct 30 12:32:37.360481 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:37.360388 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a314719d-98ed-41dd-bb49-b0813194fbec-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a314719d-98ed-41dd-bb49-b0813194fbec-maas-api-xks-ca\") pod \"a314719d-98ed-41dd-bb49-b0813194fbec\" (UID: \"a314719d-98ed-41dd-bb49-b0813194fbec\") " Oct 30 12:32:37.360481 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:37.360468 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a314719d-98ed-41dd-bb49-b0813194fbec-kube-api-access-z7qkw\" (UniqueName: \"kubernetes.io/projected/a314719d-98ed-41dd-bb49-b0813194fbec-kube-api-access-z7qkw\") pod \"a314719d-98ed-41dd-bb49-b0813194fbec\" (UID: \"a314719d-98ed-41dd-bb49-b0813194fbec\") " Oct 30 12:32:37.360724 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:37.360546 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a314719d-98ed-41dd-bb49-b0813194fbec-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a314719d-98ed-41dd-bb49-b0813194fbec-maas-api-openshift-ca\") pod \"a314719d-98ed-41dd-bb49-b0813194fbec\" (UID: \"a314719d-98ed-41dd-bb49-b0813194fbec\") " Oct 30 12:32:37.360961 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:37.360929 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a314719d-98ed-41dd-bb49-b0813194fbec-maas-api-openshift-ca" pod "a314719d-98ed-41dd-bb49-b0813194fbec" (UID: "a314719d-98ed-41dd-bb49-b0813194fbec"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:32:37.361112 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:37.361061 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a314719d-98ed-41dd-bb49-b0813194fbec-maas-api-openshift-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:32:37.363159 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:37.363133 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a314719d-98ed-41dd-bb49-b0813194fbec-maas-api-xks-ca" pod "a314719d-98ed-41dd-bb49-b0813194fbec" (UID: "a314719d-98ed-41dd-bb49-b0813194fbec"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:32:37.363350 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:37.363330 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a314719d-98ed-41dd-bb49-b0813194fbec-kube-api-access-z7qkw" pod "a314719d-98ed-41dd-bb49-b0813194fbec" (UID: "a314719d-98ed-41dd-bb49-b0813194fbec"). InnerVolumeSpecName "kube-api-access-z7qkw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:32:37.462347 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:37.462313 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a314719d-98ed-41dd-bb49-b0813194fbec-maas-api-xks-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:32:37.462347 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:37.462344 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7qkw\" (UniqueName: \"kubernetes.io/projected/a314719d-98ed-41dd-bb49-b0813194fbec-kube-api-access-z7qkw\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:32:38.111250 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:38.111223 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ac1c19b-4prjc" Oct 30 12:32:38.111407 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:38.111215 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ac1c19b-4prjc" event={"ID":"a314719d-98ed-41dd-bb49-b0813194fbec","Type":"ContainerDied","Data":"fa41b4da4c7a2653a4f16b4e6db007e4d5be7469d664e2eec0ac50d7ce388110"} Oct 30 12:32:38.111407 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:38.111360 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fa41b4da4c7a2653a4f16b4e6db007e4d5be7469d664e2eec0ac50d7ce388110" Oct 30 12:32:39.064429 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:39.064390 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c44eee-956mb"] Oct 30 12:32:39.065220 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:39.065197 4908 cpu_manager.go:414] "containerMap: removing container" podUID="a314719d-98ed-41dd-bb49-b0813194fbec" containerName="revoke-keys" Oct 30 12:32:39.065266 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:39.065223 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a314719d-98ed-41dd-bb49-b0813194fbec" containerName="revoke-keys" Oct 30 12:32:39.065393 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:39.065382 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="a314719d-98ed-41dd-bb49-b0813194fbec" containerName="revoke-keys" Oct 30 12:32:39.069587 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:39.069565 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c44eee-956mb" Oct 30 12:32:39.087443 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:39.087409 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c44eee-956mb"] Oct 30 12:32:39.092785 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:39.092550 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s75hq\"" Oct 30 12:32:39.094024 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:39.093922 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 12:32:39.094024 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:39.093999 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 12:32:39.094261 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:39.094245 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 12:32:39.190820 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:39.190781 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd471c76-9e74-48ee-8a46-4587981cfe25-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-c44eee-956mb\" (UID: \"fd471c76-9e74-48ee-8a46-4587981cfe25\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c44eee-956mb" Oct 30 12:32:39.191077 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:39.191051 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd471c76-9e74-48ee-8a46-4587981cfe25-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-c44eee-956mb\" (UID: \"fd471c76-9e74-48ee-8a46-4587981cfe25\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c44eee-956mb" Oct 30 12:32:39.191164 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:39.191095 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vpdj\" (UniqueName: \"kubernetes.io/projected/fd471c76-9e74-48ee-8a46-4587981cfe25-kube-api-access-6vpdj\") pod \"maas-api-revoke-keys-e2e-praxis-a-c44eee-956mb\" (UID: \"fd471c76-9e74-48ee-8a46-4587981cfe25\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c44eee-956mb" Oct 30 12:32:39.292518 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:39.292478 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd471c76-9e74-48ee-8a46-4587981cfe25-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-c44eee-956mb\" (UID: \"fd471c76-9e74-48ee-8a46-4587981cfe25\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c44eee-956mb" Oct 30 12:32:39.292518 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:39.292521 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vpdj\" (UniqueName: \"kubernetes.io/projected/fd471c76-9e74-48ee-8a46-4587981cfe25-kube-api-access-6vpdj\") pod \"maas-api-revoke-keys-e2e-praxis-a-c44eee-956mb\" (UID: \"fd471c76-9e74-48ee-8a46-4587981cfe25\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c44eee-956mb" Oct 30 12:32:39.292778 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:39.292639 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd471c76-9e74-48ee-8a46-4587981cfe25-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-c44eee-956mb\" (UID: \"fd471c76-9e74-48ee-8a46-4587981cfe25\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c44eee-956mb" Oct 30 12:32:39.293350 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:39.293320 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd471c76-9e74-48ee-8a46-4587981cfe25-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-c44eee-956mb\" (UID: \"fd471c76-9e74-48ee-8a46-4587981cfe25\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c44eee-956mb" Oct 30 12:32:39.295212 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:39.295187 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd471c76-9e74-48ee-8a46-4587981cfe25-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-c44eee-956mb\" (UID: \"fd471c76-9e74-48ee-8a46-4587981cfe25\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c44eee-956mb" Oct 30 12:32:39.314586 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:39.314517 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vpdj\" (UniqueName: \"kubernetes.io/projected/fd471c76-9e74-48ee-8a46-4587981cfe25-kube-api-access-6vpdj\") pod \"maas-api-revoke-keys-e2e-praxis-a-c44eee-956mb\" (UID: \"fd471c76-9e74-48ee-8a46-4587981cfe25\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c44eee-956mb" Oct 30 12:32:39.388370 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:39.388333 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c44eee-956mb" Oct 30 12:32:39.701516 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:39.695395 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c44eee-956mb"] Oct 30 12:32:40.127898 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:40.127856 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c44eee-956mb" event={"ID":"fd471c76-9e74-48ee-8a46-4587981cfe25","Type":"ContainerStarted","Data":"3de7f024f719a3244919b92e0bef6983423b60207f3fe64a0f7e55d865409966"} Oct 30 12:32:40.128295 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:40.127953 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c44eee-956mb" event={"ID":"fd471c76-9e74-48ee-8a46-4587981cfe25","Type":"ContainerStarted","Data":"c73b3937a956490453be50359189bb7ceb2ba211244ffd57d1758c5e4c2bb8ae"} Oct 30 12:32:40.166546 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:40.166488 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c44eee-956mb" podStartSLOduration=1.1664762 podStartE2EDuration="1.1664762s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:32:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:32:40.16573315 +0000 UTC m=+2316.680161913" watchObservedRunningTime="2026-10-30 12:32:40.1664762 +0000 UTC m=+2316.680904946" Oct 30 12:32:41.136687 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:41.136649 4908 generic.go:356] "Generic (PLEG): container finished" podID="fd471c76-9e74-48ee-8a46-4587981cfe25" containerID="3de7f024f719a3244919b92e0bef6983423b60207f3fe64a0f7e55d865409966" exitCode=0 Oct 30 12:32:41.137163 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:41.136731 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c44eee-956mb" event={"ID":"fd471c76-9e74-48ee-8a46-4587981cfe25","Type":"ContainerDied","Data":"3de7f024f719a3244919b92e0bef6983423b60207f3fe64a0f7e55d865409966"} Oct 30 12:32:42.228736 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:42.228698 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-qrkpg"] Oct 30 12:32:42.229283 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:42.228937 4908 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-qrkpg" podUID="8cf0a817-eff3-454d-8ddc-0a811a951647" containerName="istio-proxy" containerID="cri-o://f0026a8a3630be0abba041c0b62b4bf389e9db608c517bb959d11f4d1f27b561" gracePeriod=30 Oct 30 12:32:42.308843 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:42.308790 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38ca6809-9ncpb"] Oct 30 12:32:42.313928 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:42.313907 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38ca6809-9ncpb" Oct 30 12:32:42.336260 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:42.336209 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38ca6809-9ncpb"] Oct 30 12:32:42.388626 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:42.388328 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c44eee-956mb" Oct 30 12:32:42.451467 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:42.451388 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t95r9\" (UniqueName: \"kubernetes.io/projected/a5a7349e-bc7b-4958-b785-222620bdcfc1-kube-api-access-t95r9\") pod \"maas-api-revoke-sub-38ca6809-9ncpb\" (UID: \"a5a7349e-bc7b-4958-b785-222620bdcfc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38ca6809-9ncpb" Oct 30 12:32:42.451606 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:42.451546 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5a7349e-bc7b-4958-b785-222620bdcfc1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-38ca6809-9ncpb\" (UID: \"a5a7349e-bc7b-4958-b785-222620bdcfc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38ca6809-9ncpb" Oct 30 12:32:42.451606 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:42.451588 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5a7349e-bc7b-4958-b785-222620bdcfc1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-38ca6809-9ncpb\" (UID: \"a5a7349e-bc7b-4958-b785-222620bdcfc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38ca6809-9ncpb" Oct 30 12:32:42.552272 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:42.552240 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd471c76-9e74-48ee-8a46-4587981cfe25-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd471c76-9e74-48ee-8a46-4587981cfe25-maas-api-openshift-ca\") pod \"fd471c76-9e74-48ee-8a46-4587981cfe25\" (UID: \"fd471c76-9e74-48ee-8a46-4587981cfe25\") " Oct 30 12:32:42.552464 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:42.552408 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd471c76-9e74-48ee-8a46-4587981cfe25-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd471c76-9e74-48ee-8a46-4587981cfe25-maas-api-xks-ca\") pod \"fd471c76-9e74-48ee-8a46-4587981cfe25\" (UID: \"fd471c76-9e74-48ee-8a46-4587981cfe25\") " Oct 30 12:32:42.552542 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:42.552487 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd471c76-9e74-48ee-8a46-4587981cfe25-kube-api-access-6vpdj\" (UniqueName: \"kubernetes.io/projected/fd471c76-9e74-48ee-8a46-4587981cfe25-kube-api-access-6vpdj\") pod \"fd471c76-9e74-48ee-8a46-4587981cfe25\" (UID: \"fd471c76-9e74-48ee-8a46-4587981cfe25\") " Oct 30 12:32:42.552607 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:42.552581 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd471c76-9e74-48ee-8a46-4587981cfe25-maas-api-openshift-ca" pod "fd471c76-9e74-48ee-8a46-4587981cfe25" (UID: "fd471c76-9e74-48ee-8a46-4587981cfe25"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:32:42.552681 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:42.552664 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t95r9\" (UniqueName: \"kubernetes.io/projected/a5a7349e-bc7b-4958-b785-222620bdcfc1-kube-api-access-t95r9\") pod \"maas-api-revoke-sub-38ca6809-9ncpb\" (UID: \"a5a7349e-bc7b-4958-b785-222620bdcfc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38ca6809-9ncpb" Oct 30 12:32:42.552773 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:42.552760 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5a7349e-bc7b-4958-b785-222620bdcfc1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-38ca6809-9ncpb\" (UID: \"a5a7349e-bc7b-4958-b785-222620bdcfc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38ca6809-9ncpb" Oct 30 12:32:42.552866 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:42.552791 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5a7349e-bc7b-4958-b785-222620bdcfc1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-38ca6809-9ncpb\" (UID: \"a5a7349e-bc7b-4958-b785-222620bdcfc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38ca6809-9ncpb" Oct 30 12:32:42.553084 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:42.553060 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd471c76-9e74-48ee-8a46-4587981cfe25-maas-api-openshift-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:32:42.553586 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:42.553557 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5a7349e-bc7b-4958-b785-222620bdcfc1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-38ca6809-9ncpb\" (UID: \"a5a7349e-bc7b-4958-b785-222620bdcfc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38ca6809-9ncpb" Oct 30 12:32:42.555222 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:42.555184 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd471c76-9e74-48ee-8a46-4587981cfe25-kube-api-access-6vpdj" pod "fd471c76-9e74-48ee-8a46-4587981cfe25" (UID: "fd471c76-9e74-48ee-8a46-4587981cfe25"). InnerVolumeSpecName "kube-api-access-6vpdj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:32:42.555510 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:42.555474 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5a7349e-bc7b-4958-b785-222620bdcfc1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-38ca6809-9ncpb\" (UID: \"a5a7349e-bc7b-4958-b785-222620bdcfc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38ca6809-9ncpb" Oct 30 12:32:42.555661 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:42.555639 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd471c76-9e74-48ee-8a46-4587981cfe25-maas-api-xks-ca" pod "fd471c76-9e74-48ee-8a46-4587981cfe25" (UID: "fd471c76-9e74-48ee-8a46-4587981cfe25"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:32:42.606706 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:42.604609 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t95r9\" (UniqueName: \"kubernetes.io/projected/a5a7349e-bc7b-4958-b785-222620bdcfc1-kube-api-access-t95r9\") pod \"maas-api-revoke-sub-38ca6809-9ncpb\" (UID: \"a5a7349e-bc7b-4958-b785-222620bdcfc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38ca6809-9ncpb" Oct 30 12:32:42.655113 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:42.655076 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd471c76-9e74-48ee-8a46-4587981cfe25-maas-api-xks-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:32:42.655113 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:42.655106 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vpdj\" (UniqueName: \"kubernetes.io/projected/fd471c76-9e74-48ee-8a46-4587981cfe25-kube-api-access-6vpdj\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:32:42.684405 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:42.684367 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38ca6809-9ncpb" Oct 30 12:32:42.990644 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:42.990611 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38ca6809-9ncpb"] Oct 30 12:32:42.993913 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:32:42.993850 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5a7349e_bc7b_4958_b785_222620bdcfc1.slice/crio-693d220ca54e4140bc7cbcbf2f487be357dbbd893584a027f8db2e250fd2f78f WatchSource:0}: Error finding container 693d220ca54e4140bc7cbcbf2f487be357dbbd893584a027f8db2e250fd2f78f: Status 404 returned error can't find the container with id 693d220ca54e4140bc7cbcbf2f487be357dbbd893584a027f8db2e250fd2f78f Oct 30 12:32:43.156226 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:43.156196 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38ca6809-9ncpb" event={"ID":"a5a7349e-bc7b-4958-b785-222620bdcfc1","Type":"ContainerStarted","Data":"693d220ca54e4140bc7cbcbf2f487be357dbbd893584a027f8db2e250fd2f78f"} Oct 30 12:32:43.157871 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:43.157807 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c44eee-956mb" event={"ID":"fd471c76-9e74-48ee-8a46-4587981cfe25","Type":"ContainerDied","Data":"c73b3937a956490453be50359189bb7ceb2ba211244ffd57d1758c5e4c2bb8ae"} Oct 30 12:32:43.157991 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:43.157891 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c73b3937a956490453be50359189bb7ceb2ba211244ffd57d1758c5e4c2bb8ae" Oct 30 12:32:43.157991 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:43.157820 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c44eee-956mb" Oct 30 12:32:44.175590 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:44.175487 4908 generic.go:356] "Generic (PLEG): container finished" podID="a5a7349e-bc7b-4958-b785-222620bdcfc1" containerID="a905ebef08e66a7e5d0ea43808d2d913fd30c46433178de7e41ed403fac0af7f" exitCode=0 Oct 30 12:32:44.176087 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:44.175654 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38ca6809-9ncpb" event={"ID":"a5a7349e-bc7b-4958-b785-222620bdcfc1","Type":"ContainerDied","Data":"a905ebef08e66a7e5d0ea43808d2d913fd30c46433178de7e41ed403fac0af7f"} Oct 30 12:32:45.367802 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:45.367767 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38ca6809-9ncpb" Oct 30 12:32:45.393452 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:45.393421 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5a7349e-bc7b-4958-b785-222620bdcfc1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5a7349e-bc7b-4958-b785-222620bdcfc1-maas-api-openshift-ca\") pod \"a5a7349e-bc7b-4958-b785-222620bdcfc1\" (UID: \"a5a7349e-bc7b-4958-b785-222620bdcfc1\") " Oct 30 12:32:45.393605 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:45.393578 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5a7349e-bc7b-4958-b785-222620bdcfc1-kube-api-access-t95r9\" (UniqueName: \"kubernetes.io/projected/a5a7349e-bc7b-4958-b785-222620bdcfc1-kube-api-access-t95r9\") pod \"a5a7349e-bc7b-4958-b785-222620bdcfc1\" (UID: \"a5a7349e-bc7b-4958-b785-222620bdcfc1\") " Oct 30 12:32:45.393674 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:45.393608 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a5a7349e-bc7b-4958-b785-222620bdcfc1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5a7349e-bc7b-4958-b785-222620bdcfc1-maas-api-xks-ca\") pod \"a5a7349e-bc7b-4958-b785-222620bdcfc1\" (UID: \"a5a7349e-bc7b-4958-b785-222620bdcfc1\") " Oct 30 12:32:45.393816 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:45.393783 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5a7349e-bc7b-4958-b785-222620bdcfc1-maas-api-openshift-ca" pod "a5a7349e-bc7b-4958-b785-222620bdcfc1" (UID: "a5a7349e-bc7b-4958-b785-222620bdcfc1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:32:45.394186 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:45.394161 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5a7349e-bc7b-4958-b785-222620bdcfc1-maas-api-openshift-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:32:45.396988 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:45.396962 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5a7349e-bc7b-4958-b785-222620bdcfc1-maas-api-xks-ca" pod "a5a7349e-bc7b-4958-b785-222620bdcfc1" (UID: "a5a7349e-bc7b-4958-b785-222620bdcfc1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:32:45.397107 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:45.397000 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5a7349e-bc7b-4958-b785-222620bdcfc1-kube-api-access-t95r9" pod "a5a7349e-bc7b-4958-b785-222620bdcfc1" (UID: "a5a7349e-bc7b-4958-b785-222620bdcfc1"). InnerVolumeSpecName "kube-api-access-t95r9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:32:45.495107 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:45.495072 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5a7349e-bc7b-4958-b785-222620bdcfc1-maas-api-xks-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:32:45.495107 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:45.495105 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t95r9\" (UniqueName: \"kubernetes.io/projected/a5a7349e-bc7b-4958-b785-222620bdcfc1-kube-api-access-t95r9\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:32:46.194409 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:46.194374 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38ca6809-9ncpb" Oct 30 12:32:46.194588 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:46.194411 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38ca6809-9ncpb" event={"ID":"a5a7349e-bc7b-4958-b785-222620bdcfc1","Type":"ContainerDied","Data":"693d220ca54e4140bc7cbcbf2f487be357dbbd893584a027f8db2e250fd2f78f"} Oct 30 12:32:46.194588 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:46.194462 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="693d220ca54e4140bc7cbcbf2f487be357dbbd893584a027f8db2e250fd2f78f" Oct 30 12:32:47.336069 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:47.336036 4908 patch_prober.go:29] interesting pod/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-qrkpg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Oct 30 12:32:47.336394 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:47.336120 4908 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-qrkpg" podUID="8cf0a817-eff3-454d-8ddc-0a811a951647" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Oct 30 12:32:47.564972 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:47.564944 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-qrkpg" Oct 30 12:32:47.620672 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:47.620637 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8cf0a817-eff3-454d-8ddc-0a811a951647-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cf0a817-eff3-454d-8ddc-0a811a951647-credential-socket\") pod \"8cf0a817-eff3-454d-8ddc-0a811a951647\" (UID: \"8cf0a817-eff3-454d-8ddc-0a811a951647\") " Oct 30 12:32:47.620869 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:47.620684 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8cf0a817-eff3-454d-8ddc-0a811a951647-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cf0a817-eff3-454d-8ddc-0a811a951647-workload-socket\") pod \"8cf0a817-eff3-454d-8ddc-0a811a951647\" (UID: \"8cf0a817-eff3-454d-8ddc-0a811a951647\") " Oct 30 12:32:47.620869 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:47.620762 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8cf0a817-eff3-454d-8ddc-0a811a951647-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8cf0a817-eff3-454d-8ddc-0a811a951647-istio-envoy\") pod \"8cf0a817-eff3-454d-8ddc-0a811a951647\" (UID: \"8cf0a817-eff3-454d-8ddc-0a811a951647\") " Oct 30 12:32:47.621017 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:47.620870 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8cf0a817-eff3-454d-8ddc-0a811a951647-istio-token\" (UniqueName: \"kubernetes.io/projected/8cf0a817-eff3-454d-8ddc-0a811a951647-istio-token\") pod \"8cf0a817-eff3-454d-8ddc-0a811a951647\" (UID: \"8cf0a817-eff3-454d-8ddc-0a811a951647\") " Oct 30 12:32:47.621017 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:47.620938 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8cf0a817-eff3-454d-8ddc-0a811a951647-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8cf0a817-eff3-454d-8ddc-0a811a951647-workload-certs\") pod \"8cf0a817-eff3-454d-8ddc-0a811a951647\" (UID: \"8cf0a817-eff3-454d-8ddc-0a811a951647\") " Oct 30 12:32:47.621017 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:47.621009 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8cf0a817-eff3-454d-8ddc-0a811a951647-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8cf0a817-eff3-454d-8ddc-0a811a951647-istio-data\") pod \"8cf0a817-eff3-454d-8ddc-0a811a951647\" (UID: \"8cf0a817-eff3-454d-8ddc-0a811a951647\") " Oct 30 12:32:47.621183 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:47.621054 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8cf0a817-eff3-454d-8ddc-0a811a951647-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8cf0a817-eff3-454d-8ddc-0a811a951647-istiod-ca-cert\") pod \"8cf0a817-eff3-454d-8ddc-0a811a951647\" (UID: \"8cf0a817-eff3-454d-8ddc-0a811a951647\") " Oct 30 12:32:47.621183 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:47.621123 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8cf0a817-eff3-454d-8ddc-0a811a951647-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8cf0a817-eff3-454d-8ddc-0a811a951647-istio-podinfo\") pod \"8cf0a817-eff3-454d-8ddc-0a811a951647\" (UID: \"8cf0a817-eff3-454d-8ddc-0a811a951647\") " Oct 30 12:32:47.621288 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:47.621197 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8cf0a817-eff3-454d-8ddc-0a811a951647-kube-api-access-zw5n7\" (UniqueName: \"kubernetes.io/projected/8cf0a817-eff3-454d-8ddc-0a811a951647-kube-api-access-zw5n7\") pod \"8cf0a817-eff3-454d-8ddc-0a811a951647\" (UID: \"8cf0a817-eff3-454d-8ddc-0a811a951647\") " Oct 30 12:32:47.622733 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:47.622091 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8cf0a817-eff3-454d-8ddc-0a811a951647-workload-socket" pod "8cf0a817-eff3-454d-8ddc-0a811a951647" (UID: "8cf0a817-eff3-454d-8ddc-0a811a951647"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:32:47.622733 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:47.622646 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cf0a817-eff3-454d-8ddc-0a811a951647-istiod-ca-cert" pod "8cf0a817-eff3-454d-8ddc-0a811a951647" (UID: "8cf0a817-eff3-454d-8ddc-0a811a951647"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:32:47.622733 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:47.622709 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8cf0a817-eff3-454d-8ddc-0a811a951647-credential-socket" pod "8cf0a817-eff3-454d-8ddc-0a811a951647" (UID: "8cf0a817-eff3-454d-8ddc-0a811a951647"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:32:47.623244 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:47.623211 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8cf0a817-eff3-454d-8ddc-0a811a951647-istio-data" pod "8cf0a817-eff3-454d-8ddc-0a811a951647" (UID: "8cf0a817-eff3-454d-8ddc-0a811a951647"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:32:47.623780 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:47.623760 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8cf0a817-eff3-454d-8ddc-0a811a951647-workload-certs" pod "8cf0a817-eff3-454d-8ddc-0a811a951647" (UID: "8cf0a817-eff3-454d-8ddc-0a811a951647"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:32:47.633868 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:47.632167 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cf0a817-eff3-454d-8ddc-0a811a951647-kube-api-access-zw5n7" pod "8cf0a817-eff3-454d-8ddc-0a811a951647" (UID: "8cf0a817-eff3-454d-8ddc-0a811a951647"). InnerVolumeSpecName "kube-api-access-zw5n7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:32:47.634665 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:47.633124 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8cf0a817-eff3-454d-8ddc-0a811a951647-istio-envoy" pod "8cf0a817-eff3-454d-8ddc-0a811a951647" (UID: "8cf0a817-eff3-454d-8ddc-0a811a951647"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:32:47.634763 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:47.633250 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cf0a817-eff3-454d-8ddc-0a811a951647-istio-token" pod "8cf0a817-eff3-454d-8ddc-0a811a951647" (UID: "8cf0a817-eff3-454d-8ddc-0a811a951647"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:32:47.636061 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:47.636030 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8cf0a817-eff3-454d-8ddc-0a811a951647-istio-podinfo" pod "8cf0a817-eff3-454d-8ddc-0a811a951647" (UID: "8cf0a817-eff3-454d-8ddc-0a811a951647"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 12:32:47.722595 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:47.722566 4908 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cf0a817-eff3-454d-8ddc-0a811a951647-credential-socket\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:32:47.722595 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:47.722593 4908 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cf0a817-eff3-454d-8ddc-0a811a951647-workload-socket\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:32:47.722740 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:47.722604 4908 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8cf0a817-eff3-454d-8ddc-0a811a951647-istio-envoy\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:32:47.722740 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:47.722613 4908 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8cf0a817-eff3-454d-8ddc-0a811a951647-istio-token\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:32:47.722740 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:47.722621 4908 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8cf0a817-eff3-454d-8ddc-0a811a951647-workload-certs\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:32:47.722740 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:47.722628 4908 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8cf0a817-eff3-454d-8ddc-0a811a951647-istio-data\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:32:47.722740 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:47.722635 4908 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8cf0a817-eff3-454d-8ddc-0a811a951647-istiod-ca-cert\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:32:47.722740 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:47.722643 4908 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8cf0a817-eff3-454d-8ddc-0a811a951647-istio-podinfo\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:32:47.722740 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:47.722651 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zw5n7\" (UniqueName: \"kubernetes.io/projected/8cf0a817-eff3-454d-8ddc-0a811a951647-kube-api-access-zw5n7\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:32:48.213591 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:48.213552 4908 generic.go:356] "Generic (PLEG): container finished" podID="8cf0a817-eff3-454d-8ddc-0a811a951647" containerID="f0026a8a3630be0abba041c0b62b4bf389e9db608c517bb959d11f4d1f27b561" exitCode=0 Oct 30 12:32:48.213795 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:48.213606 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-qrkpg" event={"ID":"8cf0a817-eff3-454d-8ddc-0a811a951647","Type":"ContainerDied","Data":"f0026a8a3630be0abba041c0b62b4bf389e9db608c517bb959d11f4d1f27b561"} Oct 30 12:32:48.213795 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:48.213689 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-qrkpg" event={"ID":"8cf0a817-eff3-454d-8ddc-0a811a951647","Type":"ContainerDied","Data":"8fde4183f1a0a7afca13fc2809839e9b2028f38e2f813efb6404b78ed535b78a"} Oct 30 12:32:48.213795 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:48.213724 4908 scope.go:122] "RemoveContainer" containerID="f0026a8a3630be0abba041c0b62b4bf389e9db608c517bb959d11f4d1f27b561" Oct 30 12:32:48.213795 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:48.213618 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-qrkpg" Oct 30 12:32:48.225329 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:48.225312 4908 scope.go:122] "RemoveContainer" containerID="f0026a8a3630be0abba041c0b62b4bf389e9db608c517bb959d11f4d1f27b561" Oct 30 12:32:48.225615 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:32:48.225596 4908 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0026a8a3630be0abba041c0b62b4bf389e9db608c517bb959d11f4d1f27b561\": container with ID starting with f0026a8a3630be0abba041c0b62b4bf389e9db608c517bb959d11f4d1f27b561 not found: ID does not exist" containerID="f0026a8a3630be0abba041c0b62b4bf389e9db608c517bb959d11f4d1f27b561" Oct 30 12:32:48.225670 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:48.225623 4908 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0026a8a3630be0abba041c0b62b4bf389e9db608c517bb959d11f4d1f27b561"} err="failed to get container status \"f0026a8a3630be0abba041c0b62b4bf389e9db608c517bb959d11f4d1f27b561\": rpc error: code = NotFound desc = could not find container \"f0026a8a3630be0abba041c0b62b4bf389e9db608c517bb959d11f4d1f27b561\": container with ID starting with f0026a8a3630be0abba041c0b62b4bf389e9db608c517bb959d11f4d1f27b561 not found: ID does not exist" Oct 30 12:32:48.275210 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:48.275185 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-qrkpg"] Oct 30 12:32:48.286664 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:48.286645 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-qrkpg"] Oct 30 12:32:49.608389 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:49.608355 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8cf0a817-eff3-454d-8ddc-0a811a951647" path="/var/lib/kubelet/pods/8cf0a817-eff3-454d-8ddc-0a811a951647/volumes" Oct 30 12:32:49.699848 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:32:49.699757 4908 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vn6vv" containerName="extractor" Oct 30 12:32:50.078496 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:50.078461 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-7z8r7"] Oct 30 12:32:50.079073 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:50.079047 4908 cpu_manager.go:414] "containerMap: removing container" podUID="8cf0a817-eff3-454d-8ddc-0a811a951647" containerName="istio-proxy" Oct 30 12:32:50.079255 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:50.079232 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8cf0a817-eff3-454d-8ddc-0a811a951647" containerName="istio-proxy" Oct 30 12:32:50.079331 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:50.079289 4908 cpu_manager.go:414] "containerMap: removing container" podUID="fd471c76-9e74-48ee-8a46-4587981cfe25" containerName="revoke-keys" Oct 30 12:32:50.079331 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:50.079301 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd471c76-9e74-48ee-8a46-4587981cfe25" containerName="revoke-keys" Oct 30 12:32:50.079405 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:50.079347 4908 cpu_manager.go:414] "containerMap: removing container" podUID="a5a7349e-bc7b-4958-b785-222620bdcfc1" containerName="revoke-keys" Oct 30 12:32:50.079405 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:50.079356 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5a7349e-bc7b-4958-b785-222620bdcfc1" containerName="revoke-keys" Oct 30 12:32:50.079541 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:50.079526 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5a7349e-bc7b-4958-b785-222620bdcfc1" containerName="revoke-keys" Oct 30 12:32:50.079575 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:50.079550 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="8cf0a817-eff3-454d-8ddc-0a811a951647" containerName="istio-proxy" Oct 30 12:32:50.079575 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:50.079565 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd471c76-9e74-48ee-8a46-4587981cfe25" containerName="revoke-keys" Oct 30 12:32:50.085051 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:50.085032 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-7z8r7" Oct 30 12:32:50.116656 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:50.116623 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-7z8r7"] Oct 30 12:32:50.154715 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:50.154684 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b875062-a976-42cd-8b43-85dc197f3f4c-istio-envoy\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-7z8r7\" (UID: \"6b875062-a976-42cd-8b43-85dc197f3f4c\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-7z8r7" Oct 30 12:32:50.154874 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:50.154744 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sv9rt\" (UniqueName: \"kubernetes.io/projected/6b875062-a976-42cd-8b43-85dc197f3f4c-kube-api-access-sv9rt\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-7z8r7\" (UID: \"6b875062-a976-42cd-8b43-85dc197f3f4c\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-7z8r7" Oct 30 12:32:50.154874 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:50.154861 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b875062-a976-42cd-8b43-85dc197f3f4c-istio-data\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-7z8r7\" (UID: \"6b875062-a976-42cd-8b43-85dc197f3f4c\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-7z8r7" Oct 30 12:32:50.155009 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:50.154892 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b875062-a976-42cd-8b43-85dc197f3f4c-istio-podinfo\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-7z8r7\" (UID: \"6b875062-a976-42cd-8b43-85dc197f3f4c\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-7z8r7" Oct 30 12:32:50.155073 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:50.155048 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b875062-a976-42cd-8b43-85dc197f3f4c-workload-socket\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-7z8r7\" (UID: \"6b875062-a976-42cd-8b43-85dc197f3f4c\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-7z8r7" Oct 30 12:32:50.155133 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:50.155097 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b875062-a976-42cd-8b43-85dc197f3f4c-istio-token\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-7z8r7\" (UID: \"6b875062-a976-42cd-8b43-85dc197f3f4c\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-7z8r7" Oct 30 12:32:50.155207 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:50.155183 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b875062-a976-42cd-8b43-85dc197f3f4c-workload-certs\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-7z8r7\" (UID: \"6b875062-a976-42cd-8b43-85dc197f3f4c\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-7z8r7" Oct 30 12:32:50.155263 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:50.155250 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b875062-a976-42cd-8b43-85dc197f3f4c-istiod-ca-cert\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-7z8r7\" (UID: \"6b875062-a976-42cd-8b43-85dc197f3f4c\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-7z8r7" Oct 30 12:32:50.155324 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:50.155306 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b875062-a976-42cd-8b43-85dc197f3f4c-credential-socket\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-7z8r7\" (UID: \"6b875062-a976-42cd-8b43-85dc197f3f4c\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-7z8r7" Oct 30 12:32:50.262853 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:50.257237 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b875062-a976-42cd-8b43-85dc197f3f4c-istio-data\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-7z8r7\" (UID: \"6b875062-a976-42cd-8b43-85dc197f3f4c\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-7z8r7" Oct 30 12:32:50.262853 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:50.257320 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b875062-a976-42cd-8b43-85dc197f3f4c-istio-podinfo\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-7z8r7\" (UID: \"6b875062-a976-42cd-8b43-85dc197f3f4c\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-7z8r7" Oct 30 12:32:50.262853 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:50.257431 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b875062-a976-42cd-8b43-85dc197f3f4c-workload-socket\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-7z8r7\" (UID: \"6b875062-a976-42cd-8b43-85dc197f3f4c\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-7z8r7" Oct 30 12:32:50.262853 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:50.257472 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b875062-a976-42cd-8b43-85dc197f3f4c-istio-token\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-7z8r7\" (UID: \"6b875062-a976-42cd-8b43-85dc197f3f4c\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-7z8r7" Oct 30 12:32:50.262853 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:50.257821 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b875062-a976-42cd-8b43-85dc197f3f4c-workload-certs\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-7z8r7\" (UID: \"6b875062-a976-42cd-8b43-85dc197f3f4c\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-7z8r7" Oct 30 12:32:50.262853 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:50.257926 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b875062-a976-42cd-8b43-85dc197f3f4c-istiod-ca-cert\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-7z8r7\" (UID: \"6b875062-a976-42cd-8b43-85dc197f3f4c\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-7z8r7" Oct 30 12:32:50.262853 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:50.257993 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b875062-a976-42cd-8b43-85dc197f3f4c-credential-socket\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-7z8r7\" (UID: \"6b875062-a976-42cd-8b43-85dc197f3f4c\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-7z8r7" Oct 30 12:32:50.262853 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:50.258262 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b875062-a976-42cd-8b43-85dc197f3f4c-istio-envoy\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-7z8r7\" (UID: \"6b875062-a976-42cd-8b43-85dc197f3f4c\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-7z8r7" Oct 30 12:32:50.262853 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:50.258328 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sv9rt\" (UniqueName: \"kubernetes.io/projected/6b875062-a976-42cd-8b43-85dc197f3f4c-kube-api-access-sv9rt\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-7z8r7\" (UID: \"6b875062-a976-42cd-8b43-85dc197f3f4c\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-7z8r7" Oct 30 12:32:50.262853 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:50.259195 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b875062-a976-42cd-8b43-85dc197f3f4c-istio-data\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-7z8r7\" (UID: \"6b875062-a976-42cd-8b43-85dc197f3f4c\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-7z8r7" Oct 30 12:32:50.262853 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:50.260806 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b875062-a976-42cd-8b43-85dc197f3f4c-istiod-ca-cert\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-7z8r7\" (UID: \"6b875062-a976-42cd-8b43-85dc197f3f4c\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-7z8r7" Oct 30 12:32:50.262853 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:50.261973 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b875062-a976-42cd-8b43-85dc197f3f4c-workload-certs\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-7z8r7\" (UID: \"6b875062-a976-42cd-8b43-85dc197f3f4c\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-7z8r7" Oct 30 12:32:50.279592 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:50.266164 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b875062-a976-42cd-8b43-85dc197f3f4c-workload-socket\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-7z8r7\" (UID: \"6b875062-a976-42cd-8b43-85dc197f3f4c\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-7z8r7" Oct 30 12:32:50.279592 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:50.266385 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b875062-a976-42cd-8b43-85dc197f3f4c-credential-socket\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-7z8r7\" (UID: \"6b875062-a976-42cd-8b43-85dc197f3f4c\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-7z8r7" Oct 30 12:32:50.279592 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:50.275391 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b875062-a976-42cd-8b43-85dc197f3f4c-istio-podinfo\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-7z8r7\" (UID: \"6b875062-a976-42cd-8b43-85dc197f3f4c\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-7z8r7" Oct 30 12:32:50.279592 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:50.277334 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b875062-a976-42cd-8b43-85dc197f3f4c-istio-envoy\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-7z8r7\" (UID: \"6b875062-a976-42cd-8b43-85dc197f3f4c\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-7z8r7" Oct 30 12:32:50.306850 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:50.303808 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b875062-a976-42cd-8b43-85dc197f3f4c-istio-token\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-7z8r7\" (UID: \"6b875062-a976-42cd-8b43-85dc197f3f4c\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-7z8r7" Oct 30 12:32:50.316903 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:50.316847 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sv9rt\" (UniqueName: \"kubernetes.io/projected/6b875062-a976-42cd-8b43-85dc197f3f4c-kube-api-access-sv9rt\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-7z8r7\" (UID: \"6b875062-a976-42cd-8b43-85dc197f3f4c\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-7z8r7" Oct 30 12:32:50.400479 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:50.400400 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-7z8r7" Oct 30 12:32:50.728626 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:50.728595 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-7z8r7"] Oct 30 12:32:50.731629 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:32:50.731604 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b875062_a976_42cd_8b43_85dc197f3f4c.slice/crio-7ea80cb364e812c832d652a4bd459ce97def3eb6f964efe348734849c08b55d7 WatchSource:0}: Error finding container 7ea80cb364e812c832d652a4bd459ce97def3eb6f964efe348734849c08b55d7: Status 404 returned error can't find the container with id 7ea80cb364e812c832d652a4bd459ce97def3eb6f964efe348734849c08b55d7 Oct 30 12:32:50.734016 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:50.733981 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:32:50.734133 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:50.734058 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:32:50.734133 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:50.734104 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:32:51.241892 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:51.241858 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-7z8r7" event={"ID":"6b875062-a976-42cd-8b43-85dc197f3f4c","Type":"ContainerStarted","Data":"b6b5da560cca29c612e9861dea20b806ee92c440fc8f39dc6f5ed876e79024e2"} Oct 30 12:32:51.242039 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:51.241911 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-7z8r7" event={"ID":"6b875062-a976-42cd-8b43-85dc197f3f4c","Type":"ContainerStarted","Data":"7ea80cb364e812c832d652a4bd459ce97def3eb6f964efe348734849c08b55d7"} Oct 30 12:32:51.298314 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:51.298255 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-7z8r7" podStartSLOduration=1.2982355270000001 podStartE2EDuration="1.298235527s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:32:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:32:51.295171373 +0000 UTC m=+2327.809600143" watchObservedRunningTime="2026-10-30 12:32:51.298235527 +0000 UTC m=+2327.812664273" Oct 30 12:32:51.401492 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:51.401462 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-7z8r7" Oct 30 12:32:51.403155 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:51.403127 4908 patch_prober.go:29] interesting pod/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-7z8r7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body= Oct 30 12:32:51.403315 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:51.403181 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-7z8r7" podUID="6b875062-a976-42cd-8b43-85dc197f3f4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" Oct 30 12:32:51.653629 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:51.653552 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-9bpr4"] Oct 30 12:32:51.657662 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:51.657641 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-9bpr4" Oct 30 12:32:51.700512 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:51.700480 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-9bpr4"] Oct 30 12:32:51.704256 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:51.704228 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de431683-xg6lb"] Oct 30 12:32:51.708314 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:51.708290 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de431683-xg6lb" Oct 30 12:32:51.725391 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:51.725367 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s75hq\"" Oct 30 12:32:51.725588 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:51.725569 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 12:32:51.725718 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:51.725702 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 12:32:51.726192 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:51.726172 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 12:32:51.736037 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:51.730554 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de431683-xg6lb"] Oct 30 12:32:51.793564 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:51.793522 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f00a2bcc-a63b-48a2-91a7-332f9fee2c68-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-de431683-xg6lb\" (UID: \"f00a2bcc-a63b-48a2-91a7-332f9fee2c68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de431683-xg6lb" Oct 30 12:32:51.793956 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:51.793930 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/434eb0c5-0505-4a8e-99c5-40eab19b2674-istio-token\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-9bpr4\" (UID: \"434eb0c5-0505-4a8e-99c5-40eab19b2674\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-9bpr4" Oct 30 12:32:51.794175 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:51.794157 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lzms\" (UniqueName: \"kubernetes.io/projected/434eb0c5-0505-4a8e-99c5-40eab19b2674-kube-api-access-9lzms\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-9bpr4\" (UID: \"434eb0c5-0505-4a8e-99c5-40eab19b2674\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-9bpr4" Oct 30 12:32:51.794351 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:51.794336 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/434eb0c5-0505-4a8e-99c5-40eab19b2674-credential-socket\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-9bpr4\" (UID: \"434eb0c5-0505-4a8e-99c5-40eab19b2674\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-9bpr4" Oct 30 12:32:51.794538 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:51.794524 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/434eb0c5-0505-4a8e-99c5-40eab19b2674-istio-podinfo\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-9bpr4\" (UID: \"434eb0c5-0505-4a8e-99c5-40eab19b2674\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-9bpr4" Oct 30 12:32:51.794687 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:51.794672 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f00a2bcc-a63b-48a2-91a7-332f9fee2c68-maas-api-xks-ca\") pod \"maas-api-revoke-sub-de431683-xg6lb\" (UID: \"f00a2bcc-a63b-48a2-91a7-332f9fee2c68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de431683-xg6lb" Oct 30 12:32:51.794871 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:51.794855 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/434eb0c5-0505-4a8e-99c5-40eab19b2674-istio-data\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-9bpr4\" (UID: \"434eb0c5-0505-4a8e-99c5-40eab19b2674\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-9bpr4" Oct 30 12:32:51.795059 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:51.795043 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/434eb0c5-0505-4a8e-99c5-40eab19b2674-workload-socket\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-9bpr4\" (UID: \"434eb0c5-0505-4a8e-99c5-40eab19b2674\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-9bpr4" Oct 30 12:32:51.797948 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:51.797921 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/434eb0c5-0505-4a8e-99c5-40eab19b2674-workload-certs\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-9bpr4\" (UID: \"434eb0c5-0505-4a8e-99c5-40eab19b2674\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-9bpr4" Oct 30 12:32:51.798163 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:51.798145 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k24xx\" (UniqueName: \"kubernetes.io/projected/f00a2bcc-a63b-48a2-91a7-332f9fee2c68-kube-api-access-k24xx\") pod \"maas-api-revoke-sub-de431683-xg6lb\" (UID: \"f00a2bcc-a63b-48a2-91a7-332f9fee2c68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de431683-xg6lb" Oct 30 12:32:51.798303 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:51.798290 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/434eb0c5-0505-4a8e-99c5-40eab19b2674-istiod-ca-cert\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-9bpr4\" (UID: \"434eb0c5-0505-4a8e-99c5-40eab19b2674\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-9bpr4" Oct 30 12:32:51.798464 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:51.798449 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/434eb0c5-0505-4a8e-99c5-40eab19b2674-istio-envoy\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-9bpr4\" (UID: \"434eb0c5-0505-4a8e-99c5-40eab19b2674\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-9bpr4" Oct 30 12:32:51.900251 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:51.900212 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/434eb0c5-0505-4a8e-99c5-40eab19b2674-istio-podinfo\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-9bpr4\" (UID: \"434eb0c5-0505-4a8e-99c5-40eab19b2674\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-9bpr4" Oct 30 12:32:51.900471 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:51.900260 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f00a2bcc-a63b-48a2-91a7-332f9fee2c68-maas-api-xks-ca\") pod \"maas-api-revoke-sub-de431683-xg6lb\" (UID: \"f00a2bcc-a63b-48a2-91a7-332f9fee2c68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de431683-xg6lb" Oct 30 12:32:51.900471 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:51.900298 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/434eb0c5-0505-4a8e-99c5-40eab19b2674-istio-data\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-9bpr4\" (UID: \"434eb0c5-0505-4a8e-99c5-40eab19b2674\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-9bpr4" Oct 30 12:32:51.900471 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:51.900345 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/434eb0c5-0505-4a8e-99c5-40eab19b2674-workload-socket\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-9bpr4\" (UID: \"434eb0c5-0505-4a8e-99c5-40eab19b2674\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-9bpr4" Oct 30 12:32:51.900471 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:51.900420 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/434eb0c5-0505-4a8e-99c5-40eab19b2674-workload-certs\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-9bpr4\" (UID: \"434eb0c5-0505-4a8e-99c5-40eab19b2674\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-9bpr4" Oct 30 12:32:51.900717 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:51.900530 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k24xx\" (UniqueName: \"kubernetes.io/projected/f00a2bcc-a63b-48a2-91a7-332f9fee2c68-kube-api-access-k24xx\") pod \"maas-api-revoke-sub-de431683-xg6lb\" (UID: \"f00a2bcc-a63b-48a2-91a7-332f9fee2c68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de431683-xg6lb" Oct 30 12:32:51.900717 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:51.900560 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/434eb0c5-0505-4a8e-99c5-40eab19b2674-istiod-ca-cert\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-9bpr4\" (UID: \"434eb0c5-0505-4a8e-99c5-40eab19b2674\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-9bpr4" Oct 30 12:32:51.900717 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:51.900623 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/434eb0c5-0505-4a8e-99c5-40eab19b2674-istio-envoy\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-9bpr4\" (UID: \"434eb0c5-0505-4a8e-99c5-40eab19b2674\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-9bpr4" Oct 30 12:32:51.900717 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:51.900664 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f00a2bcc-a63b-48a2-91a7-332f9fee2c68-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-de431683-xg6lb\" (UID: \"f00a2bcc-a63b-48a2-91a7-332f9fee2c68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de431683-xg6lb" Oct 30 12:32:51.901025 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:51.900726 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/434eb0c5-0505-4a8e-99c5-40eab19b2674-istio-token\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-9bpr4\" (UID: \"434eb0c5-0505-4a8e-99c5-40eab19b2674\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-9bpr4" Oct 30 12:32:51.901025 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:51.900753 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lzms\" (UniqueName: \"kubernetes.io/projected/434eb0c5-0505-4a8e-99c5-40eab19b2674-kube-api-access-9lzms\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-9bpr4\" (UID: \"434eb0c5-0505-4a8e-99c5-40eab19b2674\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-9bpr4" Oct 30 12:32:51.901025 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:51.900803 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/434eb0c5-0505-4a8e-99c5-40eab19b2674-credential-socket\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-9bpr4\" (UID: \"434eb0c5-0505-4a8e-99c5-40eab19b2674\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-9bpr4" Oct 30 12:32:51.901192 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:51.901088 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/434eb0c5-0505-4a8e-99c5-40eab19b2674-workload-socket\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-9bpr4\" (UID: \"434eb0c5-0505-4a8e-99c5-40eab19b2674\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-9bpr4" Oct 30 12:32:51.901519 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:51.901485 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/434eb0c5-0505-4a8e-99c5-40eab19b2674-istiod-ca-cert\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-9bpr4\" (UID: \"434eb0c5-0505-4a8e-99c5-40eab19b2674\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-9bpr4" Oct 30 12:32:51.902043 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:51.902012 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f00a2bcc-a63b-48a2-91a7-332f9fee2c68-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-de431683-xg6lb\" (UID: \"f00a2bcc-a63b-48a2-91a7-332f9fee2c68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de431683-xg6lb" Oct 30 12:32:51.902158 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:51.902087 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/434eb0c5-0505-4a8e-99c5-40eab19b2674-credential-socket\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-9bpr4\" (UID: \"434eb0c5-0505-4a8e-99c5-40eab19b2674\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-9bpr4" Oct 30 12:32:51.902158 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:51.902124 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/434eb0c5-0505-4a8e-99c5-40eab19b2674-workload-certs\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-9bpr4\" (UID: \"434eb0c5-0505-4a8e-99c5-40eab19b2674\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-9bpr4" Oct 30 12:32:51.902301 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:51.902269 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/434eb0c5-0505-4a8e-99c5-40eab19b2674-istio-data\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-9bpr4\" (UID: \"434eb0c5-0505-4a8e-99c5-40eab19b2674\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-9bpr4" Oct 30 12:32:51.903356 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:51.903332 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/434eb0c5-0505-4a8e-99c5-40eab19b2674-istio-podinfo\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-9bpr4\" (UID: \"434eb0c5-0505-4a8e-99c5-40eab19b2674\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-9bpr4" Oct 30 12:32:51.904027 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:51.903975 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f00a2bcc-a63b-48a2-91a7-332f9fee2c68-maas-api-xks-ca\") pod \"maas-api-revoke-sub-de431683-xg6lb\" (UID: \"f00a2bcc-a63b-48a2-91a7-332f9fee2c68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de431683-xg6lb" Oct 30 12:32:51.904113 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:51.904041 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/434eb0c5-0505-4a8e-99c5-40eab19b2674-istio-envoy\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-9bpr4\" (UID: \"434eb0c5-0505-4a8e-99c5-40eab19b2674\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-9bpr4" Oct 30 12:32:51.923362 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:51.923339 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/434eb0c5-0505-4a8e-99c5-40eab19b2674-istio-token\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-9bpr4\" (UID: \"434eb0c5-0505-4a8e-99c5-40eab19b2674\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-9bpr4" Oct 30 12:32:51.923913 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:51.923894 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lzms\" (UniqueName: \"kubernetes.io/projected/434eb0c5-0505-4a8e-99c5-40eab19b2674-kube-api-access-9lzms\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-9bpr4\" (UID: \"434eb0c5-0505-4a8e-99c5-40eab19b2674\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-9bpr4" Oct 30 12:32:51.924028 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:51.924008 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k24xx\" (UniqueName: \"kubernetes.io/projected/f00a2bcc-a63b-48a2-91a7-332f9fee2c68-kube-api-access-k24xx\") pod \"maas-api-revoke-sub-de431683-xg6lb\" (UID: \"f00a2bcc-a63b-48a2-91a7-332f9fee2c68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de431683-xg6lb" Oct 30 12:32:51.999883 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:51.999847 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-9bpr4" Oct 30 12:32:52.039181 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:52.039156 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de431683-xg6lb" Oct 30 12:32:52.056900 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:52.056702 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec567b86-c9jqg"] Oct 30 12:32:52.101767 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:52.082070 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec567b86-c9jqg" Oct 30 12:32:52.225024 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:52.209601 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sckb5\" (UniqueName: \"kubernetes.io/projected/2263d1e9-f4ce-4743-9b3a-51ad999798ed-kube-api-access-sckb5\") pod \"maas-api-revoke-sub-ec567b86-c9jqg\" (UID: \"2263d1e9-f4ce-4743-9b3a-51ad999798ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec567b86-c9jqg" Oct 30 12:32:52.225024 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:52.209683 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2263d1e9-f4ce-4743-9b3a-51ad999798ed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec567b86-c9jqg\" (UID: \"2263d1e9-f4ce-4743-9b3a-51ad999798ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec567b86-c9jqg" Oct 30 12:32:52.225024 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:52.209818 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2263d1e9-f4ce-4743-9b3a-51ad999798ed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec567b86-c9jqg\" (UID: \"2263d1e9-f4ce-4743-9b3a-51ad999798ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec567b86-c9jqg" Oct 30 12:32:52.225290 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:52.225074 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec567b86-c9jqg"] Oct 30 12:32:52.310924 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:52.310886 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2263d1e9-f4ce-4743-9b3a-51ad999798ed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec567b86-c9jqg\" (UID: \"2263d1e9-f4ce-4743-9b3a-51ad999798ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec567b86-c9jqg" Oct 30 12:32:52.311278 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:52.311260 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sckb5\" (UniqueName: \"kubernetes.io/projected/2263d1e9-f4ce-4743-9b3a-51ad999798ed-kube-api-access-sckb5\") pod \"maas-api-revoke-sub-ec567b86-c9jqg\" (UID: \"2263d1e9-f4ce-4743-9b3a-51ad999798ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec567b86-c9jqg" Oct 30 12:32:52.313778 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:52.313750 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2263d1e9-f4ce-4743-9b3a-51ad999798ed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec567b86-c9jqg\" (UID: \"2263d1e9-f4ce-4743-9b3a-51ad999798ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec567b86-c9jqg" Oct 30 12:32:52.316244 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:52.316218 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2263d1e9-f4ce-4743-9b3a-51ad999798ed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec567b86-c9jqg\" (UID: \"2263d1e9-f4ce-4743-9b3a-51ad999798ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec567b86-c9jqg" Oct 30 12:32:52.317756 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:52.317474 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2263d1e9-f4ce-4743-9b3a-51ad999798ed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec567b86-c9jqg\" (UID: \"2263d1e9-f4ce-4743-9b3a-51ad999798ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec567b86-c9jqg" Oct 30 12:32:52.343343 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:52.343299 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sckb5\" (UniqueName: \"kubernetes.io/projected/2263d1e9-f4ce-4743-9b3a-51ad999798ed-kube-api-access-sckb5\") pod \"maas-api-revoke-sub-ec567b86-c9jqg\" (UID: \"2263d1e9-f4ce-4743-9b3a-51ad999798ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec567b86-c9jqg" Oct 30 12:32:52.351799 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:52.351746 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-9bpr4"] Oct 30 12:32:52.353658 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:32:52.353633 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod434eb0c5_0505_4a8e_99c5_40eab19b2674.slice/crio-8f4f96f3adfdf243df5628fbaaff2ecb496105e6909c34eaeb54acc0191584db WatchSource:0}: Error finding container 8f4f96f3adfdf243df5628fbaaff2ecb496105e6909c34eaeb54acc0191584db: Status 404 returned error can't find the container with id 8f4f96f3adfdf243df5628fbaaff2ecb496105e6909c34eaeb54acc0191584db Oct 30 12:32:52.356550 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:52.356509 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:32:52.356694 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:52.356671 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:32:52.356781 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:52.356744 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:32:52.402136 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:52.401022 4908 patch_prober.go:29] interesting pod/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-7z8r7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body= Oct 30 12:32:52.402136 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:52.401078 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-7z8r7" podUID="6b875062-a976-42cd-8b43-85dc197f3f4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" Oct 30 12:32:52.512619 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:52.512592 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec567b86-c9jqg" Oct 30 12:32:52.612850 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:52.611883 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de431683-xg6lb"] Oct 30 12:32:52.617782 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:32:52.617760 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf00a2bcc_a63b_48a2_91a7_332f9fee2c68.slice/crio-536affc1a4c0e0ec6b0ebf3708e010b032984d600aefe5a50ba05f59423fc38f WatchSource:0}: Error finding container 536affc1a4c0e0ec6b0ebf3708e010b032984d600aefe5a50ba05f59423fc38f: Status 404 returned error can't find the container with id 536affc1a4c0e0ec6b0ebf3708e010b032984d600aefe5a50ba05f59423fc38f Oct 30 12:32:52.897648 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:52.897619 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec567b86-c9jqg"] Oct 30 12:32:52.898105 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:32:52.897873 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2263d1e9_f4ce_4743_9b3a_51ad999798ed.slice/crio-7f8dcf2216f11db181809eadcdae0821d18df3ef612bafc7f802a2dd0ff2dbb1 WatchSource:0}: Error finding container 7f8dcf2216f11db181809eadcdae0821d18df3ef612bafc7f802a2dd0ff2dbb1: Status 404 returned error can't find the container with id 7f8dcf2216f11db181809eadcdae0821d18df3ef612bafc7f802a2dd0ff2dbb1 Oct 30 12:32:53.291849 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:53.289383 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec567b86-c9jqg" event={"ID":"2263d1e9-f4ce-4743-9b3a-51ad999798ed","Type":"ContainerStarted","Data":"7f8dcf2216f11db181809eadcdae0821d18df3ef612bafc7f802a2dd0ff2dbb1"} Oct 30 12:32:53.291849 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:53.291492 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-9bpr4" event={"ID":"434eb0c5-0505-4a8e-99c5-40eab19b2674","Type":"ContainerStarted","Data":"d7453a256dc0d4effd974255b973caf1cdcf397d33f3841cc20cecc576f9d8f4"} Oct 30 12:32:53.291849 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:53.291555 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-9bpr4" event={"ID":"434eb0c5-0505-4a8e-99c5-40eab19b2674","Type":"ContainerStarted","Data":"8f4f96f3adfdf243df5628fbaaff2ecb496105e6909c34eaeb54acc0191584db"} Oct 30 12:32:53.297854 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:53.297809 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de431683-xg6lb" event={"ID":"f00a2bcc-a63b-48a2-91a7-332f9fee2c68","Type":"ContainerStarted","Data":"f607b93c16f0bfe4273ecba14cbd3db364579343512b15408d8ebb30a4daca2b"} Oct 30 12:32:53.297993 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:53.297874 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de431683-xg6lb" event={"ID":"f00a2bcc-a63b-48a2-91a7-332f9fee2c68","Type":"ContainerStarted","Data":"536affc1a4c0e0ec6b0ebf3708e010b032984d600aefe5a50ba05f59423fc38f"} Oct 30 12:32:53.344241 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:53.344156 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-9bpr4" podStartSLOduration=2.34414401 podStartE2EDuration="2.34414401s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:32:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:32:53.340374012 +0000 UTC m=+2329.854802758" watchObservedRunningTime="2026-10-30 12:32:53.34414401 +0000 UTC m=+2329.858572756" Oct 30 12:32:53.401245 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:53.401213 4908 patch_prober.go:29] interesting pod/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-7z8r7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body= Oct 30 12:32:53.401402 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:53.401264 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-7z8r7" podUID="6b875062-a976-42cd-8b43-85dc197f3f4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" Oct 30 12:32:54.007528 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:54.007063 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-9bpr4" Oct 30 12:32:54.007528 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:54.007201 4908 patch_prober.go:29] interesting pod/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-9bpr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body= Oct 30 12:32:54.007528 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:54.007242 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-9bpr4" podUID="434eb0c5-0505-4a8e-99c5-40eab19b2674" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" Oct 30 12:32:54.326351 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:54.326248 4908 generic.go:356] "Generic (PLEG): container finished" podID="f00a2bcc-a63b-48a2-91a7-332f9fee2c68" containerID="f607b93c16f0bfe4273ecba14cbd3db364579343512b15408d8ebb30a4daca2b" exitCode=0 Oct 30 12:32:54.326541 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:54.326470 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de431683-xg6lb" event={"ID":"f00a2bcc-a63b-48a2-91a7-332f9fee2c68","Type":"ContainerDied","Data":"f607b93c16f0bfe4273ecba14cbd3db364579343512b15408d8ebb30a4daca2b"} Oct 30 12:32:54.328290 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:54.328256 4908 generic.go:356] "Generic (PLEG): container finished" podID="2263d1e9-f4ce-4743-9b3a-51ad999798ed" containerID="063f033d80f52e6559a06c7414d55b6bfbbb51177ba1cd5ef0c131dfdd1f86cd" exitCode=0 Oct 30 12:32:54.328387 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:54.328356 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec567b86-c9jqg" event={"ID":"2263d1e9-f4ce-4743-9b3a-51ad999798ed","Type":"ContainerDied","Data":"063f033d80f52e6559a06c7414d55b6bfbbb51177ba1cd5ef0c131dfdd1f86cd"} Oct 30 12:32:54.418851 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:54.407302 4908 patch_prober.go:29] interesting pod/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-7z8r7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body= Oct 30 12:32:54.418851 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:54.407360 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-7z8r7" podUID="6b875062-a976-42cd-8b43-85dc197f3f4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" Oct 30 12:32:55.000853 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:55.000240 4908 patch_prober.go:29] interesting pod/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-9bpr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body= Oct 30 12:32:55.000853 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:55.000302 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-9bpr4" podUID="434eb0c5-0505-4a8e-99c5-40eab19b2674" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" Oct 30 12:32:55.403854 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:55.403200 4908 patch_prober.go:29] interesting pod/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-7z8r7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body= Oct 30 12:32:55.403854 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:55.403276 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-7z8r7" podUID="6b875062-a976-42cd-8b43-85dc197f3f4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" Oct 30 12:32:55.744016 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:55.743875 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec567b86-c9jqg" Oct 30 12:32:55.818387 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:55.758488 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2263d1e9-f4ce-4743-9b3a-51ad999798ed-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2263d1e9-f4ce-4743-9b3a-51ad999798ed-maas-api-openshift-ca\") pod \"2263d1e9-f4ce-4743-9b3a-51ad999798ed\" (UID: \"2263d1e9-f4ce-4743-9b3a-51ad999798ed\") " Oct 30 12:32:55.818387 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:55.758618 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2263d1e9-f4ce-4743-9b3a-51ad999798ed-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2263d1e9-f4ce-4743-9b3a-51ad999798ed-maas-api-xks-ca\") pod \"2263d1e9-f4ce-4743-9b3a-51ad999798ed\" (UID: \"2263d1e9-f4ce-4743-9b3a-51ad999798ed\") " Oct 30 12:32:55.818387 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:55.758646 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2263d1e9-f4ce-4743-9b3a-51ad999798ed-kube-api-access-sckb5\" (UniqueName: \"kubernetes.io/projected/2263d1e9-f4ce-4743-9b3a-51ad999798ed-kube-api-access-sckb5\") pod \"2263d1e9-f4ce-4743-9b3a-51ad999798ed\" (UID: \"2263d1e9-f4ce-4743-9b3a-51ad999798ed\") " Oct 30 12:32:55.818387 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:55.783047 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2263d1e9-f4ce-4743-9b3a-51ad999798ed-maas-api-openshift-ca" pod "2263d1e9-f4ce-4743-9b3a-51ad999798ed" (UID: "2263d1e9-f4ce-4743-9b3a-51ad999798ed"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:32:55.818387 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:55.783672 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2263d1e9-f4ce-4743-9b3a-51ad999798ed-maas-api-openshift-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:32:55.831236 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:55.831207 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de431683-xg6lb" Oct 30 12:32:55.837884 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:55.837853 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2263d1e9-f4ce-4743-9b3a-51ad999798ed-kube-api-access-sckb5" pod "2263d1e9-f4ce-4743-9b3a-51ad999798ed" (UID: "2263d1e9-f4ce-4743-9b3a-51ad999798ed"). InnerVolumeSpecName "kube-api-access-sckb5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:32:55.838431 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:55.838404 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2263d1e9-f4ce-4743-9b3a-51ad999798ed-maas-api-xks-ca" pod "2263d1e9-f4ce-4743-9b3a-51ad999798ed" (UID: "2263d1e9-f4ce-4743-9b3a-51ad999798ed"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:32:55.930078 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:55.923559 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f00a2bcc-a63b-48a2-91a7-332f9fee2c68-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f00a2bcc-a63b-48a2-91a7-332f9fee2c68-maas-api-xks-ca\") pod \"f00a2bcc-a63b-48a2-91a7-332f9fee2c68\" (UID: \"f00a2bcc-a63b-48a2-91a7-332f9fee2c68\") " Oct 30 12:32:55.930078 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:55.923656 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f00a2bcc-a63b-48a2-91a7-332f9fee2c68-kube-api-access-k24xx\" (UniqueName: \"kubernetes.io/projected/f00a2bcc-a63b-48a2-91a7-332f9fee2c68-kube-api-access-k24xx\") pod \"f00a2bcc-a63b-48a2-91a7-332f9fee2c68\" (UID: \"f00a2bcc-a63b-48a2-91a7-332f9fee2c68\") " Oct 30 12:32:55.930078 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:55.923769 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f00a2bcc-a63b-48a2-91a7-332f9fee2c68-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f00a2bcc-a63b-48a2-91a7-332f9fee2c68-maas-api-openshift-ca\") pod \"f00a2bcc-a63b-48a2-91a7-332f9fee2c68\" (UID: \"f00a2bcc-a63b-48a2-91a7-332f9fee2c68\") " Oct 30 12:32:55.930078 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:55.924217 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2263d1e9-f4ce-4743-9b3a-51ad999798ed-maas-api-xks-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:32:55.930078 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:55.924239 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sckb5\" (UniqueName: \"kubernetes.io/projected/2263d1e9-f4ce-4743-9b3a-51ad999798ed-kube-api-access-sckb5\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:32:55.953393 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:55.949053 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f00a2bcc-a63b-48a2-91a7-332f9fee2c68-maas-api-openshift-ca" pod "f00a2bcc-a63b-48a2-91a7-332f9fee2c68" (UID: "f00a2bcc-a63b-48a2-91a7-332f9fee2c68"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:32:55.960683 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:55.960628 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f00a2bcc-a63b-48a2-91a7-332f9fee2c68-kube-api-access-k24xx" pod "f00a2bcc-a63b-48a2-91a7-332f9fee2c68" (UID: "f00a2bcc-a63b-48a2-91a7-332f9fee2c68"). InnerVolumeSpecName "kube-api-access-k24xx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:32:55.960965 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:55.960942 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f00a2bcc-a63b-48a2-91a7-332f9fee2c68-maas-api-xks-ca" pod "f00a2bcc-a63b-48a2-91a7-332f9fee2c68" (UID: "f00a2bcc-a63b-48a2-91a7-332f9fee2c68"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:32:56.018854 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:56.016496 4908 patch_prober.go:29] interesting pod/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-9bpr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body= Oct 30 12:32:56.018854 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:56.016568 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-9bpr4" podUID="434eb0c5-0505-4a8e-99c5-40eab19b2674" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" Oct 30 12:32:56.037390 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:56.037309 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f00a2bcc-a63b-48a2-91a7-332f9fee2c68-maas-api-xks-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:32:56.037390 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:56.037345 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k24xx\" (UniqueName: \"kubernetes.io/projected/f00a2bcc-a63b-48a2-91a7-332f9fee2c68-kube-api-access-k24xx\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:32:56.037390 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:56.037363 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f00a2bcc-a63b-48a2-91a7-332f9fee2c68-maas-api-openshift-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:32:56.344938 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:56.344904 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec567b86-c9jqg" Oct 30 12:32:56.345380 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:56.345352 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec567b86-c9jqg" event={"ID":"2263d1e9-f4ce-4743-9b3a-51ad999798ed","Type":"ContainerDied","Data":"7f8dcf2216f11db181809eadcdae0821d18df3ef612bafc7f802a2dd0ff2dbb1"} Oct 30 12:32:56.345536 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:56.345402 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7f8dcf2216f11db181809eadcdae0821d18df3ef612bafc7f802a2dd0ff2dbb1" Oct 30 12:32:56.346614 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:56.346590 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de431683-xg6lb" event={"ID":"f00a2bcc-a63b-48a2-91a7-332f9fee2c68","Type":"ContainerDied","Data":"536affc1a4c0e0ec6b0ebf3708e010b032984d600aefe5a50ba05f59423fc38f"} Oct 30 12:32:56.346747 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:56.346627 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="536affc1a4c0e0ec6b0ebf3708e010b032984d600aefe5a50ba05f59423fc38f" Oct 30 12:32:56.346747 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:56.346679 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de431683-xg6lb" Oct 30 12:32:56.403853 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:56.403235 4908 patch_prober.go:29] interesting pod/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-7z8r7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body= Oct 30 12:32:56.403853 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:56.403294 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-7z8r7" podUID="6b875062-a976-42cd-8b43-85dc197f3f4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" Oct 30 12:32:57.001848 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:57.001435 4908 patch_prober.go:29] interesting pod/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-9bpr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body= Oct 30 12:32:57.001848 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:57.001509 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-9bpr4" podUID="434eb0c5-0505-4a8e-99c5-40eab19b2674" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" Oct 30 12:32:57.401350 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:57.401213 4908 patch_prober.go:29] interesting pod/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-7z8r7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body= Oct 30 12:32:57.401350 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:57.401279 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-7z8r7" podUID="6b875062-a976-42cd-8b43-85dc197f3f4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" Oct 30 12:32:58.000646 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:58.000612 4908 patch_prober.go:29] interesting pod/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-9bpr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body= Oct 30 12:32:58.001117 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:58.000664 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-9bpr4" podUID="434eb0c5-0505-4a8e-99c5-40eab19b2674" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" Oct 30 12:32:58.726733 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:58.726135 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-7z8r7" Oct 30 12:32:58.727766 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:58.727739 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-7z8r7" Oct 30 12:32:58.729457 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:58.729279 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-7z8r7" Oct 30 12:32:59.000656 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:59.000567 4908 patch_prober.go:29] interesting pod/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-9bpr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body= Oct 30 12:32:59.000656 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:59.000629 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-9bpr4" podUID="434eb0c5-0505-4a8e-99c5-40eab19b2674" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" Oct 30 12:32:59.657859 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:32:59.653376 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9wwfc"] Oct 30 12:33:00.000320 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:00.000282 4908 patch_prober.go:29] interesting pod/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-9bpr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body= Oct 30 12:33:00.000519 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:00.000353 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-9bpr4" podUID="434eb0c5-0505-4a8e-99c5-40eab19b2674" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" Oct 30 12:33:01.365854 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:01.363244 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-9bpr4" Oct 30 12:33:01.365854 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:01.365164 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-9bpr4" Oct 30 12:33:01.373852 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:01.366485 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-9bpr4" Oct 30 12:33:05.151128 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:05.151096 4908 scope.go:122] "RemoveContainer" containerID="8108a5ce3adfd825095c9d98c6daf227dc11bd586371ce2746bb559968f799f1" Oct 30 12:33:06.462843 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:06.462804 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c462d5af-5zf7l"] Oct 30 12:33:06.463354 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:06.463338 4908 cpu_manager.go:414] "containerMap: removing container" podUID="2263d1e9-f4ce-4743-9b3a-51ad999798ed" containerName="revoke-keys" Oct 30 12:33:06.463410 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:06.463357 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2263d1e9-f4ce-4743-9b3a-51ad999798ed" containerName="revoke-keys" Oct 30 12:33:06.463410 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:06.463382 4908 cpu_manager.go:414] "containerMap: removing container" podUID="f00a2bcc-a63b-48a2-91a7-332f9fee2c68" containerName="revoke-keys" Oct 30 12:33:06.463410 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:06.463388 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f00a2bcc-a63b-48a2-91a7-332f9fee2c68" containerName="revoke-keys" Oct 30 12:33:06.463523 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:06.463479 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="f00a2bcc-a63b-48a2-91a7-332f9fee2c68" containerName="revoke-keys" Oct 30 12:33:06.463523 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:06.463502 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="2263d1e9-f4ce-4743-9b3a-51ad999798ed" containerName="revoke-keys" Oct 30 12:33:06.468596 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:06.468575 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c462d5af-5zf7l" Oct 30 12:33:06.487365 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:06.487336 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c462d5af-5zf7l"] Oct 30 12:33:06.492364 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:06.492251 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 12:33:06.492364 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:06.492266 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s75hq\"" Oct 30 12:33:06.492364 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:06.492340 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 12:33:06.492940 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:06.492920 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 12:33:06.608194 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:06.608156 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpxpx\" (UniqueName: \"kubernetes.io/projected/46ba99ad-dda8-44f0-951a-63074f18fa27-kube-api-access-tpxpx\") pod \"maas-api-revoke-sub-c462d5af-5zf7l\" (UID: \"46ba99ad-dda8-44f0-951a-63074f18fa27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c462d5af-5zf7l" Oct 30 12:33:06.608361 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:06.608203 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46ba99ad-dda8-44f0-951a-63074f18fa27-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c462d5af-5zf7l\" (UID: \"46ba99ad-dda8-44f0-951a-63074f18fa27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c462d5af-5zf7l" Oct 30 12:33:06.608427 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:06.608388 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46ba99ad-dda8-44f0-951a-63074f18fa27-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c462d5af-5zf7l\" (UID: \"46ba99ad-dda8-44f0-951a-63074f18fa27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c462d5af-5zf7l" Oct 30 12:33:06.709649 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:06.709614 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46ba99ad-dda8-44f0-951a-63074f18fa27-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c462d5af-5zf7l\" (UID: \"46ba99ad-dda8-44f0-951a-63074f18fa27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c462d5af-5zf7l" Oct 30 12:33:06.709790 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:06.709737 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpxpx\" (UniqueName: \"kubernetes.io/projected/46ba99ad-dda8-44f0-951a-63074f18fa27-kube-api-access-tpxpx\") pod \"maas-api-revoke-sub-c462d5af-5zf7l\" (UID: \"46ba99ad-dda8-44f0-951a-63074f18fa27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c462d5af-5zf7l" Oct 30 12:33:06.709790 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:06.709779 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46ba99ad-dda8-44f0-951a-63074f18fa27-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c462d5af-5zf7l\" (UID: \"46ba99ad-dda8-44f0-951a-63074f18fa27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c462d5af-5zf7l" Oct 30 12:33:06.710410 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:06.710387 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46ba99ad-dda8-44f0-951a-63074f18fa27-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c462d5af-5zf7l\" (UID: \"46ba99ad-dda8-44f0-951a-63074f18fa27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c462d5af-5zf7l" Oct 30 12:33:06.712050 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:06.712030 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46ba99ad-dda8-44f0-951a-63074f18fa27-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c462d5af-5zf7l\" (UID: \"46ba99ad-dda8-44f0-951a-63074f18fa27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c462d5af-5zf7l" Oct 30 12:33:06.732861 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:06.732815 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpxpx\" (UniqueName: \"kubernetes.io/projected/46ba99ad-dda8-44f0-951a-63074f18fa27-kube-api-access-tpxpx\") pod \"maas-api-revoke-sub-c462d5af-5zf7l\" (UID: \"46ba99ad-dda8-44f0-951a-63074f18fa27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c462d5af-5zf7l" Oct 30 12:33:06.782788 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:06.782763 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c462d5af-5zf7l" Oct 30 12:33:07.096803 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:07.096777 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c462d5af-5zf7l"] Oct 30 12:33:07.099533 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:33:07.099508 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46ba99ad_dda8_44f0_951a_63074f18fa27.slice/crio-d0eecab31bc3dc82f10304f09c71d44fc293efb989f770950f9915bd2cada675 WatchSource:0}: Error finding container d0eecab31bc3dc82f10304f09c71d44fc293efb989f770950f9915bd2cada675: Status 404 returned error can't find the container with id d0eecab31bc3dc82f10304f09c71d44fc293efb989f770950f9915bd2cada675 Oct 30 12:33:07.488255 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:07.488215 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c462d5af-5zf7l" event={"ID":"46ba99ad-dda8-44f0-951a-63074f18fa27","Type":"ContainerStarted","Data":"d7ab541335fe03801bb86fbc1c59365d6b92b19ef428a61f307d57026731a213"} Oct 30 12:33:07.488737 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:07.488283 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c462d5af-5zf7l" event={"ID":"46ba99ad-dda8-44f0-951a-63074f18fa27","Type":"ContainerStarted","Data":"d0eecab31bc3dc82f10304f09c71d44fc293efb989f770950f9915bd2cada675"} Oct 30 12:33:07.528798 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:07.528749 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c462d5af-5zf7l" podStartSLOduration=1.52873648 podStartE2EDuration="1.52873648s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:33:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:33:07.525165783 +0000 UTC m=+2344.039594533" watchObservedRunningTime="2026-10-30 12:33:07.52873648 +0000 UTC m=+2344.043165260" Oct 30 12:33:08.499865 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:08.499813 4908 generic.go:356] "Generic (PLEG): container finished" podID="46ba99ad-dda8-44f0-951a-63074f18fa27" containerID="d7ab541335fe03801bb86fbc1c59365d6b92b19ef428a61f307d57026731a213" exitCode=0 Oct 30 12:33:08.500340 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:08.499874 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c462d5af-5zf7l" event={"ID":"46ba99ad-dda8-44f0-951a-63074f18fa27","Type":"ContainerDied","Data":"d7ab541335fe03801bb86fbc1c59365d6b92b19ef428a61f307d57026731a213"} Oct 30 12:33:09.834092 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:09.834068 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c462d5af-5zf7l" Oct 30 12:33:09.970295 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:09.970263 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46ba99ad-dda8-44f0-951a-63074f18fa27-kube-api-access-tpxpx\" (UniqueName: \"kubernetes.io/projected/46ba99ad-dda8-44f0-951a-63074f18fa27-kube-api-access-tpxpx\") pod \"46ba99ad-dda8-44f0-951a-63074f18fa27\" (UID: \"46ba99ad-dda8-44f0-951a-63074f18fa27\") " Oct 30 12:33:09.970468 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:09.970343 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46ba99ad-dda8-44f0-951a-63074f18fa27-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46ba99ad-dda8-44f0-951a-63074f18fa27-maas-api-openshift-ca\") pod \"46ba99ad-dda8-44f0-951a-63074f18fa27\" (UID: \"46ba99ad-dda8-44f0-951a-63074f18fa27\") " Oct 30 12:33:09.970468 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:09.970418 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/46ba99ad-dda8-44f0-951a-63074f18fa27-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46ba99ad-dda8-44f0-951a-63074f18fa27-maas-api-xks-ca\") pod \"46ba99ad-dda8-44f0-951a-63074f18fa27\" (UID: \"46ba99ad-dda8-44f0-951a-63074f18fa27\") " Oct 30 12:33:09.970694 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:09.970670 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46ba99ad-dda8-44f0-951a-63074f18fa27-maas-api-openshift-ca" pod "46ba99ad-dda8-44f0-951a-63074f18fa27" (UID: "46ba99ad-dda8-44f0-951a-63074f18fa27"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:33:09.970868 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:09.970850 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46ba99ad-dda8-44f0-951a-63074f18fa27-maas-api-openshift-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:33:09.972446 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:09.972425 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46ba99ad-dda8-44f0-951a-63074f18fa27-maas-api-xks-ca" pod "46ba99ad-dda8-44f0-951a-63074f18fa27" (UID: "46ba99ad-dda8-44f0-951a-63074f18fa27"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:33:09.972498 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:09.972484 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46ba99ad-dda8-44f0-951a-63074f18fa27-kube-api-access-tpxpx" pod "46ba99ad-dda8-44f0-951a-63074f18fa27" (UID: "46ba99ad-dda8-44f0-951a-63074f18fa27"). InnerVolumeSpecName "kube-api-access-tpxpx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:33:10.072451 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:10.072392 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46ba99ad-dda8-44f0-951a-63074f18fa27-maas-api-xks-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:33:10.072451 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:10.072416 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tpxpx\" (UniqueName: \"kubernetes.io/projected/46ba99ad-dda8-44f0-951a-63074f18fa27-kube-api-access-tpxpx\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:33:10.515654 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:10.515625 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c462d5af-5zf7l" Oct 30 12:33:10.515866 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:10.515664 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c462d5af-5zf7l" event={"ID":"46ba99ad-dda8-44f0-951a-63074f18fa27","Type":"ContainerDied","Data":"d0eecab31bc3dc82f10304f09c71d44fc293efb989f770950f9915bd2cada675"} Oct 30 12:33:10.515866 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:10.515715 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d0eecab31bc3dc82f10304f09c71d44fc293efb989f770950f9915bd2cada675" Oct 30 12:33:14.469953 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:14.469910 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-0172c6-bhqbn"] Oct 30 12:33:14.470699 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:14.470679 4908 cpu_manager.go:414] "containerMap: removing container" podUID="46ba99ad-dda8-44f0-951a-63074f18fa27" containerName="revoke-keys" Oct 30 12:33:14.470745 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:14.470702 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46ba99ad-dda8-44f0-951a-63074f18fa27" containerName="revoke-keys" Oct 30 12:33:14.470899 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:14.470886 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="46ba99ad-dda8-44f0-951a-63074f18fa27" containerName="revoke-keys" Oct 30 12:33:14.474918 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:14.474899 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-0172c6-bhqbn" Oct 30 12:33:14.490974 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:14.490951 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-0172c6-bhqbn"] Oct 30 12:33:14.496893 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:14.496870 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 12:33:14.497036 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:14.496919 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 12:33:14.497036 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:14.496973 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 12:33:14.497036 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:14.496988 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s75hq\"" Oct 30 12:33:14.536804 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:14.536770 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghhw9\" (UniqueName: \"kubernetes.io/projected/2c4243b2-80bf-492e-ab85-bc9f33563bfd-kube-api-access-ghhw9\") pod \"maas-api-revoke-keys-e2e-praxis-a-0172c6-bhqbn\" (UID: \"2c4243b2-80bf-492e-ab85-bc9f33563bfd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-0172c6-bhqbn" Oct 30 12:33:14.536977 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:14.536925 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c4243b2-80bf-492e-ab85-bc9f33563bfd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-0172c6-bhqbn\" (UID: \"2c4243b2-80bf-492e-ab85-bc9f33563bfd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-0172c6-bhqbn" Oct 30 12:33:14.537046 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:14.537011 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c4243b2-80bf-492e-ab85-bc9f33563bfd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-0172c6-bhqbn\" (UID: \"2c4243b2-80bf-492e-ab85-bc9f33563bfd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-0172c6-bhqbn" Oct 30 12:33:14.638225 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:14.638199 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c4243b2-80bf-492e-ab85-bc9f33563bfd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-0172c6-bhqbn\" (UID: \"2c4243b2-80bf-492e-ab85-bc9f33563bfd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-0172c6-bhqbn" Oct 30 12:33:14.638351 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:14.638249 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c4243b2-80bf-492e-ab85-bc9f33563bfd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-0172c6-bhqbn\" (UID: \"2c4243b2-80bf-492e-ab85-bc9f33563bfd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-0172c6-bhqbn" Oct 30 12:33:14.638351 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:14.638327 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghhw9\" (UniqueName: \"kubernetes.io/projected/2c4243b2-80bf-492e-ab85-bc9f33563bfd-kube-api-access-ghhw9\") pod \"maas-api-revoke-keys-e2e-praxis-a-0172c6-bhqbn\" (UID: \"2c4243b2-80bf-492e-ab85-bc9f33563bfd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-0172c6-bhqbn" Oct 30 12:33:14.639061 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:14.639031 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c4243b2-80bf-492e-ab85-bc9f33563bfd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-0172c6-bhqbn\" (UID: \"2c4243b2-80bf-492e-ab85-bc9f33563bfd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-0172c6-bhqbn" Oct 30 12:33:14.640624 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:14.640604 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c4243b2-80bf-492e-ab85-bc9f33563bfd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-0172c6-bhqbn\" (UID: \"2c4243b2-80bf-492e-ab85-bc9f33563bfd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-0172c6-bhqbn" Oct 30 12:33:14.660374 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:14.660336 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghhw9\" (UniqueName: \"kubernetes.io/projected/2c4243b2-80bf-492e-ab85-bc9f33563bfd-kube-api-access-ghhw9\") pod \"maas-api-revoke-keys-e2e-praxis-a-0172c6-bhqbn\" (UID: \"2c4243b2-80bf-492e-ab85-bc9f33563bfd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-0172c6-bhqbn" Oct 30 12:33:14.789448 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:14.789424 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-0172c6-bhqbn" Oct 30 12:33:15.095565 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:15.093933 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-0172c6-bhqbn"] Oct 30 12:33:15.576975 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:15.576941 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-0172c6-bhqbn" event={"ID":"2c4243b2-80bf-492e-ab85-bc9f33563bfd","Type":"ContainerStarted","Data":"8b6a7259c94f59fb6e0fa357b6be5035db28a0f51158ab0790cbf48d010115e4"} Oct 30 12:33:15.577442 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:15.576994 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-0172c6-bhqbn" event={"ID":"2c4243b2-80bf-492e-ab85-bc9f33563bfd","Type":"ContainerStarted","Data":"e32eeeb3888cf48deb5146fdb2b5f6fd73ea510e9753f3533a4539c01206b20e"} Oct 30 12:33:15.618249 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:15.618183 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-0172c6-bhqbn" podStartSLOduration=1.6181641230000001 podStartE2EDuration="1.618164123s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:33:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:33:15.615571897 +0000 UTC m=+2352.130000646" watchObservedRunningTime="2026-10-30 12:33:15.618164123 +0000 UTC m=+2352.132592866" Oct 30 12:33:16.471063 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:16.470968 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c44eee-956mb"] Oct 30 12:33:16.481945 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:16.481910 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c44eee-956mb"] Oct 30 12:33:16.593486 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:16.593302 4908 generic.go:356] "Generic (PLEG): container finished" podID="2c4243b2-80bf-492e-ab85-bc9f33563bfd" containerID="8b6a7259c94f59fb6e0fa357b6be5035db28a0f51158ab0790cbf48d010115e4" exitCode=0 Oct 30 12:33:16.593486 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:16.593436 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-0172c6-bhqbn" event={"ID":"2c4243b2-80bf-492e-ab85-bc9f33563bfd","Type":"ContainerDied","Data":"8b6a7259c94f59fb6e0fa357b6be5035db28a0f51158ab0790cbf48d010115e4"} Oct 30 12:33:17.608245 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:17.608213 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd471c76-9e74-48ee-8a46-4587981cfe25" path="/var/lib/kubelet/pods/fd471c76-9e74-48ee-8a46-4587981cfe25/volumes" Oct 30 12:33:17.761085 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:17.761059 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-0172c6-bhqbn" Oct 30 12:33:17.794172 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:17.794143 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2c4243b2-80bf-492e-ab85-bc9f33563bfd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c4243b2-80bf-492e-ab85-bc9f33563bfd-maas-api-xks-ca\") pod \"2c4243b2-80bf-492e-ab85-bc9f33563bfd\" (UID: \"2c4243b2-80bf-492e-ab85-bc9f33563bfd\") " Oct 30 12:33:17.794309 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:17.794269 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c4243b2-80bf-492e-ab85-bc9f33563bfd-kube-api-access-ghhw9\" (UniqueName: \"kubernetes.io/projected/2c4243b2-80bf-492e-ab85-bc9f33563bfd-kube-api-access-ghhw9\") pod \"2c4243b2-80bf-492e-ab85-bc9f33563bfd\" (UID: \"2c4243b2-80bf-492e-ab85-bc9f33563bfd\") " Oct 30 12:33:17.794309 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:17.794304 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c4243b2-80bf-492e-ab85-bc9f33563bfd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c4243b2-80bf-492e-ab85-bc9f33563bfd-maas-api-openshift-ca\") pod \"2c4243b2-80bf-492e-ab85-bc9f33563bfd\" (UID: \"2c4243b2-80bf-492e-ab85-bc9f33563bfd\") " Oct 30 12:33:17.794780 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:17.794734 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c4243b2-80bf-492e-ab85-bc9f33563bfd-maas-api-openshift-ca" pod "2c4243b2-80bf-492e-ab85-bc9f33563bfd" (UID: "2c4243b2-80bf-492e-ab85-bc9f33563bfd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:33:17.796239 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:17.796211 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c4243b2-80bf-492e-ab85-bc9f33563bfd-maas-api-xks-ca" pod "2c4243b2-80bf-492e-ab85-bc9f33563bfd" (UID: "2c4243b2-80bf-492e-ab85-bc9f33563bfd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:33:17.796502 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:17.796481 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c4243b2-80bf-492e-ab85-bc9f33563bfd-kube-api-access-ghhw9" pod "2c4243b2-80bf-492e-ab85-bc9f33563bfd" (UID: "2c4243b2-80bf-492e-ab85-bc9f33563bfd"). InnerVolumeSpecName "kube-api-access-ghhw9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:33:17.895872 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:17.895793 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghhw9\" (UniqueName: \"kubernetes.io/projected/2c4243b2-80bf-492e-ab85-bc9f33563bfd-kube-api-access-ghhw9\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:33:17.895872 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:17.895819 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c4243b2-80bf-492e-ab85-bc9f33563bfd-maas-api-openshift-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:33:17.895872 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:17.895841 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c4243b2-80bf-492e-ab85-bc9f33563bfd-maas-api-xks-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:33:18.615793 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:18.615765 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-0172c6-bhqbn" Oct 30 12:33:18.616291 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:18.615802 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-0172c6-bhqbn" event={"ID":"2c4243b2-80bf-492e-ab85-bc9f33563bfd","Type":"ContainerDied","Data":"e32eeeb3888cf48deb5146fdb2b5f6fd73ea510e9753f3533a4539c01206b20e"} Oct 30 12:33:18.616291 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:18.615866 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e32eeeb3888cf48deb5146fdb2b5f6fd73ea510e9753f3533a4539c01206b20e" Oct 30 12:33:19.685581 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:33:19.685539 4908 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vn6vv" containerName="extractor" Oct 30 12:33:21.558603 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:21.558567 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9wwfc"] Oct 30 12:33:23.723959 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:23.723910 4908 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-512ec5/simulator-w4-512ec5-kserve-7df7674f8-8rqwb" podUID="8a5dfcc9-dec3-4bc6-b0a8-c928350b3596" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:33:24.162853 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:24.161910 4908 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-512ec5/simulator-w4-512ec5-kserve-7df7674f8-8rqwb" podUID="8a5dfcc9-dec3-4bc6-b0a8-c928350b3596" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Oct 30 12:33:24.574957 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:24.574906 4908 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6a7148/distinct-2-w3-6a7148-kserve-67bd5fb976-9nfff" podUID="6046cc02-29fe-4f59-b522-8f57f4557277" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:33:24.613942 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:24.613900 4908 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47fd99/premium-w2-47fd99-kserve-5d85f9bb95-hqkmf" podUID="fd2cbeda-dbc4-41f6-a730-e1c6f9343c5e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:33:25.925909 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:25.925874 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c72dbbd-mvm9q"] Oct 30 12:33:25.926715 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:25.926694 4908 cpu_manager.go:414] "containerMap: removing container" podUID="2c4243b2-80bf-492e-ab85-bc9f33563bfd" containerName="revoke-keys" Oct 30 12:33:25.926790 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:25.926718 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c4243b2-80bf-492e-ab85-bc9f33563bfd" containerName="revoke-keys" Oct 30 12:33:25.926925 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:25.926910 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c4243b2-80bf-492e-ab85-bc9f33563bfd" containerName="revoke-keys" Oct 30 12:33:25.939819 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:25.939789 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c72dbbd-mvm9q" Oct 30 12:33:25.956656 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:25.956625 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c72dbbd-mvm9q"] Oct 30 12:33:25.958280 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:25.958256 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 12:33:25.958406 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:25.958312 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 12:33:25.958406 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:25.958386 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s75hq\"" Oct 30 12:33:25.964555 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:25.964532 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 12:33:26.085434 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:26.085405 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ee71283-e8bf-474f-b557-49d37ad3e704-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c72dbbd-mvm9q\" (UID: \"3ee71283-e8bf-474f-b557-49d37ad3e704\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c72dbbd-mvm9q" Oct 30 12:33:26.085608 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:26.085475 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbxbw\" (UniqueName: \"kubernetes.io/projected/3ee71283-e8bf-474f-b557-49d37ad3e704-kube-api-access-gbxbw\") pod \"maas-api-revoke-sub-0c72dbbd-mvm9q\" (UID: \"3ee71283-e8bf-474f-b557-49d37ad3e704\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c72dbbd-mvm9q" Oct 30 12:33:26.085739 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:26.085710 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ee71283-e8bf-474f-b557-49d37ad3e704-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c72dbbd-mvm9q\" (UID: \"3ee71283-e8bf-474f-b557-49d37ad3e704\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c72dbbd-mvm9q" Oct 30 12:33:26.191706 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:26.191600 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ee71283-e8bf-474f-b557-49d37ad3e704-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c72dbbd-mvm9q\" (UID: \"3ee71283-e8bf-474f-b557-49d37ad3e704\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c72dbbd-mvm9q" Oct 30 12:33:26.192035 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:26.192010 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ee71283-e8bf-474f-b557-49d37ad3e704-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c72dbbd-mvm9q\" (UID: \"3ee71283-e8bf-474f-b557-49d37ad3e704\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c72dbbd-mvm9q" Oct 30 12:33:26.192250 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:26.192232 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbxbw\" (UniqueName: \"kubernetes.io/projected/3ee71283-e8bf-474f-b557-49d37ad3e704-kube-api-access-gbxbw\") pod \"maas-api-revoke-sub-0c72dbbd-mvm9q\" (UID: \"3ee71283-e8bf-474f-b557-49d37ad3e704\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c72dbbd-mvm9q" Oct 30 12:33:26.214200 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:26.214166 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ee71283-e8bf-474f-b557-49d37ad3e704-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c72dbbd-mvm9q\" (UID: \"3ee71283-e8bf-474f-b557-49d37ad3e704\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c72dbbd-mvm9q" Oct 30 12:33:26.219391 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:26.219361 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ee71283-e8bf-474f-b557-49d37ad3e704-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c72dbbd-mvm9q\" (UID: \"3ee71283-e8bf-474f-b557-49d37ad3e704\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c72dbbd-mvm9q" Oct 30 12:33:26.239420 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:26.238592 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbxbw\" (UniqueName: \"kubernetes.io/projected/3ee71283-e8bf-474f-b557-49d37ad3e704-kube-api-access-gbxbw\") pod \"maas-api-revoke-sub-0c72dbbd-mvm9q\" (UID: \"3ee71283-e8bf-474f-b557-49d37ad3e704\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c72dbbd-mvm9q" Oct 30 12:33:26.271066 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:26.271031 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c72dbbd-mvm9q" Oct 30 12:33:26.642312 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:26.642280 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c72dbbd-mvm9q"] Oct 30 12:33:26.644147 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:33:26.644119 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ee71283_e8bf_474f_b557_49d37ad3e704.slice/crio-0a20706c03ef53c50edb70933b3515933a710fc372eb1d8b518c537fe67d8543 WatchSource:0}: Error finding container 0a20706c03ef53c50edb70933b3515933a710fc372eb1d8b518c537fe67d8543: Status 404 returned error can't find the container with id 0a20706c03ef53c50edb70933b3515933a710fc372eb1d8b518c537fe67d8543 Oct 30 12:33:26.737030 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:26.736994 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c72dbbd-mvm9q" event={"ID":"3ee71283-e8bf-474f-b557-49d37ad3e704","Type":"ContainerStarted","Data":"0a20706c03ef53c50edb70933b3515933a710fc372eb1d8b518c537fe67d8543"} Oct 30 12:33:27.748780 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:27.748690 4908 generic.go:356] "Generic (PLEG): container finished" podID="3ee71283-e8bf-474f-b557-49d37ad3e704" containerID="52cb89440105cc8a315366335f4fb8101e95061ad0e6b461a0fc46c948735cd0" exitCode=0 Oct 30 12:33:27.748780 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:27.748745 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c72dbbd-mvm9q" event={"ID":"3ee71283-e8bf-474f-b557-49d37ad3e704","Type":"ContainerDied","Data":"52cb89440105cc8a315366335f4fb8101e95061ad0e6b461a0fc46c948735cd0"} Oct 30 12:33:29.010317 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:29.010256 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c72dbbd-mvm9q" Oct 30 12:33:29.165248 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:29.165202 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ee71283-e8bf-474f-b557-49d37ad3e704-kube-api-access-gbxbw\" (UniqueName: \"kubernetes.io/projected/3ee71283-e8bf-474f-b557-49d37ad3e704-kube-api-access-gbxbw\") pod \"3ee71283-e8bf-474f-b557-49d37ad3e704\" (UID: \"3ee71283-e8bf-474f-b557-49d37ad3e704\") " Oct 30 12:33:29.165422 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:29.165326 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ee71283-e8bf-474f-b557-49d37ad3e704-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ee71283-e8bf-474f-b557-49d37ad3e704-maas-api-openshift-ca\") pod \"3ee71283-e8bf-474f-b557-49d37ad3e704\" (UID: \"3ee71283-e8bf-474f-b557-49d37ad3e704\") " Oct 30 12:33:29.165422 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:29.165405 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ee71283-e8bf-474f-b557-49d37ad3e704-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ee71283-e8bf-474f-b557-49d37ad3e704-maas-api-xks-ca\") pod \"3ee71283-e8bf-474f-b557-49d37ad3e704\" (UID: \"3ee71283-e8bf-474f-b557-49d37ad3e704\") " Oct 30 12:33:29.165688 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:29.165658 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ee71283-e8bf-474f-b557-49d37ad3e704-maas-api-openshift-ca" pod "3ee71283-e8bf-474f-b557-49d37ad3e704" (UID: "3ee71283-e8bf-474f-b557-49d37ad3e704"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:33:29.165853 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:29.165824 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ee71283-e8bf-474f-b557-49d37ad3e704-maas-api-openshift-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:33:29.167301 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:29.167273 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ee71283-e8bf-474f-b557-49d37ad3e704-kube-api-access-gbxbw" pod "3ee71283-e8bf-474f-b557-49d37ad3e704" (UID: "3ee71283-e8bf-474f-b557-49d37ad3e704"). InnerVolumeSpecName "kube-api-access-gbxbw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:33:29.167474 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:29.167453 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ee71283-e8bf-474f-b557-49d37ad3e704-maas-api-xks-ca" pod "3ee71283-e8bf-474f-b557-49d37ad3e704" (UID: "3ee71283-e8bf-474f-b557-49d37ad3e704"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:33:29.266507 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:29.266441 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbxbw\" (UniqueName: \"kubernetes.io/projected/3ee71283-e8bf-474f-b557-49d37ad3e704-kube-api-access-gbxbw\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:33:29.266507 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:29.266468 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ee71283-e8bf-474f-b557-49d37ad3e704-maas-api-xks-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:33:29.766600 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:29.766571 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c72dbbd-mvm9q" Oct 30 12:33:29.766769 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:29.766601 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c72dbbd-mvm9q" event={"ID":"3ee71283-e8bf-474f-b557-49d37ad3e704","Type":"ContainerDied","Data":"0a20706c03ef53c50edb70933b3515933a710fc372eb1d8b518c537fe67d8543"} Oct 30 12:33:29.766769 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:29.766658 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0a20706c03ef53c50edb70933b3515933a710fc372eb1d8b518c537fe67d8543" Oct 30 12:33:31.078786 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:31.078741 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-021452de-nv2nq"] Oct 30 12:33:31.079363 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:31.079346 4908 cpu_manager.go:414] "containerMap: removing container" podUID="3ee71283-e8bf-474f-b557-49d37ad3e704" containerName="revoke-keys" Oct 30 12:33:31.079363 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:31.079363 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ee71283-e8bf-474f-b557-49d37ad3e704" containerName="revoke-keys" Oct 30 12:33:31.079481 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:31.079450 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ee71283-e8bf-474f-b557-49d37ad3e704" containerName="revoke-keys" Oct 30 12:33:31.082905 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:31.082883 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-021452de-nv2nq" Oct 30 12:33:31.083610 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:31.083587 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9a635aa-4e2d-4acd-920e-8bee665a8e91-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-021452de-nv2nq\" (UID: \"d9a635aa-4e2d-4acd-920e-8bee665a8e91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-021452de-nv2nq" Oct 30 12:33:31.083765 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:31.083745 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9a635aa-4e2d-4acd-920e-8bee665a8e91-maas-api-xks-ca\") pod \"maas-api-revoke-sub-021452de-nv2nq\" (UID: \"d9a635aa-4e2d-4acd-920e-8bee665a8e91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-021452de-nv2nq" Oct 30 12:33:31.083843 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:31.083787 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spwbm\" (UniqueName: \"kubernetes.io/projected/d9a635aa-4e2d-4acd-920e-8bee665a8e91-kube-api-access-spwbm\") pod \"maas-api-revoke-sub-021452de-nv2nq\" (UID: \"d9a635aa-4e2d-4acd-920e-8bee665a8e91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-021452de-nv2nq" Oct 30 12:33:31.097370 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:31.097340 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 12:33:31.108350 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:31.108326 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 12:33:31.108521 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:31.108343 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s75hq\"" Oct 30 12:33:31.108521 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:31.108388 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 12:33:31.113207 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:31.113185 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-021452de-nv2nq"] Oct 30 12:33:31.184955 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:31.184928 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9a635aa-4e2d-4acd-920e-8bee665a8e91-maas-api-xks-ca\") pod \"maas-api-revoke-sub-021452de-nv2nq\" (UID: \"d9a635aa-4e2d-4acd-920e-8bee665a8e91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-021452de-nv2nq" Oct 30 12:33:31.185098 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:31.184971 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spwbm\" (UniqueName: \"kubernetes.io/projected/d9a635aa-4e2d-4acd-920e-8bee665a8e91-kube-api-access-spwbm\") pod \"maas-api-revoke-sub-021452de-nv2nq\" (UID: \"d9a635aa-4e2d-4acd-920e-8bee665a8e91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-021452de-nv2nq" Oct 30 12:33:31.185098 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:31.185052 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9a635aa-4e2d-4acd-920e-8bee665a8e91-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-021452de-nv2nq\" (UID: \"d9a635aa-4e2d-4acd-920e-8bee665a8e91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-021452de-nv2nq" Oct 30 12:33:31.185668 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:31.185646 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9a635aa-4e2d-4acd-920e-8bee665a8e91-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-021452de-nv2nq\" (UID: \"d9a635aa-4e2d-4acd-920e-8bee665a8e91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-021452de-nv2nq" Oct 30 12:33:31.187396 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:31.187372 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9a635aa-4e2d-4acd-920e-8bee665a8e91-maas-api-xks-ca\") pod \"maas-api-revoke-sub-021452de-nv2nq\" (UID: \"d9a635aa-4e2d-4acd-920e-8bee665a8e91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-021452de-nv2nq" Oct 30 12:33:31.206926 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:31.206903 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-spwbm\" (UniqueName: \"kubernetes.io/projected/d9a635aa-4e2d-4acd-920e-8bee665a8e91-kube-api-access-spwbm\") pod \"maas-api-revoke-sub-021452de-nv2nq\" (UID: \"d9a635aa-4e2d-4acd-920e-8bee665a8e91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-021452de-nv2nq" Oct 30 12:33:31.396699 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:31.396625 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-021452de-nv2nq" Oct 30 12:33:31.704270 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:31.704192 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-021452de-nv2nq"] Oct 30 12:33:31.711712 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:33:31.711681 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9a635aa_4e2d_4acd_920e_8bee665a8e91.slice/crio-7381f1a4b795fd309833f89ef1abb834b80ef6db961a27adfb95816bf89b040d WatchSource:0}: Error finding container 7381f1a4b795fd309833f89ef1abb834b80ef6db961a27adfb95816bf89b040d: Status 404 returned error can't find the container with id 7381f1a4b795fd309833f89ef1abb834b80ef6db961a27adfb95816bf89b040d Oct 30 12:33:31.783564 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:31.783541 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-021452de-nv2nq" event={"ID":"d9a635aa-4e2d-4acd-920e-8bee665a8e91","Type":"ContainerStarted","Data":"7381f1a4b795fd309833f89ef1abb834b80ef6db961a27adfb95816bf89b040d"} Oct 30 12:33:32.793366 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:32.793277 4908 generic.go:356] "Generic (PLEG): container finished" podID="d9a635aa-4e2d-4acd-920e-8bee665a8e91" containerID="1e8b2b85b4566566c62dee4f911343ccab3ba8caed248fcd20e4805506acd8af" exitCode=0 Oct 30 12:33:32.793789 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:32.793381 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-021452de-nv2nq" event={"ID":"d9a635aa-4e2d-4acd-920e-8bee665a8e91","Type":"ContainerDied","Data":"1e8b2b85b4566566c62dee4f911343ccab3ba8caed248fcd20e4805506acd8af"} Oct 30 12:33:33.962546 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:33.962523 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-021452de-nv2nq" Oct 30 12:33:34.015919 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:34.015890 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9a635aa-4e2d-4acd-920e-8bee665a8e91-kube-api-access-spwbm\" (UniqueName: \"kubernetes.io/projected/d9a635aa-4e2d-4acd-920e-8bee665a8e91-kube-api-access-spwbm\") pod \"d9a635aa-4e2d-4acd-920e-8bee665a8e91\" (UID: \"d9a635aa-4e2d-4acd-920e-8bee665a8e91\") " Oct 30 12:33:34.016094 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:34.015931 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d9a635aa-4e2d-4acd-920e-8bee665a8e91-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9a635aa-4e2d-4acd-920e-8bee665a8e91-maas-api-xks-ca\") pod \"d9a635aa-4e2d-4acd-920e-8bee665a8e91\" (UID: \"d9a635aa-4e2d-4acd-920e-8bee665a8e91\") " Oct 30 12:33:34.016094 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:34.015998 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9a635aa-4e2d-4acd-920e-8bee665a8e91-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9a635aa-4e2d-4acd-920e-8bee665a8e91-maas-api-openshift-ca\") pod \"d9a635aa-4e2d-4acd-920e-8bee665a8e91\" (UID: \"d9a635aa-4e2d-4acd-920e-8bee665a8e91\") " Oct 30 12:33:34.016406 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:34.016379 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9a635aa-4e2d-4acd-920e-8bee665a8e91-maas-api-openshift-ca" pod "d9a635aa-4e2d-4acd-920e-8bee665a8e91" (UID: "d9a635aa-4e2d-4acd-920e-8bee665a8e91"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:33:34.016627 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:34.016608 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9a635aa-4e2d-4acd-920e-8bee665a8e91-maas-api-openshift-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:33:34.018056 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:34.018033 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9a635aa-4e2d-4acd-920e-8bee665a8e91-maas-api-xks-ca" pod "d9a635aa-4e2d-4acd-920e-8bee665a8e91" (UID: "d9a635aa-4e2d-4acd-920e-8bee665a8e91"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:33:34.018514 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:34.018497 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9a635aa-4e2d-4acd-920e-8bee665a8e91-kube-api-access-spwbm" pod "d9a635aa-4e2d-4acd-920e-8bee665a8e91" (UID: "d9a635aa-4e2d-4acd-920e-8bee665a8e91"). InnerVolumeSpecName "kube-api-access-spwbm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:33:34.117707 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:34.117625 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-spwbm\" (UniqueName: \"kubernetes.io/projected/d9a635aa-4e2d-4acd-920e-8bee665a8e91-kube-api-access-spwbm\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:33:34.117707 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:34.117657 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9a635aa-4e2d-4acd-920e-8bee665a8e91-maas-api-xks-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:33:34.817669 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:34.817637 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-021452de-nv2nq" Oct 30 12:33:34.817944 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:34.817632 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-021452de-nv2nq" event={"ID":"d9a635aa-4e2d-4acd-920e-8bee665a8e91","Type":"ContainerDied","Data":"7381f1a4b795fd309833f89ef1abb834b80ef6db961a27adfb95816bf89b040d"} Oct 30 12:33:34.817944 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:34.817785 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7381f1a4b795fd309833f89ef1abb834b80ef6db961a27adfb95816bf89b040d" Oct 30 12:33:35.724458 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:35.724422 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-0172c6-bhqbn"] Oct 30 12:33:35.735166 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:35.735137 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-0172c6-bhqbn"] Oct 30 12:33:37.630551 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:37.630514 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c4243b2-80bf-492e-ab85-bc9f33563bfd" path="/var/lib/kubelet/pods/2c4243b2-80bf-492e-ab85-bc9f33563bfd/volumes" Oct 30 12:33:41.532962 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:41.532909 4908 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f920f7/simulator-w0-f920f7-kserve-599b7ff45f-g79d7" podUID="a85e562f-d655-4c22-b4f4-a299a4e494a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:33:41.642977 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:41.642936 4908 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6a7148/premium-w3-6a7148-kserve-67b976c888-tgd6c" podUID="13de4441-e7f6-4cbd-97d9-0f079d11cb8e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 12:33:44.081884 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:44.081844 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2461e8-k7lcr"] Oct 30 12:33:44.082677 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:44.082658 4908 cpu_manager.go:414] "containerMap: removing container" podUID="d9a635aa-4e2d-4acd-920e-8bee665a8e91" containerName="revoke-keys" Oct 30 12:33:44.082734 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:44.082681 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9a635aa-4e2d-4acd-920e-8bee665a8e91" containerName="revoke-keys" Oct 30 12:33:44.082865 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:44.082851 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9a635aa-4e2d-4acd-920e-8bee665a8e91" containerName="revoke-keys" Oct 30 12:33:44.088285 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:44.088265 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2461e8-k7lcr" Oct 30 12:33:44.102741 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:44.102718 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2461e8-k7lcr"] Oct 30 12:33:44.108655 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:44.108547 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 12:33:44.108655 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:44.108622 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 12:33:44.115191 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:44.115172 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 12:33:44.115439 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:44.115417 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s75hq\"" Oct 30 12:33:44.140993 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:44.140951 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdd00e00-df22-48ae-8621-954570963f3d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-2461e8-k7lcr\" (UID: \"fdd00e00-df22-48ae-8621-954570963f3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2461e8-k7lcr" Oct 30 12:33:44.141151 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:44.141131 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdd00e00-df22-48ae-8621-954570963f3d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-2461e8-k7lcr\" (UID: \"fdd00e00-df22-48ae-8621-954570963f3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2461e8-k7lcr" Oct 30 12:33:44.141229 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:44.141207 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nz7ht\" (UniqueName: \"kubernetes.io/projected/fdd00e00-df22-48ae-8621-954570963f3d-kube-api-access-nz7ht\") pod \"maas-api-revoke-keys-e2e-praxis-b-2461e8-k7lcr\" (UID: \"fdd00e00-df22-48ae-8621-954570963f3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2461e8-k7lcr" Oct 30 12:33:44.242905 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:44.242857 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdd00e00-df22-48ae-8621-954570963f3d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-2461e8-k7lcr\" (UID: \"fdd00e00-df22-48ae-8621-954570963f3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2461e8-k7lcr" Oct 30 12:33:44.243079 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:44.242933 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nz7ht\" (UniqueName: \"kubernetes.io/projected/fdd00e00-df22-48ae-8621-954570963f3d-kube-api-access-nz7ht\") pod \"maas-api-revoke-keys-e2e-praxis-b-2461e8-k7lcr\" (UID: \"fdd00e00-df22-48ae-8621-954570963f3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2461e8-k7lcr" Oct 30 12:33:44.243079 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:44.243020 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdd00e00-df22-48ae-8621-954570963f3d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-2461e8-k7lcr\" (UID: \"fdd00e00-df22-48ae-8621-954570963f3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2461e8-k7lcr" Oct 30 12:33:44.243614 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:44.243593 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdd00e00-df22-48ae-8621-954570963f3d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-2461e8-k7lcr\" (UID: \"fdd00e00-df22-48ae-8621-954570963f3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2461e8-k7lcr" Oct 30 12:33:44.245116 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:44.245091 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdd00e00-df22-48ae-8621-954570963f3d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-2461e8-k7lcr\" (UID: \"fdd00e00-df22-48ae-8621-954570963f3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2461e8-k7lcr" Oct 30 12:33:44.264507 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:44.264476 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nz7ht\" (UniqueName: \"kubernetes.io/projected/fdd00e00-df22-48ae-8621-954570963f3d-kube-api-access-nz7ht\") pod \"maas-api-revoke-keys-e2e-praxis-b-2461e8-k7lcr\" (UID: \"fdd00e00-df22-48ae-8621-954570963f3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2461e8-k7lcr" Oct 30 12:33:44.405806 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:44.405727 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2461e8-k7lcr" Oct 30 12:33:44.723224 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:44.723196 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2461e8-k7lcr"] Oct 30 12:33:44.724990 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:33:44.724966 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdd00e00_df22_48ae_8621_954570963f3d.slice/crio-b1adae4b46b2eb5a913c472261e27c6699e3648b125bebcb817f7268a71e8fbf WatchSource:0}: Error finding container b1adae4b46b2eb5a913c472261e27c6699e3648b125bebcb817f7268a71e8fbf: Status 404 returned error can't find the container with id b1adae4b46b2eb5a913c472261e27c6699e3648b125bebcb817f7268a71e8fbf Oct 30 12:33:44.945095 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:44.945057 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2461e8-k7lcr" event={"ID":"fdd00e00-df22-48ae-8621-954570963f3d","Type":"ContainerStarted","Data":"b1adae4b46b2eb5a913c472261e27c6699e3648b125bebcb817f7268a71e8fbf"} Oct 30 12:33:45.694124 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:45.694092 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df6b0f62-lt5g8"] Oct 30 12:33:45.698234 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:45.698213 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df6b0f62-lt5g8" Oct 30 12:33:45.729140 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:45.729114 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df6b0f62-lt5g8"] Oct 30 12:33:45.760874 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:45.760822 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef65b9d1-cff6-4ada-af9e-1f21372a65bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-df6b0f62-lt5g8\" (UID: \"ef65b9d1-cff6-4ada-af9e-1f21372a65bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df6b0f62-lt5g8" Oct 30 12:33:45.761020 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:45.760989 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhfk6\" (UniqueName: \"kubernetes.io/projected/ef65b9d1-cff6-4ada-af9e-1f21372a65bd-kube-api-access-mhfk6\") pod \"maas-api-revoke-sub-df6b0f62-lt5g8\" (UID: \"ef65b9d1-cff6-4ada-af9e-1f21372a65bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df6b0f62-lt5g8" Oct 30 12:33:45.761087 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:45.761036 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef65b9d1-cff6-4ada-af9e-1f21372a65bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-df6b0f62-lt5g8\" (UID: \"ef65b9d1-cff6-4ada-af9e-1f21372a65bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df6b0f62-lt5g8" Oct 30 12:33:45.863448 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:45.863211 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhfk6\" (UniqueName: \"kubernetes.io/projected/ef65b9d1-cff6-4ada-af9e-1f21372a65bd-kube-api-access-mhfk6\") pod \"maas-api-revoke-sub-df6b0f62-lt5g8\" (UID: \"ef65b9d1-cff6-4ada-af9e-1f21372a65bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df6b0f62-lt5g8" Oct 30 12:33:45.863760 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:45.863734 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef65b9d1-cff6-4ada-af9e-1f21372a65bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-df6b0f62-lt5g8\" (UID: \"ef65b9d1-cff6-4ada-af9e-1f21372a65bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df6b0f62-lt5g8" Oct 30 12:33:45.864385 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:45.863929 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef65b9d1-cff6-4ada-af9e-1f21372a65bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-df6b0f62-lt5g8\" (UID: \"ef65b9d1-cff6-4ada-af9e-1f21372a65bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df6b0f62-lt5g8" Oct 30 12:33:45.864893 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:45.864869 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef65b9d1-cff6-4ada-af9e-1f21372a65bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-df6b0f62-lt5g8\" (UID: \"ef65b9d1-cff6-4ada-af9e-1f21372a65bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df6b0f62-lt5g8" Oct 30 12:33:45.869639 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:45.869616 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef65b9d1-cff6-4ada-af9e-1f21372a65bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-df6b0f62-lt5g8\" (UID: \"ef65b9d1-cff6-4ada-af9e-1f21372a65bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df6b0f62-lt5g8" Oct 30 12:33:45.885578 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:45.885554 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhfk6\" (UniqueName: \"kubernetes.io/projected/ef65b9d1-cff6-4ada-af9e-1f21372a65bd-kube-api-access-mhfk6\") pod \"maas-api-revoke-sub-df6b0f62-lt5g8\" (UID: \"ef65b9d1-cff6-4ada-af9e-1f21372a65bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df6b0f62-lt5g8" Oct 30 12:33:45.954255 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:45.954227 4908 generic.go:356] "Generic (PLEG): container finished" podID="fdd00e00-df22-48ae-8621-954570963f3d" containerID="fd840c81d5911f4fc6eee3c84fbfcb6cfd846b56198021948f75292d4fb42f89" exitCode=0 Oct 30 12:33:45.954364 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:45.954295 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2461e8-k7lcr" event={"ID":"fdd00e00-df22-48ae-8621-954570963f3d","Type":"ContainerDied","Data":"fd840c81d5911f4fc6eee3c84fbfcb6cfd846b56198021948f75292d4fb42f89"} Oct 30 12:33:46.012005 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:46.011962 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df6b0f62-lt5g8" Oct 30 12:33:46.317069 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:46.317041 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df6b0f62-lt5g8"] Oct 30 12:33:46.319625 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:33:46.319596 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef65b9d1_cff6_4ada_af9e_1f21372a65bd.slice/crio-9b56961a5f49e4dea0431bbdd45cf50eb8a780d513f328a57b10d27c7085f6f1 WatchSource:0}: Error finding container 9b56961a5f49e4dea0431bbdd45cf50eb8a780d513f328a57b10d27c7085f6f1: Status 404 returned error can't find the container with id 9b56961a5f49e4dea0431bbdd45cf50eb8a780d513f328a57b10d27c7085f6f1 Oct 30 12:33:46.964966 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:46.964862 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df6b0f62-lt5g8" event={"ID":"ef65b9d1-cff6-4ada-af9e-1f21372a65bd","Type":"ContainerStarted","Data":"c6f14a6e89ea764022f4ecb30fdb1926cc0b0c9f8ae5b51c6833bf7c350ae129"} Oct 30 12:33:46.964966 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:46.964930 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df6b0f62-lt5g8" event={"ID":"ef65b9d1-cff6-4ada-af9e-1f21372a65bd","Type":"ContainerStarted","Data":"9b56961a5f49e4dea0431bbdd45cf50eb8a780d513f328a57b10d27c7085f6f1"} Oct 30 12:33:47.003179 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:47.003124 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df6b0f62-lt5g8" podStartSLOduration=2.003110147 podStartE2EDuration="2.003110147s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:33:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:33:47.002376702 +0000 UTC m=+2383.516805512" watchObservedRunningTime="2026-10-30 12:33:47.003110147 +0000 UTC m=+2383.517538893" Oct 30 12:33:47.162083 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:47.162055 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2461e8-k7lcr" Oct 30 12:33:47.281096 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:47.281066 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdd00e00-df22-48ae-8621-954570963f3d-kube-api-access-nz7ht\" (UniqueName: \"kubernetes.io/projected/fdd00e00-df22-48ae-8621-954570963f3d-kube-api-access-nz7ht\") pod \"fdd00e00-df22-48ae-8621-954570963f3d\" (UID: \"fdd00e00-df22-48ae-8621-954570963f3d\") " Oct 30 12:33:47.281263 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:47.281103 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fdd00e00-df22-48ae-8621-954570963f3d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdd00e00-df22-48ae-8621-954570963f3d-maas-api-xks-ca\") pod \"fdd00e00-df22-48ae-8621-954570963f3d\" (UID: \"fdd00e00-df22-48ae-8621-954570963f3d\") " Oct 30 12:33:47.281263 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:47.281164 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fdd00e00-df22-48ae-8621-954570963f3d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdd00e00-df22-48ae-8621-954570963f3d-maas-api-openshift-ca\") pod \"fdd00e00-df22-48ae-8621-954570963f3d\" (UID: \"fdd00e00-df22-48ae-8621-954570963f3d\") " Oct 30 12:33:47.281644 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:47.281615 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdd00e00-df22-48ae-8621-954570963f3d-maas-api-openshift-ca" pod "fdd00e00-df22-48ae-8621-954570963f3d" (UID: "fdd00e00-df22-48ae-8621-954570963f3d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:33:47.283260 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:47.283229 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fdd00e00-df22-48ae-8621-954570963f3d-maas-api-xks-ca" pod "fdd00e00-df22-48ae-8621-954570963f3d" (UID: "fdd00e00-df22-48ae-8621-954570963f3d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:33:47.283391 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:47.283310 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdd00e00-df22-48ae-8621-954570963f3d-kube-api-access-nz7ht" pod "fdd00e00-df22-48ae-8621-954570963f3d" (UID: "fdd00e00-df22-48ae-8621-954570963f3d"). InnerVolumeSpecName "kube-api-access-nz7ht". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:33:47.382534 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:47.382496 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nz7ht\" (UniqueName: \"kubernetes.io/projected/fdd00e00-df22-48ae-8621-954570963f3d-kube-api-access-nz7ht\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:33:47.382534 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:47.382524 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdd00e00-df22-48ae-8621-954570963f3d-maas-api-xks-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:33:47.382534 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:47.382534 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdd00e00-df22-48ae-8621-954570963f3d-maas-api-openshift-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:33:47.412772 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:47.412743 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c983309c-82j7m"] Oct 30 12:33:47.413304 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:47.413288 4908 cpu_manager.go:414] "containerMap: removing container" podUID="fdd00e00-df22-48ae-8621-954570963f3d" containerName="revoke-keys" Oct 30 12:33:47.413351 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:47.413306 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdd00e00-df22-48ae-8621-954570963f3d" containerName="revoke-keys" Oct 30 12:33:47.413414 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:47.413405 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="fdd00e00-df22-48ae-8621-954570963f3d" containerName="revoke-keys" Oct 30 12:33:47.416926 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:47.416909 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c983309c-82j7m" Oct 30 12:33:47.435178 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:47.435155 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c983309c-82j7m"] Oct 30 12:33:47.484014 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:47.483989 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f29949d-d1fa-4455-91ed-f3fbdb3ddefd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c983309c-82j7m\" (UID: \"5f29949d-d1fa-4455-91ed-f3fbdb3ddefd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c983309c-82j7m" Oct 30 12:33:47.484179 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:47.484159 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f29949d-d1fa-4455-91ed-f3fbdb3ddefd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c983309c-82j7m\" (UID: \"5f29949d-d1fa-4455-91ed-f3fbdb3ddefd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c983309c-82j7m" Oct 30 12:33:47.484259 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:47.484227 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hrn9\" (UniqueName: \"kubernetes.io/projected/5f29949d-d1fa-4455-91ed-f3fbdb3ddefd-kube-api-access-8hrn9\") pod \"maas-api-revoke-sub-c983309c-82j7m\" (UID: \"5f29949d-d1fa-4455-91ed-f3fbdb3ddefd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c983309c-82j7m" Oct 30 12:33:47.585463 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:47.585381 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f29949d-d1fa-4455-91ed-f3fbdb3ddefd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c983309c-82j7m\" (UID: \"5f29949d-d1fa-4455-91ed-f3fbdb3ddefd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c983309c-82j7m" Oct 30 12:33:47.585463 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:47.585440 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hrn9\" (UniqueName: \"kubernetes.io/projected/5f29949d-d1fa-4455-91ed-f3fbdb3ddefd-kube-api-access-8hrn9\") pod \"maas-api-revoke-sub-c983309c-82j7m\" (UID: \"5f29949d-d1fa-4455-91ed-f3fbdb3ddefd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c983309c-82j7m" Oct 30 12:33:47.585672 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:47.585554 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f29949d-d1fa-4455-91ed-f3fbdb3ddefd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c983309c-82j7m\" (UID: \"5f29949d-d1fa-4455-91ed-f3fbdb3ddefd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c983309c-82j7m" Oct 30 12:33:47.586404 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:47.586375 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f29949d-d1fa-4455-91ed-f3fbdb3ddefd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c983309c-82j7m\" (UID: \"5f29949d-d1fa-4455-91ed-f3fbdb3ddefd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c983309c-82j7m" Oct 30 12:33:47.588007 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:47.587983 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f29949d-d1fa-4455-91ed-f3fbdb3ddefd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c983309c-82j7m\" (UID: \"5f29949d-d1fa-4455-91ed-f3fbdb3ddefd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c983309c-82j7m" Oct 30 12:33:47.628251 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:47.623177 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hrn9\" (UniqueName: \"kubernetes.io/projected/5f29949d-d1fa-4455-91ed-f3fbdb3ddefd-kube-api-access-8hrn9\") pod \"maas-api-revoke-sub-c983309c-82j7m\" (UID: \"5f29949d-d1fa-4455-91ed-f3fbdb3ddefd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c983309c-82j7m" Oct 30 12:33:47.730477 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:47.730444 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c983309c-82j7m" Oct 30 12:33:47.989638 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:47.989596 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2461e8-k7lcr" event={"ID":"fdd00e00-df22-48ae-8621-954570963f3d","Type":"ContainerDied","Data":"b1adae4b46b2eb5a913c472261e27c6699e3648b125bebcb817f7268a71e8fbf"} Oct 30 12:33:47.990163 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:47.989696 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b1adae4b46b2eb5a913c472261e27c6699e3648b125bebcb817f7268a71e8fbf" Oct 30 12:33:47.990163 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:47.989764 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2461e8-k7lcr" Oct 30 12:33:47.992150 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:47.992109 4908 generic.go:356] "Generic (PLEG): container finished" podID="ef65b9d1-cff6-4ada-af9e-1f21372a65bd" containerID="c6f14a6e89ea764022f4ecb30fdb1926cc0b0c9f8ae5b51c6833bf7c350ae129" exitCode=0 Oct 30 12:33:47.992374 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:47.992325 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df6b0f62-lt5g8" event={"ID":"ef65b9d1-cff6-4ada-af9e-1f21372a65bd","Type":"ContainerDied","Data":"c6f14a6e89ea764022f4ecb30fdb1926cc0b0c9f8ae5b51c6833bf7c350ae129"} Oct 30 12:33:48.053345 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:33:48.053315 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f29949d_d1fa_4455_91ed_f3fbdb3ddefd.slice/crio-65fd269ab707c6fce29bdbb320a7caa2cca8be4b631eae314526b21dd7fea51a WatchSource:0}: Error finding container 65fd269ab707c6fce29bdbb320a7caa2cca8be4b631eae314526b21dd7fea51a: Status 404 returned error can't find the container with id 65fd269ab707c6fce29bdbb320a7caa2cca8be4b631eae314526b21dd7fea51a Oct 30 12:33:48.060146 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:48.060119 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c983309c-82j7m"] Oct 30 12:33:49.040853 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:49.040309 4908 generic.go:356] "Generic (PLEG): container finished" podID="5f29949d-d1fa-4455-91ed-f3fbdb3ddefd" containerID="e9463f386879eb8f29856437a007fd02f7c48b25e6498e368970c8c04a2b2f1e" exitCode=0 Oct 30 12:33:49.041425 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:49.040920 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c983309c-82j7m" event={"ID":"5f29949d-d1fa-4455-91ed-f3fbdb3ddefd","Type":"ContainerDied","Data":"e9463f386879eb8f29856437a007fd02f7c48b25e6498e368970c8c04a2b2f1e"} Oct 30 12:33:49.041425 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:49.040976 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c983309c-82j7m" event={"ID":"5f29949d-d1fa-4455-91ed-f3fbdb3ddefd","Type":"ContainerStarted","Data":"65fd269ab707c6fce29bdbb320a7caa2cca8be4b631eae314526b21dd7fea51a"} Oct 30 12:33:49.225264 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:49.225239 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df6b0f62-lt5g8" Oct 30 12:33:49.336660 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:49.336624 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef65b9d1-cff6-4ada-af9e-1f21372a65bd-kube-api-access-mhfk6\" (UniqueName: \"kubernetes.io/projected/ef65b9d1-cff6-4ada-af9e-1f21372a65bd-kube-api-access-mhfk6\") pod \"ef65b9d1-cff6-4ada-af9e-1f21372a65bd\" (UID: \"ef65b9d1-cff6-4ada-af9e-1f21372a65bd\") " Oct 30 12:33:49.336864 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:49.336842 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef65b9d1-cff6-4ada-af9e-1f21372a65bd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef65b9d1-cff6-4ada-af9e-1f21372a65bd-maas-api-openshift-ca\") pod \"ef65b9d1-cff6-4ada-af9e-1f21372a65bd\" (UID: \"ef65b9d1-cff6-4ada-af9e-1f21372a65bd\") " Oct 30 12:33:49.336939 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:49.336927 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef65b9d1-cff6-4ada-af9e-1f21372a65bd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef65b9d1-cff6-4ada-af9e-1f21372a65bd-maas-api-xks-ca\") pod \"ef65b9d1-cff6-4ada-af9e-1f21372a65bd\" (UID: \"ef65b9d1-cff6-4ada-af9e-1f21372a65bd\") " Oct 30 12:33:49.337200 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:49.337161 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef65b9d1-cff6-4ada-af9e-1f21372a65bd-maas-api-openshift-ca" pod "ef65b9d1-cff6-4ada-af9e-1f21372a65bd" (UID: "ef65b9d1-cff6-4ada-af9e-1f21372a65bd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:33:49.337491 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:49.337459 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef65b9d1-cff6-4ada-af9e-1f21372a65bd-maas-api-openshift-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:33:49.339168 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:49.339144 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef65b9d1-cff6-4ada-af9e-1f21372a65bd-kube-api-access-mhfk6" pod "ef65b9d1-cff6-4ada-af9e-1f21372a65bd" (UID: "ef65b9d1-cff6-4ada-af9e-1f21372a65bd"). InnerVolumeSpecName "kube-api-access-mhfk6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:33:49.339604 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:49.339583 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef65b9d1-cff6-4ada-af9e-1f21372a65bd-maas-api-xks-ca" pod "ef65b9d1-cff6-4ada-af9e-1f21372a65bd" (UID: "ef65b9d1-cff6-4ada-af9e-1f21372a65bd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:33:49.438243 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:49.438164 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef65b9d1-cff6-4ada-af9e-1f21372a65bd-maas-api-xks-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:33:49.438243 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:49.438192 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhfk6\" (UniqueName: \"kubernetes.io/projected/ef65b9d1-cff6-4ada-af9e-1f21372a65bd-kube-api-access-mhfk6\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:33:49.684996 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:33:49.684962 4908 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vn6vv" containerName="extractor" Oct 30 12:33:50.049980 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:50.049950 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df6b0f62-lt5g8" Oct 30 12:33:50.050379 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:50.049989 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df6b0f62-lt5g8" event={"ID":"ef65b9d1-cff6-4ada-af9e-1f21372a65bd","Type":"ContainerDied","Data":"9b56961a5f49e4dea0431bbdd45cf50eb8a780d513f328a57b10d27c7085f6f1"} Oct 30 12:33:50.050379 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:50.050040 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9b56961a5f49e4dea0431bbdd45cf50eb8a780d513f328a57b10d27c7085f6f1" Oct 30 12:33:50.217365 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:50.217339 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c983309c-82j7m" Oct 30 12:33:50.350550 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:50.350478 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f29949d-d1fa-4455-91ed-f3fbdb3ddefd-kube-api-access-8hrn9\" (UniqueName: \"kubernetes.io/projected/5f29949d-d1fa-4455-91ed-f3fbdb3ddefd-kube-api-access-8hrn9\") pod \"5f29949d-d1fa-4455-91ed-f3fbdb3ddefd\" (UID: \"5f29949d-d1fa-4455-91ed-f3fbdb3ddefd\") " Oct 30 12:33:50.350691 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:50.350586 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f29949d-d1fa-4455-91ed-f3fbdb3ddefd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f29949d-d1fa-4455-91ed-f3fbdb3ddefd-maas-api-xks-ca\") pod \"5f29949d-d1fa-4455-91ed-f3fbdb3ddefd\" (UID: \"5f29949d-d1fa-4455-91ed-f3fbdb3ddefd\") " Oct 30 12:33:50.350691 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:50.350622 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f29949d-d1fa-4455-91ed-f3fbdb3ddefd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f29949d-d1fa-4455-91ed-f3fbdb3ddefd-maas-api-openshift-ca\") pod \"5f29949d-d1fa-4455-91ed-f3fbdb3ddefd\" (UID: \"5f29949d-d1fa-4455-91ed-f3fbdb3ddefd\") " Oct 30 12:33:50.351038 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:50.351016 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f29949d-d1fa-4455-91ed-f3fbdb3ddefd-maas-api-openshift-ca" pod "5f29949d-d1fa-4455-91ed-f3fbdb3ddefd" (UID: "5f29949d-d1fa-4455-91ed-f3fbdb3ddefd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:33:50.352771 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:50.352749 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f29949d-d1fa-4455-91ed-f3fbdb3ddefd-kube-api-access-8hrn9" pod "5f29949d-d1fa-4455-91ed-f3fbdb3ddefd" (UID: "5f29949d-d1fa-4455-91ed-f3fbdb3ddefd"). InnerVolumeSpecName "kube-api-access-8hrn9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:33:50.352862 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:50.352768 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f29949d-d1fa-4455-91ed-f3fbdb3ddefd-maas-api-xks-ca" pod "5f29949d-d1fa-4455-91ed-f3fbdb3ddefd" (UID: "5f29949d-d1fa-4455-91ed-f3fbdb3ddefd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:33:50.452100 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:50.452074 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f29949d-d1fa-4455-91ed-f3fbdb3ddefd-maas-api-xks-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:33:50.452100 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:50.452097 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f29949d-d1fa-4455-91ed-f3fbdb3ddefd-maas-api-openshift-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:33:50.452252 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:50.452106 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hrn9\" (UniqueName: \"kubernetes.io/projected/5f29949d-d1fa-4455-91ed-f3fbdb3ddefd-kube-api-access-8hrn9\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:33:51.060225 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:51.060189 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c983309c-82j7m" event={"ID":"5f29949d-d1fa-4455-91ed-f3fbdb3ddefd","Type":"ContainerDied","Data":"65fd269ab707c6fce29bdbb320a7caa2cca8be4b631eae314526b21dd7fea51a"} Oct 30 12:33:51.060602 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:51.060245 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="65fd269ab707c6fce29bdbb320a7caa2cca8be4b631eae314526b21dd7fea51a" Oct 30 12:33:51.060602 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:51.060191 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c983309c-82j7m" Oct 30 12:33:54.943426 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:54.943394 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-a124ab-xlkkq"] Oct 30 12:33:54.943969 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:54.943953 4908 cpu_manager.go:414] "containerMap: removing container" podUID="ef65b9d1-cff6-4ada-af9e-1f21372a65bd" containerName="revoke-keys" Oct 30 12:33:54.944021 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:54.943970 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef65b9d1-cff6-4ada-af9e-1f21372a65bd" containerName="revoke-keys" Oct 30 12:33:54.944021 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:54.943992 4908 cpu_manager.go:414] "containerMap: removing container" podUID="5f29949d-d1fa-4455-91ed-f3fbdb3ddefd" containerName="revoke-keys" Oct 30 12:33:54.944021 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:54.943997 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f29949d-d1fa-4455-91ed-f3fbdb3ddefd" containerName="revoke-keys" Oct 30 12:33:54.944115 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:54.944091 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef65b9d1-cff6-4ada-af9e-1f21372a65bd" containerName="revoke-keys" Oct 30 12:33:54.944115 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:54.944105 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f29949d-d1fa-4455-91ed-f3fbdb3ddefd" containerName="revoke-keys" Oct 30 12:33:54.947774 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:54.947759 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-a124ab-xlkkq" Oct 30 12:33:54.970376 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:54.970348 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-a124ab-xlkkq"] Oct 30 12:33:54.972907 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:54.972882 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 12:33:54.973100 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:54.972971 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 12:33:54.976351 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:54.976329 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s75hq\"" Oct 30 12:33:54.976567 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:54.976544 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 12:33:55.113580 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:55.113548 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ee09d53-9417-4b5c-b787-00cdb0e56e5d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-a124ab-xlkkq\" (UID: \"9ee09d53-9417-4b5c-b787-00cdb0e56e5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-a124ab-xlkkq" Oct 30 12:33:55.113744 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:55.113596 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ee09d53-9417-4b5c-b787-00cdb0e56e5d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-a124ab-xlkkq\" (UID: \"9ee09d53-9417-4b5c-b787-00cdb0e56e5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-a124ab-xlkkq" Oct 30 12:33:55.113802 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:55.113780 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jw52z\" (UniqueName: \"kubernetes.io/projected/9ee09d53-9417-4b5c-b787-00cdb0e56e5d-kube-api-access-jw52z\") pod \"maas-api-revoke-keys-e2e-praxis-a-a124ab-xlkkq\" (UID: \"9ee09d53-9417-4b5c-b787-00cdb0e56e5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-a124ab-xlkkq" Oct 30 12:33:55.216903 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:55.215917 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ee09d53-9417-4b5c-b787-00cdb0e56e5d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-a124ab-xlkkq\" (UID: \"9ee09d53-9417-4b5c-b787-00cdb0e56e5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-a124ab-xlkkq" Oct 30 12:33:55.216903 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:55.216160 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jw52z\" (UniqueName: \"kubernetes.io/projected/9ee09d53-9417-4b5c-b787-00cdb0e56e5d-kube-api-access-jw52z\") pod \"maas-api-revoke-keys-e2e-praxis-a-a124ab-xlkkq\" (UID: \"9ee09d53-9417-4b5c-b787-00cdb0e56e5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-a124ab-xlkkq" Oct 30 12:33:55.216903 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:55.216232 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ee09d53-9417-4b5c-b787-00cdb0e56e5d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-a124ab-xlkkq\" (UID: \"9ee09d53-9417-4b5c-b787-00cdb0e56e5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-a124ab-xlkkq" Oct 30 12:33:55.217172 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:55.217077 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ee09d53-9417-4b5c-b787-00cdb0e56e5d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-a124ab-xlkkq\" (UID: \"9ee09d53-9417-4b5c-b787-00cdb0e56e5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-a124ab-xlkkq" Oct 30 12:33:55.245849 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:55.245177 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ee09d53-9417-4b5c-b787-00cdb0e56e5d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-a124ab-xlkkq\" (UID: \"9ee09d53-9417-4b5c-b787-00cdb0e56e5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-a124ab-xlkkq" Oct 30 12:33:55.265912 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:55.264465 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jw52z\" (UniqueName: \"kubernetes.io/projected/9ee09d53-9417-4b5c-b787-00cdb0e56e5d-kube-api-access-jw52z\") pod \"maas-api-revoke-keys-e2e-praxis-a-a124ab-xlkkq\" (UID: \"9ee09d53-9417-4b5c-b787-00cdb0e56e5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-a124ab-xlkkq" Oct 30 12:33:55.563699 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:55.563667 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-a124ab-xlkkq" Oct 30 12:33:55.871782 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:55.871751 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-a124ab-xlkkq"] Oct 30 12:33:55.873665 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:33:55.873633 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ee09d53_9417_4b5c_b787_00cdb0e56e5d.slice/crio-208aa9ebc15e91698d1f1f4e286753a5cfb749360ddb55a2169d567e8bfc0406 WatchSource:0}: Error finding container 208aa9ebc15e91698d1f1f4e286753a5cfb749360ddb55a2169d567e8bfc0406: Status 404 returned error can't find the container with id 208aa9ebc15e91698d1f1f4e286753a5cfb749360ddb55a2169d567e8bfc0406 Oct 30 12:33:56.101041 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:56.100991 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-a124ab-xlkkq" event={"ID":"9ee09d53-9417-4b5c-b787-00cdb0e56e5d","Type":"ContainerStarted","Data":"ad2ba82f46db36add7a67bb7a945c5037d0fcd6bd972a2d2a61dca693b5fba80"} Oct 30 12:33:56.101436 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:56.101095 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-a124ab-xlkkq" event={"ID":"9ee09d53-9417-4b5c-b787-00cdb0e56e5d","Type":"ContainerStarted","Data":"208aa9ebc15e91698d1f1f4e286753a5cfb749360ddb55a2169d567e8bfc0406"} Oct 30 12:33:56.141915 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:56.141815 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-a124ab-xlkkq" podStartSLOduration=2.1418035 podStartE2EDuration="2.1418035s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:33:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:33:56.138816453 +0000 UTC m=+2392.653245201" watchObservedRunningTime="2026-10-30 12:33:56.1418035 +0000 UTC m=+2392.656232245" Oct 30 12:33:57.111506 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:57.111414 4908 generic.go:356] "Generic (PLEG): container finished" podID="9ee09d53-9417-4b5c-b787-00cdb0e56e5d" containerID="ad2ba82f46db36add7a67bb7a945c5037d0fcd6bd972a2d2a61dca693b5fba80" exitCode=0 Oct 30 12:33:57.111506 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:57.111453 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-a124ab-xlkkq" event={"ID":"9ee09d53-9417-4b5c-b787-00cdb0e56e5d","Type":"ContainerDied","Data":"ad2ba82f46db36add7a67bb7a945c5037d0fcd6bd972a2d2a61dca693b5fba80"} Oct 30 12:33:58.287164 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:58.287137 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-a124ab-xlkkq" Oct 30 12:33:58.452556 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:58.452467 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9ee09d53-9417-4b5c-b787-00cdb0e56e5d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ee09d53-9417-4b5c-b787-00cdb0e56e5d-maas-api-xks-ca\") pod \"9ee09d53-9417-4b5c-b787-00cdb0e56e5d\" (UID: \"9ee09d53-9417-4b5c-b787-00cdb0e56e5d\") " Oct 30 12:33:58.452556 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:58.452519 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ee09d53-9417-4b5c-b787-00cdb0e56e5d-kube-api-access-jw52z\" (UniqueName: \"kubernetes.io/projected/9ee09d53-9417-4b5c-b787-00cdb0e56e5d-kube-api-access-jw52z\") pod \"9ee09d53-9417-4b5c-b787-00cdb0e56e5d\" (UID: \"9ee09d53-9417-4b5c-b787-00cdb0e56e5d\") " Oct 30 12:33:58.452788 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:58.452676 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ee09d53-9417-4b5c-b787-00cdb0e56e5d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ee09d53-9417-4b5c-b787-00cdb0e56e5d-maas-api-openshift-ca\") pod \"9ee09d53-9417-4b5c-b787-00cdb0e56e5d\" (UID: \"9ee09d53-9417-4b5c-b787-00cdb0e56e5d\") " Oct 30 12:33:58.453942 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:58.453800 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ee09d53-9417-4b5c-b787-00cdb0e56e5d-maas-api-openshift-ca" pod "9ee09d53-9417-4b5c-b787-00cdb0e56e5d" (UID: "9ee09d53-9417-4b5c-b787-00cdb0e56e5d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:33:58.468125 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:58.465061 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ee09d53-9417-4b5c-b787-00cdb0e56e5d-maas-api-xks-ca" pod "9ee09d53-9417-4b5c-b787-00cdb0e56e5d" (UID: "9ee09d53-9417-4b5c-b787-00cdb0e56e5d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:33:58.468125 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:58.465201 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ee09d53-9417-4b5c-b787-00cdb0e56e5d-kube-api-access-jw52z" pod "9ee09d53-9417-4b5c-b787-00cdb0e56e5d" (UID: "9ee09d53-9417-4b5c-b787-00cdb0e56e5d"). InnerVolumeSpecName "kube-api-access-jw52z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:33:58.557175 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:58.556920 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ee09d53-9417-4b5c-b787-00cdb0e56e5d-maas-api-openshift-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:33:58.557175 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:58.556970 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ee09d53-9417-4b5c-b787-00cdb0e56e5d-maas-api-xks-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:33:58.557175 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:58.557013 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jw52z\" (UniqueName: \"kubernetes.io/projected/9ee09d53-9417-4b5c-b787-00cdb0e56e5d-kube-api-access-jw52z\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:33:59.131661 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:59.131631 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-a124ab-xlkkq" Oct 30 12:33:59.131921 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:59.131669 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-a124ab-xlkkq" event={"ID":"9ee09d53-9417-4b5c-b787-00cdb0e56e5d","Type":"ContainerDied","Data":"208aa9ebc15e91698d1f1f4e286753a5cfb749360ddb55a2169d567e8bfc0406"} Oct 30 12:33:59.131921 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:33:59.131713 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="208aa9ebc15e91698d1f1f4e286753a5cfb749360ddb55a2169d567e8bfc0406" Oct 30 12:34:01.171397 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:01.171346 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02be0005-mxjlm"] Oct 30 12:34:01.172161 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:01.172138 4908 cpu_manager.go:414] "containerMap: removing container" podUID="9ee09d53-9417-4b5c-b787-00cdb0e56e5d" containerName="revoke-keys" Oct 30 12:34:01.172161 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:01.172160 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ee09d53-9417-4b5c-b787-00cdb0e56e5d" containerName="revoke-keys" Oct 30 12:34:01.172331 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:01.172307 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ee09d53-9417-4b5c-b787-00cdb0e56e5d" containerName="revoke-keys" Oct 30 12:34:01.176142 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:01.176121 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02be0005-mxjlm" Oct 30 12:34:01.201467 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:01.201431 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02be0005-mxjlm"] Oct 30 12:34:01.205544 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:01.205498 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 12:34:01.206233 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:01.206199 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s75hq\"" Oct 30 12:34:01.206366 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:01.206259 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 12:34:01.206366 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:01.206320 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 12:34:01.300089 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:01.300051 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5g4g\" (UniqueName: \"kubernetes.io/projected/383e8645-c168-4e9c-859a-55a9781dbfb1-kube-api-access-g5g4g\") pod \"maas-api-revoke-sub-02be0005-mxjlm\" (UID: \"383e8645-c168-4e9c-859a-55a9781dbfb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02be0005-mxjlm" Oct 30 12:34:01.300267 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:01.300123 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/383e8645-c168-4e9c-859a-55a9781dbfb1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-02be0005-mxjlm\" (UID: \"383e8645-c168-4e9c-859a-55a9781dbfb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02be0005-mxjlm" Oct 30 12:34:01.300412 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:01.300389 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/383e8645-c168-4e9c-859a-55a9781dbfb1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-02be0005-mxjlm\" (UID: \"383e8645-c168-4e9c-859a-55a9781dbfb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02be0005-mxjlm" Oct 30 12:34:01.401187 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:01.401158 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/383e8645-c168-4e9c-859a-55a9781dbfb1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-02be0005-mxjlm\" (UID: \"383e8645-c168-4e9c-859a-55a9781dbfb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02be0005-mxjlm" Oct 30 12:34:01.401322 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:01.401282 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/383e8645-c168-4e9c-859a-55a9781dbfb1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-02be0005-mxjlm\" (UID: \"383e8645-c168-4e9c-859a-55a9781dbfb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02be0005-mxjlm" Oct 30 12:34:01.401377 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:01.401331 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5g4g\" (UniqueName: \"kubernetes.io/projected/383e8645-c168-4e9c-859a-55a9781dbfb1-kube-api-access-g5g4g\") pod \"maas-api-revoke-sub-02be0005-mxjlm\" (UID: \"383e8645-c168-4e9c-859a-55a9781dbfb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02be0005-mxjlm" Oct 30 12:34:01.402038 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:01.402009 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/383e8645-c168-4e9c-859a-55a9781dbfb1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-02be0005-mxjlm\" (UID: \"383e8645-c168-4e9c-859a-55a9781dbfb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02be0005-mxjlm" Oct 30 12:34:01.403815 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:01.403785 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/383e8645-c168-4e9c-859a-55a9781dbfb1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-02be0005-mxjlm\" (UID: \"383e8645-c168-4e9c-859a-55a9781dbfb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02be0005-mxjlm" Oct 30 12:34:01.424084 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:01.424028 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5g4g\" (UniqueName: \"kubernetes.io/projected/383e8645-c168-4e9c-859a-55a9781dbfb1-kube-api-access-g5g4g\") pod \"maas-api-revoke-sub-02be0005-mxjlm\" (UID: \"383e8645-c168-4e9c-859a-55a9781dbfb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02be0005-mxjlm" Oct 30 12:34:01.490562 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:01.490535 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02be0005-mxjlm" Oct 30 12:34:01.801674 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:01.801640 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02be0005-mxjlm"] Oct 30 12:34:01.802810 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:34:01.802773 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod383e8645_c168_4e9c_859a_55a9781dbfb1.slice/crio-0f9c861ea8fe8347c1caae5d0f5181a928e60adb7298a08e65def64881284795 WatchSource:0}: Error finding container 0f9c861ea8fe8347c1caae5d0f5181a928e60adb7298a08e65def64881284795: Status 404 returned error can't find the container with id 0f9c861ea8fe8347c1caae5d0f5181a928e60adb7298a08e65def64881284795 Oct 30 12:34:02.161890 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:02.161819 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02be0005-mxjlm" event={"ID":"383e8645-c168-4e9c-859a-55a9781dbfb1","Type":"ContainerStarted","Data":"0f9c861ea8fe8347c1caae5d0f5181a928e60adb7298a08e65def64881284795"} Oct 30 12:34:02.175429 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:02.175406 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9wwfc"] Oct 30 12:34:03.177771 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:03.177686 4908 generic.go:356] "Generic (PLEG): container finished" podID="383e8645-c168-4e9c-859a-55a9781dbfb1" containerID="1803ee80b5b4152f1cdcbbb792546378663092f01ba6dd3c93494fcc87ee07e3" exitCode=0 Oct 30 12:34:03.180960 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:03.178215 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02be0005-mxjlm" event={"ID":"383e8645-c168-4e9c-859a-55a9781dbfb1","Type":"ContainerDied","Data":"1803ee80b5b4152f1cdcbbb792546378663092f01ba6dd3c93494fcc87ee07e3"} Oct 30 12:34:04.532128 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:04.530920 4908 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f920f7/simulator-w0-f920f7-kserve-599b7ff45f-g79d7" podUID="a85e562f-d655-4c22-b4f4-a299a4e494a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:34:04.559581 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:04.558103 4908 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47fd99/distinct-2-w2-47fd99-kserve-77cf84b469-zxs8s" podUID="baf44d3d-42e6-43f7-a0ad-f0f9932320c1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 30 12:34:04.634152 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:04.634091 4908 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6a7148/premium-w3-6a7148-kserve-67b976c888-tgd6c" podUID="13de4441-e7f6-4cbd-97d9-0f079d11cb8e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:34:04.675869 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:04.671911 4908 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47fd99/premium-w2-47fd99-kserve-5d85f9bb95-hqkmf" podUID="fd2cbeda-dbc4-41f6-a730-e1c6f9343c5e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Oct 30 12:34:04.679969 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:04.679918 4908 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6a7148/distinct-2-w3-6a7148-kserve-67bd5fb976-9nfff" podUID="6046cc02-29fe-4f59-b522-8f57f4557277" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 12:34:05.209536 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:05.169080 4908 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-512ec5/simulator-w4-512ec5-kserve-7df7674f8-8rqwb" podUID="8a5dfcc9-dec3-4bc6-b0a8-c928350b3596" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:34:05.273752 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:05.238334 4908 scope.go:122] "RemoveContainer" containerID="d5c465473f67485ed98ced3fdacdaa9f07b2b028baba5d69a2d5042af9ee7018" Oct 30 12:34:05.350484 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:05.350074 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02be0005-mxjlm" Oct 30 12:34:05.358858 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:05.352177 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02be0005-mxjlm" event={"ID":"383e8645-c168-4e9c-859a-55a9781dbfb1","Type":"ContainerDied","Data":"0f9c861ea8fe8347c1caae5d0f5181a928e60adb7298a08e65def64881284795"} Oct 30 12:34:05.358858 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:05.352236 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0f9c861ea8fe8347c1caae5d0f5181a928e60adb7298a08e65def64881284795" Oct 30 12:34:05.416343 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:05.416308 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/383e8645-c168-4e9c-859a-55a9781dbfb1-kube-api-access-g5g4g\" (UniqueName: \"kubernetes.io/projected/383e8645-c168-4e9c-859a-55a9781dbfb1-kube-api-access-g5g4g\") pod \"383e8645-c168-4e9c-859a-55a9781dbfb1\" (UID: \"383e8645-c168-4e9c-859a-55a9781dbfb1\") " Oct 30 12:34:05.456879 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:05.455870 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/383e8645-c168-4e9c-859a-55a9781dbfb1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/383e8645-c168-4e9c-859a-55a9781dbfb1-maas-api-openshift-ca\") pod \"383e8645-c168-4e9c-859a-55a9781dbfb1\" (UID: \"383e8645-c168-4e9c-859a-55a9781dbfb1\") " Oct 30 12:34:05.456879 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:05.455998 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/383e8645-c168-4e9c-859a-55a9781dbfb1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/383e8645-c168-4e9c-859a-55a9781dbfb1-maas-api-xks-ca\") pod \"383e8645-c168-4e9c-859a-55a9781dbfb1\" (UID: \"383e8645-c168-4e9c-859a-55a9781dbfb1\") " Oct 30 12:34:05.457308 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:05.457285 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/383e8645-c168-4e9c-859a-55a9781dbfb1-maas-api-openshift-ca" pod "383e8645-c168-4e9c-859a-55a9781dbfb1" (UID: "383e8645-c168-4e9c-859a-55a9781dbfb1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:34:05.563863 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:05.540917 4908 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f920f7/simulator-w0-f920f7-kserve-599b7ff45f-g79d7" podUID="a85e562f-d655-4c22-b4f4-a299a4e494a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 12:34:05.563863 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:05.542350 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f920f7/simulator-w0-f920f7-kserve-599b7ff45f-g79d7" Oct 30 12:34:05.567327 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:05.567300 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/383e8645-c168-4e9c-859a-55a9781dbfb1-maas-api-openshift-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:34:05.595395 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:05.595371 4908 scope.go:122] "RemoveContainer" containerID="5113b55b6db549ba7f8b132585f80665e0806f06f066153003d4e26e12d6f037" Oct 30 12:34:05.596264 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:05.596243 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f920f7/simulator-w0-f920f7-kserve-599b7ff45f-g79d7" Oct 30 12:34:05.695345 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:05.695248 4908 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6a7148/distinct-2-w3-6a7148-kserve-67bd5fb976-9nfff" podUID="6046cc02-29fe-4f59-b522-8f57f4557277" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": context deadline exceeded" Oct 30 12:34:05.701077 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:05.701027 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/383e8645-c168-4e9c-859a-55a9781dbfb1-kube-api-access-g5g4g" pod "383e8645-c168-4e9c-859a-55a9781dbfb1" (UID: "383e8645-c168-4e9c-859a-55a9781dbfb1"). InnerVolumeSpecName "kube-api-access-g5g4g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:34:05.810851 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:05.806362 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/383e8645-c168-4e9c-859a-55a9781dbfb1-maas-api-xks-ca" pod "383e8645-c168-4e9c-859a-55a9781dbfb1" (UID: "383e8645-c168-4e9c-859a-55a9781dbfb1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:34:05.836850 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:05.823170 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g5g4g\" (UniqueName: \"kubernetes.io/projected/383e8645-c168-4e9c-859a-55a9781dbfb1-kube-api-access-g5g4g\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:34:05.836850 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:05.823205 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/383e8645-c168-4e9c-859a-55a9781dbfb1-maas-api-xks-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:34:05.915707 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:05.915673 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-6a7148/distinct-2-w3-6a7148-kserve-67bd5fb976-9nfff" Oct 30 12:34:05.939609 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:05.939581 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-6a7148/distinct-2-w3-6a7148-kserve-67bd5fb976-9nfff" Oct 30 12:34:06.017138 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:34:06.016913 4908 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5c465473f67485ed98ced3fdacdaa9f07b2b028baba5d69a2d5042af9ee7018\": container with ID starting with d5c465473f67485ed98ced3fdacdaa9f07b2b028baba5d69a2d5042af9ee7018 not found: ID does not exist" containerID="d5c465473f67485ed98ced3fdacdaa9f07b2b028baba5d69a2d5042af9ee7018" Oct 30 12:34:06.017356 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:34:06.017328 4908 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="get container status: runtime container status: rpc error: code = NotFound desc = could not find container \"d5c465473f67485ed98ced3fdacdaa9f07b2b028baba5d69a2d5042af9ee7018\": container with ID starting with d5c465473f67485ed98ced3fdacdaa9f07b2b028baba5d69a2d5042af9ee7018 not found: ID does not exist" Oct 30 12:34:06.038030 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:06.038000 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2461e8-k7lcr"] Oct 30 12:34:06.040551 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:06.040523 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2461e8-k7lcr"] Oct 30 12:34:06.097771 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:06.097550 4908 scope.go:122] "RemoveContainer" containerID="e27fdac5987b431a869150937e48f5b2d65f4e7bf4aa1568302dae692a7ca88d" Oct 30 12:34:06.299129 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:06.299095 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-c44eee-openshift-default-6758d9545-k5l44"] Oct 30 12:34:06.302580 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:06.302411 4908 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-c44eee-openshift-default-6758d9545-k5l44" podUID="fcefbe36-2c1b-43bf-aec5-4e2def37ef97" containerName="istio-proxy" containerID="cri-o://62e582122a2594963d6956e0f8456fce7f85096e0dd11a8432c4dde509a543b0" gracePeriod=30 Oct 30 12:34:06.362905 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:06.362819 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02be0005-mxjlm" Oct 30 12:34:06.395405 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:06.395378 4908 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-plczt_a70ee11c-c20e-4a4f-8476-ac5eb02816d0/console-operator/3.log" Oct 30 12:34:06.401845 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:06.401815 4908 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b8bxz_644e19b3-10ad-4145-9ed3-595d19ed9b07/ovn-acl-logging/0.log" Oct 30 12:34:07.654869 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:07.654728 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fdd00e00-df22-48ae-8621-954570963f3d" path="/var/lib/kubelet/pods/fdd00e00-df22-48ae-8621-954570963f3d/volumes" Oct 30 12:34:08.605967 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:08.605930 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bcafdbfc-ljf77"] Oct 30 12:34:08.606678 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:08.606658 4908 cpu_manager.go:414] "containerMap: removing container" podUID="383e8645-c168-4e9c-859a-55a9781dbfb1" containerName="revoke-keys" Oct 30 12:34:08.606771 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:08.606679 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="383e8645-c168-4e9c-859a-55a9781dbfb1" containerName="revoke-keys" Oct 30 12:34:08.606888 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:08.606875 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="383e8645-c168-4e9c-859a-55a9781dbfb1" containerName="revoke-keys" Oct 30 12:34:08.611987 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:08.611959 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcafdbfc-ljf77" Oct 30 12:34:08.626590 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:08.626566 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bcafdbfc-ljf77"] Oct 30 12:34:08.632517 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:08.632497 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 12:34:08.632654 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:08.632517 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 12:34:08.632654 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:08.632555 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s75hq\"" Oct 30 12:34:08.632654 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:08.632574 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 12:34:08.705085 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:08.705019 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe1dc6d4-2233-47d2-a637-f5d8531a8b4f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bcafdbfc-ljf77\" (UID: \"fe1dc6d4-2233-47d2-a637-f5d8531a8b4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcafdbfc-ljf77" Oct 30 12:34:08.705754 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:08.705719 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe1dc6d4-2233-47d2-a637-f5d8531a8b4f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bcafdbfc-ljf77\" (UID: \"fe1dc6d4-2233-47d2-a637-f5d8531a8b4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcafdbfc-ljf77" Oct 30 12:34:08.711505 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:08.706056 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g866w\" (UniqueName: \"kubernetes.io/projected/fe1dc6d4-2233-47d2-a637-f5d8531a8b4f-kube-api-access-g866w\") pod \"maas-api-revoke-sub-bcafdbfc-ljf77\" (UID: \"fe1dc6d4-2233-47d2-a637-f5d8531a8b4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcafdbfc-ljf77" Oct 30 12:34:08.814445 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:08.814412 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe1dc6d4-2233-47d2-a637-f5d8531a8b4f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bcafdbfc-ljf77\" (UID: \"fe1dc6d4-2233-47d2-a637-f5d8531a8b4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcafdbfc-ljf77" Oct 30 12:34:08.814602 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:08.814485 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g866w\" (UniqueName: \"kubernetes.io/projected/fe1dc6d4-2233-47d2-a637-f5d8531a8b4f-kube-api-access-g866w\") pod \"maas-api-revoke-sub-bcafdbfc-ljf77\" (UID: \"fe1dc6d4-2233-47d2-a637-f5d8531a8b4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcafdbfc-ljf77" Oct 30 12:34:08.814648 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:08.814612 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe1dc6d4-2233-47d2-a637-f5d8531a8b4f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bcafdbfc-ljf77\" (UID: \"fe1dc6d4-2233-47d2-a637-f5d8531a8b4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcafdbfc-ljf77" Oct 30 12:34:08.815247 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:08.815219 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe1dc6d4-2233-47d2-a637-f5d8531a8b4f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bcafdbfc-ljf77\" (UID: \"fe1dc6d4-2233-47d2-a637-f5d8531a8b4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcafdbfc-ljf77" Oct 30 12:34:08.817152 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:08.817126 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe1dc6d4-2233-47d2-a637-f5d8531a8b4f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bcafdbfc-ljf77\" (UID: \"fe1dc6d4-2233-47d2-a637-f5d8531a8b4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcafdbfc-ljf77" Oct 30 12:34:08.836548 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:08.836519 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g866w\" (UniqueName: \"kubernetes.io/projected/fe1dc6d4-2233-47d2-a637-f5d8531a8b4f-kube-api-access-g866w\") pod \"maas-api-revoke-sub-bcafdbfc-ljf77\" (UID: \"fe1dc6d4-2233-47d2-a637-f5d8531a8b4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcafdbfc-ljf77" Oct 30 12:34:08.927070 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:08.927001 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcafdbfc-ljf77" Oct 30 12:34:09.228586 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:09.228551 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bcafdbfc-ljf77"] Oct 30 12:34:09.230410 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:34:09.230386 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe1dc6d4_2233_47d2_a637_f5d8531a8b4f.slice/crio-ceb0db2495bca55b2cd21fa153c23486d413ce10a14ad6546c4dc1acef291402 WatchSource:0}: Error finding container ceb0db2495bca55b2cd21fa153c23486d413ce10a14ad6546c4dc1acef291402: Status 404 returned error can't find the container with id ceb0db2495bca55b2cd21fa153c23486d413ce10a14ad6546c4dc1acef291402 Oct 30 12:34:09.391721 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:09.391688 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcafdbfc-ljf77" event={"ID":"fe1dc6d4-2233-47d2-a637-f5d8531a8b4f","Type":"ContainerStarted","Data":"ceb0db2495bca55b2cd21fa153c23486d413ce10a14ad6546c4dc1acef291402"} Oct 30 12:34:10.438014 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:10.437972 4908 generic.go:356] "Generic (PLEG): container finished" podID="fe1dc6d4-2233-47d2-a637-f5d8531a8b4f" containerID="6f92403d6f4fd8df4b95daf93fcbbdef2b920c12c4fd17cd088688ab45f1ae31" exitCode=0 Oct 30 12:34:10.438999 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:10.438937 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcafdbfc-ljf77" event={"ID":"fe1dc6d4-2233-47d2-a637-f5d8531a8b4f","Type":"ContainerDied","Data":"6f92403d6f4fd8df4b95daf93fcbbdef2b920c12c4fd17cd088688ab45f1ae31"} Oct 30 12:34:11.449362 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:11.449333 4908 generic.go:356] "Generic (PLEG): container finished" podID="fcefbe36-2c1b-43bf-aec5-4e2def37ef97" containerID="62e582122a2594963d6956e0f8456fce7f85096e0dd11a8432c4dde509a543b0" exitCode=0 Oct 30 12:34:11.449730 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:11.449402 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-c44eee-openshift-default-6758d9545-k5l44" event={"ID":"fcefbe36-2c1b-43bf-aec5-4e2def37ef97","Type":"ContainerDied","Data":"62e582122a2594963d6956e0f8456fce7f85096e0dd11a8432c4dde509a543b0"} Oct 30 12:34:11.449730 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:11.449463 4908 scope.go:122] "RemoveContainer" containerID="a83eb4ce815d8daf3833499153def814577c34c6a9b1c3b022fa405292eda2c4" Oct 30 12:34:11.644081 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:11.644056 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-c44eee-openshift-default-6758d9545-k5l44" Oct 30 12:34:11.646140 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:11.646119 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcafdbfc-ljf77" Oct 30 12:34:11.743121 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:11.743090 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fcefbe36-2c1b-43bf-aec5-4e2def37ef97-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcefbe36-2c1b-43bf-aec5-4e2def37ef97-istio-podinfo\") pod \"fcefbe36-2c1b-43bf-aec5-4e2def37ef97\" (UID: \"fcefbe36-2c1b-43bf-aec5-4e2def37ef97\") " Oct 30 12:34:11.743271 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:11.743154 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fcefbe36-2c1b-43bf-aec5-4e2def37ef97-kube-api-access-tjdlk\" (UniqueName: \"kubernetes.io/projected/fcefbe36-2c1b-43bf-aec5-4e2def37ef97-kube-api-access-tjdlk\") pod \"fcefbe36-2c1b-43bf-aec5-4e2def37ef97\" (UID: \"fcefbe36-2c1b-43bf-aec5-4e2def37ef97\") " Oct 30 12:34:11.743271 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:11.743186 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcefbe36-2c1b-43bf-aec5-4e2def37ef97-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcefbe36-2c1b-43bf-aec5-4e2def37ef97-istio-data\") pod \"fcefbe36-2c1b-43bf-aec5-4e2def37ef97\" (UID: \"fcefbe36-2c1b-43bf-aec5-4e2def37ef97\") " Oct 30 12:34:11.743271 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:11.743230 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcefbe36-2c1b-43bf-aec5-4e2def37ef97-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcefbe36-2c1b-43bf-aec5-4e2def37ef97-istio-envoy\") pod \"fcefbe36-2c1b-43bf-aec5-4e2def37ef97\" (UID: \"fcefbe36-2c1b-43bf-aec5-4e2def37ef97\") " Oct 30 12:34:11.743271 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:11.743257 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe1dc6d4-2233-47d2-a637-f5d8531a8b4f-kube-api-access-g866w\" (UniqueName: \"kubernetes.io/projected/fe1dc6d4-2233-47d2-a637-f5d8531a8b4f-kube-api-access-g866w\") pod \"fe1dc6d4-2233-47d2-a637-f5d8531a8b4f\" (UID: \"fe1dc6d4-2233-47d2-a637-f5d8531a8b4f\") " Oct 30 12:34:11.743509 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:11.743323 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcefbe36-2c1b-43bf-aec5-4e2def37ef97-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcefbe36-2c1b-43bf-aec5-4e2def37ef97-credential-socket\") pod \"fcefbe36-2c1b-43bf-aec5-4e2def37ef97\" (UID: \"fcefbe36-2c1b-43bf-aec5-4e2def37ef97\") " Oct 30 12:34:11.743509 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:11.743379 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fe1dc6d4-2233-47d2-a637-f5d8531a8b4f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe1dc6d4-2233-47d2-a637-f5d8531a8b4f-maas-api-xks-ca\") pod \"fe1dc6d4-2233-47d2-a637-f5d8531a8b4f\" (UID: \"fe1dc6d4-2233-47d2-a637-f5d8531a8b4f\") " Oct 30 12:34:11.743509 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:11.743414 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe1dc6d4-2233-47d2-a637-f5d8531a8b4f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe1dc6d4-2233-47d2-a637-f5d8531a8b4f-maas-api-openshift-ca\") pod \"fe1dc6d4-2233-47d2-a637-f5d8531a8b4f\" (UID: \"fe1dc6d4-2233-47d2-a637-f5d8531a8b4f\") " Oct 30 12:34:11.743509 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:11.743456 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fcefbe36-2c1b-43bf-aec5-4e2def37ef97-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcefbe36-2c1b-43bf-aec5-4e2def37ef97-istiod-ca-cert\") pod \"fcefbe36-2c1b-43bf-aec5-4e2def37ef97\" (UID: \"fcefbe36-2c1b-43bf-aec5-4e2def37ef97\") " Oct 30 12:34:11.743710 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:11.743516 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcefbe36-2c1b-43bf-aec5-4e2def37ef97-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcefbe36-2c1b-43bf-aec5-4e2def37ef97-workload-socket\") pod \"fcefbe36-2c1b-43bf-aec5-4e2def37ef97\" (UID: \"fcefbe36-2c1b-43bf-aec5-4e2def37ef97\") " Oct 30 12:34:11.743710 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:11.743567 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcefbe36-2c1b-43bf-aec5-4e2def37ef97-istio-data" pod "fcefbe36-2c1b-43bf-aec5-4e2def37ef97" (UID: "fcefbe36-2c1b-43bf-aec5-4e2def37ef97"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:34:11.743710 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:11.743588 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fcefbe36-2c1b-43bf-aec5-4e2def37ef97-istio-token\" (UniqueName: \"kubernetes.io/projected/fcefbe36-2c1b-43bf-aec5-4e2def37ef97-istio-token\") pod \"fcefbe36-2c1b-43bf-aec5-4e2def37ef97\" (UID: \"fcefbe36-2c1b-43bf-aec5-4e2def37ef97\") " Oct 30 12:34:11.743710 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:11.743636 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcefbe36-2c1b-43bf-aec5-4e2def37ef97-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcefbe36-2c1b-43bf-aec5-4e2def37ef97-workload-certs\") pod \"fcefbe36-2c1b-43bf-aec5-4e2def37ef97\" (UID: \"fcefbe36-2c1b-43bf-aec5-4e2def37ef97\") " Oct 30 12:34:11.743939 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:11.743785 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcefbe36-2c1b-43bf-aec5-4e2def37ef97-credential-socket" pod "fcefbe36-2c1b-43bf-aec5-4e2def37ef97" (UID: "fcefbe36-2c1b-43bf-aec5-4e2def37ef97"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:34:11.744016 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:11.743995 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcefbe36-2c1b-43bf-aec5-4e2def37ef97-istiod-ca-cert" pod "fcefbe36-2c1b-43bf-aec5-4e2def37ef97" (UID: "fcefbe36-2c1b-43bf-aec5-4e2def37ef97"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:34:11.744074 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:11.744008 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcefbe36-2c1b-43bf-aec5-4e2def37ef97-workload-certs" pod "fcefbe36-2c1b-43bf-aec5-4e2def37ef97" (UID: "fcefbe36-2c1b-43bf-aec5-4e2def37ef97"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:34:11.744232 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:11.744210 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcefbe36-2c1b-43bf-aec5-4e2def37ef97-workload-socket" pod "fcefbe36-2c1b-43bf-aec5-4e2def37ef97" (UID: "fcefbe36-2c1b-43bf-aec5-4e2def37ef97"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:34:11.744301 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:11.744243 4908 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcefbe36-2c1b-43bf-aec5-4e2def37ef97-credential-socket\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:34:11.744301 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:11.744264 4908 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcefbe36-2c1b-43bf-aec5-4e2def37ef97-istiod-ca-cert\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:34:11.744301 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:11.744279 4908 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcefbe36-2c1b-43bf-aec5-4e2def37ef97-workload-certs\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:34:11.744301 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:11.744292 4908 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcefbe36-2c1b-43bf-aec5-4e2def37ef97-istio-data\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:34:11.744443 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:11.744424 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe1dc6d4-2233-47d2-a637-f5d8531a8b4f-maas-api-openshift-ca" pod "fe1dc6d4-2233-47d2-a637-f5d8531a8b4f" (UID: "fe1dc6d4-2233-47d2-a637-f5d8531a8b4f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:34:11.748393 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:11.745965 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcefbe36-2c1b-43bf-aec5-4e2def37ef97-kube-api-access-tjdlk" pod "fcefbe36-2c1b-43bf-aec5-4e2def37ef97" (UID: "fcefbe36-2c1b-43bf-aec5-4e2def37ef97"). InnerVolumeSpecName "kube-api-access-tjdlk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:34:11.748393 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:11.747205 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcefbe36-2c1b-43bf-aec5-4e2def37ef97-istio-token" pod "fcefbe36-2c1b-43bf-aec5-4e2def37ef97" (UID: "fcefbe36-2c1b-43bf-aec5-4e2def37ef97"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:34:11.748745 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:11.748724 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcefbe36-2c1b-43bf-aec5-4e2def37ef97-istio-envoy" pod "fcefbe36-2c1b-43bf-aec5-4e2def37ef97" (UID: "fcefbe36-2c1b-43bf-aec5-4e2def37ef97"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:34:11.751201 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:11.751178 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fcefbe36-2c1b-43bf-aec5-4e2def37ef97-istio-podinfo" pod "fcefbe36-2c1b-43bf-aec5-4e2def37ef97" (UID: "fcefbe36-2c1b-43bf-aec5-4e2def37ef97"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 12:34:11.751201 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:11.751184 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe1dc6d4-2233-47d2-a637-f5d8531a8b4f-kube-api-access-g866w" pod "fe1dc6d4-2233-47d2-a637-f5d8531a8b4f" (UID: "fe1dc6d4-2233-47d2-a637-f5d8531a8b4f"). InnerVolumeSpecName "kube-api-access-g866w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:34:11.752312 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:11.752286 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe1dc6d4-2233-47d2-a637-f5d8531a8b4f-maas-api-xks-ca" pod "fe1dc6d4-2233-47d2-a637-f5d8531a8b4f" (UID: "fe1dc6d4-2233-47d2-a637-f5d8531a8b4f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:34:11.845034 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:11.845006 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjdlk\" (UniqueName: \"kubernetes.io/projected/fcefbe36-2c1b-43bf-aec5-4e2def37ef97-kube-api-access-tjdlk\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:34:11.845034 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:11.845030 4908 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcefbe36-2c1b-43bf-aec5-4e2def37ef97-istio-envoy\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:34:11.845176 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:11.845040 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g866w\" (UniqueName: \"kubernetes.io/projected/fe1dc6d4-2233-47d2-a637-f5d8531a8b4f-kube-api-access-g866w\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:34:11.845176 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:11.845049 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe1dc6d4-2233-47d2-a637-f5d8531a8b4f-maas-api-xks-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:34:11.845176 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:11.845058 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe1dc6d4-2233-47d2-a637-f5d8531a8b4f-maas-api-openshift-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:34:11.845176 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:11.845065 4908 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcefbe36-2c1b-43bf-aec5-4e2def37ef97-workload-socket\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:34:11.845176 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:11.845075 4908 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fcefbe36-2c1b-43bf-aec5-4e2def37ef97-istio-token\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:34:11.845176 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:11.845083 4908 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcefbe36-2c1b-43bf-aec5-4e2def37ef97-istio-podinfo\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:34:11.851145 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:11.851084 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-5ce7c4-hctqm"] Oct 30 12:34:11.851678 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:11.851659 4908 cpu_manager.go:414] "containerMap: removing container" podUID="fcefbe36-2c1b-43bf-aec5-4e2def37ef97" containerName="istio-proxy" Oct 30 12:34:11.851770 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:11.851679 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fcefbe36-2c1b-43bf-aec5-4e2def37ef97" containerName="istio-proxy" Oct 30 12:34:11.851770 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:11.851720 4908 cpu_manager.go:414] "containerMap: removing container" podUID="fe1dc6d4-2233-47d2-a637-f5d8531a8b4f" containerName="revoke-keys" Oct 30 12:34:11.851770 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:11.851729 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe1dc6d4-2233-47d2-a637-f5d8531a8b4f" containerName="revoke-keys" Oct 30 12:34:11.851950 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:11.851913 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe1dc6d4-2233-47d2-a637-f5d8531a8b4f" containerName="revoke-keys" Oct 30 12:34:11.851950 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:11.851932 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="fcefbe36-2c1b-43bf-aec5-4e2def37ef97" containerName="istio-proxy" Oct 30 12:34:11.856882 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:11.856862 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-5ce7c4-hctqm" Oct 30 12:34:11.876807 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:11.876781 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-5ce7c4-hctqm"] Oct 30 12:34:11.946422 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:11.946395 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c84680a-8b23-48bc-b048-bf3b3bcbe10f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-5ce7c4-hctqm\" (UID: \"7c84680a-8b23-48bc-b048-bf3b3bcbe10f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-5ce7c4-hctqm" Oct 30 12:34:11.946570 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:11.946435 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c84680a-8b23-48bc-b048-bf3b3bcbe10f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-5ce7c4-hctqm\" (UID: \"7c84680a-8b23-48bc-b048-bf3b3bcbe10f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-5ce7c4-hctqm" Oct 30 12:34:11.946782 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:11.946753 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjf5x\" (UniqueName: \"kubernetes.io/projected/7c84680a-8b23-48bc-b048-bf3b3bcbe10f-kube-api-access-rjf5x\") pod \"maas-api-revoke-keys-e2e-praxis-b-5ce7c4-hctqm\" (UID: \"7c84680a-8b23-48bc-b048-bf3b3bcbe10f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-5ce7c4-hctqm" Oct 30 12:34:12.047599 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:12.047565 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjf5x\" (UniqueName: \"kubernetes.io/projected/7c84680a-8b23-48bc-b048-bf3b3bcbe10f-kube-api-access-rjf5x\") pod \"maas-api-revoke-keys-e2e-praxis-b-5ce7c4-hctqm\" (UID: \"7c84680a-8b23-48bc-b048-bf3b3bcbe10f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-5ce7c4-hctqm" Oct 30 12:34:12.047766 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:12.047606 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c84680a-8b23-48bc-b048-bf3b3bcbe10f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-5ce7c4-hctqm\" (UID: \"7c84680a-8b23-48bc-b048-bf3b3bcbe10f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-5ce7c4-hctqm" Oct 30 12:34:12.047766 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:12.047631 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c84680a-8b23-48bc-b048-bf3b3bcbe10f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-5ce7c4-hctqm\" (UID: \"7c84680a-8b23-48bc-b048-bf3b3bcbe10f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-5ce7c4-hctqm" Oct 30 12:34:12.048365 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:12.048342 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c84680a-8b23-48bc-b048-bf3b3bcbe10f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-5ce7c4-hctqm\" (UID: \"7c84680a-8b23-48bc-b048-bf3b3bcbe10f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-5ce7c4-hctqm" Oct 30 12:34:12.049928 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:12.049907 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c84680a-8b23-48bc-b048-bf3b3bcbe10f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-5ce7c4-hctqm\" (UID: \"7c84680a-8b23-48bc-b048-bf3b3bcbe10f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-5ce7c4-hctqm" Oct 30 12:34:12.070881 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:12.070818 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjf5x\" (UniqueName: \"kubernetes.io/projected/7c84680a-8b23-48bc-b048-bf3b3bcbe10f-kube-api-access-rjf5x\") pod \"maas-api-revoke-keys-e2e-praxis-b-5ce7c4-hctqm\" (UID: \"7c84680a-8b23-48bc-b048-bf3b3bcbe10f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-5ce7c4-hctqm" Oct 30 12:34:12.191355 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:12.182422 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-5ce7c4-hctqm" Oct 30 12:34:12.460058 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:12.459802 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-c44eee-openshift-default-6758d9545-k5l44" event={"ID":"fcefbe36-2c1b-43bf-aec5-4e2def37ef97","Type":"ContainerDied","Data":"0975b689c4da48002bde7bbfc79624d329b14dc985db7a2348b82efe7f1d30fd"} Oct 30 12:34:12.460058 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:12.459910 4908 scope.go:122] "RemoveContainer" containerID="62e582122a2594963d6956e0f8456fce7f85096e0dd11a8432c4dde509a543b0" Oct 30 12:34:12.460661 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:12.459774 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-c44eee-openshift-default-6758d9545-k5l44" Oct 30 12:34:12.462862 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:12.462817 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcafdbfc-ljf77" event={"ID":"fe1dc6d4-2233-47d2-a637-f5d8531a8b4f","Type":"ContainerDied","Data":"ceb0db2495bca55b2cd21fa153c23486d413ce10a14ad6546c4dc1acef291402"} Oct 30 12:34:12.462862 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:12.462861 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcafdbfc-ljf77" Oct 30 12:34:12.463054 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:12.462880 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ceb0db2495bca55b2cd21fa153c23486d413ce10a14ad6546c4dc1acef291402" Oct 30 12:34:12.506802 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:12.506776 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-5ce7c4-hctqm"] Oct 30 12:34:12.510884 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:34:12.510853 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c84680a_8b23_48bc_b048_bf3b3bcbe10f.slice/crio-5b94e2477652a13c96cbbfb89a989eece26914d580f46f3c52edb4ec49e902b9 WatchSource:0}: Error finding container 5b94e2477652a13c96cbbfb89a989eece26914d580f46f3c52edb4ec49e902b9: Status 404 returned error can't find the container with id 5b94e2477652a13c96cbbfb89a989eece26914d580f46f3c52edb4ec49e902b9 Oct 30 12:34:12.521241 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:12.521220 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-c44eee-openshift-default-6758d9545-k5l44"] Oct 30 12:34:12.526770 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:12.526747 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-c44eee-openshift-default-6758d9545-k5l44"] Oct 30 12:34:13.478156 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:13.478120 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-5ce7c4-hctqm" event={"ID":"7c84680a-8b23-48bc-b048-bf3b3bcbe10f","Type":"ContainerStarted","Data":"da1e8c404a3b9dad86ffc2d42241694e2d9447ed9a58343cab19b55e1c918b34"} Oct 30 12:34:13.478739 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:13.478709 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-5ce7c4-hctqm" event={"ID":"7c84680a-8b23-48bc-b048-bf3b3bcbe10f","Type":"ContainerStarted","Data":"5b94e2477652a13c96cbbfb89a989eece26914d580f46f3c52edb4ec49e902b9"} Oct 30 12:34:13.524855 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:13.524773 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-5ce7c4-hctqm" podStartSLOduration=2.5247561530000002 podStartE2EDuration="2.524756153s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:34:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:34:13.520767991 +0000 UTC m=+2410.035196750" watchObservedRunningTime="2026-10-30 12:34:13.524756153 +0000 UTC m=+2410.039184899" Oct 30 12:34:13.670851 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:13.669820 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fcefbe36-2c1b-43bf-aec5-4e2def37ef97" path="/var/lib/kubelet/pods/fcefbe36-2c1b-43bf-aec5-4e2def37ef97/volumes" Oct 30 12:34:14.489135 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:14.489099 4908 generic.go:356] "Generic (PLEG): container finished" podID="7c84680a-8b23-48bc-b048-bf3b3bcbe10f" containerID="da1e8c404a3b9dad86ffc2d42241694e2d9447ed9a58343cab19b55e1c918b34" exitCode=0 Oct 30 12:34:14.489508 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:14.489176 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-5ce7c4-hctqm" event={"ID":"7c84680a-8b23-48bc-b048-bf3b3bcbe10f","Type":"ContainerDied","Data":"da1e8c404a3b9dad86ffc2d42241694e2d9447ed9a58343cab19b55e1c918b34"} Oct 30 12:34:15.666316 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:15.666287 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-5ce7c4-hctqm" Oct 30 12:34:15.706306 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:15.706271 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c84680a-8b23-48bc-b048-bf3b3bcbe10f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c84680a-8b23-48bc-b048-bf3b3bcbe10f-maas-api-openshift-ca\") pod \"7c84680a-8b23-48bc-b048-bf3b3bcbe10f\" (UID: \"7c84680a-8b23-48bc-b048-bf3b3bcbe10f\") " Oct 30 12:34:15.706451 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:15.706382 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c84680a-8b23-48bc-b048-bf3b3bcbe10f-kube-api-access-rjf5x\" (UniqueName: \"kubernetes.io/projected/7c84680a-8b23-48bc-b048-bf3b3bcbe10f-kube-api-access-rjf5x\") pod \"7c84680a-8b23-48bc-b048-bf3b3bcbe10f\" (UID: \"7c84680a-8b23-48bc-b048-bf3b3bcbe10f\") " Oct 30 12:34:15.706518 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:15.706486 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c84680a-8b23-48bc-b048-bf3b3bcbe10f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c84680a-8b23-48bc-b048-bf3b3bcbe10f-maas-api-xks-ca\") pod \"7c84680a-8b23-48bc-b048-bf3b3bcbe10f\" (UID: \"7c84680a-8b23-48bc-b048-bf3b3bcbe10f\") " Oct 30 12:34:15.706634 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:15.706609 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c84680a-8b23-48bc-b048-bf3b3bcbe10f-maas-api-openshift-ca" pod "7c84680a-8b23-48bc-b048-bf3b3bcbe10f" (UID: "7c84680a-8b23-48bc-b048-bf3b3bcbe10f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:34:15.707000 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:15.706969 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c84680a-8b23-48bc-b048-bf3b3bcbe10f-maas-api-openshift-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:34:15.708565 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:15.708536 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c84680a-8b23-48bc-b048-bf3b3bcbe10f-maas-api-xks-ca" pod "7c84680a-8b23-48bc-b048-bf3b3bcbe10f" (UID: "7c84680a-8b23-48bc-b048-bf3b3bcbe10f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:34:15.708695 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:15.708674 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c84680a-8b23-48bc-b048-bf3b3bcbe10f-kube-api-access-rjf5x" pod "7c84680a-8b23-48bc-b048-bf3b3bcbe10f" (UID: "7c84680a-8b23-48bc-b048-bf3b3bcbe10f"). InnerVolumeSpecName "kube-api-access-rjf5x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:34:15.808605 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:15.808523 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rjf5x\" (UniqueName: \"kubernetes.io/projected/7c84680a-8b23-48bc-b048-bf3b3bcbe10f-kube-api-access-rjf5x\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:34:15.808605 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:15.808557 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c84680a-8b23-48bc-b048-bf3b3bcbe10f-maas-api-xks-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:34:15.823998 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:15.823967 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8faf1aa4-lr8fq"] Oct 30 12:34:15.824644 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:15.824624 4908 cpu_manager.go:414] "containerMap: removing container" podUID="fcefbe36-2c1b-43bf-aec5-4e2def37ef97" containerName="istio-proxy" Oct 30 12:34:15.824644 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:15.824643 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fcefbe36-2c1b-43bf-aec5-4e2def37ef97" containerName="istio-proxy" Oct 30 12:34:15.824792 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:15.824661 4908 cpu_manager.go:414] "containerMap: removing container" podUID="7c84680a-8b23-48bc-b048-bf3b3bcbe10f" containerName="revoke-keys" Oct 30 12:34:15.824792 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:15.824668 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c84680a-8b23-48bc-b048-bf3b3bcbe10f" containerName="revoke-keys" Oct 30 12:34:15.824792 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:15.824762 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c84680a-8b23-48bc-b048-bf3b3bcbe10f" containerName="revoke-keys" Oct 30 12:34:15.824792 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:15.824774 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="fcefbe36-2c1b-43bf-aec5-4e2def37ef97" containerName="istio-proxy" Oct 30 12:34:15.829219 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:15.829200 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8faf1aa4-lr8fq" Oct 30 12:34:15.846303 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:15.846278 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8faf1aa4-lr8fq"] Oct 30 12:34:15.909566 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:15.909534 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36caef27-f0ce-486f-ab9c-ef667e8b56dc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8faf1aa4-lr8fq\" (UID: \"36caef27-f0ce-486f-ab9c-ef667e8b56dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8faf1aa4-lr8fq" Oct 30 12:34:15.909710 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:15.909591 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36caef27-f0ce-486f-ab9c-ef667e8b56dc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8faf1aa4-lr8fq\" (UID: \"36caef27-f0ce-486f-ab9c-ef667e8b56dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8faf1aa4-lr8fq" Oct 30 12:34:15.909710 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:15.909658 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kxkk\" (UniqueName: \"kubernetes.io/projected/36caef27-f0ce-486f-ab9c-ef667e8b56dc-kube-api-access-6kxkk\") pod \"maas-api-revoke-sub-8faf1aa4-lr8fq\" (UID: \"36caef27-f0ce-486f-ab9c-ef667e8b56dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8faf1aa4-lr8fq" Oct 30 12:34:16.010874 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:16.010840 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36caef27-f0ce-486f-ab9c-ef667e8b56dc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8faf1aa4-lr8fq\" (UID: \"36caef27-f0ce-486f-ab9c-ef667e8b56dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8faf1aa4-lr8fq" Oct 30 12:34:16.011027 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:16.010891 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kxkk\" (UniqueName: \"kubernetes.io/projected/36caef27-f0ce-486f-ab9c-ef667e8b56dc-kube-api-access-6kxkk\") pod \"maas-api-revoke-sub-8faf1aa4-lr8fq\" (UID: \"36caef27-f0ce-486f-ab9c-ef667e8b56dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8faf1aa4-lr8fq" Oct 30 12:34:16.011072 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:16.011033 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36caef27-f0ce-486f-ab9c-ef667e8b56dc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8faf1aa4-lr8fq\" (UID: \"36caef27-f0ce-486f-ab9c-ef667e8b56dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8faf1aa4-lr8fq" Oct 30 12:34:16.011662 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:16.011635 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36caef27-f0ce-486f-ab9c-ef667e8b56dc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8faf1aa4-lr8fq\" (UID: \"36caef27-f0ce-486f-ab9c-ef667e8b56dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8faf1aa4-lr8fq" Oct 30 12:34:16.013164 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:16.013142 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36caef27-f0ce-486f-ab9c-ef667e8b56dc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8faf1aa4-lr8fq\" (UID: \"36caef27-f0ce-486f-ab9c-ef667e8b56dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8faf1aa4-lr8fq" Oct 30 12:34:16.056648 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:16.056609 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kxkk\" (UniqueName: \"kubernetes.io/projected/36caef27-f0ce-486f-ab9c-ef667e8b56dc-kube-api-access-6kxkk\") pod \"maas-api-revoke-sub-8faf1aa4-lr8fq\" (UID: \"36caef27-f0ce-486f-ab9c-ef667e8b56dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8faf1aa4-lr8fq" Oct 30 12:34:16.147247 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:16.147175 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8faf1aa4-lr8fq" Oct 30 12:34:16.355317 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:16.355281 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-34ce52bf-fxf5g"] Oct 30 12:34:16.360467 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:16.360442 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34ce52bf-fxf5g" Oct 30 12:34:16.397442 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:16.397369 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-34ce52bf-fxf5g"] Oct 30 12:34:16.415771 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:16.415740 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvpmv\" (UniqueName: \"kubernetes.io/projected/d84d4552-9bf6-427b-bd5b-78ee31a3f99d-kube-api-access-vvpmv\") pod \"maas-api-revoke-sub-34ce52bf-fxf5g\" (UID: \"d84d4552-9bf6-427b-bd5b-78ee31a3f99d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34ce52bf-fxf5g" Oct 30 12:34:16.415984 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:16.415965 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d84d4552-9bf6-427b-bd5b-78ee31a3f99d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-34ce52bf-fxf5g\" (UID: \"d84d4552-9bf6-427b-bd5b-78ee31a3f99d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34ce52bf-fxf5g" Oct 30 12:34:16.416079 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:16.416043 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d84d4552-9bf6-427b-bd5b-78ee31a3f99d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-34ce52bf-fxf5g\" (UID: \"d84d4552-9bf6-427b-bd5b-78ee31a3f99d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34ce52bf-fxf5g" Oct 30 12:34:16.497521 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:16.497489 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8faf1aa4-lr8fq"] Oct 30 12:34:16.500082 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:34:16.500052 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36caef27_f0ce_486f_ab9c_ef667e8b56dc.slice/crio-56607cde3e47750ab402ee4442c542b7dc5361ecbd4cb28384f97d7dccffc48a WatchSource:0}: Error finding container 56607cde3e47750ab402ee4442c542b7dc5361ecbd4cb28384f97d7dccffc48a: Status 404 returned error can't find the container with id 56607cde3e47750ab402ee4442c542b7dc5361ecbd4cb28384f97d7dccffc48a Oct 30 12:34:16.511192 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:16.511172 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-5ce7c4-hctqm" Oct 30 12:34:16.511282 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:16.511213 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-5ce7c4-hctqm" event={"ID":"7c84680a-8b23-48bc-b048-bf3b3bcbe10f","Type":"ContainerDied","Data":"5b94e2477652a13c96cbbfb89a989eece26914d580f46f3c52edb4ec49e902b9"} Oct 30 12:34:16.511282 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:16.511269 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5b94e2477652a13c96cbbfb89a989eece26914d580f46f3c52edb4ec49e902b9" Oct 30 12:34:16.516860 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:16.516818 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d84d4552-9bf6-427b-bd5b-78ee31a3f99d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-34ce52bf-fxf5g\" (UID: \"d84d4552-9bf6-427b-bd5b-78ee31a3f99d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34ce52bf-fxf5g" Oct 30 12:34:16.516982 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:16.516904 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d84d4552-9bf6-427b-bd5b-78ee31a3f99d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-34ce52bf-fxf5g\" (UID: \"d84d4552-9bf6-427b-bd5b-78ee31a3f99d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34ce52bf-fxf5g" Oct 30 12:34:16.516982 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:16.516951 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvpmv\" (UniqueName: \"kubernetes.io/projected/d84d4552-9bf6-427b-bd5b-78ee31a3f99d-kube-api-access-vvpmv\") pod \"maas-api-revoke-sub-34ce52bf-fxf5g\" (UID: \"d84d4552-9bf6-427b-bd5b-78ee31a3f99d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34ce52bf-fxf5g" Oct 30 12:34:16.517482 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:16.517462 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d84d4552-9bf6-427b-bd5b-78ee31a3f99d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-34ce52bf-fxf5g\" (UID: \"d84d4552-9bf6-427b-bd5b-78ee31a3f99d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34ce52bf-fxf5g" Oct 30 12:34:16.519717 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:16.519698 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d84d4552-9bf6-427b-bd5b-78ee31a3f99d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-34ce52bf-fxf5g\" (UID: \"d84d4552-9bf6-427b-bd5b-78ee31a3f99d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34ce52bf-fxf5g" Oct 30 12:34:16.550864 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:16.550846 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvpmv\" (UniqueName: \"kubernetes.io/projected/d84d4552-9bf6-427b-bd5b-78ee31a3f99d-kube-api-access-vvpmv\") pod \"maas-api-revoke-sub-34ce52bf-fxf5g\" (UID: \"d84d4552-9bf6-427b-bd5b-78ee31a3f99d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34ce52bf-fxf5g" Oct 30 12:34:16.676983 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:16.676176 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34ce52bf-fxf5g" Oct 30 12:34:17.158112 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:17.157405 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-34ce52bf-fxf5g"] Oct 30 12:34:17.792041 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:17.791994 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34ce52bf-fxf5g" event={"ID":"d84d4552-9bf6-427b-bd5b-78ee31a3f99d","Type":"ContainerStarted","Data":"dc5a75c5f0e6fedb3d86977d91cf37eccbe853322036c03ecc0092bff6d0f93f"} Oct 30 12:34:17.792524 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:17.792507 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8faf1aa4-lr8fq" event={"ID":"36caef27-f0ce-486f-ab9c-ef667e8b56dc","Type":"ContainerStarted","Data":"59e678f3dd2c69efaf7de237c0915a73554ee8f8dce52c612aa1f025b424326b"} Oct 30 12:34:17.792608 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:17.792598 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8faf1aa4-lr8fq" event={"ID":"36caef27-f0ce-486f-ab9c-ef667e8b56dc","Type":"ContainerStarted","Data":"56607cde3e47750ab402ee4442c542b7dc5361ecbd4cb28384f97d7dccffc48a"} Oct 30 12:34:17.840859 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:17.826016 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8faf1aa4-lr8fq" podStartSLOduration=2.825995437 podStartE2EDuration="2.825995437s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:34:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:34:17.824439173 +0000 UTC m=+2414.338867924" watchObservedRunningTime="2026-10-30 12:34:17.825995437 +0000 UTC m=+2414.340424185" Oct 30 12:34:18.626951 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:18.626900 4908 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6a7148/premium-w3-6a7148-kserve-67b976c888-tgd6c" podUID="13de4441-e7f6-4cbd-97d9-0f079d11cb8e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:34:18.666136 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:18.664412 4908 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f920f7/simulator-w0-f920f7-kserve-599b7ff45f-g79d7" podUID="a85e562f-d655-4c22-b4f4-a299a4e494a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:34:18.666136 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:18.664793 4908 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47fd99/premium-w2-47fd99-kserve-5d85f9bb95-hqkmf" podUID="fd2cbeda-dbc4-41f6-a730-e1c6f9343c5e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:34:18.984915 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:18.950079 4908 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6a7148/distinct-2-w3-6a7148-kserve-67bd5fb976-9nfff" podUID="6046cc02-29fe-4f59-b522-8f57f4557277" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:34:19.012919 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:19.012348 4908 generic.go:356] "Generic (PLEG): container finished" podID="36caef27-f0ce-486f-ab9c-ef667e8b56dc" containerID="59e678f3dd2c69efaf7de237c0915a73554ee8f8dce52c612aa1f025b424326b" exitCode=0 Oct 30 12:34:19.012919 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:19.012483 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8faf1aa4-lr8fq" event={"ID":"36caef27-f0ce-486f-ab9c-ef667e8b56dc","Type":"ContainerDied","Data":"59e678f3dd2c69efaf7de237c0915a73554ee8f8dce52c612aa1f025b424326b"} Oct 30 12:34:19.025470 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:19.025050 4908 generic.go:356] "Generic (PLEG): container finished" podID="d84d4552-9bf6-427b-bd5b-78ee31a3f99d" containerID="70a9909ffcbfff07b707e392236fb1c69b410da8c98ee9b99b74499e83b908da" exitCode=0 Oct 30 12:34:19.025470 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:19.025289 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34ce52bf-fxf5g" event={"ID":"d84d4552-9bf6-427b-bd5b-78ee31a3f99d","Type":"ContainerDied","Data":"70a9909ffcbfff07b707e392236fb1c69b410da8c98ee9b99b74499e83b908da"} Oct 30 12:34:19.046853 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:19.042079 4908 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-47fd99/premium-w2-47fd99-kserve-5d85f9bb95-hqkmf" podUID="fd2cbeda-dbc4-41f6-a730-e1c6f9343c5e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": context deadline exceeded" Oct 30 12:34:19.528852 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:19.504301 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0eee5157-k46dv"] Oct 30 12:34:19.528852 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:19.524083 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0eee5157-k46dv" Oct 30 12:34:19.723815 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:19.711977 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0eee5157-k46dv"] Oct 30 12:34:19.745418 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:19.744314 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d27421c-c8a1-41ef-85bd-9c18fd5d6aba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0eee5157-k46dv\" (UID: \"7d27421c-c8a1-41ef-85bd-9c18fd5d6aba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0eee5157-k46dv" Oct 30 12:34:19.745418 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:19.744370 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d27421c-c8a1-41ef-85bd-9c18fd5d6aba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0eee5157-k46dv\" (UID: \"7d27421c-c8a1-41ef-85bd-9c18fd5d6aba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0eee5157-k46dv" Oct 30 12:34:19.745418 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:19.744508 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppwxl\" (UniqueName: \"kubernetes.io/projected/7d27421c-c8a1-41ef-85bd-9c18fd5d6aba-kube-api-access-ppwxl\") pod \"maas-api-revoke-sub-0eee5157-k46dv\" (UID: \"7d27421c-c8a1-41ef-85bd-9c18fd5d6aba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0eee5157-k46dv" Oct 30 12:34:19.865893 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:19.857906 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d27421c-c8a1-41ef-85bd-9c18fd5d6aba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0eee5157-k46dv\" (UID: \"7d27421c-c8a1-41ef-85bd-9c18fd5d6aba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0eee5157-k46dv" Oct 30 12:34:19.865893 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:19.857963 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d27421c-c8a1-41ef-85bd-9c18fd5d6aba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0eee5157-k46dv\" (UID: \"7d27421c-c8a1-41ef-85bd-9c18fd5d6aba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0eee5157-k46dv" Oct 30 12:34:19.865893 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:19.858063 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppwxl\" (UniqueName: \"kubernetes.io/projected/7d27421c-c8a1-41ef-85bd-9c18fd5d6aba-kube-api-access-ppwxl\") pod \"maas-api-revoke-sub-0eee5157-k46dv\" (UID: \"7d27421c-c8a1-41ef-85bd-9c18fd5d6aba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0eee5157-k46dv" Oct 30 12:34:19.865893 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:19.859311 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d27421c-c8a1-41ef-85bd-9c18fd5d6aba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0eee5157-k46dv\" (UID: \"7d27421c-c8a1-41ef-85bd-9c18fd5d6aba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0eee5157-k46dv" Oct 30 12:34:19.957851 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:34:19.956998 4908 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vn6vv" containerName="extractor" Oct 30 12:34:19.959899 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:19.959632 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d27421c-c8a1-41ef-85bd-9c18fd5d6aba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0eee5157-k46dv\" (UID: \"7d27421c-c8a1-41ef-85bd-9c18fd5d6aba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0eee5157-k46dv" Oct 30 12:34:20.014665 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:20.013555 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppwxl\" (UniqueName: \"kubernetes.io/projected/7d27421c-c8a1-41ef-85bd-9c18fd5d6aba-kube-api-access-ppwxl\") pod \"maas-api-revoke-sub-0eee5157-k46dv\" (UID: \"7d27421c-c8a1-41ef-85bd-9c18fd5d6aba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0eee5157-k46dv" Oct 30 12:34:20.170960 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:20.170875 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0eee5157-k46dv" Oct 30 12:34:20.366325 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:20.366294 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34ce52bf-fxf5g" Oct 30 12:34:20.369593 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:20.369469 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8faf1aa4-lr8fq" Oct 30 12:34:20.476553 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:20.476520 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d84d4552-9bf6-427b-bd5b-78ee31a3f99d-kube-api-access-vvpmv\" (UniqueName: \"kubernetes.io/projected/d84d4552-9bf6-427b-bd5b-78ee31a3f99d-kube-api-access-vvpmv\") pod \"d84d4552-9bf6-427b-bd5b-78ee31a3f99d\" (UID: \"d84d4552-9bf6-427b-bd5b-78ee31a3f99d\") " Oct 30 12:34:20.476728 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:20.476616 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36caef27-f0ce-486f-ab9c-ef667e8b56dc-kube-api-access-6kxkk\" (UniqueName: \"kubernetes.io/projected/36caef27-f0ce-486f-ab9c-ef667e8b56dc-kube-api-access-6kxkk\") pod \"36caef27-f0ce-486f-ab9c-ef667e8b56dc\" (UID: \"36caef27-f0ce-486f-ab9c-ef667e8b56dc\") " Oct 30 12:34:20.476728 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:20.476706 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36caef27-f0ce-486f-ab9c-ef667e8b56dc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36caef27-f0ce-486f-ab9c-ef667e8b56dc-maas-api-openshift-ca\") pod \"36caef27-f0ce-486f-ab9c-ef667e8b56dc\" (UID: \"36caef27-f0ce-486f-ab9c-ef667e8b56dc\") " Oct 30 12:34:20.476876 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:20.476770 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d84d4552-9bf6-427b-bd5b-78ee31a3f99d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d84d4552-9bf6-427b-bd5b-78ee31a3f99d-maas-api-xks-ca\") pod \"d84d4552-9bf6-427b-bd5b-78ee31a3f99d\" (UID: \"d84d4552-9bf6-427b-bd5b-78ee31a3f99d\") " Oct 30 12:34:20.476944 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:20.476891 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d84d4552-9bf6-427b-bd5b-78ee31a3f99d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d84d4552-9bf6-427b-bd5b-78ee31a3f99d-maas-api-openshift-ca\") pod \"d84d4552-9bf6-427b-bd5b-78ee31a3f99d\" (UID: \"d84d4552-9bf6-427b-bd5b-78ee31a3f99d\") " Oct 30 12:34:20.477003 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:20.476985 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/36caef27-f0ce-486f-ab9c-ef667e8b56dc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36caef27-f0ce-486f-ab9c-ef667e8b56dc-maas-api-xks-ca\") pod \"36caef27-f0ce-486f-ab9c-ef667e8b56dc\" (UID: \"36caef27-f0ce-486f-ab9c-ef667e8b56dc\") " Oct 30 12:34:20.477332 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:20.477300 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36caef27-f0ce-486f-ab9c-ef667e8b56dc-maas-api-openshift-ca" pod "36caef27-f0ce-486f-ab9c-ef667e8b56dc" (UID: "36caef27-f0ce-486f-ab9c-ef667e8b56dc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:34:20.477671 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:20.477632 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36caef27-f0ce-486f-ab9c-ef667e8b56dc-maas-api-openshift-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:34:20.478351 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:20.478323 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d84d4552-9bf6-427b-bd5b-78ee31a3f99d-maas-api-openshift-ca" pod "d84d4552-9bf6-427b-bd5b-78ee31a3f99d" (UID: "d84d4552-9bf6-427b-bd5b-78ee31a3f99d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:34:20.480249 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:20.480207 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d84d4552-9bf6-427b-bd5b-78ee31a3f99d-maas-api-xks-ca" pod "d84d4552-9bf6-427b-bd5b-78ee31a3f99d" (UID: "d84d4552-9bf6-427b-bd5b-78ee31a3f99d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:34:20.480686 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:20.480664 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36caef27-f0ce-486f-ab9c-ef667e8b56dc-kube-api-access-6kxkk" pod "36caef27-f0ce-486f-ab9c-ef667e8b56dc" (UID: "36caef27-f0ce-486f-ab9c-ef667e8b56dc"). InnerVolumeSpecName "kube-api-access-6kxkk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:34:20.480893 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:20.480873 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d84d4552-9bf6-427b-bd5b-78ee31a3f99d-kube-api-access-vvpmv" pod "d84d4552-9bf6-427b-bd5b-78ee31a3f99d" (UID: "d84d4552-9bf6-427b-bd5b-78ee31a3f99d"). InnerVolumeSpecName "kube-api-access-vvpmv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:34:20.481010 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:20.480993 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36caef27-f0ce-486f-ab9c-ef667e8b56dc-maas-api-xks-ca" pod "36caef27-f0ce-486f-ab9c-ef667e8b56dc" (UID: "36caef27-f0ce-486f-ab9c-ef667e8b56dc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:34:20.539233 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:20.539204 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0eee5157-k46dv"] Oct 30 12:34:20.540678 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:34:20.540652 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d27421c_c8a1_41ef_85bd_9c18fd5d6aba.slice/crio-096f6d49f8cc67df281de8224fd213efb4764ede31940d6c571432d5306c681a WatchSource:0}: Error finding container 096f6d49f8cc67df281de8224fd213efb4764ede31940d6c571432d5306c681a: Status 404 returned error can't find the container with id 096f6d49f8cc67df281de8224fd213efb4764ede31940d6c571432d5306c681a Oct 30 12:34:20.580700 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:20.580671 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6kxkk\" (UniqueName: \"kubernetes.io/projected/36caef27-f0ce-486f-ab9c-ef667e8b56dc-kube-api-access-6kxkk\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:34:20.580887 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:20.580704 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d84d4552-9bf6-427b-bd5b-78ee31a3f99d-maas-api-xks-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:34:20.580887 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:20.580722 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d84d4552-9bf6-427b-bd5b-78ee31a3f99d-maas-api-openshift-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:34:20.580887 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:20.580735 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36caef27-f0ce-486f-ab9c-ef667e8b56dc-maas-api-xks-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:34:20.580887 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:20.580748 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvpmv\" (UniqueName: \"kubernetes.io/projected/d84d4552-9bf6-427b-bd5b-78ee31a3f99d-kube-api-access-vvpmv\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:34:20.638808 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:20.638779 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-a124ab-xlkkq"] Oct 30 12:34:20.644389 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:20.644362 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-a124ab-xlkkq"] Oct 30 12:34:21.080476 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:21.080390 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8faf1aa4-lr8fq" event={"ID":"36caef27-f0ce-486f-ab9c-ef667e8b56dc","Type":"ContainerDied","Data":"56607cde3e47750ab402ee4442c542b7dc5361ecbd4cb28384f97d7dccffc48a"} Oct 30 12:34:21.080476 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:21.080444 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="56607cde3e47750ab402ee4442c542b7dc5361ecbd4cb28384f97d7dccffc48a" Oct 30 12:34:21.080476 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:21.080402 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8faf1aa4-lr8fq" Oct 30 12:34:21.082331 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:21.082300 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0eee5157-k46dv" event={"ID":"7d27421c-c8a1-41ef-85bd-9c18fd5d6aba","Type":"ContainerStarted","Data":"83e1584662a5ce905bf88e8c81ec8c3490697b4bf544fde13f9b1c2c697329e7"} Oct 30 12:34:21.082467 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:21.082353 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0eee5157-k46dv" event={"ID":"7d27421c-c8a1-41ef-85bd-9c18fd5d6aba","Type":"ContainerStarted","Data":"096f6d49f8cc67df281de8224fd213efb4764ede31940d6c571432d5306c681a"} Oct 30 12:34:21.084606 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:21.084561 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34ce52bf-fxf5g" event={"ID":"d84d4552-9bf6-427b-bd5b-78ee31a3f99d","Type":"ContainerDied","Data":"dc5a75c5f0e6fedb3d86977d91cf37eccbe853322036c03ecc0092bff6d0f93f"} Oct 30 12:34:21.084724 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:21.084626 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dc5a75c5f0e6fedb3d86977d91cf37eccbe853322036c03ecc0092bff6d0f93f" Oct 30 12:34:21.084724 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:21.084662 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34ce52bf-fxf5g" Oct 30 12:34:21.121548 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:21.121511 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0eee5157-k46dv" podStartSLOduration=2.121498608 podStartE2EDuration="2.121498608s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:34:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:34:21.119397729 +0000 UTC m=+2417.633826484" watchObservedRunningTime="2026-10-30 12:34:21.121498608 +0000 UTC m=+2417.635927355" Oct 30 12:34:21.607436 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:21.607402 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ee09d53-9417-4b5c-b787-00cdb0e56e5d" path="/var/lib/kubelet/pods/9ee09d53-9417-4b5c-b787-00cdb0e56e5d/volumes" Oct 30 12:34:22.096361 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:22.096323 4908 generic.go:356] "Generic (PLEG): container finished" podID="7d27421c-c8a1-41ef-85bd-9c18fd5d6aba" containerID="83e1584662a5ce905bf88e8c81ec8c3490697b4bf544fde13f9b1c2c697329e7" exitCode=0 Oct 30 12:34:22.096893 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:22.096410 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0eee5157-k46dv" event={"ID":"7d27421c-c8a1-41ef-85bd-9c18fd5d6aba","Type":"ContainerDied","Data":"83e1584662a5ce905bf88e8c81ec8c3490697b4bf544fde13f9b1c2c697329e7"} Oct 30 12:34:23.279221 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:23.279192 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0eee5157-k46dv" Oct 30 12:34:23.343126 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:23.343096 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d27421c-c8a1-41ef-85bd-9c18fd5d6aba-kube-api-access-ppwxl\" (UniqueName: \"kubernetes.io/projected/7d27421c-c8a1-41ef-85bd-9c18fd5d6aba-kube-api-access-ppwxl\") pod \"7d27421c-c8a1-41ef-85bd-9c18fd5d6aba\" (UID: \"7d27421c-c8a1-41ef-85bd-9c18fd5d6aba\") " Oct 30 12:34:23.343293 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:23.343184 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d27421c-c8a1-41ef-85bd-9c18fd5d6aba-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d27421c-c8a1-41ef-85bd-9c18fd5d6aba-maas-api-xks-ca\") pod \"7d27421c-c8a1-41ef-85bd-9c18fd5d6aba\" (UID: \"7d27421c-c8a1-41ef-85bd-9c18fd5d6aba\") " Oct 30 12:34:23.343293 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:23.343285 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d27421c-c8a1-41ef-85bd-9c18fd5d6aba-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d27421c-c8a1-41ef-85bd-9c18fd5d6aba-maas-api-openshift-ca\") pod \"7d27421c-c8a1-41ef-85bd-9c18fd5d6aba\" (UID: \"7d27421c-c8a1-41ef-85bd-9c18fd5d6aba\") " Oct 30 12:34:23.343663 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:23.343634 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d27421c-c8a1-41ef-85bd-9c18fd5d6aba-maas-api-openshift-ca" pod "7d27421c-c8a1-41ef-85bd-9c18fd5d6aba" (UID: "7d27421c-c8a1-41ef-85bd-9c18fd5d6aba"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:34:23.343882 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:23.343864 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d27421c-c8a1-41ef-85bd-9c18fd5d6aba-maas-api-openshift-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:34:23.345183 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:23.345160 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d27421c-c8a1-41ef-85bd-9c18fd5d6aba-maas-api-xks-ca" pod "7d27421c-c8a1-41ef-85bd-9c18fd5d6aba" (UID: "7d27421c-c8a1-41ef-85bd-9c18fd5d6aba"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:34:23.345609 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:23.345590 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d27421c-c8a1-41ef-85bd-9c18fd5d6aba-kube-api-access-ppwxl" pod "7d27421c-c8a1-41ef-85bd-9c18fd5d6aba" (UID: "7d27421c-c8a1-41ef-85bd-9c18fd5d6aba"). InnerVolumeSpecName "kube-api-access-ppwxl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:34:23.444877 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:23.444850 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ppwxl\" (UniqueName: \"kubernetes.io/projected/7d27421c-c8a1-41ef-85bd-9c18fd5d6aba-kube-api-access-ppwxl\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:34:23.444877 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:23.444875 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d27421c-c8a1-41ef-85bd-9c18fd5d6aba-maas-api-xks-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:34:24.122022 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:24.121995 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0eee5157-k46dv" Oct 30 12:34:24.122204 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:24.121985 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0eee5157-k46dv" event={"ID":"7d27421c-c8a1-41ef-85bd-9c18fd5d6aba","Type":"ContainerDied","Data":"096f6d49f8cc67df281de8224fd213efb4764ede31940d6c571432d5306c681a"} Oct 30 12:34:24.122204 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:24.122149 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="096f6d49f8cc67df281de8224fd213efb4764ede31940d6c571432d5306c681a" Oct 30 12:34:28.934583 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:28.934543 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cc08f1-p577p"] Oct 30 12:34:28.935677 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:28.935646 4908 cpu_manager.go:414] "containerMap: removing container" podUID="d84d4552-9bf6-427b-bd5b-78ee31a3f99d" containerName="revoke-keys" Oct 30 12:34:28.935677 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:28.935669 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d84d4552-9bf6-427b-bd5b-78ee31a3f99d" containerName="revoke-keys" Oct 30 12:34:28.935677 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:28.935682 4908 cpu_manager.go:414] "containerMap: removing container" podUID="36caef27-f0ce-486f-ab9c-ef667e8b56dc" containerName="revoke-keys" Oct 30 12:34:28.935966 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:28.935689 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36caef27-f0ce-486f-ab9c-ef667e8b56dc" containerName="revoke-keys" Oct 30 12:34:28.935966 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:28.935720 4908 cpu_manager.go:414] "containerMap: removing container" podUID="7d27421c-c8a1-41ef-85bd-9c18fd5d6aba" containerName="revoke-keys" Oct 30 12:34:28.935966 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:28.935728 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d27421c-c8a1-41ef-85bd-9c18fd5d6aba" containerName="revoke-keys" Oct 30 12:34:28.935966 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:28.935900 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="d84d4552-9bf6-427b-bd5b-78ee31a3f99d" containerName="revoke-keys" Oct 30 12:34:28.935966 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:28.935919 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="36caef27-f0ce-486f-ab9c-ef667e8b56dc" containerName="revoke-keys" Oct 30 12:34:28.935966 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:28.935931 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d27421c-c8a1-41ef-85bd-9c18fd5d6aba" containerName="revoke-keys" Oct 30 12:34:28.941876 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:28.941858 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cc08f1-p577p" Oct 30 12:34:28.958803 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:28.958775 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cc08f1-p577p"] Oct 30 12:34:28.962923 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:28.962876 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 12:34:28.963067 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:28.962991 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 12:34:28.963632 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:28.963613 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 12:34:28.968594 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:28.968558 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s75hq\"" Oct 30 12:34:29.035314 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:29.035287 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7hfl\" (UniqueName: \"kubernetes.io/projected/9f095c6c-697c-4b5f-b8ad-7c145b4759c1-kube-api-access-f7hfl\") pod \"maas-api-revoke-keys-e2e-praxis-b-cc08f1-p577p\" (UID: \"9f095c6c-697c-4b5f-b8ad-7c145b4759c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cc08f1-p577p" Oct 30 12:34:29.035491 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:29.035345 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f095c6c-697c-4b5f-b8ad-7c145b4759c1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-cc08f1-p577p\" (UID: \"9f095c6c-697c-4b5f-b8ad-7c145b4759c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cc08f1-p577p" Oct 30 12:34:29.035491 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:29.035462 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f095c6c-697c-4b5f-b8ad-7c145b4759c1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-cc08f1-p577p\" (UID: \"9f095c6c-697c-4b5f-b8ad-7c145b4759c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cc08f1-p577p" Oct 30 12:34:29.136821 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:29.136782 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7hfl\" (UniqueName: \"kubernetes.io/projected/9f095c6c-697c-4b5f-b8ad-7c145b4759c1-kube-api-access-f7hfl\") pod \"maas-api-revoke-keys-e2e-praxis-b-cc08f1-p577p\" (UID: \"9f095c6c-697c-4b5f-b8ad-7c145b4759c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cc08f1-p577p" Oct 30 12:34:29.137022 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:29.136881 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f095c6c-697c-4b5f-b8ad-7c145b4759c1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-cc08f1-p577p\" (UID: \"9f095c6c-697c-4b5f-b8ad-7c145b4759c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cc08f1-p577p" Oct 30 12:34:29.137022 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:29.136951 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f095c6c-697c-4b5f-b8ad-7c145b4759c1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-cc08f1-p577p\" (UID: \"9f095c6c-697c-4b5f-b8ad-7c145b4759c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cc08f1-p577p" Oct 30 12:34:29.137538 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:29.137516 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f095c6c-697c-4b5f-b8ad-7c145b4759c1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-cc08f1-p577p\" (UID: \"9f095c6c-697c-4b5f-b8ad-7c145b4759c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cc08f1-p577p" Oct 30 12:34:29.139349 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:29.139325 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f095c6c-697c-4b5f-b8ad-7c145b4759c1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-cc08f1-p577p\" (UID: \"9f095c6c-697c-4b5f-b8ad-7c145b4759c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cc08f1-p577p" Oct 30 12:34:29.167722 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:29.167689 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7hfl\" (UniqueName: \"kubernetes.io/projected/9f095c6c-697c-4b5f-b8ad-7c145b4759c1-kube-api-access-f7hfl\") pod \"maas-api-revoke-keys-e2e-praxis-b-cc08f1-p577p\" (UID: \"9f095c6c-697c-4b5f-b8ad-7c145b4759c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cc08f1-p577p" Oct 30 12:34:29.264099 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:29.264053 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cc08f1-p577p" Oct 30 12:34:29.736745 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:29.736715 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cc08f1-p577p"] Oct 30 12:34:29.738695 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:34:29.738609 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f095c6c_697c_4b5f_b8ad_7c145b4759c1.slice/crio-548e9ee769b4c25afa21972dbd0a29ecf9be222a11c2539cb2e7b960378590b3 WatchSource:0}: Error finding container 548e9ee769b4c25afa21972dbd0a29ecf9be222a11c2539cb2e7b960378590b3: Status 404 returned error can't find the container with id 548e9ee769b4c25afa21972dbd0a29ecf9be222a11c2539cb2e7b960378590b3 Oct 30 12:34:30.209615 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:30.209579 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cc08f1-p577p" event={"ID":"9f095c6c-697c-4b5f-b8ad-7c145b4759c1","Type":"ContainerStarted","Data":"8e78d324ec999e4eab99acf45aa7f25e8679f7ba42377aac33ffff04ca120681"} Oct 30 12:34:30.210059 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:30.209645 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cc08f1-p577p" event={"ID":"9f095c6c-697c-4b5f-b8ad-7c145b4759c1","Type":"ContainerStarted","Data":"548e9ee769b4c25afa21972dbd0a29ecf9be222a11c2539cb2e7b960378590b3"} Oct 30 12:34:30.251291 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:30.251241 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cc08f1-p577p" podStartSLOduration=2.251228868 podStartE2EDuration="2.251228868s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:34:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:34:30.246578838 +0000 UTC m=+2426.761007576" watchObservedRunningTime="2026-10-30 12:34:30.251228868 +0000 UTC m=+2426.765657607" Oct 30 12:34:31.223044 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:31.223012 4908 generic.go:356] "Generic (PLEG): container finished" podID="9f095c6c-697c-4b5f-b8ad-7c145b4759c1" containerID="8e78d324ec999e4eab99acf45aa7f25e8679f7ba42377aac33ffff04ca120681" exitCode=0 Oct 30 12:34:31.223415 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:31.223088 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cc08f1-p577p" event={"ID":"9f095c6c-697c-4b5f-b8ad-7c145b4759c1","Type":"ContainerDied","Data":"8e78d324ec999e4eab99acf45aa7f25e8679f7ba42377aac33ffff04ca120681"} Oct 30 12:34:32.374757 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:32.374728 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cc08f1-p577p" Oct 30 12:34:32.490607 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:32.490531 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f095c6c-697c-4b5f-b8ad-7c145b4759c1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f095c6c-697c-4b5f-b8ad-7c145b4759c1-maas-api-xks-ca\") pod \"9f095c6c-697c-4b5f-b8ad-7c145b4759c1\" (UID: \"9f095c6c-697c-4b5f-b8ad-7c145b4759c1\") " Oct 30 12:34:32.490754 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:32.490677 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f095c6c-697c-4b5f-b8ad-7c145b4759c1-kube-api-access-f7hfl\" (UniqueName: \"kubernetes.io/projected/9f095c6c-697c-4b5f-b8ad-7c145b4759c1-kube-api-access-f7hfl\") pod \"9f095c6c-697c-4b5f-b8ad-7c145b4759c1\" (UID: \"9f095c6c-697c-4b5f-b8ad-7c145b4759c1\") " Oct 30 12:34:32.490754 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:32.490727 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f095c6c-697c-4b5f-b8ad-7c145b4759c1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f095c6c-697c-4b5f-b8ad-7c145b4759c1-maas-api-openshift-ca\") pod \"9f095c6c-697c-4b5f-b8ad-7c145b4759c1\" (UID: \"9f095c6c-697c-4b5f-b8ad-7c145b4759c1\") " Oct 30 12:34:32.491199 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:32.491170 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f095c6c-697c-4b5f-b8ad-7c145b4759c1-maas-api-openshift-ca" pod "9f095c6c-697c-4b5f-b8ad-7c145b4759c1" (UID: "9f095c6c-697c-4b5f-b8ad-7c145b4759c1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:34:32.493165 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:32.493141 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f095c6c-697c-4b5f-b8ad-7c145b4759c1-maas-api-xks-ca" pod "9f095c6c-697c-4b5f-b8ad-7c145b4759c1" (UID: "9f095c6c-697c-4b5f-b8ad-7c145b4759c1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:34:32.493276 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:32.493194 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f095c6c-697c-4b5f-b8ad-7c145b4759c1-kube-api-access-f7hfl" pod "9f095c6c-697c-4b5f-b8ad-7c145b4759c1" (UID: "9f095c6c-697c-4b5f-b8ad-7c145b4759c1"). InnerVolumeSpecName "kube-api-access-f7hfl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:34:32.591717 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:32.591689 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f095c6c-697c-4b5f-b8ad-7c145b4759c1-maas-api-xks-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:34:32.591717 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:32.591717 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7hfl\" (UniqueName: \"kubernetes.io/projected/9f095c6c-697c-4b5f-b8ad-7c145b4759c1-kube-api-access-f7hfl\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:34:32.591969 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:32.591727 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f095c6c-697c-4b5f-b8ad-7c145b4759c1-maas-api-openshift-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:34:33.243322 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:33.243295 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cc08f1-p577p" Oct 30 12:34:33.243477 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:33.243328 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cc08f1-p577p" event={"ID":"9f095c6c-697c-4b5f-b8ad-7c145b4759c1","Type":"ContainerDied","Data":"548e9ee769b4c25afa21972dbd0a29ecf9be222a11c2539cb2e7b960378590b3"} Oct 30 12:34:33.243477 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:33.243383 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="548e9ee769b4c25afa21972dbd0a29ecf9be222a11c2539cb2e7b960378590b3" Oct 30 12:34:34.415517 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:34.415464 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9wwfc"] Oct 30 12:34:36.423926 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:36.423882 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-5ce7c4-hctqm"] Oct 30 12:34:36.439796 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:36.439764 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-5ce7c4-hctqm"] Oct 30 12:34:37.553694 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:37.553647 4908 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47fd99/distinct-2-w2-47fd99-kserve-77cf84b469-zxs8s" podUID="baf44d3d-42e6-43f7-a0ad-f0f9932320c1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 30 12:34:37.617950 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:37.617908 4908 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47fd99/premium-w2-47fd99-kserve-5d85f9bb95-hqkmf" podUID="fd2cbeda-dbc4-41f6-a730-e1c6f9343c5e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Oct 30 12:34:37.627495 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:37.627452 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c84680a-8b23-48bc-b048-bf3b3bcbe10f" path="/var/lib/kubelet/pods/7c84680a-8b23-48bc-b048-bf3b3bcbe10f/volumes" Oct 30 12:34:37.634086 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:37.634056 4908 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6a7148/premium-w3-6a7148-kserve-67b976c888-tgd6c" podUID="13de4441-e7f6-4cbd-97d9-0f079d11cb8e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 12:34:38.016610 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:38.015914 4908 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6a7148/distinct-2-w3-6a7148-kserve-67bd5fb976-9nfff" podUID="6046cc02-29fe-4f59-b522-8f57f4557277" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:34:38.203877 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:38.199920 4908 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-512ec5/simulator-w4-512ec5-kserve-7df7674f8-8rqwb" podUID="8a5dfcc9-dec3-4bc6-b0a8-c928350b3596" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 12:34:38.625191 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:38.625128 4908 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47fd99/premium-w2-47fd99-kserve-5d85f9bb95-hqkmf" podUID="fd2cbeda-dbc4-41f6-a730-e1c6f9343c5e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:34:38.628868 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:38.626339 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-47fd99/premium-w2-47fd99-kserve-5d85f9bb95-hqkmf" Oct 30 12:34:38.809492 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:38.809460 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-47fd99/premium-w2-47fd99-kserve-5d85f9bb95-hqkmf" Oct 30 12:34:39.160353 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:39.160317 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-sqv2j"] Oct 30 12:34:39.160867 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:39.160812 4908 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-sqv2j" podUID="8f110318-3125-4a76-ba6e-b9b0e6861051" containerName="istio-proxy" containerID="cri-o://c090aea8285182d5fe43510bf0bb33c033249b4dc80999f1896128a59d9c7615" gracePeriod=30 Oct 30 12:34:44.444318 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:44.444263 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-sqv2j" Oct 30 12:34:44.506852 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:44.506739 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f110318-3125-4a76-ba6e-b9b0e6861051-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f110318-3125-4a76-ba6e-b9b0e6861051-istio-envoy\") pod \"8f110318-3125-4a76-ba6e-b9b0e6861051\" (UID: \"8f110318-3125-4a76-ba6e-b9b0e6861051\") " Oct 30 12:34:44.507226 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:44.507081 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f110318-3125-4a76-ba6e-b9b0e6861051-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f110318-3125-4a76-ba6e-b9b0e6861051-workload-certs\") pod \"8f110318-3125-4a76-ba6e-b9b0e6861051\" (UID: \"8f110318-3125-4a76-ba6e-b9b0e6861051\") " Oct 30 12:34:44.507226 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:44.507133 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f110318-3125-4a76-ba6e-b9b0e6861051-istio-token\" (UniqueName: \"kubernetes.io/projected/8f110318-3125-4a76-ba6e-b9b0e6861051-istio-token\") pod \"8f110318-3125-4a76-ba6e-b9b0e6861051\" (UID: \"8f110318-3125-4a76-ba6e-b9b0e6861051\") " Oct 30 12:34:44.507226 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:44.507165 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f110318-3125-4a76-ba6e-b9b0e6861051-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f110318-3125-4a76-ba6e-b9b0e6861051-workload-socket\") pod \"8f110318-3125-4a76-ba6e-b9b0e6861051\" (UID: \"8f110318-3125-4a76-ba6e-b9b0e6861051\") " Oct 30 12:34:44.507468 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:44.507441 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f110318-3125-4a76-ba6e-b9b0e6861051-kube-api-access-7qhjn\" (UniqueName: \"kubernetes.io/projected/8f110318-3125-4a76-ba6e-b9b0e6861051-kube-api-access-7qhjn\") pod \"8f110318-3125-4a76-ba6e-b9b0e6861051\" (UID: \"8f110318-3125-4a76-ba6e-b9b0e6861051\") " Oct 30 12:34:44.507604 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:44.507541 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8f110318-3125-4a76-ba6e-b9b0e6861051-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f110318-3125-4a76-ba6e-b9b0e6861051-istio-podinfo\") pod \"8f110318-3125-4a76-ba6e-b9b0e6861051\" (UID: \"8f110318-3125-4a76-ba6e-b9b0e6861051\") " Oct 30 12:34:44.507604 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:44.507593 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f110318-3125-4a76-ba6e-b9b0e6861051-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f110318-3125-4a76-ba6e-b9b0e6861051-credential-socket\") pod \"8f110318-3125-4a76-ba6e-b9b0e6861051\" (UID: \"8f110318-3125-4a76-ba6e-b9b0e6861051\") " Oct 30 12:34:44.507721 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:44.507660 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f110318-3125-4a76-ba6e-b9b0e6861051-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f110318-3125-4a76-ba6e-b9b0e6861051-istiod-ca-cert\") pod \"8f110318-3125-4a76-ba6e-b9b0e6861051\" (UID: \"8f110318-3125-4a76-ba6e-b9b0e6861051\") " Oct 30 12:34:44.507721 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:44.507710 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f110318-3125-4a76-ba6e-b9b0e6861051-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f110318-3125-4a76-ba6e-b9b0e6861051-istio-data\") pod \"8f110318-3125-4a76-ba6e-b9b0e6861051\" (UID: \"8f110318-3125-4a76-ba6e-b9b0e6861051\") " Oct 30 12:34:44.510842 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:44.509096 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f110318-3125-4a76-ba6e-b9b0e6861051-istio-data" pod "8f110318-3125-4a76-ba6e-b9b0e6861051" (UID: "8f110318-3125-4a76-ba6e-b9b0e6861051"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:34:44.511350 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:44.511323 4908 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f110318-3125-4a76-ba6e-b9b0e6861051-istio-data\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:34:44.512156 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:44.512126 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f110318-3125-4a76-ba6e-b9b0e6861051-workload-certs" pod "8f110318-3125-4a76-ba6e-b9b0e6861051" (UID: "8f110318-3125-4a76-ba6e-b9b0e6861051"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:34:44.513486 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:44.513231 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f110318-3125-4a76-ba6e-b9b0e6861051-workload-socket" pod "8f110318-3125-4a76-ba6e-b9b0e6861051" (UID: "8f110318-3125-4a76-ba6e-b9b0e6861051"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:34:44.514430 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:44.514163 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f110318-3125-4a76-ba6e-b9b0e6861051-credential-socket" pod "8f110318-3125-4a76-ba6e-b9b0e6861051" (UID: "8f110318-3125-4a76-ba6e-b9b0e6861051"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:34:44.514746 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:44.514721 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f110318-3125-4a76-ba6e-b9b0e6861051-istiod-ca-cert" pod "8f110318-3125-4a76-ba6e-b9b0e6861051" (UID: "8f110318-3125-4a76-ba6e-b9b0e6861051"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:34:44.516919 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:44.516884 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f110318-3125-4a76-ba6e-b9b0e6861051-kube-api-access-7qhjn" pod "8f110318-3125-4a76-ba6e-b9b0e6861051" (UID: "8f110318-3125-4a76-ba6e-b9b0e6861051"). InnerVolumeSpecName "kube-api-access-7qhjn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:34:44.518860 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:44.518819 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8f110318-3125-4a76-ba6e-b9b0e6861051-istio-podinfo" pod "8f110318-3125-4a76-ba6e-b9b0e6861051" (UID: "8f110318-3125-4a76-ba6e-b9b0e6861051"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 12:34:44.518973 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:44.518954 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f110318-3125-4a76-ba6e-b9b0e6861051-istio-token" pod "8f110318-3125-4a76-ba6e-b9b0e6861051" (UID: "8f110318-3125-4a76-ba6e-b9b0e6861051"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:34:44.519743 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:44.519718 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f110318-3125-4a76-ba6e-b9b0e6861051-istio-envoy" pod "8f110318-3125-4a76-ba6e-b9b0e6861051" (UID: "8f110318-3125-4a76-ba6e-b9b0e6861051"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:34:44.613223 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:44.613192 4908 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f110318-3125-4a76-ba6e-b9b0e6861051-istio-envoy\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:34:44.613223 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:44.613221 4908 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f110318-3125-4a76-ba6e-b9b0e6861051-workload-certs\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:34:44.613223 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:44.613230 4908 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f110318-3125-4a76-ba6e-b9b0e6861051-istio-token\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:34:44.613566 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:44.613239 4908 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f110318-3125-4a76-ba6e-b9b0e6861051-workload-socket\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:34:44.613566 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:44.613248 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7qhjn\" (UniqueName: \"kubernetes.io/projected/8f110318-3125-4a76-ba6e-b9b0e6861051-kube-api-access-7qhjn\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:34:44.613566 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:44.613258 4908 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f110318-3125-4a76-ba6e-b9b0e6861051-istio-podinfo\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:34:44.613566 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:44.613268 4908 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f110318-3125-4a76-ba6e-b9b0e6861051-credential-socket\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:34:44.613566 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:44.613277 4908 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f110318-3125-4a76-ba6e-b9b0e6861051-istiod-ca-cert\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:34:44.705045 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:44.704949 4908 generic.go:356] "Generic (PLEG): container finished" podID="8f110318-3125-4a76-ba6e-b9b0e6861051" containerID="c090aea8285182d5fe43510bf0bb33c033249b4dc80999f1896128a59d9c7615" exitCode=0 Oct 30 12:34:44.705045 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:44.705002 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-sqv2j" Oct 30 12:34:44.705045 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:44.705027 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-sqv2j" event={"ID":"8f110318-3125-4a76-ba6e-b9b0e6861051","Type":"ContainerDied","Data":"c090aea8285182d5fe43510bf0bb33c033249b4dc80999f1896128a59d9c7615"} Oct 30 12:34:44.705315 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:44.705084 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-sqv2j" event={"ID":"8f110318-3125-4a76-ba6e-b9b0e6861051","Type":"ContainerDied","Data":"706623eddc8ecad79b4a616106169db48c0729848a9b86584db6e8398dbe766e"} Oct 30 12:34:44.705315 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:44.705106 4908 scope.go:122] "RemoveContainer" containerID="c090aea8285182d5fe43510bf0bb33c033249b4dc80999f1896128a59d9c7615" Oct 30 12:34:44.716675 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:44.716660 4908 scope.go:122] "RemoveContainer" containerID="c090aea8285182d5fe43510bf0bb33c033249b4dc80999f1896128a59d9c7615" Oct 30 12:34:44.716966 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:34:44.716948 4908 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c090aea8285182d5fe43510bf0bb33c033249b4dc80999f1896128a59d9c7615\": container with ID starting with c090aea8285182d5fe43510bf0bb33c033249b4dc80999f1896128a59d9c7615 not found: ID does not exist" containerID="c090aea8285182d5fe43510bf0bb33c033249b4dc80999f1896128a59d9c7615" Oct 30 12:34:44.717047 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:44.716975 4908 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c090aea8285182d5fe43510bf0bb33c033249b4dc80999f1896128a59d9c7615"} err="failed to get container status \"c090aea8285182d5fe43510bf0bb33c033249b4dc80999f1896128a59d9c7615\": rpc error: code = NotFound desc = could not find container \"c090aea8285182d5fe43510bf0bb33c033249b4dc80999f1896128a59d9c7615\": container with ID starting with c090aea8285182d5fe43510bf0bb33c033249b4dc80999f1896128a59d9c7615 not found: ID does not exist" Oct 30 12:34:44.756156 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:44.756134 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-sqv2j"] Oct 30 12:34:44.760155 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:44.760135 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-sqv2j"] Oct 30 12:34:45.609696 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:45.609658 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f110318-3125-4a76-ba6e-b9b0e6861051" path="/var/lib/kubelet/pods/8f110318-3125-4a76-ba6e-b9b0e6861051/volumes" Oct 30 12:34:49.685911 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:34:49.685806 4908 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vn6vv" containerName="extractor" Oct 30 12:34:59.049128 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:59.049087 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cc08f1-p577p"] Oct 30 12:34:59.059582 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:59.059548 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cc08f1-p577p"] Oct 30 12:34:59.605173 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:59.605142 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f095c6c-697c-4b5f-b8ad-7c145b4759c1" path="/var/lib/kubelet/pods/9f095c6c-697c-4b5f-b8ad-7c145b4759c1/volumes" Oct 30 12:34:59.723850 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:34:59.717368 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9wwfc"] Oct 30 12:35:06.203044 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:35:06.203001 4908 scope.go:122] "RemoveContainer" containerID="55ce4b63e49998770c3873f059cf32fd6309f5b9925105333d0c8a92306fac87" Oct 30 12:35:06.213963 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:35:06.213942 4908 scope.go:122] "RemoveContainer" containerID="707421c0567db4bb146230e62929c3806e97fdfce9a03f1fed1dd9cdb867f52c" Oct 30 12:35:06.223378 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:35:06.223357 4908 scope.go:122] "RemoveContainer" containerID="82f91830fc682e19f2aa68ab350ae2d4c1f778741183968de927afd0517578bc" Oct 30 12:35:12.799057 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:35:12.799023 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bvpc2"] Oct 30 12:35:12.800100 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:35:12.800076 4908 cpu_manager.go:414] "containerMap: removing container" podUID="8f110318-3125-4a76-ba6e-b9b0e6861051" containerName="istio-proxy" Oct 30 12:35:12.800100 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:35:12.800100 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f110318-3125-4a76-ba6e-b9b0e6861051" containerName="istio-proxy" Oct 30 12:35:12.800297 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:35:12.800186 4908 cpu_manager.go:414] "containerMap: removing container" podUID="9f095c6c-697c-4b5f-b8ad-7c145b4759c1" containerName="revoke-keys" Oct 30 12:35:12.800297 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:35:12.800196 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f095c6c-697c-4b5f-b8ad-7c145b4759c1" containerName="revoke-keys" Oct 30 12:35:12.800469 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:35:12.800453 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f095c6c-697c-4b5f-b8ad-7c145b4759c1" containerName="revoke-keys" Oct 30 12:35:12.800535 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:35:12.800478 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f110318-3125-4a76-ba6e-b9b0e6861051" containerName="istio-proxy" Oct 30 12:35:12.809766 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:35:12.809742 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bvpc2" Oct 30 12:35:12.835431 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:35:12.835402 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bvpc2"] Oct 30 12:35:12.943769 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:35:12.943731 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79ee2692-b905-4301-9389-ec07c74cc426-istio-token\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bvpc2\" (UID: \"79ee2692-b905-4301-9389-ec07c74cc426\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bvpc2" Oct 30 12:35:12.943769 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:35:12.943772 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkpth\" (UniqueName: \"kubernetes.io/projected/79ee2692-b905-4301-9389-ec07c74cc426-kube-api-access-wkpth\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bvpc2\" (UID: \"79ee2692-b905-4301-9389-ec07c74cc426\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bvpc2" Oct 30 12:35:12.943991 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:35:12.943960 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79ee2692-b905-4301-9389-ec07c74cc426-istio-envoy\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bvpc2\" (UID: \"79ee2692-b905-4301-9389-ec07c74cc426\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bvpc2" Oct 30 12:35:12.944031 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:35:12.943990 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79ee2692-b905-4301-9389-ec07c74cc426-istio-data\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bvpc2\" (UID: \"79ee2692-b905-4301-9389-ec07c74cc426\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bvpc2" Oct 30 12:35:12.944112 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:35:12.944094 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79ee2692-b905-4301-9389-ec07c74cc426-workload-certs\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bvpc2\" (UID: \"79ee2692-b905-4301-9389-ec07c74cc426\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bvpc2" Oct 30 12:35:12.944152 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:35:12.944126 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79ee2692-b905-4301-9389-ec07c74cc426-credential-socket\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bvpc2\" (UID: \"79ee2692-b905-4301-9389-ec07c74cc426\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bvpc2" Oct 30 12:35:12.944197 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:35:12.944159 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79ee2692-b905-4301-9389-ec07c74cc426-workload-socket\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bvpc2\" (UID: \"79ee2692-b905-4301-9389-ec07c74cc426\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bvpc2" Oct 30 12:35:12.944234 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:35:12.944208 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79ee2692-b905-4301-9389-ec07c74cc426-istio-podinfo\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bvpc2\" (UID: \"79ee2692-b905-4301-9389-ec07c74cc426\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bvpc2" Oct 30 12:35:12.944270 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:35:12.944246 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79ee2692-b905-4301-9389-ec07c74cc426-istiod-ca-cert\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bvpc2\" (UID: \"79ee2692-b905-4301-9389-ec07c74cc426\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bvpc2" Oct 30 12:35:13.044965 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:35:13.044930 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79ee2692-b905-4301-9389-ec07c74cc426-istio-envoy\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bvpc2\" (UID: \"79ee2692-b905-4301-9389-ec07c74cc426\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bvpc2" Oct 30 12:35:13.044965 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:35:13.044963 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79ee2692-b905-4301-9389-ec07c74cc426-istio-data\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bvpc2\" (UID: \"79ee2692-b905-4301-9389-ec07c74cc426\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bvpc2" Oct 30 12:35:13.045194 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:35:13.045011 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79ee2692-b905-4301-9389-ec07c74cc426-workload-certs\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bvpc2\" (UID: \"79ee2692-b905-4301-9389-ec07c74cc426\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bvpc2" Oct 30 12:35:13.045194 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:35:13.045034 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79ee2692-b905-4301-9389-ec07c74cc426-credential-socket\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bvpc2\" (UID: \"79ee2692-b905-4301-9389-ec07c74cc426\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bvpc2" Oct 30 12:35:13.045194 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:35:13.045055 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79ee2692-b905-4301-9389-ec07c74cc426-workload-socket\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bvpc2\" (UID: \"79ee2692-b905-4301-9389-ec07c74cc426\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bvpc2" Oct 30 12:35:13.045194 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:35:13.045081 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79ee2692-b905-4301-9389-ec07c74cc426-istio-podinfo\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bvpc2\" (UID: \"79ee2692-b905-4301-9389-ec07c74cc426\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bvpc2" Oct 30 12:35:13.045194 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:35:13.045129 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79ee2692-b905-4301-9389-ec07c74cc426-istiod-ca-cert\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bvpc2\" (UID: \"79ee2692-b905-4301-9389-ec07c74cc426\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bvpc2" Oct 30 12:35:13.045479 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:35:13.045232 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79ee2692-b905-4301-9389-ec07c74cc426-istio-token\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bvpc2\" (UID: \"79ee2692-b905-4301-9389-ec07c74cc426\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bvpc2" Oct 30 12:35:13.045479 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:35:13.045265 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkpth\" (UniqueName: \"kubernetes.io/projected/79ee2692-b905-4301-9389-ec07c74cc426-kube-api-access-wkpth\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bvpc2\" (UID: \"79ee2692-b905-4301-9389-ec07c74cc426\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bvpc2" Oct 30 12:35:13.045586 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:35:13.045553 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79ee2692-b905-4301-9389-ec07c74cc426-credential-socket\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bvpc2\" (UID: \"79ee2692-b905-4301-9389-ec07c74cc426\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bvpc2" Oct 30 12:35:13.045649 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:35:13.045579 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79ee2692-b905-4301-9389-ec07c74cc426-workload-socket\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bvpc2\" (UID: \"79ee2692-b905-4301-9389-ec07c74cc426\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bvpc2" Oct 30 12:35:13.045759 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:35:13.045736 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79ee2692-b905-4301-9389-ec07c74cc426-istio-data\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bvpc2\" (UID: \"79ee2692-b905-4301-9389-ec07c74cc426\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bvpc2" Oct 30 12:35:13.045812 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:35:13.045770 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79ee2692-b905-4301-9389-ec07c74cc426-workload-certs\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bvpc2\" (UID: \"79ee2692-b905-4301-9389-ec07c74cc426\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bvpc2" Oct 30 12:35:13.046094 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:35:13.046076 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79ee2692-b905-4301-9389-ec07c74cc426-istiod-ca-cert\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bvpc2\" (UID: \"79ee2692-b905-4301-9389-ec07c74cc426\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bvpc2" Oct 30 12:35:13.047428 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:35:13.047408 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79ee2692-b905-4301-9389-ec07c74cc426-istio-envoy\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bvpc2\" (UID: \"79ee2692-b905-4301-9389-ec07c74cc426\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bvpc2" Oct 30 12:35:13.047528 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:35:13.047516 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79ee2692-b905-4301-9389-ec07c74cc426-istio-podinfo\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bvpc2\" (UID: \"79ee2692-b905-4301-9389-ec07c74cc426\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bvpc2" Oct 30 12:35:13.069070 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:35:13.069008 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79ee2692-b905-4301-9389-ec07c74cc426-istio-token\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bvpc2\" (UID: \"79ee2692-b905-4301-9389-ec07c74cc426\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bvpc2" Oct 30 12:35:13.069242 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:35:13.069224 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkpth\" (UniqueName: \"kubernetes.io/projected/79ee2692-b905-4301-9389-ec07c74cc426-kube-api-access-wkpth\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bvpc2\" (UID: \"79ee2692-b905-4301-9389-ec07c74cc426\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bvpc2" Oct 30 12:35:13.127977 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:35:13.127954 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bvpc2" Oct 30 12:35:13.448849 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:35:13.448811 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bvpc2"] Oct 30 12:35:13.450218 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:35:13.450194 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79ee2692_b905_4301_9389_ec07c74cc426.slice/crio-69ba7d0a2926683cb0f55aa46f1075d701d0a02c41224a1af0d7d4d02838db03 WatchSource:0}: Error finding container 69ba7d0a2926683cb0f55aa46f1075d701d0a02c41224a1af0d7d4d02838db03: Status 404 returned error can't find the container with id 69ba7d0a2926683cb0f55aa46f1075d701d0a02c41224a1af0d7d4d02838db03 Oct 30 12:35:13.452515 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:35:13.452484 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:35:13.452592 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:35:13.452546 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:35:13.452592 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:35:13.452575 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:35:14.057926 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:35:14.057667 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bvpc2" event={"ID":"79ee2692-b905-4301-9389-ec07c74cc426","Type":"ContainerStarted","Data":"3639b0a7ab88cc99f973404e1dcbc8c51205dac754ff2a91721109e2e0bec131"} Oct 30 12:35:14.057926 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:35:14.057718 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bvpc2" event={"ID":"79ee2692-b905-4301-9389-ec07c74cc426","Type":"ContainerStarted","Data":"69ba7d0a2926683cb0f55aa46f1075d701d0a02c41224a1af0d7d4d02838db03"} Oct 30 12:35:14.128521 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:35:14.128484 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bvpc2" Oct 30 12:35:14.130675 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:35:14.130641 4908 patch_prober.go:29] interesting pod/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bvpc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" start-of-body= Oct 30 12:35:14.130874 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:35:14.130695 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bvpc2" podUID="79ee2692-b905-4301-9389-ec07c74cc426" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" Oct 30 12:35:14.142677 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:35:14.142632 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bvpc2" podStartSLOduration=2.142619082 podStartE2EDuration="2.142619082s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:35:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:35:14.1292516 +0000 UTC m=+2470.643680365" watchObservedRunningTime="2026-10-30 12:35:14.142619082 +0000 UTC m=+2470.657047827" Oct 30 12:35:14.438809 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:35:14.438719 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bfc84a5-5vg6b"] Oct 30 12:35:14.443051 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:35:14.443025 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bfc84a5-5vg6b" Oct 30 12:35:14.467857 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:35:14.465910 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 12:35:14.467857 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:35:14.466224 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s75hq\"" Oct 30 12:35:14.467857 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:35:14.466314 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 12:35:14.467857 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:35:14.466338 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 12:35:14.486847 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:35:14.477088 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bfc84a5-5vg6b"] Oct 30 12:35:14.585789 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:35:14.585756 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2baaaf0-ce6b-48ee-93d3-5a34e266d5b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5bfc84a5-5vg6b\" (UID: \"f2baaaf0-ce6b-48ee-93d3-5a34e266d5b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bfc84a5-5vg6b" Oct 30 12:35:14.586028 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:35:14.585854 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnwlb\" (UniqueName: \"kubernetes.io/projected/f2baaaf0-ce6b-48ee-93d3-5a34e266d5b1-kube-api-access-tnwlb\") pod \"maas-api-revoke-sub-5bfc84a5-5vg6b\" (UID: \"f2baaaf0-ce6b-48ee-93d3-5a34e266d5b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bfc84a5-5vg6b" Oct 30 12:35:14.586028 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:35:14.585884 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2baaaf0-ce6b-48ee-93d3-5a34e266d5b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5bfc84a5-5vg6b\" (UID: \"f2baaaf0-ce6b-48ee-93d3-5a34e266d5b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bfc84a5-5vg6b" Oct 30 12:35:14.690898 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:35:14.687180 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2baaaf0-ce6b-48ee-93d3-5a34e266d5b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5bfc84a5-5vg6b\" (UID: \"f2baaaf0-ce6b-48ee-93d3-5a34e266d5b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bfc84a5-5vg6b" Oct 30 12:35:14.690898 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:35:14.687293 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnwlb\" (UniqueName: \"kubernetes.io/projected/f2baaaf0-ce6b-48ee-93d3-5a34e266d5b1-kube-api-access-tnwlb\") pod \"maas-api-revoke-sub-5bfc84a5-5vg6b\" (UID: \"f2baaaf0-ce6b-48ee-93d3-5a34e266d5b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bfc84a5-5vg6b" Oct 30 12:35:14.690898 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:35:14.687327 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2baaaf0-ce6b-48ee-93d3-5a34e266d5b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5bfc84a5-5vg6b\" (UID: \"f2baaaf0-ce6b-48ee-93d3-5a34e266d5b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bfc84a5-5vg6b" Oct 30 12:35:14.690898 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:35:14.688750 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2baaaf0-ce6b-48ee-93d3-5a34e266d5b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5bfc84a5-5vg6b\" (UID: \"f2baaaf0-ce6b-48ee-93d3-5a34e266d5b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bfc84a5-5vg6b" Oct 30 12:35:14.700052 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:35:14.700020 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2baaaf0-ce6b-48ee-93d3-5a34e266d5b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5bfc84a5-5vg6b\" (UID: \"f2baaaf0-ce6b-48ee-93d3-5a34e266d5b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bfc84a5-5vg6b" Oct 30 12:35:14.711323 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:35:14.711293 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnwlb\" (UniqueName: \"kubernetes.io/projected/f2baaaf0-ce6b-48ee-93d3-5a34e266d5b1-kube-api-access-tnwlb\") pod \"maas-api-revoke-sub-5bfc84a5-5vg6b\" (UID: \"f2baaaf0-ce6b-48ee-93d3-5a34e266d5b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bfc84a5-5vg6b" Oct 30 12:35:14.757127 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:35:14.757098 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bfc84a5-5vg6b" Oct 30 12:35:15.129682 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:35:15.129462 4908 patch_prober.go:29] interesting pod/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bvpc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" start-of-body= Oct 30 12:35:15.129682 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:35:15.129518 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bvpc2" podUID="79ee2692-b905-4301-9389-ec07c74cc426" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" Oct 30 12:35:15.209635 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:35:15.207356 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bfc84a5-5vg6b"] Oct 30 12:35:16.119891 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:35:16.119680 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bfc84a5-5vg6b" event={"ID":"f2baaaf0-ce6b-48ee-93d3-5a34e266d5b1","Type":"ContainerStarted","Data":"9c80293cd6952df5a1087f76b7ae664bb4011a5677d69af423338e81237f4d29"} Oct 30 12:35:16.119891 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:35:16.119763 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bfc84a5-5vg6b" event={"ID":"f2baaaf0-ce6b-48ee-93d3-5a34e266d5b1","Type":"ContainerStarted","Data":"4b08750173fc286d45c4f6d8832f3628fabfed788b911c8520e96b5b68e1b767"} Oct 30 12:35:16.128398 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:35:16.128369 4908 patch_prober.go:29] interesting pod/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bvpc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" start-of-body= Oct 30 12:35:16.128529 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:35:16.128428 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bvpc2" podUID="79ee2692-b905-4301-9389-ec07c74cc426" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" Oct 30 12:35:16.184016 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:35:16.183940 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bfc84a5-5vg6b" podStartSLOduration=2.183920652 podStartE2EDuration="2.183920652s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:35:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:35:16.171251578 +0000 UTC m=+2472.685680325" watchObservedRunningTime="2026-10-30 12:35:16.183920652 +0000 UTC m=+2472.698349398" Oct 30 12:35:17.128474 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:35:17.128435 4908 patch_prober.go:29] interesting pod/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bvpc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" start-of-body= Oct 30 12:35:17.128687 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:35:17.128492 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bvpc2" podUID="79ee2692-b905-4301-9389-ec07c74cc426" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" Oct 30 12:35:17.141456 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:35:17.140453 4908 generic.go:356] "Generic (PLEG): container finished" podID="f2baaaf0-ce6b-48ee-93d3-5a34e266d5b1" containerID="9c80293cd6952df5a1087f76b7ae664bb4011a5677d69af423338e81237f4d29" exitCode=0 Oct 30 12:35:17.141456 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:35:17.140714 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bfc84a5-5vg6b" event={"ID":"f2baaaf0-ce6b-48ee-93d3-5a34e266d5b1","Type":"ContainerDied","Data":"9c80293cd6952df5a1087f76b7ae664bb4011a5677d69af423338e81237f4d29"} Oct 30 12:35:18.129246 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:35:18.129208 4908 patch_prober.go:29] interesting pod/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bvpc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" start-of-body= Oct 30 12:35:18.129702 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:35:18.129276 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bvpc2" podUID="79ee2692-b905-4301-9389-ec07c74cc426" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" Oct 30 12:35:18.523092 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:35:18.523064 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bfc84a5-5vg6b" Oct 30 12:35:18.633964 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:35:18.633927 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2baaaf0-ce6b-48ee-93d3-5a34e266d5b1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2baaaf0-ce6b-48ee-93d3-5a34e266d5b1-maas-api-openshift-ca\") pod \"f2baaaf0-ce6b-48ee-93d3-5a34e266d5b1\" (UID: \"f2baaaf0-ce6b-48ee-93d3-5a34e266d5b1\") " Oct 30 12:35:18.634134 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:35:18.633988 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2baaaf0-ce6b-48ee-93d3-5a34e266d5b1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2baaaf0-ce6b-48ee-93d3-5a34e266d5b1-maas-api-xks-ca\") pod \"f2baaaf0-ce6b-48ee-93d3-5a34e266d5b1\" (UID: \"f2baaaf0-ce6b-48ee-93d3-5a34e266d5b1\") " Oct 30 12:35:18.634201 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:35:18.634141 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2baaaf0-ce6b-48ee-93d3-5a34e266d5b1-kube-api-access-tnwlb\" (UniqueName: \"kubernetes.io/projected/f2baaaf0-ce6b-48ee-93d3-5a34e266d5b1-kube-api-access-tnwlb\") pod \"f2baaaf0-ce6b-48ee-93d3-5a34e266d5b1\" (UID: \"f2baaaf0-ce6b-48ee-93d3-5a34e266d5b1\") " Oct 30 12:35:18.635162 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:35:18.635116 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2baaaf0-ce6b-48ee-93d3-5a34e266d5b1-maas-api-openshift-ca" pod "f2baaaf0-ce6b-48ee-93d3-5a34e266d5b1" (UID: "f2baaaf0-ce6b-48ee-93d3-5a34e266d5b1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:35:18.637403 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:35:18.637363 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2baaaf0-ce6b-48ee-93d3-5a34e266d5b1-kube-api-access-tnwlb" pod "f2baaaf0-ce6b-48ee-93d3-5a34e266d5b1" (UID: "f2baaaf0-ce6b-48ee-93d3-5a34e266d5b1"). InnerVolumeSpecName "kube-api-access-tnwlb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:35:18.637965 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:35:18.637943 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2baaaf0-ce6b-48ee-93d3-5a34e266d5b1-maas-api-xks-ca" pod "f2baaaf0-ce6b-48ee-93d3-5a34e266d5b1" (UID: "f2baaaf0-ce6b-48ee-93d3-5a34e266d5b1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:35:18.736416 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:35:18.736388 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2baaaf0-ce6b-48ee-93d3-5a34e266d5b1-maas-api-openshift-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:35:18.736416 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:35:18.736414 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2baaaf0-ce6b-48ee-93d3-5a34e266d5b1-maas-api-xks-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:35:18.736598 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:35:18.736423 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tnwlb\" (UniqueName: \"kubernetes.io/projected/f2baaaf0-ce6b-48ee-93d3-5a34e266d5b1-kube-api-access-tnwlb\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:35:19.128513 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:35:19.128421 4908 patch_prober.go:29] interesting pod/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bvpc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" start-of-body= Oct 30 12:35:19.128513 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:35:19.128486 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bvpc2" podUID="79ee2692-b905-4301-9389-ec07c74cc426" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" Oct 30 12:35:19.163853 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:35:19.159100 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bfc84a5-5vg6b" event={"ID":"f2baaaf0-ce6b-48ee-93d3-5a34e266d5b1","Type":"ContainerDied","Data":"4b08750173fc286d45c4f6d8832f3628fabfed788b911c8520e96b5b68e1b767"} Oct 30 12:35:19.163853 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:35:19.159159 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4b08750173fc286d45c4f6d8832f3628fabfed788b911c8520e96b5b68e1b767" Oct 30 12:35:19.163853 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:35:19.159222 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bfc84a5-5vg6b" Oct 30 12:35:19.686583 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:35:19.686543 4908 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vn6vv" containerName="extractor" Oct 30 12:35:20.128950 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:35:20.128912 4908 patch_prober.go:29] interesting pod/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bvpc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" start-of-body= Oct 30 12:35:20.129132 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:35:20.129016 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bvpc2" podUID="79ee2692-b905-4301-9389-ec07c74cc426" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" Oct 30 12:35:21.132414 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:35:21.132384 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bvpc2" Oct 30 12:35:21.132881 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:35:21.132858 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bvpc2" Oct 30 12:35:21.133569 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:35:21.133553 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bvpc2" Oct 30 12:35:23.080852 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:35:23.078375 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9wwfc"] Oct 30 12:35:26.058229 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:35:26.058188 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-5d90d8-openshift-default-577df8c9d9-cjd27"] Oct 30 12:35:26.060099 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:35:26.060067 4908 cpu_manager.go:414] "containerMap: removing container" podUID="f2baaaf0-ce6b-48ee-93d3-5a34e266d5b1" containerName="revoke-keys" Oct 30 12:35:26.060099 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:35:26.060100 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2baaaf0-ce6b-48ee-93d3-5a34e266d5b1" containerName="revoke-keys" Oct 30 12:35:26.060531 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:35:26.060509 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2baaaf0-ce6b-48ee-93d3-5a34e266d5b1" containerName="revoke-keys" Oct 30 12:35:26.065149 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:35:26.065123 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-5d90d8-openshift-default-577df8c9d9-cjd27" Oct 30 12:35:26.091973 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:35:26.091947 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-5d90d8-openshift-default-dockercfg-9jgg9\"" Oct 30 12:35:26.101483 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:35:26.101341 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-5d90d8-openshift-default-577df8c9d9-cjd27"] Oct 30 12:35:26.180109 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:35:26.180078 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5526e66b-b972-4e0f-9a11-d9a9b8fdf0d1-istio-token\") pod \"e2e-explicit-5d90d8-openshift-default-577df8c9d9-cjd27\" (UID: \"5526e66b-b972-4e0f-9a11-d9a9b8fdf0d1\") " pod="openshift-ingress/e2e-explicit-5d90d8-openshift-default-577df8c9d9-cjd27" Oct 30 12:35:26.180272 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:35:26.180120 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5526e66b-b972-4e0f-9a11-d9a9b8fdf0d1-credential-socket\") pod \"e2e-explicit-5d90d8-openshift-default-577df8c9d9-cjd27\" (UID: \"5526e66b-b972-4e0f-9a11-d9a9b8fdf0d1\") " pod="openshift-ingress/e2e-explicit-5d90d8-openshift-default-577df8c9d9-cjd27" Oct 30 12:35:26.180272 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:35:26.180158 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5526e66b-b972-4e0f-9a11-d9a9b8fdf0d1-workload-socket\") pod \"e2e-explicit-5d90d8-openshift-default-577df8c9d9-cjd27\" (UID: \"5526e66b-b972-4e0f-9a11-d9a9b8fdf0d1\") " pod="openshift-ingress/e2e-explicit-5d90d8-openshift-default-577df8c9d9-cjd27" Oct 30 12:35:26.180272 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:35:26.180192 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5526e66b-b972-4e0f-9a11-d9a9b8fdf0d1-istio-data\") pod \"e2e-explicit-5d90d8-openshift-default-577df8c9d9-cjd27\" (UID: \"5526e66b-b972-4e0f-9a11-d9a9b8fdf0d1\") " pod="openshift-ingress/e2e-explicit-5d90d8-openshift-default-577df8c9d9-cjd27" Oct 30 12:35:26.180272 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:35:26.180221 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5526e66b-b972-4e0f-9a11-d9a9b8fdf0d1-istiod-ca-cert\") pod \"e2e-explicit-5d90d8-openshift-default-577df8c9d9-cjd27\" (UID: \"5526e66b-b972-4e0f-9a11-d9a9b8fdf0d1\") " pod="openshift-ingress/e2e-explicit-5d90d8-openshift-default-577df8c9d9-cjd27" Oct 30 12:35:26.180502 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:35:26.180275 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5526e66b-b972-4e0f-9a11-d9a9b8fdf0d1-workload-certs\") pod \"e2e-explicit-5d90d8-openshift-default-577df8c9d9-cjd27\" (UID: \"5526e66b-b972-4e0f-9a11-d9a9b8fdf0d1\") " pod="openshift-ingress/e2e-explicit-5d90d8-openshift-default-577df8c9d9-cjd27" Oct 30 12:35:26.180502 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:35:26.180329 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5526e66b-b972-4e0f-9a11-d9a9b8fdf0d1-istio-podinfo\") pod \"e2e-explicit-5d90d8-openshift-default-577df8c9d9-cjd27\" (UID: \"5526e66b-b972-4e0f-9a11-d9a9b8fdf0d1\") " pod="openshift-ingress/e2e-explicit-5d90d8-openshift-default-577df8c9d9-cjd27" Oct 30 12:35:26.180502 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:35:26.180353 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5526e66b-b972-4e0f-9a11-d9a9b8fdf0d1-istio-envoy\") pod \"e2e-explicit-5d90d8-openshift-default-577df8c9d9-cjd27\" (UID: \"5526e66b-b972-4e0f-9a11-d9a9b8fdf0d1\") " pod="openshift-ingress/e2e-explicit-5d90d8-openshift-default-577df8c9d9-cjd27" Oct 30 12:35:26.180502 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:35:26.180370 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k28xn\" (UniqueName: \"kubernetes.io/projected/5526e66b-b972-4e0f-9a11-d9a9b8fdf0d1-kube-api-access-k28xn\") pod \"e2e-explicit-5d90d8-openshift-default-577df8c9d9-cjd27\" (UID: \"5526e66b-b972-4e0f-9a11-d9a9b8fdf0d1\") " pod="openshift-ingress/e2e-explicit-5d90d8-openshift-default-577df8c9d9-cjd27" Oct 30 12:35:26.288856 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:35:26.285591 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5526e66b-b972-4e0f-9a11-d9a9b8fdf0d1-istio-podinfo\") pod \"e2e-explicit-5d90d8-openshift-default-577df8c9d9-cjd27\" (UID: \"5526e66b-b972-4e0f-9a11-d9a9b8fdf0d1\") " pod="openshift-ingress/e2e-explicit-5d90d8-openshift-default-577df8c9d9-cjd27" Oct 30 12:35:26.288856 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:35:26.285656 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5526e66b-b972-4e0f-9a11-d9a9b8fdf0d1-istio-envoy\") pod \"e2e-explicit-5d90d8-openshift-default-577df8c9d9-cjd27\" (UID: \"5526e66b-b972-4e0f-9a11-d9a9b8fdf0d1\") " pod="openshift-ingress/e2e-explicit-5d90d8-openshift-default-577df8c9d9-cjd27" Oct 30 12:35:26.288856 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:35:26.285685 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k28xn\" (UniqueName: \"kubernetes.io/projected/5526e66b-b972-4e0f-9a11-d9a9b8fdf0d1-kube-api-access-k28xn\") pod \"e2e-explicit-5d90d8-openshift-default-577df8c9d9-cjd27\" (UID: \"5526e66b-b972-4e0f-9a11-d9a9b8fdf0d1\") " pod="openshift-ingress/e2e-explicit-5d90d8-openshift-default-577df8c9d9-cjd27" Oct 30 12:35:26.288856 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:35:26.285729 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5526e66b-b972-4e0f-9a11-d9a9b8fdf0d1-istio-token\") pod \"e2e-explicit-5d90d8-openshift-default-577df8c9d9-cjd27\" (UID: \"5526e66b-b972-4e0f-9a11-d9a9b8fdf0d1\") " pod="openshift-ingress/e2e-explicit-5d90d8-openshift-default-577df8c9d9-cjd27" Oct 30 12:35:26.288856 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:35:26.285766 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5526e66b-b972-4e0f-9a11-d9a9b8fdf0d1-credential-socket\") pod \"e2e-explicit-5d90d8-openshift-default-577df8c9d9-cjd27\" (UID: \"5526e66b-b972-4e0f-9a11-d9a9b8fdf0d1\") " pod="openshift-ingress/e2e-explicit-5d90d8-openshift-default-577df8c9d9-cjd27" Oct 30 12:35:26.288856 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:35:26.285885 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5526e66b-b972-4e0f-9a11-d9a9b8fdf0d1-workload-socket\") pod \"e2e-explicit-5d90d8-openshift-default-577df8c9d9-cjd27\" (UID: \"5526e66b-b972-4e0f-9a11-d9a9b8fdf0d1\") " pod="openshift-ingress/e2e-explicit-5d90d8-openshift-default-577df8c9d9-cjd27" Oct 30 12:35:26.288856 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:35:26.285945 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5526e66b-b972-4e0f-9a11-d9a9b8fdf0d1-istio-data\") pod \"e2e-explicit-5d90d8-openshift-default-577df8c9d9-cjd27\" (UID: \"5526e66b-b972-4e0f-9a11-d9a9b8fdf0d1\") " pod="openshift-ingress/e2e-explicit-5d90d8-openshift-default-577df8c9d9-cjd27" Oct 30 12:35:26.288856 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:35:26.285993 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5526e66b-b972-4e0f-9a11-d9a9b8fdf0d1-istiod-ca-cert\") pod \"e2e-explicit-5d90d8-openshift-default-577df8c9d9-cjd27\" (UID: \"5526e66b-b972-4e0f-9a11-d9a9b8fdf0d1\") " pod="openshift-ingress/e2e-explicit-5d90d8-openshift-default-577df8c9d9-cjd27" Oct 30 12:35:26.288856 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:35:26.286086 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5526e66b-b972-4e0f-9a11-d9a9b8fdf0d1-workload-certs\") pod \"e2e-explicit-5d90d8-openshift-default-577df8c9d9-cjd27\" (UID: \"5526e66b-b972-4e0f-9a11-d9a9b8fdf0d1\") " pod="openshift-ingress/e2e-explicit-5d90d8-openshift-default-577df8c9d9-cjd27" Oct 30 12:35:26.288856 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:35:26.286541 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5526e66b-b972-4e0f-9a11-d9a9b8fdf0d1-workload-certs\") pod \"e2e-explicit-5d90d8-openshift-default-577df8c9d9-cjd27\" (UID: \"5526e66b-b972-4e0f-9a11-d9a9b8fdf0d1\") " pod="openshift-ingress/e2e-explicit-5d90d8-openshift-default-577df8c9d9-cjd27" Oct 30 12:35:26.291191 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:35:26.291160 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5526e66b-b972-4e0f-9a11-d9a9b8fdf0d1-credential-socket\") pod \"e2e-explicit-5d90d8-openshift-default-577df8c9d9-cjd27\" (UID: \"5526e66b-b972-4e0f-9a11-d9a9b8fdf0d1\") " pod="openshift-ingress/e2e-explicit-5d90d8-openshift-default-577df8c9d9-cjd27" Oct 30 12:35:26.292322 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:35:26.292298 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5526e66b-b972-4e0f-9a11-d9a9b8fdf0d1-istio-data\") pod \"e2e-explicit-5d90d8-openshift-default-577df8c9d9-cjd27\" (UID: \"5526e66b-b972-4e0f-9a11-d9a9b8fdf0d1\") " pod="openshift-ingress/e2e-explicit-5d90d8-openshift-default-577df8c9d9-cjd27" Oct 30 12:35:26.292741 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:35:26.292718 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5526e66b-b972-4e0f-9a11-d9a9b8fdf0d1-workload-socket\") pod \"e2e-explicit-5d90d8-openshift-default-577df8c9d9-cjd27\" (UID: \"5526e66b-b972-4e0f-9a11-d9a9b8fdf0d1\") " pod="openshift-ingress/e2e-explicit-5d90d8-openshift-default-577df8c9d9-cjd27" Oct 30 12:35:26.293533 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:35:26.293507 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5526e66b-b972-4e0f-9a11-d9a9b8fdf0d1-istiod-ca-cert\") pod \"e2e-explicit-5d90d8-openshift-default-577df8c9d9-cjd27\" (UID: \"5526e66b-b972-4e0f-9a11-d9a9b8fdf0d1\") " pod="openshift-ingress/e2e-explicit-5d90d8-openshift-default-577df8c9d9-cjd27" Oct 30 12:35:26.328892 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:35:26.326704 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5526e66b-b972-4e0f-9a11-d9a9b8fdf0d1-istio-podinfo\") pod \"e2e-explicit-5d90d8-openshift-default-577df8c9d9-cjd27\" (UID: \"5526e66b-b972-4e0f-9a11-d9a9b8fdf0d1\") " pod="openshift-ingress/e2e-explicit-5d90d8-openshift-default-577df8c9d9-cjd27" Oct 30 12:35:26.329517 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:35:26.329489 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5526e66b-b972-4e0f-9a11-d9a9b8fdf0d1-istio-envoy\") pod \"e2e-explicit-5d90d8-openshift-default-577df8c9d9-cjd27\" (UID: \"5526e66b-b972-4e0f-9a11-d9a9b8fdf0d1\") " pod="openshift-ingress/e2e-explicit-5d90d8-openshift-default-577df8c9d9-cjd27" Oct 30 12:35:26.333961 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:35:26.333934 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5526e66b-b972-4e0f-9a11-d9a9b8fdf0d1-istio-token\") pod \"e2e-explicit-5d90d8-openshift-default-577df8c9d9-cjd27\" (UID: \"5526e66b-b972-4e0f-9a11-d9a9b8fdf0d1\") " pod="openshift-ingress/e2e-explicit-5d90d8-openshift-default-577df8c9d9-cjd27" Oct 30 12:35:26.334649 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:35:26.334630 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k28xn\" (UniqueName: \"kubernetes.io/projected/5526e66b-b972-4e0f-9a11-d9a9b8fdf0d1-kube-api-access-k28xn\") pod \"e2e-explicit-5d90d8-openshift-default-577df8c9d9-cjd27\" (UID: \"5526e66b-b972-4e0f-9a11-d9a9b8fdf0d1\") " pod="openshift-ingress/e2e-explicit-5d90d8-openshift-default-577df8c9d9-cjd27" Oct 30 12:35:26.383103 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:35:26.382821 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-5d90d8-openshift-default-577df8c9d9-cjd27" Oct 30 12:35:26.734342 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:35:26.734317 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-5d90d8-openshift-default-577df8c9d9-cjd27"] Oct 30 12:35:26.736992 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:35:26.736962 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5526e66b_b972_4e0f_9a11_d9a9b8fdf0d1.slice/crio-ef6c51bb7199166cf06ca14b83db4d5acfce9fad49d7a8dbe2d31346acce71bb WatchSource:0}: Error finding container ef6c51bb7199166cf06ca14b83db4d5acfce9fad49d7a8dbe2d31346acce71bb: Status 404 returned error can't find the container with id ef6c51bb7199166cf06ca14b83db4d5acfce9fad49d7a8dbe2d31346acce71bb Oct 30 12:35:26.739562 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:35:26.739529 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:35:26.739649 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:35:26.739594 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:35:26.739649 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:35:26.739624 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:35:27.290786 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:35:27.290749 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-5d90d8-openshift-default-577df8c9d9-cjd27" event={"ID":"5526e66b-b972-4e0f-9a11-d9a9b8fdf0d1","Type":"ContainerStarted","Data":"121bdcccd0dfbb5ad3e498b082c7ba5eeb2ee0f946a00128883ab8b2f3b68dc6"} Oct 30 12:35:27.291233 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:35:27.290804 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-5d90d8-openshift-default-577df8c9d9-cjd27" event={"ID":"5526e66b-b972-4e0f-9a11-d9a9b8fdf0d1","Type":"ContainerStarted","Data":"ef6c51bb7199166cf06ca14b83db4d5acfce9fad49d7a8dbe2d31346acce71bb"} Oct 30 12:35:27.346572 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:35:27.346511 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-5d90d8-openshift-default-577df8c9d9-cjd27" podStartSLOduration=1.346496332 podStartE2EDuration="1.346496332s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:35:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:35:27.344301053 +0000 UTC m=+2483.858729799" watchObservedRunningTime="2026-10-30 12:35:27.346496332 +0000 UTC m=+2483.860925079" Oct 30 12:35:27.383902 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:35:27.383869 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-5d90d8-openshift-default-577df8c9d9-cjd27" Oct 30 12:35:27.391131 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:35:27.391108 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-5d90d8-openshift-default-577df8c9d9-cjd27" Oct 30 12:35:28.300313 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:35:28.300276 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-5d90d8-openshift-default-577df8c9d9-cjd27" Oct 30 12:35:28.301291 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:35:28.301271 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-5d90d8-openshift-default-577df8c9d9-cjd27" Oct 30 12:35:49.684708 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:35:49.684657 4908 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vn6vv" containerName="extractor" Oct 30 12:36:05.893642 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:05.893600 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9wwfc"] Oct 30 12:36:06.296211 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:06.296178 4908 scope.go:122] "RemoveContainer" containerID="c27b1d681498e5c17ad86cf45797658d107f7cebb6ede02f548b51cf192a3387" Oct 30 12:36:06.820111 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:06.820081 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-2675c0/auth-test-model-2675c0-kserve-6bdcd4c665-qspzk"] Oct 30 12:36:06.825459 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:06.825438 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-2675c0/auth-test-model-2675c0-kserve-6bdcd4c665-qspzk" Oct 30 12:36:06.832309 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:06.832282 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-2675c0/auth-test-model-2675c0-kserve-6bdcd4c665-qspzk"] Oct 30 12:36:06.850548 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:06.850527 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-2675c0\"/\"kube-root-ca.crt\"" Oct 30 12:36:06.850681 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:06.850537 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-2675c0\"/\"openshift-service-ca.crt\"" Oct 30 12:36:06.850681 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:06.850537 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-2675c0\"/\"default-dockercfg-zmqp5\"" Oct 30 12:36:06.850681 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:06.850578 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-2675c0\"/\"auth-test-model-2675c0-kserve-self-signed-certs\"" Oct 30 12:36:06.900811 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:06.900777 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8595d8d1-88b3-412e-9b02-bed8ed21f57b-tmp-dir\") pod \"auth-test-model-2675c0-kserve-6bdcd4c665-qspzk\" (UID: \"8595d8d1-88b3-412e-9b02-bed8ed21f57b\") " pod="ai-tenant-e2e-shared-a-w3-2675c0/auth-test-model-2675c0-kserve-6bdcd4c665-qspzk" Oct 30 12:36:06.901246 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:06.900952 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h96c8\" (UniqueName: \"kubernetes.io/projected/8595d8d1-88b3-412e-9b02-bed8ed21f57b-kube-api-access-h96c8\") pod \"auth-test-model-2675c0-kserve-6bdcd4c665-qspzk\" (UID: \"8595d8d1-88b3-412e-9b02-bed8ed21f57b\") " pod="ai-tenant-e2e-shared-a-w3-2675c0/auth-test-model-2675c0-kserve-6bdcd4c665-qspzk" Oct 30 12:36:06.901246 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:06.901071 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8595d8d1-88b3-412e-9b02-bed8ed21f57b-tls-certs\") pod \"auth-test-model-2675c0-kserve-6bdcd4c665-qspzk\" (UID: \"8595d8d1-88b3-412e-9b02-bed8ed21f57b\") " pod="ai-tenant-e2e-shared-a-w3-2675c0/auth-test-model-2675c0-kserve-6bdcd4c665-qspzk" Oct 30 12:36:06.901246 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:06.901106 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8595d8d1-88b3-412e-9b02-bed8ed21f57b-model-cache\") pod \"auth-test-model-2675c0-kserve-6bdcd4c665-qspzk\" (UID: \"8595d8d1-88b3-412e-9b02-bed8ed21f57b\") " pod="ai-tenant-e2e-shared-a-w3-2675c0/auth-test-model-2675c0-kserve-6bdcd4c665-qspzk" Oct 30 12:36:06.901375 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:06.901278 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8595d8d1-88b3-412e-9b02-bed8ed21f57b-home\") pod \"auth-test-model-2675c0-kserve-6bdcd4c665-qspzk\" (UID: \"8595d8d1-88b3-412e-9b02-bed8ed21f57b\") " pod="ai-tenant-e2e-shared-a-w3-2675c0/auth-test-model-2675c0-kserve-6bdcd4c665-qspzk" Oct 30 12:36:06.901375 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:06.901331 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8595d8d1-88b3-412e-9b02-bed8ed21f57b-dshm\") pod \"auth-test-model-2675c0-kserve-6bdcd4c665-qspzk\" (UID: \"8595d8d1-88b3-412e-9b02-bed8ed21f57b\") " pod="ai-tenant-e2e-shared-a-w3-2675c0/auth-test-model-2675c0-kserve-6bdcd4c665-qspzk" Oct 30 12:36:07.002392 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:07.002348 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h96c8\" (UniqueName: \"kubernetes.io/projected/8595d8d1-88b3-412e-9b02-bed8ed21f57b-kube-api-access-h96c8\") pod \"auth-test-model-2675c0-kserve-6bdcd4c665-qspzk\" (UID: \"8595d8d1-88b3-412e-9b02-bed8ed21f57b\") " pod="ai-tenant-e2e-shared-a-w3-2675c0/auth-test-model-2675c0-kserve-6bdcd4c665-qspzk" Oct 30 12:36:07.002604 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:07.002434 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8595d8d1-88b3-412e-9b02-bed8ed21f57b-tls-certs\") pod \"auth-test-model-2675c0-kserve-6bdcd4c665-qspzk\" (UID: \"8595d8d1-88b3-412e-9b02-bed8ed21f57b\") " pod="ai-tenant-e2e-shared-a-w3-2675c0/auth-test-model-2675c0-kserve-6bdcd4c665-qspzk" Oct 30 12:36:07.002604 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:07.002464 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8595d8d1-88b3-412e-9b02-bed8ed21f57b-model-cache\") pod \"auth-test-model-2675c0-kserve-6bdcd4c665-qspzk\" (UID: \"8595d8d1-88b3-412e-9b02-bed8ed21f57b\") " pod="ai-tenant-e2e-shared-a-w3-2675c0/auth-test-model-2675c0-kserve-6bdcd4c665-qspzk" Oct 30 12:36:07.002748 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:07.002614 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8595d8d1-88b3-412e-9b02-bed8ed21f57b-home\") pod \"auth-test-model-2675c0-kserve-6bdcd4c665-qspzk\" (UID: \"8595d8d1-88b3-412e-9b02-bed8ed21f57b\") " pod="ai-tenant-e2e-shared-a-w3-2675c0/auth-test-model-2675c0-kserve-6bdcd4c665-qspzk" Oct 30 12:36:07.002748 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:07.002672 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8595d8d1-88b3-412e-9b02-bed8ed21f57b-dshm\") pod \"auth-test-model-2675c0-kserve-6bdcd4c665-qspzk\" (UID: \"8595d8d1-88b3-412e-9b02-bed8ed21f57b\") " pod="ai-tenant-e2e-shared-a-w3-2675c0/auth-test-model-2675c0-kserve-6bdcd4c665-qspzk" Oct 30 12:36:07.003055 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:07.003026 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8595d8d1-88b3-412e-9b02-bed8ed21f57b-tmp-dir\") pod \"auth-test-model-2675c0-kserve-6bdcd4c665-qspzk\" (UID: \"8595d8d1-88b3-412e-9b02-bed8ed21f57b\") " pod="ai-tenant-e2e-shared-a-w3-2675c0/auth-test-model-2675c0-kserve-6bdcd4c665-qspzk" Oct 30 12:36:07.003148 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:07.003081 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8595d8d1-88b3-412e-9b02-bed8ed21f57b-model-cache\") pod \"auth-test-model-2675c0-kserve-6bdcd4c665-qspzk\" (UID: \"8595d8d1-88b3-412e-9b02-bed8ed21f57b\") " pod="ai-tenant-e2e-shared-a-w3-2675c0/auth-test-model-2675c0-kserve-6bdcd4c665-qspzk" Oct 30 12:36:07.003401 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:07.003382 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8595d8d1-88b3-412e-9b02-bed8ed21f57b-home\") pod \"auth-test-model-2675c0-kserve-6bdcd4c665-qspzk\" (UID: \"8595d8d1-88b3-412e-9b02-bed8ed21f57b\") " pod="ai-tenant-e2e-shared-a-w3-2675c0/auth-test-model-2675c0-kserve-6bdcd4c665-qspzk" Oct 30 12:36:07.003525 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:07.003407 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8595d8d1-88b3-412e-9b02-bed8ed21f57b-tmp-dir\") pod \"auth-test-model-2675c0-kserve-6bdcd4c665-qspzk\" (UID: \"8595d8d1-88b3-412e-9b02-bed8ed21f57b\") " pod="ai-tenant-e2e-shared-a-w3-2675c0/auth-test-model-2675c0-kserve-6bdcd4c665-qspzk" Oct 30 12:36:07.005870 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:07.005421 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8595d8d1-88b3-412e-9b02-bed8ed21f57b-dshm\") pod \"auth-test-model-2675c0-kserve-6bdcd4c665-qspzk\" (UID: \"8595d8d1-88b3-412e-9b02-bed8ed21f57b\") " pod="ai-tenant-e2e-shared-a-w3-2675c0/auth-test-model-2675c0-kserve-6bdcd4c665-qspzk" Oct 30 12:36:07.006014 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:07.005991 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8595d8d1-88b3-412e-9b02-bed8ed21f57b-tls-certs\") pod \"auth-test-model-2675c0-kserve-6bdcd4c665-qspzk\" (UID: \"8595d8d1-88b3-412e-9b02-bed8ed21f57b\") " pod="ai-tenant-e2e-shared-a-w3-2675c0/auth-test-model-2675c0-kserve-6bdcd4c665-qspzk" Oct 30 12:36:07.027159 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:07.027099 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h96c8\" (UniqueName: \"kubernetes.io/projected/8595d8d1-88b3-412e-9b02-bed8ed21f57b-kube-api-access-h96c8\") pod \"auth-test-model-2675c0-kserve-6bdcd4c665-qspzk\" (UID: \"8595d8d1-88b3-412e-9b02-bed8ed21f57b\") " pod="ai-tenant-e2e-shared-a-w3-2675c0/auth-test-model-2675c0-kserve-6bdcd4c665-qspzk" Oct 30 12:36:07.166917 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:07.150992 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-2675c0/auth-test-model-2675c0-kserve-6bdcd4c665-qspzk" Oct 30 12:36:07.543267 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:07.543236 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-2675c0/auth-test-model-2675c0-kserve-6bdcd4c665-qspzk"] Oct 30 12:36:07.545462 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:36:07.545429 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8595d8d1_88b3_412e_9b02_bed8ed21f57b.slice/crio-e6159d5e6ac341d6bfcbac109d13152dae04ab219a14ca62b308c4e4ca36d5be WatchSource:0}: Error finding container e6159d5e6ac341d6bfcbac109d13152dae04ab219a14ca62b308c4e4ca36d5be: Status 404 returned error can't find the container with id e6159d5e6ac341d6bfcbac109d13152dae04ab219a14ca62b308c4e4ca36d5be Oct 30 12:36:07.788243 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:07.788198 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-2675c0/auth-test-model-2675c0-kserve-6bdcd4c665-qspzk" event={"ID":"8595d8d1-88b3-412e-9b02-bed8ed21f57b","Type":"ContainerStarted","Data":"cf1eedddf7102b0cc04a9b3dfd986b12770f4083d02a70c039a39b0965907318"} Oct 30 12:36:07.788418 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:07.788253 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-2675c0/auth-test-model-2675c0-kserve-6bdcd4c665-qspzk" event={"ID":"8595d8d1-88b3-412e-9b02-bed8ed21f57b","Type":"ContainerStarted","Data":"e6159d5e6ac341d6bfcbac109d13152dae04ab219a14ca62b308c4e4ca36d5be"} Oct 30 12:36:07.858368 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:07.858275 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-2675c0/auth-test-model-2675c0-kserve-6bdcd4c665-qspzk" podStartSLOduration=1.8582612410000001 podStartE2EDuration="1.858261241s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:36:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:36:07.856804456 +0000 UTC m=+2524.371233203" watchObservedRunningTime="2026-10-30 12:36:07.858261241 +0000 UTC m=+2524.372690041" Oct 30 12:36:08.151879 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:08.151759 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-2675c0/auth-test-model-2675c0-kserve-6bdcd4c665-qspzk" Oct 30 12:36:10.471089 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:10.471051 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-pzxkx"] Oct 30 12:36:10.475596 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:10.475572 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-pzxkx" Oct 30 12:36:10.489510 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:10.489484 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-9d45fb-openshift-default-dockercfg-xd9x9\"" Oct 30 12:36:10.506282 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:10.506253 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-pzxkx"] Oct 30 12:36:10.563882 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:10.563855 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b4543f5-ebb4-4ebc-94f8-fa451750afa6-istio-envoy\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-pzxkx\" (UID: \"8b4543f5-ebb4-4ebc-94f8-fa451750afa6\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-pzxkx" Oct 30 12:36:10.564021 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:10.563894 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b4543f5-ebb4-4ebc-94f8-fa451750afa6-istio-podinfo\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-pzxkx\" (UID: \"8b4543f5-ebb4-4ebc-94f8-fa451750afa6\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-pzxkx" Oct 30 12:36:10.564021 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:10.563919 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b4543f5-ebb4-4ebc-94f8-fa451750afa6-credential-socket\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-pzxkx\" (UID: \"8b4543f5-ebb4-4ebc-94f8-fa451750afa6\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-pzxkx" Oct 30 12:36:10.564102 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:10.564049 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b4543f5-ebb4-4ebc-94f8-fa451750afa6-istiod-ca-cert\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-pzxkx\" (UID: \"8b4543f5-ebb4-4ebc-94f8-fa451750afa6\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-pzxkx" Oct 30 12:36:10.564102 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:10.564070 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b4543f5-ebb4-4ebc-94f8-fa451750afa6-istio-data\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-pzxkx\" (UID: \"8b4543f5-ebb4-4ebc-94f8-fa451750afa6\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-pzxkx" Oct 30 12:36:10.564102 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:10.564088 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b4543f5-ebb4-4ebc-94f8-fa451750afa6-istio-token\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-pzxkx\" (UID: \"8b4543f5-ebb4-4ebc-94f8-fa451750afa6\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-pzxkx" Oct 30 12:36:10.564298 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:10.564274 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b4543f5-ebb4-4ebc-94f8-fa451750afa6-workload-socket\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-pzxkx\" (UID: \"8b4543f5-ebb4-4ebc-94f8-fa451750afa6\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-pzxkx" Oct 30 12:36:10.564362 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:10.564320 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfvth\" (UniqueName: \"kubernetes.io/projected/8b4543f5-ebb4-4ebc-94f8-fa451750afa6-kube-api-access-lfvth\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-pzxkx\" (UID: \"8b4543f5-ebb4-4ebc-94f8-fa451750afa6\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-pzxkx" Oct 30 12:36:10.564417 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:10.564363 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b4543f5-ebb4-4ebc-94f8-fa451750afa6-workload-certs\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-pzxkx\" (UID: \"8b4543f5-ebb4-4ebc-94f8-fa451750afa6\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-pzxkx" Oct 30 12:36:10.665728 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:10.665685 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b4543f5-ebb4-4ebc-94f8-fa451750afa6-istiod-ca-cert\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-pzxkx\" (UID: \"8b4543f5-ebb4-4ebc-94f8-fa451750afa6\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-pzxkx" Oct 30 12:36:10.665728 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:10.665730 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b4543f5-ebb4-4ebc-94f8-fa451750afa6-istio-data\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-pzxkx\" (UID: \"8b4543f5-ebb4-4ebc-94f8-fa451750afa6\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-pzxkx" Oct 30 12:36:10.665959 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:10.665754 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b4543f5-ebb4-4ebc-94f8-fa451750afa6-istio-token\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-pzxkx\" (UID: \"8b4543f5-ebb4-4ebc-94f8-fa451750afa6\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-pzxkx" Oct 30 12:36:10.665959 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:10.665919 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b4543f5-ebb4-4ebc-94f8-fa451750afa6-workload-socket\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-pzxkx\" (UID: \"8b4543f5-ebb4-4ebc-94f8-fa451750afa6\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-pzxkx" Oct 30 12:36:10.665959 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:10.665946 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfvth\" (UniqueName: \"kubernetes.io/projected/8b4543f5-ebb4-4ebc-94f8-fa451750afa6-kube-api-access-lfvth\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-pzxkx\" (UID: \"8b4543f5-ebb4-4ebc-94f8-fa451750afa6\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-pzxkx" Oct 30 12:36:10.666121 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:10.665976 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b4543f5-ebb4-4ebc-94f8-fa451750afa6-workload-certs\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-pzxkx\" (UID: \"8b4543f5-ebb4-4ebc-94f8-fa451750afa6\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-pzxkx" Oct 30 12:36:10.666121 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:10.666063 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b4543f5-ebb4-4ebc-94f8-fa451750afa6-istio-envoy\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-pzxkx\" (UID: \"8b4543f5-ebb4-4ebc-94f8-fa451750afa6\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-pzxkx" Oct 30 12:36:10.666235 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:10.666119 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b4543f5-ebb4-4ebc-94f8-fa451750afa6-istio-data\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-pzxkx\" (UID: \"8b4543f5-ebb4-4ebc-94f8-fa451750afa6\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-pzxkx" Oct 30 12:36:10.666296 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:10.666276 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b4543f5-ebb4-4ebc-94f8-fa451750afa6-istio-podinfo\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-pzxkx\" (UID: \"8b4543f5-ebb4-4ebc-94f8-fa451750afa6\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-pzxkx" Oct 30 12:36:10.666356 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:10.666342 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b4543f5-ebb4-4ebc-94f8-fa451750afa6-credential-socket\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-pzxkx\" (UID: \"8b4543f5-ebb4-4ebc-94f8-fa451750afa6\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-pzxkx" Oct 30 12:36:10.666859 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:10.666460 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b4543f5-ebb4-4ebc-94f8-fa451750afa6-workload-socket\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-pzxkx\" (UID: \"8b4543f5-ebb4-4ebc-94f8-fa451750afa6\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-pzxkx" Oct 30 12:36:10.666859 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:10.666686 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b4543f5-ebb4-4ebc-94f8-fa451750afa6-credential-socket\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-pzxkx\" (UID: \"8b4543f5-ebb4-4ebc-94f8-fa451750afa6\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-pzxkx" Oct 30 12:36:10.666859 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:10.666808 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b4543f5-ebb4-4ebc-94f8-fa451750afa6-istiod-ca-cert\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-pzxkx\" (UID: \"8b4543f5-ebb4-4ebc-94f8-fa451750afa6\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-pzxkx" Oct 30 12:36:10.667089 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:10.666934 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b4543f5-ebb4-4ebc-94f8-fa451750afa6-workload-certs\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-pzxkx\" (UID: \"8b4543f5-ebb4-4ebc-94f8-fa451750afa6\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-pzxkx" Oct 30 12:36:10.668775 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:10.668743 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b4543f5-ebb4-4ebc-94f8-fa451750afa6-istio-envoy\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-pzxkx\" (UID: \"8b4543f5-ebb4-4ebc-94f8-fa451750afa6\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-pzxkx" Oct 30 12:36:10.669417 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:10.669393 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b4543f5-ebb4-4ebc-94f8-fa451750afa6-istio-podinfo\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-pzxkx\" (UID: \"8b4543f5-ebb4-4ebc-94f8-fa451750afa6\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-pzxkx" Oct 30 12:36:10.688130 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:10.688105 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b4543f5-ebb4-4ebc-94f8-fa451750afa6-istio-token\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-pzxkx\" (UID: \"8b4543f5-ebb4-4ebc-94f8-fa451750afa6\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-pzxkx" Oct 30 12:36:10.688410 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:10.688388 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfvth\" (UniqueName: \"kubernetes.io/projected/8b4543f5-ebb4-4ebc-94f8-fa451750afa6-kube-api-access-lfvth\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-pzxkx\" (UID: \"8b4543f5-ebb4-4ebc-94f8-fa451750afa6\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-pzxkx" Oct 30 12:36:10.795279 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:10.795242 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-pzxkx" Oct 30 12:36:11.149970 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:11.149939 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-pzxkx"] Oct 30 12:36:11.150199 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:36:11.150177 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b4543f5_ebb4_4ebc_94f8_fa451750afa6.slice/crio-2a2269ede944158044bc0b2f6fdb63daf2080094f8075741188772671269e8db WatchSource:0}: Error finding container 2a2269ede944158044bc0b2f6fdb63daf2080094f8075741188772671269e8db: Status 404 returned error can't find the container with id 2a2269ede944158044bc0b2f6fdb63daf2080094f8075741188772671269e8db Oct 30 12:36:11.152805 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:11.152767 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:36:11.152950 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:11.152875 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:36:11.152950 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:11.152924 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:36:11.832407 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:11.832250 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-pzxkx" event={"ID":"8b4543f5-ebb4-4ebc-94f8-fa451750afa6","Type":"ContainerStarted","Data":"fe633ac50288577e8ff75c53d3a4ce7a91bce30d42c38ed3407dd57504d013d9"} Oct 30 12:36:11.832407 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:11.832315 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-pzxkx" event={"ID":"8b4543f5-ebb4-4ebc-94f8-fa451750afa6","Type":"ContainerStarted","Data":"2a2269ede944158044bc0b2f6fdb63daf2080094f8075741188772671269e8db"} Oct 30 12:36:11.906881 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:11.904419 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-pzxkx" podStartSLOduration=1.904396336 podStartE2EDuration="1.904396336s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:36:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:36:11.890284173 +0000 UTC m=+2528.404712919" watchObservedRunningTime="2026-10-30 12:36:11.904396336 +0000 UTC m=+2528.418825082" Oct 30 12:36:12.797855 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:12.797145 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-pzxkx" Oct 30 12:36:12.800313 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:12.800274 4908 patch_prober.go:29] interesting pod/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-pzxkx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" start-of-body= Oct 30 12:36:12.800578 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:12.800534 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-pzxkx" podUID="8b4543f5-ebb4-4ebc-94f8-fa451750afa6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" Oct 30 12:36:13.795747 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:13.795712 4908 patch_prober.go:29] interesting pod/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-pzxkx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" start-of-body= Oct 30 12:36:13.796151 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:13.795764 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-pzxkx" podUID="8b4543f5-ebb4-4ebc-94f8-fa451750afa6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" Oct 30 12:36:14.796286 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:14.796249 4908 patch_prober.go:29] interesting pod/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-pzxkx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" start-of-body= Oct 30 12:36:14.796821 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:14.796323 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-pzxkx" podUID="8b4543f5-ebb4-4ebc-94f8-fa451750afa6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" Oct 30 12:36:15.796429 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:15.796387 4908 patch_prober.go:29] interesting pod/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-pzxkx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" start-of-body= Oct 30 12:36:15.796941 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:15.796457 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-pzxkx" podUID="8b4543f5-ebb4-4ebc-94f8-fa451750afa6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" Oct 30 12:36:16.796037 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:16.796003 4908 patch_prober.go:29] interesting pod/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-pzxkx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" start-of-body= Oct 30 12:36:16.796250 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:16.796056 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-pzxkx" podUID="8b4543f5-ebb4-4ebc-94f8-fa451750afa6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" Oct 30 12:36:17.151663 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:17.151578 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-2675c0/auth-test-model-2675c0-kserve-6bdcd4c665-qspzk" Oct 30 12:36:17.193846 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:17.193795 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-2675c0/auth-test-model-2675c0-kserve-6bdcd4c665-qspzk" Oct 30 12:36:17.800540 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:17.800511 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-pzxkx" Oct 30 12:36:17.800969 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:17.800950 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-pzxkx" Oct 30 12:36:17.801712 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:17.801688 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-pzxkx" Oct 30 12:36:17.959026 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:17.958993 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-2675c0/auth-test-model-2675c0-kserve-6bdcd4c665-qspzk" Oct 30 12:36:19.032028 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:19.031992 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-34a210df-s2hkm"] Oct 30 12:36:19.041971 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:19.041944 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-34a210df-s2hkm"] Oct 30 12:36:19.606393 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:19.606364 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8898760f-3f4d-4a4d-bb30-69744a87e78f" path="/var/lib/kubelet/pods/8898760f-3f4d-4a4d-bb30-69744a87e78f/volumes" Oct 30 12:36:19.728502 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:36:19.728460 4908 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vn6vv" containerName="extractor" Oct 30 12:36:30.376212 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:30.376172 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w2-812990/auth-test-model-812990-kserve-849f9f7d57-w28mb"] Oct 30 12:36:30.382374 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:30.382347 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-812990/auth-test-model-812990-kserve-849f9f7d57-w28mb" Oct 30 12:36:30.401682 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:30.401653 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-812990/auth-test-model-812990-kserve-849f9f7d57-w28mb"] Oct 30 12:36:30.407573 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:30.407543 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-812990\"/\"auth-test-model-812990-kserve-self-signed-certs\"" Oct 30 12:36:30.407895 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:30.407873 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-812990\"/\"openshift-service-ca.crt\"" Oct 30 12:36:30.408193 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:30.408169 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-812990\"/\"kube-root-ca.crt\"" Oct 30 12:36:30.408329 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:30.408313 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-812990\"/\"default-dockercfg-5jbbp\"" Oct 30 12:36:30.433525 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:30.433499 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d547033-e804-49e8-b329-b96c8ee1c2b1-dshm\") pod \"auth-test-model-812990-kserve-849f9f7d57-w28mb\" (UID: \"5d547033-e804-49e8-b329-b96c8ee1c2b1\") " pod="ai-tenant-e2e-shared-a-w2-812990/auth-test-model-812990-kserve-849f9f7d57-w28mb" Oct 30 12:36:30.433626 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:30.433537 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d547033-e804-49e8-b329-b96c8ee1c2b1-tmp-dir\") pod \"auth-test-model-812990-kserve-849f9f7d57-w28mb\" (UID: \"5d547033-e804-49e8-b329-b96c8ee1c2b1\") " pod="ai-tenant-e2e-shared-a-w2-812990/auth-test-model-812990-kserve-849f9f7d57-w28mb" Oct 30 12:36:30.433707 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:30.433651 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzllg\" (UniqueName: \"kubernetes.io/projected/5d547033-e804-49e8-b329-b96c8ee1c2b1-kube-api-access-wzllg\") pod \"auth-test-model-812990-kserve-849f9f7d57-w28mb\" (UID: \"5d547033-e804-49e8-b329-b96c8ee1c2b1\") " pod="ai-tenant-e2e-shared-a-w2-812990/auth-test-model-812990-kserve-849f9f7d57-w28mb" Oct 30 12:36:30.433769 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:30.433715 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d547033-e804-49e8-b329-b96c8ee1c2b1-model-cache\") pod \"auth-test-model-812990-kserve-849f9f7d57-w28mb\" (UID: \"5d547033-e804-49e8-b329-b96c8ee1c2b1\") " pod="ai-tenant-e2e-shared-a-w2-812990/auth-test-model-812990-kserve-849f9f7d57-w28mb" Oct 30 12:36:30.433873 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:30.433856 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d547033-e804-49e8-b329-b96c8ee1c2b1-home\") pod \"auth-test-model-812990-kserve-849f9f7d57-w28mb\" (UID: \"5d547033-e804-49e8-b329-b96c8ee1c2b1\") " pod="ai-tenant-e2e-shared-a-w2-812990/auth-test-model-812990-kserve-849f9f7d57-w28mb" Oct 30 12:36:30.433988 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:30.433974 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d547033-e804-49e8-b329-b96c8ee1c2b1-tls-certs\") pod \"auth-test-model-812990-kserve-849f9f7d57-w28mb\" (UID: \"5d547033-e804-49e8-b329-b96c8ee1c2b1\") " pod="ai-tenant-e2e-shared-a-w2-812990/auth-test-model-812990-kserve-849f9f7d57-w28mb" Oct 30 12:36:30.535374 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:30.535334 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d547033-e804-49e8-b329-b96c8ee1c2b1-dshm\") pod \"auth-test-model-812990-kserve-849f9f7d57-w28mb\" (UID: \"5d547033-e804-49e8-b329-b96c8ee1c2b1\") " pod="ai-tenant-e2e-shared-a-w2-812990/auth-test-model-812990-kserve-849f9f7d57-w28mb" Oct 30 12:36:30.535374 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:30.535381 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d547033-e804-49e8-b329-b96c8ee1c2b1-tmp-dir\") pod \"auth-test-model-812990-kserve-849f9f7d57-w28mb\" (UID: \"5d547033-e804-49e8-b329-b96c8ee1c2b1\") " pod="ai-tenant-e2e-shared-a-w2-812990/auth-test-model-812990-kserve-849f9f7d57-w28mb" Oct 30 12:36:30.535663 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:30.535428 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzllg\" (UniqueName: \"kubernetes.io/projected/5d547033-e804-49e8-b329-b96c8ee1c2b1-kube-api-access-wzllg\") pod \"auth-test-model-812990-kserve-849f9f7d57-w28mb\" (UID: \"5d547033-e804-49e8-b329-b96c8ee1c2b1\") " pod="ai-tenant-e2e-shared-a-w2-812990/auth-test-model-812990-kserve-849f9f7d57-w28mb" Oct 30 12:36:30.535663 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:30.535479 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d547033-e804-49e8-b329-b96c8ee1c2b1-model-cache\") pod \"auth-test-model-812990-kserve-849f9f7d57-w28mb\" (UID: \"5d547033-e804-49e8-b329-b96c8ee1c2b1\") " pod="ai-tenant-e2e-shared-a-w2-812990/auth-test-model-812990-kserve-849f9f7d57-w28mb" Oct 30 12:36:30.535663 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:30.535564 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d547033-e804-49e8-b329-b96c8ee1c2b1-home\") pod \"auth-test-model-812990-kserve-849f9f7d57-w28mb\" (UID: \"5d547033-e804-49e8-b329-b96c8ee1c2b1\") " pod="ai-tenant-e2e-shared-a-w2-812990/auth-test-model-812990-kserve-849f9f7d57-w28mb" Oct 30 12:36:30.535875 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:30.535682 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d547033-e804-49e8-b329-b96c8ee1c2b1-tls-certs\") pod \"auth-test-model-812990-kserve-849f9f7d57-w28mb\" (UID: \"5d547033-e804-49e8-b329-b96c8ee1c2b1\") " pod="ai-tenant-e2e-shared-a-w2-812990/auth-test-model-812990-kserve-849f9f7d57-w28mb" Oct 30 12:36:30.535875 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:30.535863 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d547033-e804-49e8-b329-b96c8ee1c2b1-tmp-dir\") pod \"auth-test-model-812990-kserve-849f9f7d57-w28mb\" (UID: \"5d547033-e804-49e8-b329-b96c8ee1c2b1\") " pod="ai-tenant-e2e-shared-a-w2-812990/auth-test-model-812990-kserve-849f9f7d57-w28mb" Oct 30 12:36:30.536062 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:30.536043 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d547033-e804-49e8-b329-b96c8ee1c2b1-model-cache\") pod \"auth-test-model-812990-kserve-849f9f7d57-w28mb\" (UID: \"5d547033-e804-49e8-b329-b96c8ee1c2b1\") " pod="ai-tenant-e2e-shared-a-w2-812990/auth-test-model-812990-kserve-849f9f7d57-w28mb" Oct 30 12:36:30.536158 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:30.536135 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d547033-e804-49e8-b329-b96c8ee1c2b1-home\") pod \"auth-test-model-812990-kserve-849f9f7d57-w28mb\" (UID: \"5d547033-e804-49e8-b329-b96c8ee1c2b1\") " pod="ai-tenant-e2e-shared-a-w2-812990/auth-test-model-812990-kserve-849f9f7d57-w28mb" Oct 30 12:36:30.538490 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:30.538464 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d547033-e804-49e8-b329-b96c8ee1c2b1-dshm\") pod \"auth-test-model-812990-kserve-849f9f7d57-w28mb\" (UID: \"5d547033-e804-49e8-b329-b96c8ee1c2b1\") " pod="ai-tenant-e2e-shared-a-w2-812990/auth-test-model-812990-kserve-849f9f7d57-w28mb" Oct 30 12:36:30.539027 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:30.539004 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d547033-e804-49e8-b329-b96c8ee1c2b1-tls-certs\") pod \"auth-test-model-812990-kserve-849f9f7d57-w28mb\" (UID: \"5d547033-e804-49e8-b329-b96c8ee1c2b1\") " pod="ai-tenant-e2e-shared-a-w2-812990/auth-test-model-812990-kserve-849f9f7d57-w28mb" Oct 30 12:36:30.559372 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:30.559347 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzllg\" (UniqueName: \"kubernetes.io/projected/5d547033-e804-49e8-b329-b96c8ee1c2b1-kube-api-access-wzllg\") pod \"auth-test-model-812990-kserve-849f9f7d57-w28mb\" (UID: \"5d547033-e804-49e8-b329-b96c8ee1c2b1\") " pod="ai-tenant-e2e-shared-a-w2-812990/auth-test-model-812990-kserve-849f9f7d57-w28mb" Oct 30 12:36:30.726638 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:30.726261 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-812990/auth-test-model-812990-kserve-849f9f7d57-w28mb" Oct 30 12:36:31.091076 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:36:31.091036 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5d547033_e804_49e8_b329_b96c8ee1c2b1.slice/crio-4f5399777e4899e06749446eea5bd45b97ae9615b9dcaae008d1949287bd28be WatchSource:0}: Error finding container 4f5399777e4899e06749446eea5bd45b97ae9615b9dcaae008d1949287bd28be: Status 404 returned error can't find the container with id 4f5399777e4899e06749446eea5bd45b97ae9615b9dcaae008d1949287bd28be Oct 30 12:36:31.091076 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:31.091049 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-812990/auth-test-model-812990-kserve-849f9f7d57-w28mb"] Oct 30 12:36:31.147396 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:31.147371 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-812990/auth-test-model-812990-kserve-849f9f7d57-w28mb" event={"ID":"5d547033-e804-49e8-b329-b96c8ee1c2b1","Type":"ContainerStarted","Data":"4f5399777e4899e06749446eea5bd45b97ae9615b9dcaae008d1949287bd28be"} Oct 30 12:36:32.159309 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:32.159254 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-812990/auth-test-model-812990-kserve-849f9f7d57-w28mb" event={"ID":"5d547033-e804-49e8-b329-b96c8ee1c2b1","Type":"ContainerStarted","Data":"b0b9a92cec6f91aa0278818ee74303bdac7ccc339ba4136464ab4b46a6698505"} Oct 30 12:36:32.204052 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:32.203995 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w2-812990/auth-test-model-812990-kserve-849f9f7d57-w28mb" podStartSLOduration=2.203980685 podStartE2EDuration="2.203980685s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:36:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:36:32.19861059 +0000 UTC m=+2548.713039355" watchObservedRunningTime="2026-10-30 12:36:32.203980685 +0000 UTC m=+2548.718409428" Oct 30 12:36:32.409895 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:32.404039 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9wwfc"] Oct 30 12:36:32.727670 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:32.727642 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-812990/auth-test-model-812990-kserve-849f9f7d57-w28mb" Oct 30 12:36:33.171693 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:33.171612 4908 generic.go:356] "Generic (PLEG): container finished" podID="5526e66b-b972-4e0f-9a11-d9a9b8fdf0d1" containerID="121bdcccd0dfbb5ad3e498b082c7ba5eeb2ee0f946a00128883ab8b2f3b68dc6" exitCode=0 Oct 30 12:36:33.171693 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:33.171679 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-5d90d8-openshift-default-577df8c9d9-cjd27" event={"ID":"5526e66b-b972-4e0f-9a11-d9a9b8fdf0d1","Type":"ContainerDied","Data":"121bdcccd0dfbb5ad3e498b082c7ba5eeb2ee0f946a00128883ab8b2f3b68dc6"} Oct 30 12:36:33.172251 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:33.172233 4908 scope.go:122] "RemoveContainer" containerID="121bdcccd0dfbb5ad3e498b082c7ba5eeb2ee0f946a00128883ab8b2f3b68dc6" Oct 30 12:36:33.174028 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:33.173999 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:36:33.174152 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:33.174084 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:36:33.174152 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:33.174132 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:36:34.195922 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:34.195881 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-5d90d8-openshift-default-577df8c9d9-cjd27" event={"ID":"5526e66b-b972-4e0f-9a11-d9a9b8fdf0d1","Type":"ContainerStarted","Data":"8ef34bcb2159c7b9304fb55ce105e3d72d971d08763b9e9a6e7f00ef08275b93"} Oct 30 12:36:34.385850 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:34.385160 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-5d90d8-openshift-default-577df8c9d9-cjd27" Oct 30 12:36:34.385850 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:34.385279 4908 patch_prober.go:29] interesting pod/e2e-explicit-5d90d8-openshift-default-577df8c9d9-cjd27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" start-of-body= Oct 30 12:36:34.385850 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:34.385344 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-5d90d8-openshift-default-577df8c9d9-cjd27" podUID="5526e66b-b972-4e0f-9a11-d9a9b8fdf0d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" Oct 30 12:36:35.388851 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:35.388150 4908 patch_prober.go:29] interesting pod/e2e-explicit-5d90d8-openshift-default-577df8c9d9-cjd27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" start-of-body= Oct 30 12:36:35.388851 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:35.388226 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-5d90d8-openshift-default-577df8c9d9-cjd27" podUID="5526e66b-b972-4e0f-9a11-d9a9b8fdf0d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" Oct 30 12:36:36.033768 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:36.033734 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-264ed6ef-jvzzz"] Oct 30 12:36:36.043438 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:36.042811 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-264ed6ef-jvzzz"] Oct 30 12:36:36.384865 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:36.384224 4908 patch_prober.go:29] interesting pod/e2e-explicit-5d90d8-openshift-default-577df8c9d9-cjd27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" start-of-body= Oct 30 12:36:36.384865 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:36.384301 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-5d90d8-openshift-default-577df8c9d9-cjd27" podUID="5526e66b-b972-4e0f-9a11-d9a9b8fdf0d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" Oct 30 12:36:36.491369 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:36.491330 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w1-0bda67/auth-test-model-0bda67-kserve-686fd886d5-kngvr"] Oct 30 12:36:36.500281 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:36.500251 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-0bda67/auth-test-model-0bda67-kserve-686fd886d5-kngvr" Oct 30 12:36:36.527819 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:36.527775 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-0bda67/auth-test-model-0bda67-kserve-686fd886d5-kngvr"] Oct 30 12:36:36.532813 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:36.532791 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-0bda67\"/\"default-dockercfg-lgvqc\"" Oct 30 12:36:36.533926 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:36.533047 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-0bda67\"/\"openshift-service-ca.crt\"" Oct 30 12:36:36.534063 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:36.533178 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-0bda67\"/\"auth-test-model-0bda67-kserve-self-signed-certs\"" Oct 30 12:36:36.534159 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:36.533264 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-0bda67\"/\"kube-root-ca.crt\"" Oct 30 12:36:36.638855 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:36.638736 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3577ee1a-5102-41e3-99b2-03dc9618ae0b-dshm\") pod \"auth-test-model-0bda67-kserve-686fd886d5-kngvr\" (UID: \"3577ee1a-5102-41e3-99b2-03dc9618ae0b\") " pod="ai-tenant-e2e-shared-a-w1-0bda67/auth-test-model-0bda67-kserve-686fd886d5-kngvr" Oct 30 12:36:36.638855 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:36.638789 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3577ee1a-5102-41e3-99b2-03dc9618ae0b-model-cache\") pod \"auth-test-model-0bda67-kserve-686fd886d5-kngvr\" (UID: \"3577ee1a-5102-41e3-99b2-03dc9618ae0b\") " pod="ai-tenant-e2e-shared-a-w1-0bda67/auth-test-model-0bda67-kserve-686fd886d5-kngvr" Oct 30 12:36:36.639069 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:36.639022 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3577ee1a-5102-41e3-99b2-03dc9618ae0b-tmp-dir\") pod \"auth-test-model-0bda67-kserve-686fd886d5-kngvr\" (UID: \"3577ee1a-5102-41e3-99b2-03dc9618ae0b\") " pod="ai-tenant-e2e-shared-a-w1-0bda67/auth-test-model-0bda67-kserve-686fd886d5-kngvr" Oct 30 12:36:36.639136 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:36.639119 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3577ee1a-5102-41e3-99b2-03dc9618ae0b-home\") pod \"auth-test-model-0bda67-kserve-686fd886d5-kngvr\" (UID: \"3577ee1a-5102-41e3-99b2-03dc9618ae0b\") " pod="ai-tenant-e2e-shared-a-w1-0bda67/auth-test-model-0bda67-kserve-686fd886d5-kngvr" Oct 30 12:36:36.639215 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:36.639194 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkg2f\" (UniqueName: \"kubernetes.io/projected/3577ee1a-5102-41e3-99b2-03dc9618ae0b-kube-api-access-dkg2f\") pod \"auth-test-model-0bda67-kserve-686fd886d5-kngvr\" (UID: \"3577ee1a-5102-41e3-99b2-03dc9618ae0b\") " pod="ai-tenant-e2e-shared-a-w1-0bda67/auth-test-model-0bda67-kserve-686fd886d5-kngvr" Oct 30 12:36:36.639314 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:36.639276 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3577ee1a-5102-41e3-99b2-03dc9618ae0b-tls-certs\") pod \"auth-test-model-0bda67-kserve-686fd886d5-kngvr\" (UID: \"3577ee1a-5102-41e3-99b2-03dc9618ae0b\") " pod="ai-tenant-e2e-shared-a-w1-0bda67/auth-test-model-0bda67-kserve-686fd886d5-kngvr" Oct 30 12:36:36.740308 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:36.740268 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3577ee1a-5102-41e3-99b2-03dc9618ae0b-tmp-dir\") pod \"auth-test-model-0bda67-kserve-686fd886d5-kngvr\" (UID: \"3577ee1a-5102-41e3-99b2-03dc9618ae0b\") " pod="ai-tenant-e2e-shared-a-w1-0bda67/auth-test-model-0bda67-kserve-686fd886d5-kngvr" Oct 30 12:36:36.740507 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:36.740364 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3577ee1a-5102-41e3-99b2-03dc9618ae0b-home\") pod \"auth-test-model-0bda67-kserve-686fd886d5-kngvr\" (UID: \"3577ee1a-5102-41e3-99b2-03dc9618ae0b\") " pod="ai-tenant-e2e-shared-a-w1-0bda67/auth-test-model-0bda67-kserve-686fd886d5-kngvr" Oct 30 12:36:36.740507 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:36.740425 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkg2f\" (UniqueName: \"kubernetes.io/projected/3577ee1a-5102-41e3-99b2-03dc9618ae0b-kube-api-access-dkg2f\") pod \"auth-test-model-0bda67-kserve-686fd886d5-kngvr\" (UID: \"3577ee1a-5102-41e3-99b2-03dc9618ae0b\") " pod="ai-tenant-e2e-shared-a-w1-0bda67/auth-test-model-0bda67-kserve-686fd886d5-kngvr" Oct 30 12:36:36.740507 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:36.740465 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3577ee1a-5102-41e3-99b2-03dc9618ae0b-tls-certs\") pod \"auth-test-model-0bda67-kserve-686fd886d5-kngvr\" (UID: \"3577ee1a-5102-41e3-99b2-03dc9618ae0b\") " pod="ai-tenant-e2e-shared-a-w1-0bda67/auth-test-model-0bda67-kserve-686fd886d5-kngvr" Oct 30 12:36:36.740760 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:36.740620 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3577ee1a-5102-41e3-99b2-03dc9618ae0b-dshm\") pod \"auth-test-model-0bda67-kserve-686fd886d5-kngvr\" (UID: \"3577ee1a-5102-41e3-99b2-03dc9618ae0b\") " pod="ai-tenant-e2e-shared-a-w1-0bda67/auth-test-model-0bda67-kserve-686fd886d5-kngvr" Oct 30 12:36:36.740760 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:36.740653 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3577ee1a-5102-41e3-99b2-03dc9618ae0b-model-cache\") pod \"auth-test-model-0bda67-kserve-686fd886d5-kngvr\" (UID: \"3577ee1a-5102-41e3-99b2-03dc9618ae0b\") " pod="ai-tenant-e2e-shared-a-w1-0bda67/auth-test-model-0bda67-kserve-686fd886d5-kngvr" Oct 30 12:36:36.740760 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:36.740712 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3577ee1a-5102-41e3-99b2-03dc9618ae0b-tmp-dir\") pod \"auth-test-model-0bda67-kserve-686fd886d5-kngvr\" (UID: \"3577ee1a-5102-41e3-99b2-03dc9618ae0b\") " pod="ai-tenant-e2e-shared-a-w1-0bda67/auth-test-model-0bda67-kserve-686fd886d5-kngvr" Oct 30 12:36:36.740760 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:36.740725 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3577ee1a-5102-41e3-99b2-03dc9618ae0b-home\") pod \"auth-test-model-0bda67-kserve-686fd886d5-kngvr\" (UID: \"3577ee1a-5102-41e3-99b2-03dc9618ae0b\") " pod="ai-tenant-e2e-shared-a-w1-0bda67/auth-test-model-0bda67-kserve-686fd886d5-kngvr" Oct 30 12:36:36.741106 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:36.741083 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3577ee1a-5102-41e3-99b2-03dc9618ae0b-model-cache\") pod \"auth-test-model-0bda67-kserve-686fd886d5-kngvr\" (UID: \"3577ee1a-5102-41e3-99b2-03dc9618ae0b\") " pod="ai-tenant-e2e-shared-a-w1-0bda67/auth-test-model-0bda67-kserve-686fd886d5-kngvr" Oct 30 12:36:36.743519 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:36.743485 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3577ee1a-5102-41e3-99b2-03dc9618ae0b-dshm\") pod \"auth-test-model-0bda67-kserve-686fd886d5-kngvr\" (UID: \"3577ee1a-5102-41e3-99b2-03dc9618ae0b\") " pod="ai-tenant-e2e-shared-a-w1-0bda67/auth-test-model-0bda67-kserve-686fd886d5-kngvr" Oct 30 12:36:36.743853 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:36.743813 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3577ee1a-5102-41e3-99b2-03dc9618ae0b-tls-certs\") pod \"auth-test-model-0bda67-kserve-686fd886d5-kngvr\" (UID: \"3577ee1a-5102-41e3-99b2-03dc9618ae0b\") " pod="ai-tenant-e2e-shared-a-w1-0bda67/auth-test-model-0bda67-kserve-686fd886d5-kngvr" Oct 30 12:36:36.770862 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:36.769541 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkg2f\" (UniqueName: \"kubernetes.io/projected/3577ee1a-5102-41e3-99b2-03dc9618ae0b-kube-api-access-dkg2f\") pod \"auth-test-model-0bda67-kserve-686fd886d5-kngvr\" (UID: \"3577ee1a-5102-41e3-99b2-03dc9618ae0b\") " pod="ai-tenant-e2e-shared-a-w1-0bda67/auth-test-model-0bda67-kserve-686fd886d5-kngvr" Oct 30 12:36:36.847851 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:36.843300 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-0bda67/auth-test-model-0bda67-kserve-686fd886d5-kngvr" Oct 30 12:36:37.294144 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:36:37.294106 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3577ee1a_5102_41e3_99b2_03dc9618ae0b.slice/crio-c5fbde98ba89424aeee3171677e41ffee1bfdd090e88c505d17e40cdd247b90d WatchSource:0}: Error finding container c5fbde98ba89424aeee3171677e41ffee1bfdd090e88c505d17e40cdd247b90d: Status 404 returned error can't find the container with id c5fbde98ba89424aeee3171677e41ffee1bfdd090e88c505d17e40cdd247b90d Oct 30 12:36:37.294699 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:37.294676 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-0bda67/auth-test-model-0bda67-kserve-686fd886d5-kngvr"] Oct 30 12:36:37.389850 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:37.389668 4908 patch_prober.go:29] interesting pod/e2e-explicit-5d90d8-openshift-default-577df8c9d9-cjd27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" start-of-body= Oct 30 12:36:37.389850 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:37.389719 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-5d90d8-openshift-default-577df8c9d9-cjd27" podUID="5526e66b-b972-4e0f-9a11-d9a9b8fdf0d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" Oct 30 12:36:37.608118 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:37.608041 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6fe995aa-83ea-4d67-b827-780415870c80" path="/var/lib/kubelet/pods/6fe995aa-83ea-4d67-b827-780415870c80/volumes" Oct 30 12:36:38.304265 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:38.304223 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-0bda67/auth-test-model-0bda67-kserve-686fd886d5-kngvr" event={"ID":"3577ee1a-5102-41e3-99b2-03dc9618ae0b","Type":"ContainerStarted","Data":"dddddc5235b7fc4ff7f05bd97ed2adcd2c67a24af57a8df200c45abb9f8d855b"} Oct 30 12:36:38.304446 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:38.304273 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-0bda67/auth-test-model-0bda67-kserve-686fd886d5-kngvr" event={"ID":"3577ee1a-5102-41e3-99b2-03dc9618ae0b","Type":"ContainerStarted","Data":"c5fbde98ba89424aeee3171677e41ffee1bfdd090e88c505d17e40cdd247b90d"} Oct 30 12:36:38.347218 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:38.347173 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w1-0bda67/auth-test-model-0bda67-kserve-686fd886d5-kngvr" podStartSLOduration=2.34715986 podStartE2EDuration="2.34715986s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:36:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:36:38.342160861 +0000 UTC m=+2554.856589626" watchObservedRunningTime="2026-10-30 12:36:38.34715986 +0000 UTC m=+2554.861588631" Oct 30 12:36:38.385204 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:38.385170 4908 patch_prober.go:29] interesting pod/e2e-explicit-5d90d8-openshift-default-577df8c9d9-cjd27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" start-of-body= Oct 30 12:36:38.385349 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:38.385233 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-5d90d8-openshift-default-577df8c9d9-cjd27" podUID="5526e66b-b972-4e0f-9a11-d9a9b8fdf0d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" Oct 30 12:36:38.843776 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:38.843743 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w1-0bda67/auth-test-model-0bda67-kserve-686fd886d5-kngvr" Oct 30 12:36:39.383998 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:39.383966 4908 patch_prober.go:29] interesting pod/e2e-explicit-5d90d8-openshift-default-577df8c9d9-cjd27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" start-of-body= Oct 30 12:36:39.384205 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:39.384016 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-5d90d8-openshift-default-577df8c9d9-cjd27" podUID="5526e66b-b972-4e0f-9a11-d9a9b8fdf0d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" Oct 30 12:36:40.389268 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:40.389233 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-5d90d8-openshift-default-577df8c9d9-cjd27" Oct 30 12:36:40.389748 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:40.389734 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-5d90d8-openshift-default-577df8c9d9-cjd27" Oct 30 12:36:40.390385 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:40.390362 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-5d90d8-openshift-default-577df8c9d9-cjd27" Oct 30 12:36:40.727488 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:40.727458 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w2-812990/auth-test-model-812990-kserve-849f9f7d57-w28mb" Oct 30 12:36:40.741395 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:40.741369 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w2-812990/auth-test-model-812990-kserve-849f9f7d57-w28mb" Oct 30 12:36:41.343115 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:41.343086 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-812990/auth-test-model-812990-kserve-849f9f7d57-w28mb" Oct 30 12:36:46.844128 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:46.844053 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w1-0bda67/auth-test-model-0bda67-kserve-686fd886d5-kngvr" Oct 30 12:36:46.856660 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:46.856632 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w1-0bda67/auth-test-model-0bda67-kserve-686fd886d5-kngvr" Oct 30 12:36:47.401716 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:47.401681 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w1-0bda67/auth-test-model-0bda67-kserve-686fd886d5-kngvr" Oct 30 12:36:49.037533 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:49.037484 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f4d3590-7lc6p"] Oct 30 12:36:49.048327 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:49.048296 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f4d3590-7lc6p"] Oct 30 12:36:49.606946 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:49.606907 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ace46019-d978-4707-ab77-e1b4dc45dc32" path="/var/lib/kubelet/pods/ace46019-d978-4707-ab77-e1b4dc45dc32/volumes" Oct 30 12:36:49.685505 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:36:49.685467 4908 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vn6vv" containerName="extractor" Oct 30 12:36:52.092466 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:52.092430 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-5d90d8/explicit-model-5d90d8-kserve-5c8b447959-6ngsf"] Oct 30 12:36:52.099309 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:52.099287 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-5d90d8/explicit-model-5d90d8-kserve-5c8b447959-6ngsf" Oct 30 12:36:52.103956 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:52.103931 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-5d90d8/explicit-model-5d90d8-kserve-5c8b447959-6ngsf"] Oct 30 12:36:52.113168 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:52.113144 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-5d90d8\"/\"default-dockercfg-kmz6r\"" Oct 30 12:36:52.124334 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:52.124312 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-5d90d8\"/\"explicit-model-5d90d8-kserve-self-signed-certs\"" Oct 30 12:36:52.149088 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:52.149064 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-5d90d8\"/\"openshift-service-ca.crt\"" Oct 30 12:36:52.149204 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:52.149184 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-5d90d8\"/\"kube-root-ca.crt\"" Oct 30 12:36:52.210976 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:52.210939 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7r9w\" (UniqueName: \"kubernetes.io/projected/34857a8b-c29b-43f0-8b2c-561cb869947c-kube-api-access-t7r9w\") pod \"explicit-model-5d90d8-kserve-5c8b447959-6ngsf\" (UID: \"34857a8b-c29b-43f0-8b2c-561cb869947c\") " pod="ai-tenant-e2e-explicit-5d90d8/explicit-model-5d90d8-kserve-5c8b447959-6ngsf" Oct 30 12:36:52.211174 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:52.211024 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34857a8b-c29b-43f0-8b2c-561cb869947c-dshm\") pod \"explicit-model-5d90d8-kserve-5c8b447959-6ngsf\" (UID: \"34857a8b-c29b-43f0-8b2c-561cb869947c\") " pod="ai-tenant-e2e-explicit-5d90d8/explicit-model-5d90d8-kserve-5c8b447959-6ngsf" Oct 30 12:36:52.211174 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:52.211145 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34857a8b-c29b-43f0-8b2c-561cb869947c-tmp-dir\") pod \"explicit-model-5d90d8-kserve-5c8b447959-6ngsf\" (UID: \"34857a8b-c29b-43f0-8b2c-561cb869947c\") " pod="ai-tenant-e2e-explicit-5d90d8/explicit-model-5d90d8-kserve-5c8b447959-6ngsf" Oct 30 12:36:52.211317 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:52.211194 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34857a8b-c29b-43f0-8b2c-561cb869947c-tls-certs\") pod \"explicit-model-5d90d8-kserve-5c8b447959-6ngsf\" (UID: \"34857a8b-c29b-43f0-8b2c-561cb869947c\") " pod="ai-tenant-e2e-explicit-5d90d8/explicit-model-5d90d8-kserve-5c8b447959-6ngsf" Oct 30 12:36:52.211381 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:52.211335 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34857a8b-c29b-43f0-8b2c-561cb869947c-home\") pod \"explicit-model-5d90d8-kserve-5c8b447959-6ngsf\" (UID: \"34857a8b-c29b-43f0-8b2c-561cb869947c\") " pod="ai-tenant-e2e-explicit-5d90d8/explicit-model-5d90d8-kserve-5c8b447959-6ngsf" Oct 30 12:36:52.211493 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:52.211467 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34857a8b-c29b-43f0-8b2c-561cb869947c-model-cache\") pod \"explicit-model-5d90d8-kserve-5c8b447959-6ngsf\" (UID: \"34857a8b-c29b-43f0-8b2c-561cb869947c\") " pod="ai-tenant-e2e-explicit-5d90d8/explicit-model-5d90d8-kserve-5c8b447959-6ngsf" Oct 30 12:36:52.312907 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:52.312862 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7r9w\" (UniqueName: \"kubernetes.io/projected/34857a8b-c29b-43f0-8b2c-561cb869947c-kube-api-access-t7r9w\") pod \"explicit-model-5d90d8-kserve-5c8b447959-6ngsf\" (UID: \"34857a8b-c29b-43f0-8b2c-561cb869947c\") " pod="ai-tenant-e2e-explicit-5d90d8/explicit-model-5d90d8-kserve-5c8b447959-6ngsf" Oct 30 12:36:52.313120 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:52.312939 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34857a8b-c29b-43f0-8b2c-561cb869947c-dshm\") pod \"explicit-model-5d90d8-kserve-5c8b447959-6ngsf\" (UID: \"34857a8b-c29b-43f0-8b2c-561cb869947c\") " pod="ai-tenant-e2e-explicit-5d90d8/explicit-model-5d90d8-kserve-5c8b447959-6ngsf" Oct 30 12:36:52.313120 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:52.313001 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34857a8b-c29b-43f0-8b2c-561cb869947c-tmp-dir\") pod \"explicit-model-5d90d8-kserve-5c8b447959-6ngsf\" (UID: \"34857a8b-c29b-43f0-8b2c-561cb869947c\") " pod="ai-tenant-e2e-explicit-5d90d8/explicit-model-5d90d8-kserve-5c8b447959-6ngsf" Oct 30 12:36:52.313120 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:52.313030 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34857a8b-c29b-43f0-8b2c-561cb869947c-tls-certs\") pod \"explicit-model-5d90d8-kserve-5c8b447959-6ngsf\" (UID: \"34857a8b-c29b-43f0-8b2c-561cb869947c\") " pod="ai-tenant-e2e-explicit-5d90d8/explicit-model-5d90d8-kserve-5c8b447959-6ngsf" Oct 30 12:36:52.313283 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:52.313128 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34857a8b-c29b-43f0-8b2c-561cb869947c-home\") pod \"explicit-model-5d90d8-kserve-5c8b447959-6ngsf\" (UID: \"34857a8b-c29b-43f0-8b2c-561cb869947c\") " pod="ai-tenant-e2e-explicit-5d90d8/explicit-model-5d90d8-kserve-5c8b447959-6ngsf" Oct 30 12:36:52.313283 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:52.313211 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34857a8b-c29b-43f0-8b2c-561cb869947c-model-cache\") pod \"explicit-model-5d90d8-kserve-5c8b447959-6ngsf\" (UID: \"34857a8b-c29b-43f0-8b2c-561cb869947c\") " pod="ai-tenant-e2e-explicit-5d90d8/explicit-model-5d90d8-kserve-5c8b447959-6ngsf" Oct 30 12:36:52.313688 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:52.313659 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34857a8b-c29b-43f0-8b2c-561cb869947c-home\") pod \"explicit-model-5d90d8-kserve-5c8b447959-6ngsf\" (UID: \"34857a8b-c29b-43f0-8b2c-561cb869947c\") " pod="ai-tenant-e2e-explicit-5d90d8/explicit-model-5d90d8-kserve-5c8b447959-6ngsf" Oct 30 12:36:52.313881 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:52.313851 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34857a8b-c29b-43f0-8b2c-561cb869947c-tmp-dir\") pod \"explicit-model-5d90d8-kserve-5c8b447959-6ngsf\" (UID: \"34857a8b-c29b-43f0-8b2c-561cb869947c\") " pod="ai-tenant-e2e-explicit-5d90d8/explicit-model-5d90d8-kserve-5c8b447959-6ngsf" Oct 30 12:36:52.314025 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:52.313674 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34857a8b-c29b-43f0-8b2c-561cb869947c-model-cache\") pod \"explicit-model-5d90d8-kserve-5c8b447959-6ngsf\" (UID: \"34857a8b-c29b-43f0-8b2c-561cb869947c\") " pod="ai-tenant-e2e-explicit-5d90d8/explicit-model-5d90d8-kserve-5c8b447959-6ngsf" Oct 30 12:36:52.323540 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:52.323512 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34857a8b-c29b-43f0-8b2c-561cb869947c-tls-certs\") pod \"explicit-model-5d90d8-kserve-5c8b447959-6ngsf\" (UID: \"34857a8b-c29b-43f0-8b2c-561cb869947c\") " pod="ai-tenant-e2e-explicit-5d90d8/explicit-model-5d90d8-kserve-5c8b447959-6ngsf" Oct 30 12:36:52.342687 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:52.342576 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7r9w\" (UniqueName: \"kubernetes.io/projected/34857a8b-c29b-43f0-8b2c-561cb869947c-kube-api-access-t7r9w\") pod \"explicit-model-5d90d8-kserve-5c8b447959-6ngsf\" (UID: \"34857a8b-c29b-43f0-8b2c-561cb869947c\") " pod="ai-tenant-e2e-explicit-5d90d8/explicit-model-5d90d8-kserve-5c8b447959-6ngsf" Oct 30 12:36:52.368271 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:52.368226 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34857a8b-c29b-43f0-8b2c-561cb869947c-dshm\") pod \"explicit-model-5d90d8-kserve-5c8b447959-6ngsf\" (UID: \"34857a8b-c29b-43f0-8b2c-561cb869947c\") " pod="ai-tenant-e2e-explicit-5d90d8/explicit-model-5d90d8-kserve-5c8b447959-6ngsf" Oct 30 12:36:52.421378 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:52.421104 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-5d90d8/explicit-model-5d90d8-kserve-5c8b447959-6ngsf" Oct 30 12:36:52.796054 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:52.796022 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-5d90d8/explicit-model-5d90d8-kserve-5c8b447959-6ngsf"] Oct 30 12:36:52.797116 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:36:52.797089 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod34857a8b_c29b_43f0_8b2c_561cb869947c.slice/crio-87695e419fb935638815b8fff05e174aa701b9d70bf63c335cce1f5ff7c7d23c WatchSource:0}: Error finding container 87695e419fb935638815b8fff05e174aa701b9d70bf63c335cce1f5ff7c7d23c: Status 404 returned error can't find the container with id 87695e419fb935638815b8fff05e174aa701b9d70bf63c335cce1f5ff7c7d23c Oct 30 12:36:53.517592 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:53.517550 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-5d90d8/explicit-model-5d90d8-kserve-5c8b447959-6ngsf" event={"ID":"34857a8b-c29b-43f0-8b2c-561cb869947c","Type":"ContainerStarted","Data":"5f8dfce082e8ba8928ab045b744c39a8cd3e33f9092e8ea83ce5cdd194f7dabe"} Oct 30 12:36:53.518024 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:53.517604 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-5d90d8/explicit-model-5d90d8-kserve-5c8b447959-6ngsf" event={"ID":"34857a8b-c29b-43f0-8b2c-561cb869947c","Type":"ContainerStarted","Data":"87695e419fb935638815b8fff05e174aa701b9d70bf63c335cce1f5ff7c7d23c"} Oct 30 12:36:53.562858 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:53.562801 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-5d90d8/explicit-model-5d90d8-kserve-5c8b447959-6ngsf" podStartSLOduration=1.5627873700000001 podStartE2EDuration="1.56278737s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:36:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:36:53.556545911 +0000 UTC m=+2570.070974662" watchObservedRunningTime="2026-10-30 12:36:53.56278737 +0000 UTC m=+2570.077216127" Oct 30 12:36:54.423376 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:54.423345 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-5d90d8/explicit-model-5d90d8-kserve-5c8b447959-6ngsf" Oct 30 12:36:55.031024 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:55.030990 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9933f47-s6tcr"] Oct 30 12:36:55.041611 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:55.041584 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9933f47-s6tcr"] Oct 30 12:36:55.593477 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:55.593437 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-4t45f"] Oct 30 12:36:55.598490 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:55.598462 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-4t45f" Oct 30 12:36:55.611350 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:55.611318 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="452ea436-3165-416d-93c1-6563af3cd263" path="/var/lib/kubelet/pods/452ea436-3165-416d-93c1-6563af3cd263/volumes" Oct 30 12:36:55.624301 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:55.624271 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-4t45f"] Oct 30 12:36:55.665888 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:55.665849 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/866116c2-e247-4ac0-9af7-72fadb1d5c31-istio-data\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-4t45f\" (UID: \"866116c2-e247-4ac0-9af7-72fadb1d5c31\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-4t45f" Oct 30 12:36:55.666073 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:55.665929 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/866116c2-e247-4ac0-9af7-72fadb1d5c31-istiod-ca-cert\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-4t45f\" (UID: \"866116c2-e247-4ac0-9af7-72fadb1d5c31\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-4t45f" Oct 30 12:36:55.666073 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:55.666060 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/866116c2-e247-4ac0-9af7-72fadb1d5c31-istio-envoy\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-4t45f\" (UID: \"866116c2-e247-4ac0-9af7-72fadb1d5c31\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-4t45f" Oct 30 12:36:55.666216 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:55.666120 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znhkn\" (UniqueName: \"kubernetes.io/projected/866116c2-e247-4ac0-9af7-72fadb1d5c31-kube-api-access-znhkn\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-4t45f\" (UID: \"866116c2-e247-4ac0-9af7-72fadb1d5c31\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-4t45f" Oct 30 12:36:55.666216 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:55.666149 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/866116c2-e247-4ac0-9af7-72fadb1d5c31-workload-socket\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-4t45f\" (UID: \"866116c2-e247-4ac0-9af7-72fadb1d5c31\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-4t45f" Oct 30 12:36:55.666216 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:55.666195 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/866116c2-e247-4ac0-9af7-72fadb1d5c31-istio-podinfo\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-4t45f\" (UID: \"866116c2-e247-4ac0-9af7-72fadb1d5c31\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-4t45f" Oct 30 12:36:55.666882 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:55.666508 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/866116c2-e247-4ac0-9af7-72fadb1d5c31-credential-socket\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-4t45f\" (UID: \"866116c2-e247-4ac0-9af7-72fadb1d5c31\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-4t45f" Oct 30 12:36:55.666989 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:55.666915 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/866116c2-e247-4ac0-9af7-72fadb1d5c31-istio-token\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-4t45f\" (UID: \"866116c2-e247-4ac0-9af7-72fadb1d5c31\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-4t45f" Oct 30 12:36:55.667111 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:55.667093 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/866116c2-e247-4ac0-9af7-72fadb1d5c31-workload-certs\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-4t45f\" (UID: \"866116c2-e247-4ac0-9af7-72fadb1d5c31\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-4t45f" Oct 30 12:36:55.782726 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:55.782679 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/866116c2-e247-4ac0-9af7-72fadb1d5c31-workload-certs\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-4t45f\" (UID: \"866116c2-e247-4ac0-9af7-72fadb1d5c31\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-4t45f" Oct 30 12:36:55.783092 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:55.783065 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/866116c2-e247-4ac0-9af7-72fadb1d5c31-istio-data\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-4t45f\" (UID: \"866116c2-e247-4ac0-9af7-72fadb1d5c31\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-4t45f" Oct 30 12:36:55.783301 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:55.783286 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/866116c2-e247-4ac0-9af7-72fadb1d5c31-istiod-ca-cert\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-4t45f\" (UID: \"866116c2-e247-4ac0-9af7-72fadb1d5c31\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-4t45f" Oct 30 12:36:55.783490 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:55.783475 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/866116c2-e247-4ac0-9af7-72fadb1d5c31-istio-envoy\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-4t45f\" (UID: \"866116c2-e247-4ac0-9af7-72fadb1d5c31\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-4t45f" Oct 30 12:36:55.783640 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:55.783622 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znhkn\" (UniqueName: \"kubernetes.io/projected/866116c2-e247-4ac0-9af7-72fadb1d5c31-kube-api-access-znhkn\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-4t45f\" (UID: \"866116c2-e247-4ac0-9af7-72fadb1d5c31\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-4t45f" Oct 30 12:36:55.783744 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:55.783732 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/866116c2-e247-4ac0-9af7-72fadb1d5c31-workload-socket\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-4t45f\" (UID: \"866116c2-e247-4ac0-9af7-72fadb1d5c31\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-4t45f" Oct 30 12:36:55.783867 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:55.783852 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/866116c2-e247-4ac0-9af7-72fadb1d5c31-istio-podinfo\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-4t45f\" (UID: \"866116c2-e247-4ac0-9af7-72fadb1d5c31\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-4t45f" Oct 30 12:36:55.784005 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:55.783986 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/866116c2-e247-4ac0-9af7-72fadb1d5c31-credential-socket\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-4t45f\" (UID: \"866116c2-e247-4ac0-9af7-72fadb1d5c31\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-4t45f" Oct 30 12:36:55.784134 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:55.784118 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/866116c2-e247-4ac0-9af7-72fadb1d5c31-istio-token\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-4t45f\" (UID: \"866116c2-e247-4ac0-9af7-72fadb1d5c31\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-4t45f" Oct 30 12:36:55.809849 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:55.785070 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/866116c2-e247-4ac0-9af7-72fadb1d5c31-workload-certs\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-4t45f\" (UID: \"866116c2-e247-4ac0-9af7-72fadb1d5c31\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-4t45f" Oct 30 12:36:55.810633 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:55.785408 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/866116c2-e247-4ac0-9af7-72fadb1d5c31-istio-data\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-4t45f\" (UID: \"866116c2-e247-4ac0-9af7-72fadb1d5c31\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-4t45f" Oct 30 12:36:55.810867 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:55.786192 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/866116c2-e247-4ac0-9af7-72fadb1d5c31-istiod-ca-cert\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-4t45f\" (UID: \"866116c2-e247-4ac0-9af7-72fadb1d5c31\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-4t45f" Oct 30 12:36:55.811016 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:55.792730 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/866116c2-e247-4ac0-9af7-72fadb1d5c31-istio-envoy\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-4t45f\" (UID: \"866116c2-e247-4ac0-9af7-72fadb1d5c31\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-4t45f" Oct 30 12:36:55.811128 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:55.792987 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/866116c2-e247-4ac0-9af7-72fadb1d5c31-credential-socket\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-4t45f\" (UID: \"866116c2-e247-4ac0-9af7-72fadb1d5c31\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-4t45f" Oct 30 12:36:55.811226 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:55.793248 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/866116c2-e247-4ac0-9af7-72fadb1d5c31-workload-socket\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-4t45f\" (UID: \"866116c2-e247-4ac0-9af7-72fadb1d5c31\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-4t45f" Oct 30 12:36:55.811314 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:55.797440 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/866116c2-e247-4ac0-9af7-72fadb1d5c31-istio-podinfo\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-4t45f\" (UID: \"866116c2-e247-4ac0-9af7-72fadb1d5c31\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-4t45f" Oct 30 12:36:55.811411 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:55.810506 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/866116c2-e247-4ac0-9af7-72fadb1d5c31-istio-token\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-4t45f\" (UID: \"866116c2-e247-4ac0-9af7-72fadb1d5c31\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-4t45f" Oct 30 12:36:55.825850 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:55.818814 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-znhkn\" (UniqueName: \"kubernetes.io/projected/866116c2-e247-4ac0-9af7-72fadb1d5c31-kube-api-access-znhkn\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-4t45f\" (UID: \"866116c2-e247-4ac0-9af7-72fadb1d5c31\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-4t45f" Oct 30 12:36:55.928780 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:55.928695 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-4t45f" Oct 30 12:36:56.253864 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:56.253822 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-4t45f"] Oct 30 12:36:56.255811 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:36:56.255780 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod866116c2_e247_4ac0_9af7_72fadb1d5c31.slice/crio-9691bf79baf35af330c47ce01ebb6674e115c356c8690bd63e09e489d7a86bbe WatchSource:0}: Error finding container 9691bf79baf35af330c47ce01ebb6674e115c356c8690bd63e09e489d7a86bbe: Status 404 returned error can't find the container with id 9691bf79baf35af330c47ce01ebb6674e115c356c8690bd63e09e489d7a86bbe Oct 30 12:36:56.258545 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:56.258513 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:36:56.258652 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:56.258576 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:36:56.258652 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:56.258604 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:36:56.546528 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:56.546440 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-4t45f" event={"ID":"866116c2-e247-4ac0-9af7-72fadb1d5c31","Type":"ContainerStarted","Data":"3acad7e3b6b9e387e2a22c8720e16f2e28039762db8bc1c8c76d7a0083cf4fe8"} Oct 30 12:36:56.546528 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:56.546499 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-4t45f" event={"ID":"866116c2-e247-4ac0-9af7-72fadb1d5c31","Type":"ContainerStarted","Data":"9691bf79baf35af330c47ce01ebb6674e115c356c8690bd63e09e489d7a86bbe"} Oct 30 12:36:56.595106 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:56.595024 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-4t45f" podStartSLOduration=1.595005835 podStartE2EDuration="1.595005835s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:36:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:36:56.588678652 +0000 UTC m=+2573.103107435" watchObservedRunningTime="2026-10-30 12:36:56.595005835 +0000 UTC m=+2573.109434583" Oct 30 12:36:56.929530 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:56.929453 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-4t45f" Oct 30 12:36:57.930906 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:57.930871 4908 patch_prober.go:29] interesting pod/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-4t45f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body= Oct 30 12:36:57.931403 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:57.930923 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-4t45f" podUID="866116c2-e247-4ac0-9af7-72fadb1d5c31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" Oct 30 12:36:58.929161 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:58.929126 4908 patch_prober.go:29] interesting pod/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-4t45f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body= Oct 30 12:36:58.929329 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:58.929183 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-4t45f" podUID="866116c2-e247-4ac0-9af7-72fadb1d5c31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" Oct 30 12:36:59.929176 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:59.929137 4908 patch_prober.go:29] interesting pod/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-4t45f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body= Oct 30 12:36:59.929640 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:36:59.929199 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-4t45f" podUID="866116c2-e247-4ac0-9af7-72fadb1d5c31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" Oct 30 12:37:00.929349 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:00.929306 4908 patch_prober.go:29] interesting pod/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-4t45f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body= Oct 30 12:37:00.929749 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:00.929372 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-4t45f" podUID="866116c2-e247-4ac0-9af7-72fadb1d5c31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" Oct 30 12:37:01.929436 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:01.929399 4908 patch_prober.go:29] interesting pod/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-4t45f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body= Oct 30 12:37:01.929869 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:01.929481 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-4t45f" podUID="866116c2-e247-4ac0-9af7-72fadb1d5c31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" Oct 30 12:37:02.035250 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:02.035216 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a32072a-9gttn"] Oct 30 12:37:02.042577 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:02.042549 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a32072a-9gttn"] Oct 30 12:37:02.422626 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:02.422592 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-5d90d8/explicit-model-5d90d8-kserve-5c8b447959-6ngsf" Oct 30 12:37:02.435652 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:02.435628 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-5d90d8/explicit-model-5d90d8-kserve-5c8b447959-6ngsf" Oct 30 12:37:02.714126 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:02.714048 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-5d90d8/explicit-model-5d90d8-kserve-5c8b447959-6ngsf" Oct 30 12:37:02.934409 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:02.934374 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-4t45f" Oct 30 12:37:02.934928 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:02.934750 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-4t45f" Oct 30 12:37:02.935535 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:02.935514 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-4t45f" Oct 30 12:37:03.610843 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:03.610792 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f16f29d-fe4d-4fbd-91d7-c07a188222b7" path="/var/lib/kubelet/pods/8f16f29d-fe4d-4fbd-91d7-c07a188222b7/volumes" Oct 30 12:37:05.031674 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:05.031639 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-539475a0-b9vp6"] Oct 30 12:37:05.043238 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:05.042966 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-539475a0-b9vp6"] Oct 30 12:37:05.607731 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:05.607692 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68c31061-4f8a-434e-863f-e1199f5b7c1b" path="/var/lib/kubelet/pods/68c31061-4f8a-434e-863f-e1199f5b7c1b/volumes" Oct 30 12:37:05.639208 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:05.639173 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w4-327e31/auth-test-model-327e31-kserve-587f6fcd8f-r8vm4"] Oct 30 12:37:05.645761 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:05.645737 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-327e31/auth-test-model-327e31-kserve-587f6fcd8f-r8vm4" Oct 30 12:37:05.658157 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:05.658133 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-327e31/auth-test-model-327e31-kserve-587f6fcd8f-r8vm4"] Oct 30 12:37:05.664079 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:05.664051 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-327e31\"/\"default-dockercfg-pk8xt\"" Oct 30 12:37:05.664204 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:05.664080 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-327e31\"/\"openshift-service-ca.crt\"" Oct 30 12:37:05.664322 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:05.664304 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-327e31\"/\"kube-root-ca.crt\"" Oct 30 12:37:05.672942 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:05.672894 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-327e31\"/\"auth-test-model-327e31-kserve-self-signed-certs\"" Oct 30 12:37:05.759704 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:05.759664 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zc8g2\" (UniqueName: \"kubernetes.io/projected/f69baca9-270e-4b25-a96d-a771ffc429a8-kube-api-access-zc8g2\") pod \"auth-test-model-327e31-kserve-587f6fcd8f-r8vm4\" (UID: \"f69baca9-270e-4b25-a96d-a771ffc429a8\") " pod="ai-tenant-e2e-shared-a-w4-327e31/auth-test-model-327e31-kserve-587f6fcd8f-r8vm4" Oct 30 12:37:05.759981 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:05.759863 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f69baca9-270e-4b25-a96d-a771ffc429a8-model-cache\") pod \"auth-test-model-327e31-kserve-587f6fcd8f-r8vm4\" (UID: \"f69baca9-270e-4b25-a96d-a771ffc429a8\") " pod="ai-tenant-e2e-shared-a-w4-327e31/auth-test-model-327e31-kserve-587f6fcd8f-r8vm4" Oct 30 12:37:05.760063 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:05.760007 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f69baca9-270e-4b25-a96d-a771ffc429a8-home\") pod \"auth-test-model-327e31-kserve-587f6fcd8f-r8vm4\" (UID: \"f69baca9-270e-4b25-a96d-a771ffc429a8\") " pod="ai-tenant-e2e-shared-a-w4-327e31/auth-test-model-327e31-kserve-587f6fcd8f-r8vm4" Oct 30 12:37:05.760127 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:05.760103 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f69baca9-270e-4b25-a96d-a771ffc429a8-tmp-dir\") pod \"auth-test-model-327e31-kserve-587f6fcd8f-r8vm4\" (UID: \"f69baca9-270e-4b25-a96d-a771ffc429a8\") " pod="ai-tenant-e2e-shared-a-w4-327e31/auth-test-model-327e31-kserve-587f6fcd8f-r8vm4" Oct 30 12:37:05.760252 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:05.760229 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f69baca9-270e-4b25-a96d-a771ffc429a8-dshm\") pod \"auth-test-model-327e31-kserve-587f6fcd8f-r8vm4\" (UID: \"f69baca9-270e-4b25-a96d-a771ffc429a8\") " pod="ai-tenant-e2e-shared-a-w4-327e31/auth-test-model-327e31-kserve-587f6fcd8f-r8vm4" Oct 30 12:37:05.760320 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:05.760264 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f69baca9-270e-4b25-a96d-a771ffc429a8-tls-certs\") pod \"auth-test-model-327e31-kserve-587f6fcd8f-r8vm4\" (UID: \"f69baca9-270e-4b25-a96d-a771ffc429a8\") " pod="ai-tenant-e2e-shared-a-w4-327e31/auth-test-model-327e31-kserve-587f6fcd8f-r8vm4" Oct 30 12:37:05.867776 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:05.867675 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f69baca9-270e-4b25-a96d-a771ffc429a8-dshm\") pod \"auth-test-model-327e31-kserve-587f6fcd8f-r8vm4\" (UID: \"f69baca9-270e-4b25-a96d-a771ffc429a8\") " pod="ai-tenant-e2e-shared-a-w4-327e31/auth-test-model-327e31-kserve-587f6fcd8f-r8vm4" Oct 30 12:37:05.868466 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:05.868443 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f69baca9-270e-4b25-a96d-a771ffc429a8-tls-certs\") pod \"auth-test-model-327e31-kserve-587f6fcd8f-r8vm4\" (UID: \"f69baca9-270e-4b25-a96d-a771ffc429a8\") " pod="ai-tenant-e2e-shared-a-w4-327e31/auth-test-model-327e31-kserve-587f6fcd8f-r8vm4" Oct 30 12:37:05.868679 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:05.868660 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zc8g2\" (UniqueName: \"kubernetes.io/projected/f69baca9-270e-4b25-a96d-a771ffc429a8-kube-api-access-zc8g2\") pod \"auth-test-model-327e31-kserve-587f6fcd8f-r8vm4\" (UID: \"f69baca9-270e-4b25-a96d-a771ffc429a8\") " pod="ai-tenant-e2e-shared-a-w4-327e31/auth-test-model-327e31-kserve-587f6fcd8f-r8vm4" Oct 30 12:37:05.868907 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:05.868892 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f69baca9-270e-4b25-a96d-a771ffc429a8-model-cache\") pod \"auth-test-model-327e31-kserve-587f6fcd8f-r8vm4\" (UID: \"f69baca9-270e-4b25-a96d-a771ffc429a8\") " pod="ai-tenant-e2e-shared-a-w4-327e31/auth-test-model-327e31-kserve-587f6fcd8f-r8vm4" Oct 30 12:37:05.869146 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:05.869129 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f69baca9-270e-4b25-a96d-a771ffc429a8-home\") pod \"auth-test-model-327e31-kserve-587f6fcd8f-r8vm4\" (UID: \"f69baca9-270e-4b25-a96d-a771ffc429a8\") " pod="ai-tenant-e2e-shared-a-w4-327e31/auth-test-model-327e31-kserve-587f6fcd8f-r8vm4" Oct 30 12:37:05.869331 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:05.869313 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f69baca9-270e-4b25-a96d-a771ffc429a8-tmp-dir\") pod \"auth-test-model-327e31-kserve-587f6fcd8f-r8vm4\" (UID: \"f69baca9-270e-4b25-a96d-a771ffc429a8\") " pod="ai-tenant-e2e-shared-a-w4-327e31/auth-test-model-327e31-kserve-587f6fcd8f-r8vm4" Oct 30 12:37:05.869941 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:05.869918 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f69baca9-270e-4b25-a96d-a771ffc429a8-tmp-dir\") pod \"auth-test-model-327e31-kserve-587f6fcd8f-r8vm4\" (UID: \"f69baca9-270e-4b25-a96d-a771ffc429a8\") " pod="ai-tenant-e2e-shared-a-w4-327e31/auth-test-model-327e31-kserve-587f6fcd8f-r8vm4" Oct 30 12:37:05.884065 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:05.876181 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f69baca9-270e-4b25-a96d-a771ffc429a8-home\") pod \"auth-test-model-327e31-kserve-587f6fcd8f-r8vm4\" (UID: \"f69baca9-270e-4b25-a96d-a771ffc429a8\") " pod="ai-tenant-e2e-shared-a-w4-327e31/auth-test-model-327e31-kserve-587f6fcd8f-r8vm4" Oct 30 12:37:05.884247 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:05.876797 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f69baca9-270e-4b25-a96d-a771ffc429a8-model-cache\") pod \"auth-test-model-327e31-kserve-587f6fcd8f-r8vm4\" (UID: \"f69baca9-270e-4b25-a96d-a771ffc429a8\") " pod="ai-tenant-e2e-shared-a-w4-327e31/auth-test-model-327e31-kserve-587f6fcd8f-r8vm4" Oct 30 12:37:05.884364 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:05.883955 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f69baca9-270e-4b25-a96d-a771ffc429a8-dshm\") pod \"auth-test-model-327e31-kserve-587f6fcd8f-r8vm4\" (UID: \"f69baca9-270e-4b25-a96d-a771ffc429a8\") " pod="ai-tenant-e2e-shared-a-w4-327e31/auth-test-model-327e31-kserve-587f6fcd8f-r8vm4" Oct 30 12:37:05.909861 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:05.901315 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f69baca9-270e-4b25-a96d-a771ffc429a8-tls-certs\") pod \"auth-test-model-327e31-kserve-587f6fcd8f-r8vm4\" (UID: \"f69baca9-270e-4b25-a96d-a771ffc429a8\") " pod="ai-tenant-e2e-shared-a-w4-327e31/auth-test-model-327e31-kserve-587f6fcd8f-r8vm4" Oct 30 12:37:05.925044 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:05.925011 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zc8g2\" (UniqueName: \"kubernetes.io/projected/f69baca9-270e-4b25-a96d-a771ffc429a8-kube-api-access-zc8g2\") pod \"auth-test-model-327e31-kserve-587f6fcd8f-r8vm4\" (UID: \"f69baca9-270e-4b25-a96d-a771ffc429a8\") " pod="ai-tenant-e2e-shared-a-w4-327e31/auth-test-model-327e31-kserve-587f6fcd8f-r8vm4" Oct 30 12:37:05.967422 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:05.967112 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-327e31/auth-test-model-327e31-kserve-587f6fcd8f-r8vm4" Oct 30 12:37:06.277437 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:06.277402 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9wwfc"] Oct 30 12:37:06.325737 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:06.325706 4908 scope.go:122] "RemoveContainer" containerID="de5ed5804305f8cc691e17d69815d55fced91baf12befade01b898e662ecae53" Oct 30 12:37:06.335795 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:06.335768 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-327e31/auth-test-model-327e31-kserve-587f6fcd8f-r8vm4"] Oct 30 12:37:06.337990 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:06.337968 4908 scope.go:122] "RemoveContainer" containerID="44db5df2fb480b1dada1a43f5a4c9c79487c30adb602551a4d4d0e81861e70a8" Oct 30 12:37:06.339687 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:37:06.339664 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf69baca9_270e_4b25_a96d_a771ffc429a8.slice/crio-89ba5dbb380d8c684f342f2ad44455204f5d54f519b72c48ed74a1ac2bbf135f WatchSource:0}: Error finding container 89ba5dbb380d8c684f342f2ad44455204f5d54f519b72c48ed74a1ac2bbf135f: Status 404 returned error can't find the container with id 89ba5dbb380d8c684f342f2ad44455204f5d54f519b72c48ed74a1ac2bbf135f Oct 30 12:37:06.348203 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:06.348180 4908 scope.go:122] "RemoveContainer" containerID="0ce70383eebd9ee67e2385e6e92bc8729117ac8644d52c1934fc407301016acf" Oct 30 12:37:06.359883 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:06.359858 4908 scope.go:122] "RemoveContainer" containerID="dc96eb27757db022136d47e820181a67052a243a8e9dec4714e8c5ad365a8b6d" Oct 30 12:37:06.743082 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:06.743047 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-327e31/auth-test-model-327e31-kserve-587f6fcd8f-r8vm4" event={"ID":"f69baca9-270e-4b25-a96d-a771ffc429a8","Type":"ContainerStarted","Data":"8c11f5fadeed0c1a30749fb2a7b31b36f774caf860cdfbefc2770c765061d2f8"} Oct 30 12:37:06.743277 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:06.743100 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-327e31/auth-test-model-327e31-kserve-587f6fcd8f-r8vm4" event={"ID":"f69baca9-270e-4b25-a96d-a771ffc429a8","Type":"ContainerStarted","Data":"89ba5dbb380d8c684f342f2ad44455204f5d54f519b72c48ed74a1ac2bbf135f"} Oct 30 12:37:06.793037 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:06.792992 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w4-327e31/auth-test-model-327e31-kserve-587f6fcd8f-r8vm4" podStartSLOduration=1.792977255 podStartE2EDuration="1.792977255s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:37:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:37:06.787401128 +0000 UTC m=+2583.301829876" watchObservedRunningTime="2026-10-30 12:37:06.792977255 +0000 UTC m=+2583.307405998" Oct 30 12:37:06.980853 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:06.968940 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-327e31/auth-test-model-327e31-kserve-587f6fcd8f-r8vm4" Oct 30 12:37:12.029663 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:12.029627 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f894569-vs6m2"] Oct 30 12:37:12.040360 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:12.040333 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f894569-vs6m2"] Oct 30 12:37:13.608739 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:13.608704 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c090356f-b5ff-4678-aab5-db6f893b2cb1" path="/var/lib/kubelet/pods/c090356f-b5ff-4678-aab5-db6f893b2cb1/volumes" Oct 30 12:37:14.430598 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:14.430555 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-5d90d8/explicit-model-5d90d8-kserve-5c8b447959-6ngsf"] Oct 30 12:37:14.737128 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:14.737097 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-5d90d8-tt4p6"] Oct 30 12:37:14.742557 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:14.742536 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-5d90d8-tt4p6" Oct 30 12:37:14.761394 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:14.761366 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 12:37:14.761591 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:14.761568 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s75hq\"" Oct 30 12:37:14.762450 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:14.762428 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-5d90d8-tt4p6"] Oct 30 12:37:14.837597 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:14.833424 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 12:37:14.837597 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:14.833510 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 12:37:14.843484 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:14.843453 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gd678\" (UniqueName: \"kubernetes.io/projected/55581cb0-176e-4f35-8a10-a31bba1b60a4-kube-api-access-gd678\") pod \"maas-api-revoke-keys-e2e-explicit-5d90d8-tt4p6\" (UID: \"55581cb0-176e-4f35-8a10-a31bba1b60a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-5d90d8-tt4p6" Oct 30 12:37:14.853848 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:14.853647 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55581cb0-176e-4f35-8a10-a31bba1b60a4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-explicit-5d90d8-tt4p6\" (UID: \"55581cb0-176e-4f35-8a10-a31bba1b60a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-5d90d8-tt4p6" Oct 30 12:37:14.854082 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:14.854018 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55581cb0-176e-4f35-8a10-a31bba1b60a4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-explicit-5d90d8-tt4p6\" (UID: \"55581cb0-176e-4f35-8a10-a31bba1b60a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-5d90d8-tt4p6" Oct 30 12:37:14.954727 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:14.954694 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gd678\" (UniqueName: \"kubernetes.io/projected/55581cb0-176e-4f35-8a10-a31bba1b60a4-kube-api-access-gd678\") pod \"maas-api-revoke-keys-e2e-explicit-5d90d8-tt4p6\" (UID: \"55581cb0-176e-4f35-8a10-a31bba1b60a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-5d90d8-tt4p6" Oct 30 12:37:14.954899 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:14.954735 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55581cb0-176e-4f35-8a10-a31bba1b60a4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-explicit-5d90d8-tt4p6\" (UID: \"55581cb0-176e-4f35-8a10-a31bba1b60a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-5d90d8-tt4p6" Oct 30 12:37:14.955006 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:14.954988 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55581cb0-176e-4f35-8a10-a31bba1b60a4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-explicit-5d90d8-tt4p6\" (UID: \"55581cb0-176e-4f35-8a10-a31bba1b60a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-5d90d8-tt4p6" Oct 30 12:37:14.955594 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:14.955571 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55581cb0-176e-4f35-8a10-a31bba1b60a4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-explicit-5d90d8-tt4p6\" (UID: \"55581cb0-176e-4f35-8a10-a31bba1b60a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-5d90d8-tt4p6" Oct 30 12:37:14.957286 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:14.957264 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55581cb0-176e-4f35-8a10-a31bba1b60a4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-explicit-5d90d8-tt4p6\" (UID: \"55581cb0-176e-4f35-8a10-a31bba1b60a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-5d90d8-tt4p6" Oct 30 12:37:14.976792 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:14.976753 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gd678\" (UniqueName: \"kubernetes.io/projected/55581cb0-176e-4f35-8a10-a31bba1b60a4-kube-api-access-gd678\") pod \"maas-api-revoke-keys-e2e-explicit-5d90d8-tt4p6\" (UID: \"55581cb0-176e-4f35-8a10-a31bba1b60a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-5d90d8-tt4p6" Oct 30 12:37:15.057896 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:15.057824 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-5d90d8-tt4p6" Oct 30 12:37:15.362875 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:15.362842 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-5d90d8-tt4p6"] Oct 30 12:37:15.898487 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:15.898450 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-5d90d8-tt4p6" event={"ID":"55581cb0-176e-4f35-8a10-a31bba1b60a4","Type":"ContainerStarted","Data":"afeae25d85adb6805166772afe122092971f5d6d811cb082ee3b7dd28f40508f"} Oct 30 12:37:15.898933 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:15.898511 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-5d90d8-tt4p6" event={"ID":"55581cb0-176e-4f35-8a10-a31bba1b60a4","Type":"ContainerStarted","Data":"24250570b805d5f0792fc41ba6348c8251ae78d5beddc93b9a72b41249492e67"} Oct 30 12:37:15.938329 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:15.938242 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-5d90d8-tt4p6" podStartSLOduration=1.93822947 podStartE2EDuration="1.93822947s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:37:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:37:15.935582398 +0000 UTC m=+2592.450011143" watchObservedRunningTime="2026-10-30 12:37:15.93822947 +0000 UTC m=+2592.452658217" Oct 30 12:37:15.968026 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:15.968002 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w4-327e31/auth-test-model-327e31-kserve-587f6fcd8f-r8vm4" Oct 30 12:37:15.980751 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:15.980726 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w4-327e31/auth-test-model-327e31-kserve-587f6fcd8f-r8vm4" Oct 30 12:37:16.910032 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:16.909999 4908 generic.go:356] "Generic (PLEG): container finished" podID="55581cb0-176e-4f35-8a10-a31bba1b60a4" containerID="afeae25d85adb6805166772afe122092971f5d6d811cb082ee3b7dd28f40508f" exitCode=0 Oct 30 12:37:16.910551 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:16.910067 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-5d90d8-tt4p6" event={"ID":"55581cb0-176e-4f35-8a10-a31bba1b60a4","Type":"ContainerDied","Data":"afeae25d85adb6805166772afe122092971f5d6d811cb082ee3b7dd28f40508f"} Oct 30 12:37:16.927627 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:16.927601 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w4-327e31/auth-test-model-327e31-kserve-587f6fcd8f-r8vm4" Oct 30 12:37:18.106039 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:18.106007 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-5d90d8-tt4p6" Oct 30 12:37:18.203398 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:18.203319 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55581cb0-176e-4f35-8a10-a31bba1b60a4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55581cb0-176e-4f35-8a10-a31bba1b60a4-maas-api-openshift-ca\") pod \"55581cb0-176e-4f35-8a10-a31bba1b60a4\" (UID: \"55581cb0-176e-4f35-8a10-a31bba1b60a4\") " Oct 30 12:37:18.203650 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:18.203629 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55581cb0-176e-4f35-8a10-a31bba1b60a4-kube-api-access-gd678\" (UniqueName: \"kubernetes.io/projected/55581cb0-176e-4f35-8a10-a31bba1b60a4-kube-api-access-gd678\") pod \"55581cb0-176e-4f35-8a10-a31bba1b60a4\" (UID: \"55581cb0-176e-4f35-8a10-a31bba1b60a4\") " Oct 30 12:37:18.203783 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:18.203752 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/55581cb0-176e-4f35-8a10-a31bba1b60a4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55581cb0-176e-4f35-8a10-a31bba1b60a4-maas-api-xks-ca\") pod \"55581cb0-176e-4f35-8a10-a31bba1b60a4\" (UID: \"55581cb0-176e-4f35-8a10-a31bba1b60a4\") " Oct 30 12:37:18.205181 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:18.205156 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55581cb0-176e-4f35-8a10-a31bba1b60a4-maas-api-openshift-ca" pod "55581cb0-176e-4f35-8a10-a31bba1b60a4" (UID: "55581cb0-176e-4f35-8a10-a31bba1b60a4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:37:18.207554 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:18.207520 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55581cb0-176e-4f35-8a10-a31bba1b60a4-kube-api-access-gd678" pod "55581cb0-176e-4f35-8a10-a31bba1b60a4" (UID: "55581cb0-176e-4f35-8a10-a31bba1b60a4"). InnerVolumeSpecName "kube-api-access-gd678". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:37:18.207990 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:18.207960 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55581cb0-176e-4f35-8a10-a31bba1b60a4-maas-api-xks-ca" pod "55581cb0-176e-4f35-8a10-a31bba1b60a4" (UID: "55581cb0-176e-4f35-8a10-a31bba1b60a4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:37:18.323932 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:18.314983 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55581cb0-176e-4f35-8a10-a31bba1b60a4-maas-api-openshift-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:37:18.323932 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:18.315025 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gd678\" (UniqueName: \"kubernetes.io/projected/55581cb0-176e-4f35-8a10-a31bba1b60a4-kube-api-access-gd678\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:37:18.323932 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:18.315039 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55581cb0-176e-4f35-8a10-a31bba1b60a4-maas-api-xks-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:37:18.931942 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:18.931912 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-5d90d8-tt4p6" Oct 30 12:37:18.932109 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:18.931945 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-5d90d8-tt4p6" event={"ID":"55581cb0-176e-4f35-8a10-a31bba1b60a4","Type":"ContainerDied","Data":"24250570b805d5f0792fc41ba6348c8251ae78d5beddc93b9a72b41249492e67"} Oct 30 12:37:18.932109 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:18.931995 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="24250570b805d5f0792fc41ba6348c8251ae78d5beddc93b9a72b41249492e67" Oct 30 12:37:19.038560 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:19.038529 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00d131b1-sv5v8"] Oct 30 12:37:19.043094 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:19.043068 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00d131b1-sv5v8"] Oct 30 12:37:19.608220 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:19.608186 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="856d3aa7-ff0f-4df5-ab38-369fb6a8757d" path="/var/lib/kubelet/pods/856d3aa7-ff0f-4df5-ab38-369fb6a8757d/volumes" Oct 30 12:37:19.685214 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:37:19.685166 4908 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vn6vv" containerName="extractor" Oct 30 12:37:19.822005 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:19.821972 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-2s2dq"] Oct 30 12:37:19.822772 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:19.822753 4908 cpu_manager.go:414] "containerMap: removing container" podUID="55581cb0-176e-4f35-8a10-a31bba1b60a4" containerName="revoke-keys" Oct 30 12:37:19.822862 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:19.822774 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55581cb0-176e-4f35-8a10-a31bba1b60a4" containerName="revoke-keys" Oct 30 12:37:19.822912 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:19.822893 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="55581cb0-176e-4f35-8a10-a31bba1b60a4" containerName="revoke-keys" Oct 30 12:37:19.826798 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:19.826777 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-2s2dq" Oct 30 12:37:19.840869 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:19.840846 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-0bda67-openshift-default-dockercfg-7n4kl\"" Oct 30 12:37:19.858403 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:19.858337 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-2s2dq"] Oct 30 12:37:19.943415 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:19.943381 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3729561-3035-459d-a451-290595a76f5d-istio-token\") pod \"e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-2s2dq\" (UID: \"c3729561-3035-459d-a451-290595a76f5d\") " pod="openshift-ingress/e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-2s2dq" Oct 30 12:37:19.943628 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:19.943475 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3729561-3035-459d-a451-290595a76f5d-istio-data\") pod \"e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-2s2dq\" (UID: \"c3729561-3035-459d-a451-290595a76f5d\") " pod="openshift-ingress/e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-2s2dq" Oct 30 12:37:19.943628 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:19.943512 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3729561-3035-459d-a451-290595a76f5d-istio-podinfo\") pod \"e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-2s2dq\" (UID: \"c3729561-3035-459d-a451-290595a76f5d\") " pod="openshift-ingress/e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-2s2dq" Oct 30 12:37:19.943752 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:19.943620 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3729561-3035-459d-a451-290595a76f5d-istiod-ca-cert\") pod \"e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-2s2dq\" (UID: \"c3729561-3035-459d-a451-290595a76f5d\") " pod="openshift-ingress/e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-2s2dq" Oct 30 12:37:19.943752 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:19.943671 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3729561-3035-459d-a451-290595a76f5d-credential-socket\") pod \"e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-2s2dq\" (UID: \"c3729561-3035-459d-a451-290595a76f5d\") " pod="openshift-ingress/e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-2s2dq" Oct 30 12:37:19.943752 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:19.943714 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3729561-3035-459d-a451-290595a76f5d-workload-socket\") pod \"e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-2s2dq\" (UID: \"c3729561-3035-459d-a451-290595a76f5d\") " pod="openshift-ingress/e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-2s2dq" Oct 30 12:37:19.943928 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:19.943759 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wkv7\" (UniqueName: \"kubernetes.io/projected/c3729561-3035-459d-a451-290595a76f5d-kube-api-access-6wkv7\") pod \"e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-2s2dq\" (UID: \"c3729561-3035-459d-a451-290595a76f5d\") " pod="openshift-ingress/e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-2s2dq" Oct 30 12:37:19.943928 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:19.943907 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3729561-3035-459d-a451-290595a76f5d-workload-certs\") pod \"e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-2s2dq\" (UID: \"c3729561-3035-459d-a451-290595a76f5d\") " pod="openshift-ingress/e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-2s2dq" Oct 30 12:37:19.944033 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:19.943945 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3729561-3035-459d-a451-290595a76f5d-istio-envoy\") pod \"e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-2s2dq\" (UID: \"c3729561-3035-459d-a451-290595a76f5d\") " pod="openshift-ingress/e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-2s2dq" Oct 30 12:37:20.044933 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:20.044879 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3729561-3035-459d-a451-290595a76f5d-istio-data\") pod \"e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-2s2dq\" (UID: \"c3729561-3035-459d-a451-290595a76f5d\") " pod="openshift-ingress/e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-2s2dq" Oct 30 12:37:20.045191 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:20.044939 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3729561-3035-459d-a451-290595a76f5d-istio-podinfo\") pod \"e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-2s2dq\" (UID: \"c3729561-3035-459d-a451-290595a76f5d\") " pod="openshift-ingress/e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-2s2dq" Oct 30 12:37:20.045191 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:20.044986 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3729561-3035-459d-a451-290595a76f5d-istiod-ca-cert\") pod \"e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-2s2dq\" (UID: \"c3729561-3035-459d-a451-290595a76f5d\") " pod="openshift-ingress/e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-2s2dq" Oct 30 12:37:20.045191 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:20.045030 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3729561-3035-459d-a451-290595a76f5d-credential-socket\") pod \"e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-2s2dq\" (UID: \"c3729561-3035-459d-a451-290595a76f5d\") " pod="openshift-ingress/e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-2s2dq" Oct 30 12:37:20.045191 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:20.045077 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3729561-3035-459d-a451-290595a76f5d-workload-socket\") pod \"e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-2s2dq\" (UID: \"c3729561-3035-459d-a451-290595a76f5d\") " pod="openshift-ingress/e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-2s2dq" Oct 30 12:37:20.045191 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:20.045130 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wkv7\" (UniqueName: \"kubernetes.io/projected/c3729561-3035-459d-a451-290595a76f5d-kube-api-access-6wkv7\") pod \"e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-2s2dq\" (UID: \"c3729561-3035-459d-a451-290595a76f5d\") " pod="openshift-ingress/e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-2s2dq" Oct 30 12:37:20.045408 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:20.045262 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3729561-3035-459d-a451-290595a76f5d-workload-certs\") pod \"e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-2s2dq\" (UID: \"c3729561-3035-459d-a451-290595a76f5d\") " pod="openshift-ingress/e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-2s2dq" Oct 30 12:37:20.045408 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:20.045307 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3729561-3035-459d-a451-290595a76f5d-istio-envoy\") pod \"e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-2s2dq\" (UID: \"c3729561-3035-459d-a451-290595a76f5d\") " pod="openshift-ingress/e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-2s2dq" Oct 30 12:37:20.045485 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:20.045422 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3729561-3035-459d-a451-290595a76f5d-istio-token\") pod \"e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-2s2dq\" (UID: \"c3729561-3035-459d-a451-290595a76f5d\") " pod="openshift-ingress/e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-2s2dq" Oct 30 12:37:20.046522 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:20.046371 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3729561-3035-459d-a451-290595a76f5d-credential-socket\") pod \"e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-2s2dq\" (UID: \"c3729561-3035-459d-a451-290595a76f5d\") " pod="openshift-ingress/e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-2s2dq" Oct 30 12:37:20.046522 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:20.046405 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3729561-3035-459d-a451-290595a76f5d-workload-socket\") pod \"e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-2s2dq\" (UID: \"c3729561-3035-459d-a451-290595a76f5d\") " pod="openshift-ingress/e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-2s2dq" Oct 30 12:37:20.046678 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:20.046563 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3729561-3035-459d-a451-290595a76f5d-istio-data\") pod \"e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-2s2dq\" (UID: \"c3729561-3035-459d-a451-290595a76f5d\") " pod="openshift-ingress/e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-2s2dq" Oct 30 12:37:20.046720 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:20.046667 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3729561-3035-459d-a451-290595a76f5d-istiod-ca-cert\") pod \"e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-2s2dq\" (UID: \"c3729561-3035-459d-a451-290595a76f5d\") " pod="openshift-ingress/e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-2s2dq" Oct 30 12:37:20.046960 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:20.046939 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3729561-3035-459d-a451-290595a76f5d-workload-certs\") pod \"e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-2s2dq\" (UID: \"c3729561-3035-459d-a451-290595a76f5d\") " pod="openshift-ingress/e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-2s2dq" Oct 30 12:37:20.049120 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:20.049094 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3729561-3035-459d-a451-290595a76f5d-istio-envoy\") pod \"e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-2s2dq\" (UID: \"c3729561-3035-459d-a451-290595a76f5d\") " pod="openshift-ingress/e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-2s2dq" Oct 30 12:37:20.050751 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:20.050727 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3729561-3035-459d-a451-290595a76f5d-istio-podinfo\") pod \"e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-2s2dq\" (UID: \"c3729561-3035-459d-a451-290595a76f5d\") " pod="openshift-ingress/e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-2s2dq" Oct 30 12:37:20.067583 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:20.067561 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3729561-3035-459d-a451-290595a76f5d-istio-token\") pod \"e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-2s2dq\" (UID: \"c3729561-3035-459d-a451-290595a76f5d\") " pod="openshift-ingress/e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-2s2dq" Oct 30 12:37:20.067700 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:20.067679 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wkv7\" (UniqueName: \"kubernetes.io/projected/c3729561-3035-459d-a451-290595a76f5d-kube-api-access-6wkv7\") pod \"e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-2s2dq\" (UID: \"c3729561-3035-459d-a451-290595a76f5d\") " pod="openshift-ingress/e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-2s2dq" Oct 30 12:37:20.141169 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:20.141105 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-2s2dq" Oct 30 12:37:20.476140 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:20.472134 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-2s2dq"] Oct 30 12:37:20.480972 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:20.480937 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:37:20.481090 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:20.481017 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:37:20.481090 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:20.481062 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:37:20.952219 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:20.952185 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-2s2dq" event={"ID":"c3729561-3035-459d-a451-290595a76f5d","Type":"ContainerStarted","Data":"99bb35caa0eb730f715e9c61b8621115a44b26a4d229d207dd2110d6c2ecd2f4"} Oct 30 12:37:20.952685 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:20.952237 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-2s2dq" event={"ID":"c3729561-3035-459d-a451-290595a76f5d","Type":"ContainerStarted","Data":"1647e26cab206e76c296dd423ceb9b823471400fe90ef364391a09b367e279a4"} Oct 30 12:37:21.009621 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:21.009558 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-2s2dq" podStartSLOduration=2.00954227 podStartE2EDuration="2.00954227s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:37:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:37:21.006481844 +0000 UTC m=+2597.520910601" watchObservedRunningTime="2026-10-30 12:37:21.00954227 +0000 UTC m=+2597.523971015" Oct 30 12:37:21.141978 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:21.141939 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-2s2dq" Oct 30 12:37:21.143081 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:21.143060 4908 patch_prober.go:29] interesting pod/e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-2s2dq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body= Oct 30 12:37:21.143157 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:21.143105 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-2s2dq" podUID="c3729561-3035-459d-a451-290595a76f5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" Oct 30 12:37:22.141575 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:22.141538 4908 patch_prober.go:29] interesting pod/e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-2s2dq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body= Oct 30 12:37:22.142070 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:22.141614 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-2s2dq" podUID="c3729561-3035-459d-a451-290595a76f5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" Oct 30 12:37:23.142115 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:23.142080 4908 patch_prober.go:29] interesting pod/e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-2s2dq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body= Oct 30 12:37:23.142540 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:23.142134 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-2s2dq" podUID="c3729561-3035-459d-a451-290595a76f5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" Oct 30 12:37:24.141652 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:24.141610 4908 patch_prober.go:29] interesting pod/e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-2s2dq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body= Oct 30 12:37:24.141893 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:24.141683 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-2s2dq" podUID="c3729561-3035-459d-a451-290595a76f5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" Oct 30 12:37:25.141876 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:25.141838 4908 patch_prober.go:29] interesting pod/e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-2s2dq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body= Oct 30 12:37:25.142347 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:25.141896 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-2s2dq" podUID="c3729561-3035-459d-a451-290595a76f5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" Oct 30 12:37:26.049190 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:26.049153 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-460afc00-5s8cg"] Oct 30 12:37:26.084851 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:26.081260 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-460afc00-5s8cg"] Oct 30 12:37:26.141452 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:26.141416 4908 patch_prober.go:29] interesting pod/e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-2s2dq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body= Oct 30 12:37:26.141625 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:26.141490 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-2s2dq" podUID="c3729561-3035-459d-a451-290595a76f5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" Oct 30 12:37:27.146235 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:27.146204 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-2s2dq" Oct 30 12:37:27.146693 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:27.146669 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-2s2dq" Oct 30 12:37:27.147571 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:27.147551 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-2s2dq" Oct 30 12:37:27.609599 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:27.609565 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f008d28-b03a-4c7c-8a89-c3f9a03a4a5f" path="/var/lib/kubelet/pods/1f008d28-b03a-4c7c-8a89-c3f9a03a4a5f/volumes" Oct 30 12:37:29.461528 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:29.461493 4908 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-5d90d8/explicit-model-5d90d8-kserve-5c8b447959-6ngsf" podUID="34857a8b-c29b-43f0-8b2c-561cb869947c" containerName="main" containerID="cri-o://5f8dfce082e8ba8928ab045b744c39a8cd3e33f9092e8ea83ce5cdd194f7dabe" gracePeriod=45 Oct 30 12:37:29.741960 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:29.741936 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-5d90d8/explicit-model-5d90d8-kserve-5c8b447959-6ngsf" Oct 30 12:37:29.861860 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:29.858238 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34857a8b-c29b-43f0-8b2c-561cb869947c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34857a8b-c29b-43f0-8b2c-561cb869947c-model-cache\") pod \"34857a8b-c29b-43f0-8b2c-561cb869947c\" (UID: \"34857a8b-c29b-43f0-8b2c-561cb869947c\") " Oct 30 12:37:29.861860 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:29.858322 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34857a8b-c29b-43f0-8b2c-561cb869947c-home\" (UniqueName: \"kubernetes.io/empty-dir/34857a8b-c29b-43f0-8b2c-561cb869947c-home\") pod \"34857a8b-c29b-43f0-8b2c-561cb869947c\" (UID: \"34857a8b-c29b-43f0-8b2c-561cb869947c\") " Oct 30 12:37:29.861860 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:29.858408 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34857a8b-c29b-43f0-8b2c-561cb869947c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34857a8b-c29b-43f0-8b2c-561cb869947c-tmp-dir\") pod \"34857a8b-c29b-43f0-8b2c-561cb869947c\" (UID: \"34857a8b-c29b-43f0-8b2c-561cb869947c\") " Oct 30 12:37:29.861860 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:29.858512 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34857a8b-c29b-43f0-8b2c-561cb869947c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/34857a8b-c29b-43f0-8b2c-561cb869947c-dshm\") pod \"34857a8b-c29b-43f0-8b2c-561cb869947c\" (UID: \"34857a8b-c29b-43f0-8b2c-561cb869947c\") " Oct 30 12:37:29.861860 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:29.858546 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/34857a8b-c29b-43f0-8b2c-561cb869947c-tls-certs\" (UniqueName: \"kubernetes.io/secret/34857a8b-c29b-43f0-8b2c-561cb869947c-tls-certs\") pod \"34857a8b-c29b-43f0-8b2c-561cb869947c\" (UID: \"34857a8b-c29b-43f0-8b2c-561cb869947c\") " Oct 30 12:37:29.861860 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:29.858730 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34857a8b-c29b-43f0-8b2c-561cb869947c-kube-api-access-t7r9w\" (UniqueName: \"kubernetes.io/projected/34857a8b-c29b-43f0-8b2c-561cb869947c-kube-api-access-t7r9w\") pod \"34857a8b-c29b-43f0-8b2c-561cb869947c\" (UID: \"34857a8b-c29b-43f0-8b2c-561cb869947c\") " Oct 30 12:37:29.861860 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:29.859773 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34857a8b-c29b-43f0-8b2c-561cb869947c-tmp-dir" pod "34857a8b-c29b-43f0-8b2c-561cb869947c" (UID: "34857a8b-c29b-43f0-8b2c-561cb869947c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:37:29.861860 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:29.860033 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34857a8b-c29b-43f0-8b2c-561cb869947c-model-cache" pod "34857a8b-c29b-43f0-8b2c-561cb869947c" (UID: "34857a8b-c29b-43f0-8b2c-561cb869947c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:37:29.861860 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:29.860252 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34857a8b-c29b-43f0-8b2c-561cb869947c-home" pod "34857a8b-c29b-43f0-8b2c-561cb869947c" (UID: "34857a8b-c29b-43f0-8b2c-561cb869947c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:37:29.884349 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:29.884313 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34857a8b-c29b-43f0-8b2c-561cb869947c-kube-api-access-t7r9w" pod "34857a8b-c29b-43f0-8b2c-561cb869947c" (UID: "34857a8b-c29b-43f0-8b2c-561cb869947c"). InnerVolumeSpecName "kube-api-access-t7r9w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:37:29.895209 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:29.895178 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34857a8b-c29b-43f0-8b2c-561cb869947c-tls-certs" pod "34857a8b-c29b-43f0-8b2c-561cb869947c" (UID: "34857a8b-c29b-43f0-8b2c-561cb869947c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:37:29.895356 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:29.895298 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34857a8b-c29b-43f0-8b2c-561cb869947c-dshm" pod "34857a8b-c29b-43f0-8b2c-561cb869947c" (UID: "34857a8b-c29b-43f0-8b2c-561cb869947c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:37:29.961103 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:29.961078 4908 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34857a8b-c29b-43f0-8b2c-561cb869947c-dshm\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:37:29.961103 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:29.961102 4908 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34857a8b-c29b-43f0-8b2c-561cb869947c-tls-certs\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:37:29.961250 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:29.961112 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7r9w\" (UniqueName: \"kubernetes.io/projected/34857a8b-c29b-43f0-8b2c-561cb869947c-kube-api-access-t7r9w\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:37:29.961250 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:29.961121 4908 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34857a8b-c29b-43f0-8b2c-561cb869947c-model-cache\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:37:29.961250 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:29.961130 4908 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34857a8b-c29b-43f0-8b2c-561cb869947c-home\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:37:29.961250 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:29.961138 4908 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34857a8b-c29b-43f0-8b2c-561cb869947c-tmp-dir\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:37:30.096044 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:30.096015 4908 generic.go:356] "Generic (PLEG): container finished" podID="34857a8b-c29b-43f0-8b2c-561cb869947c" containerID="5f8dfce082e8ba8928ab045b744c39a8cd3e33f9092e8ea83ce5cdd194f7dabe" exitCode=0 Oct 30 12:37:30.096227 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:30.096076 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-5d90d8/explicit-model-5d90d8-kserve-5c8b447959-6ngsf" Oct 30 12:37:30.096227 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:30.096100 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-5d90d8/explicit-model-5d90d8-kserve-5c8b447959-6ngsf" event={"ID":"34857a8b-c29b-43f0-8b2c-561cb869947c","Type":"ContainerDied","Data":"5f8dfce082e8ba8928ab045b744c39a8cd3e33f9092e8ea83ce5cdd194f7dabe"} Oct 30 12:37:30.096227 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:30.096144 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-5d90d8/explicit-model-5d90d8-kserve-5c8b447959-6ngsf" event={"ID":"34857a8b-c29b-43f0-8b2c-561cb869947c","Type":"ContainerDied","Data":"87695e419fb935638815b8fff05e174aa701b9d70bf63c335cce1f5ff7c7d23c"} Oct 30 12:37:30.096227 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:30.096163 4908 scope.go:122] "RemoveContainer" containerID="5f8dfce082e8ba8928ab045b744c39a8cd3e33f9092e8ea83ce5cdd194f7dabe" Oct 30 12:37:30.108819 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:30.108802 4908 scope.go:122] "RemoveContainer" containerID="5f8dfce082e8ba8928ab045b744c39a8cd3e33f9092e8ea83ce5cdd194f7dabe" Oct 30 12:37:30.109132 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:37:30.109108 4908 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f8dfce082e8ba8928ab045b744c39a8cd3e33f9092e8ea83ce5cdd194f7dabe\": container with ID starting with 5f8dfce082e8ba8928ab045b744c39a8cd3e33f9092e8ea83ce5cdd194f7dabe not found: ID does not exist" containerID="5f8dfce082e8ba8928ab045b744c39a8cd3e33f9092e8ea83ce5cdd194f7dabe" Oct 30 12:37:30.109197 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:30.109139 4908 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f8dfce082e8ba8928ab045b744c39a8cd3e33f9092e8ea83ce5cdd194f7dabe"} err="failed to get container status \"5f8dfce082e8ba8928ab045b744c39a8cd3e33f9092e8ea83ce5cdd194f7dabe\": rpc error: code = NotFound desc = could not find container \"5f8dfce082e8ba8928ab045b744c39a8cd3e33f9092e8ea83ce5cdd194f7dabe\": container with ID starting with 5f8dfce082e8ba8928ab045b744c39a8cd3e33f9092e8ea83ce5cdd194f7dabe not found: ID does not exist" Oct 30 12:37:30.146928 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:30.146897 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-5d90d8/explicit-model-5d90d8-kserve-5c8b447959-6ngsf"] Oct 30 12:37:30.151399 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:30.151376 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-5d90d8/explicit-model-5d90d8-kserve-5c8b447959-6ngsf"] Oct 30 12:37:30.703563 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:30.703516 4908 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-explicit-5d90d8/explicit-model-5d90d8-kserve-5c8b447959-6ngsf" podUID="34857a8b-c29b-43f0-8b2c-561cb869947c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.208:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:37:31.614558 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:31.613363 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34857a8b-c29b-43f0-8b2c-561cb869947c" path="/var/lib/kubelet/pods/34857a8b-c29b-43f0-8b2c-561cb869947c/volumes" Oct 30 12:37:38.031970 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:38.031931 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ac1c19b-4prjc"] Oct 30 12:37:38.042175 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:38.042145 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ac1c19b-4prjc"] Oct 30 12:37:38.758351 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:38.758314 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-5d90d8-tt4p6"] Oct 30 12:37:38.818849 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:38.818548 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-5d90d8-tt4p6"] Oct 30 12:37:39.607886 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:39.607844 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55581cb0-176e-4f35-8a10-a31bba1b60a4" path="/var/lib/kubelet/pods/55581cb0-176e-4f35-8a10-a31bba1b60a4/volumes" Oct 30 12:37:39.608515 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:39.608492 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a314719d-98ed-41dd-bb49-b0813194fbec" path="/var/lib/kubelet/pods/a314719d-98ed-41dd-bb49-b0813194fbec/volumes" Oct 30 12:37:43.972080 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:43.972046 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-ljzls"] Oct 30 12:37:43.972583 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:43.972283 4908 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-ljzls" podUID="6f0ed460-7572-4518-af84-589e8ac4ad29" containerName="istio-proxy" containerID="cri-o://3da236f86a561f2d61503e62ccb9ed7e5586bb94851e79be1e2b35e011b9361c" gracePeriod=30 Oct 30 12:37:44.775957 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:44.775917 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-sn5mv"] Oct 30 12:37:44.776191 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:44.776155 4908 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-sn5mv" podUID="a2806f1c-03ec-42ee-a217-040d43646d9d" containerName="istio-proxy" containerID="cri-o://89a39b6d77b062b4f7662480aab1f806c19ab69a85a13fdff103025375f3876e" gracePeriod=30 Oct 30 12:37:46.011291 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:46.011249 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-5d90d8-openshift-default-577df8c9d9-cjd27"] Oct 30 12:37:46.011783 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:46.011522 4908 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-5d90d8-openshift-default-577df8c9d9-cjd27" podUID="5526e66b-b972-4e0f-9a11-d9a9b8fdf0d1" containerName="istio-proxy" containerID="cri-o://8ef34bcb2159c7b9304fb55ce105e3d72d971d08763b9e9a6e7f00ef08275b93" gracePeriod=30 Oct 30 12:37:46.040887 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:46.040855 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38ca6809-9ncpb"] Oct 30 12:37:46.045377 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:46.045350 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38ca6809-9ncpb"] Oct 30 12:37:47.550402 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:47.550364 4908 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-sn5mv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 12:37:47.550928 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:47.550436 4908 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-sn5mv" podUID="a2806f1c-03ec-42ee-a217-040d43646d9d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 12:37:47.643850 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:47.642998 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5a7349e-bc7b-4958-b785-222620bdcfc1" path="/var/lib/kubelet/pods/a5a7349e-bc7b-4958-b785-222620bdcfc1/volumes" Oct 30 12:37:48.750762 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:48.750718 4908 patch_prober.go:29] interesting pod/e2e-worker-w0-f920f7-openshift-default-7c9767c97-ljzls container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 12:37:48.751316 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:48.750804 4908 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-ljzls" podUID="6f0ed460-7572-4518-af84-589e8ac4ad29" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 12:37:49.432125 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:49.432097 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-ljzls" Oct 30 12:37:49.462536 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:49.462502 4908 generic.go:356] "Generic (PLEG): container finished" podID="6f0ed460-7572-4518-af84-589e8ac4ad29" containerID="3da236f86a561f2d61503e62ccb9ed7e5586bb94851e79be1e2b35e011b9361c" exitCode=0 Oct 30 12:37:49.462676 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:49.462552 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-ljzls" Oct 30 12:37:49.462676 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:49.462585 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-ljzls" event={"ID":"6f0ed460-7572-4518-af84-589e8ac4ad29","Type":"ContainerDied","Data":"3da236f86a561f2d61503e62ccb9ed7e5586bb94851e79be1e2b35e011b9361c"} Oct 30 12:37:49.462785 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:49.462686 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-ljzls" event={"ID":"6f0ed460-7572-4518-af84-589e8ac4ad29","Type":"ContainerDied","Data":"84a434992effc5097e4fe1485d6adbe254eee5b8ed2e24a6cbca95c0b8a8e7ad"} Oct 30 12:37:49.462785 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:49.462719 4908 scope.go:122] "RemoveContainer" containerID="3da236f86a561f2d61503e62ccb9ed7e5586bb94851e79be1e2b35e011b9361c" Oct 30 12:37:49.475675 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:49.475641 4908 scope.go:122] "RemoveContainer" containerID="3da236f86a561f2d61503e62ccb9ed7e5586bb94851e79be1e2b35e011b9361c" Oct 30 12:37:49.479089 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:37:49.479063 4908 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3da236f86a561f2d61503e62ccb9ed7e5586bb94851e79be1e2b35e011b9361c\": container with ID starting with 3da236f86a561f2d61503e62ccb9ed7e5586bb94851e79be1e2b35e011b9361c not found: ID does not exist" containerID="3da236f86a561f2d61503e62ccb9ed7e5586bb94851e79be1e2b35e011b9361c" Oct 30 12:37:49.479243 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:49.479100 4908 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3da236f86a561f2d61503e62ccb9ed7e5586bb94851e79be1e2b35e011b9361c"} err="failed to get container status \"3da236f86a561f2d61503e62ccb9ed7e5586bb94851e79be1e2b35e011b9361c\": rpc error: code = NotFound desc = could not find container \"3da236f86a561f2d61503e62ccb9ed7e5586bb94851e79be1e2b35e011b9361c\": container with ID starting with 3da236f86a561f2d61503e62ccb9ed7e5586bb94851e79be1e2b35e011b9361c not found: ID does not exist" Oct 30 12:37:49.533949 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:49.533914 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f0ed460-7572-4518-af84-589e8ac4ad29-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f0ed460-7572-4518-af84-589e8ac4ad29-istio-envoy\") pod \"6f0ed460-7572-4518-af84-589e8ac4ad29\" (UID: \"6f0ed460-7572-4518-af84-589e8ac4ad29\") " Oct 30 12:37:49.534119 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:49.533985 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f0ed460-7572-4518-af84-589e8ac4ad29-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f0ed460-7572-4518-af84-589e8ac4ad29-credential-socket\") pod \"6f0ed460-7572-4518-af84-589e8ac4ad29\" (UID: \"6f0ed460-7572-4518-af84-589e8ac4ad29\") " Oct 30 12:37:49.534119 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:49.534068 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f0ed460-7572-4518-af84-589e8ac4ad29-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f0ed460-7572-4518-af84-589e8ac4ad29-istiod-ca-cert\") pod \"6f0ed460-7572-4518-af84-589e8ac4ad29\" (UID: \"6f0ed460-7572-4518-af84-589e8ac4ad29\") " Oct 30 12:37:49.534209 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:49.534120 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f0ed460-7572-4518-af84-589e8ac4ad29-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f0ed460-7572-4518-af84-589e8ac4ad29-workload-socket\") pod \"6f0ed460-7572-4518-af84-589e8ac4ad29\" (UID: \"6f0ed460-7572-4518-af84-589e8ac4ad29\") " Oct 30 12:37:49.534209 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:49.534148 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f0ed460-7572-4518-af84-589e8ac4ad29-kube-api-access-pfphq\" (UniqueName: \"kubernetes.io/projected/6f0ed460-7572-4518-af84-589e8ac4ad29-kube-api-access-pfphq\") pod \"6f0ed460-7572-4518-af84-589e8ac4ad29\" (UID: \"6f0ed460-7572-4518-af84-589e8ac4ad29\") " Oct 30 12:37:49.534562 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:49.534244 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f0ed460-7572-4518-af84-589e8ac4ad29-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f0ed460-7572-4518-af84-589e8ac4ad29-workload-certs\") pod \"6f0ed460-7572-4518-af84-589e8ac4ad29\" (UID: \"6f0ed460-7572-4518-af84-589e8ac4ad29\") " Oct 30 12:37:49.534562 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:49.534381 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f0ed460-7572-4518-af84-589e8ac4ad29-credential-socket" pod "6f0ed460-7572-4518-af84-589e8ac4ad29" (UID: "6f0ed460-7572-4518-af84-589e8ac4ad29"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:37:49.534562 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:49.534460 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f0ed460-7572-4518-af84-589e8ac4ad29-workload-socket" pod "6f0ed460-7572-4518-af84-589e8ac4ad29" (UID: "6f0ed460-7572-4518-af84-589e8ac4ad29"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:37:49.534562 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:49.534533 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f0ed460-7572-4518-af84-589e8ac4ad29-istio-token\" (UniqueName: \"kubernetes.io/projected/6f0ed460-7572-4518-af84-589e8ac4ad29-istio-token\") pod \"6f0ed460-7572-4518-af84-589e8ac4ad29\" (UID: \"6f0ed460-7572-4518-af84-589e8ac4ad29\") " Oct 30 12:37:49.534803 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:49.534584 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f0ed460-7572-4518-af84-589e8ac4ad29-istiod-ca-cert" pod "6f0ed460-7572-4518-af84-589e8ac4ad29" (UID: "6f0ed460-7572-4518-af84-589e8ac4ad29"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:37:49.534803 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:49.534599 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6f0ed460-7572-4518-af84-589e8ac4ad29-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f0ed460-7572-4518-af84-589e8ac4ad29-istio-podinfo\") pod \"6f0ed460-7572-4518-af84-589e8ac4ad29\" (UID: \"6f0ed460-7572-4518-af84-589e8ac4ad29\") " Oct 30 12:37:49.534803 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:49.534658 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f0ed460-7572-4518-af84-589e8ac4ad29-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f0ed460-7572-4518-af84-589e8ac4ad29-istio-data\") pod \"6f0ed460-7572-4518-af84-589e8ac4ad29\" (UID: \"6f0ed460-7572-4518-af84-589e8ac4ad29\") " Oct 30 12:37:49.534991 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:49.534946 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f0ed460-7572-4518-af84-589e8ac4ad29-istio-data" pod "6f0ed460-7572-4518-af84-589e8ac4ad29" (UID: "6f0ed460-7572-4518-af84-589e8ac4ad29"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:37:49.535116 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:49.535085 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f0ed460-7572-4518-af84-589e8ac4ad29-workload-certs" pod "6f0ed460-7572-4518-af84-589e8ac4ad29" (UID: "6f0ed460-7572-4518-af84-589e8ac4ad29"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:37:49.535326 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:49.535289 4908 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f0ed460-7572-4518-af84-589e8ac4ad29-workload-certs\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:37:49.535326 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:49.535315 4908 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f0ed460-7572-4518-af84-589e8ac4ad29-istio-data\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:37:49.535466 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:49.535329 4908 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f0ed460-7572-4518-af84-589e8ac4ad29-credential-socket\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:37:49.535466 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:49.535344 4908 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f0ed460-7572-4518-af84-589e8ac4ad29-istiod-ca-cert\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:37:49.535466 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:49.535358 4908 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f0ed460-7572-4518-af84-589e8ac4ad29-workload-socket\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:37:49.537001 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:49.536972 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6f0ed460-7572-4518-af84-589e8ac4ad29-istio-podinfo" pod "6f0ed460-7572-4518-af84-589e8ac4ad29" (UID: "6f0ed460-7572-4518-af84-589e8ac4ad29"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 12:37:49.537238 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:49.537219 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f0ed460-7572-4518-af84-589e8ac4ad29-istio-envoy" pod "6f0ed460-7572-4518-af84-589e8ac4ad29" (UID: "6f0ed460-7572-4518-af84-589e8ac4ad29"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:37:49.537307 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:49.537236 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f0ed460-7572-4518-af84-589e8ac4ad29-istio-token" pod "6f0ed460-7572-4518-af84-589e8ac4ad29" (UID: "6f0ed460-7572-4518-af84-589e8ac4ad29"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:37:49.537725 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:49.537703 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f0ed460-7572-4518-af84-589e8ac4ad29-kube-api-access-pfphq" pod "6f0ed460-7572-4518-af84-589e8ac4ad29" (UID: "6f0ed460-7572-4518-af84-589e8ac4ad29"). InnerVolumeSpecName "kube-api-access-pfphq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:37:49.636450 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:49.636425 4908 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f0ed460-7572-4518-af84-589e8ac4ad29-istio-envoy\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:37:49.636450 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:49.636450 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pfphq\" (UniqueName: \"kubernetes.io/projected/6f0ed460-7572-4518-af84-589e8ac4ad29-kube-api-access-pfphq\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:37:49.636597 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:49.636460 4908 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f0ed460-7572-4518-af84-589e8ac4ad29-istio-token\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:37:49.636597 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:49.636468 4908 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f0ed460-7572-4518-af84-589e8ac4ad29-istio-podinfo\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:37:49.686141 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:37:49.686030 4908 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vn6vv" containerName="extractor" Oct 30 12:37:49.803794 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:49.803765 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-ljzls"] Oct 30 12:37:49.814311 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:49.814287 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-ljzls"] Oct 30 12:37:50.043768 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:50.043743 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-sn5mv" Oct 30 12:37:50.142858 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:50.142816 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a2806f1c-03ec-42ee-a217-040d43646d9d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2806f1c-03ec-42ee-a217-040d43646d9d-istio-podinfo\") pod \"a2806f1c-03ec-42ee-a217-040d43646d9d\" (UID: \"a2806f1c-03ec-42ee-a217-040d43646d9d\") " Oct 30 12:37:50.143021 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:50.142882 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2806f1c-03ec-42ee-a217-040d43646d9d-kube-api-access-5cvvg\" (UniqueName: \"kubernetes.io/projected/a2806f1c-03ec-42ee-a217-040d43646d9d-kube-api-access-5cvvg\") pod \"a2806f1c-03ec-42ee-a217-040d43646d9d\" (UID: \"a2806f1c-03ec-42ee-a217-040d43646d9d\") " Oct 30 12:37:50.143021 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:50.142941 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2806f1c-03ec-42ee-a217-040d43646d9d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2806f1c-03ec-42ee-a217-040d43646d9d-workload-certs\") pod \"a2806f1c-03ec-42ee-a217-040d43646d9d\" (UID: \"a2806f1c-03ec-42ee-a217-040d43646d9d\") " Oct 30 12:37:50.143021 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:50.142962 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2806f1c-03ec-42ee-a217-040d43646d9d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2806f1c-03ec-42ee-a217-040d43646d9d-istio-data\") pod \"a2806f1c-03ec-42ee-a217-040d43646d9d\" (UID: \"a2806f1c-03ec-42ee-a217-040d43646d9d\") " Oct 30 12:37:50.143197 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:50.143035 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2806f1c-03ec-42ee-a217-040d43646d9d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2806f1c-03ec-42ee-a217-040d43646d9d-workload-socket\") pod \"a2806f1c-03ec-42ee-a217-040d43646d9d\" (UID: \"a2806f1c-03ec-42ee-a217-040d43646d9d\") " Oct 30 12:37:50.143197 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:50.143084 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2806f1c-03ec-42ee-a217-040d43646d9d-istio-token\" (UniqueName: \"kubernetes.io/projected/a2806f1c-03ec-42ee-a217-040d43646d9d-istio-token\") pod \"a2806f1c-03ec-42ee-a217-040d43646d9d\" (UID: \"a2806f1c-03ec-42ee-a217-040d43646d9d\") " Oct 30 12:37:50.143197 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:50.143112 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2806f1c-03ec-42ee-a217-040d43646d9d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2806f1c-03ec-42ee-a217-040d43646d9d-credential-socket\") pod \"a2806f1c-03ec-42ee-a217-040d43646d9d\" (UID: \"a2806f1c-03ec-42ee-a217-040d43646d9d\") " Oct 30 12:37:50.143197 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:50.143195 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2806f1c-03ec-42ee-a217-040d43646d9d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2806f1c-03ec-42ee-a217-040d43646d9d-istiod-ca-cert\") pod \"a2806f1c-03ec-42ee-a217-040d43646d9d\" (UID: \"a2806f1c-03ec-42ee-a217-040d43646d9d\") " Oct 30 12:37:50.143432 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:50.143250 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2806f1c-03ec-42ee-a217-040d43646d9d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2806f1c-03ec-42ee-a217-040d43646d9d-istio-envoy\") pod \"a2806f1c-03ec-42ee-a217-040d43646d9d\" (UID: \"a2806f1c-03ec-42ee-a217-040d43646d9d\") " Oct 30 12:37:50.143432 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:50.143285 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2806f1c-03ec-42ee-a217-040d43646d9d-workload-certs" pod "a2806f1c-03ec-42ee-a217-040d43646d9d" (UID: "a2806f1c-03ec-42ee-a217-040d43646d9d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:37:50.143432 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:50.143366 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2806f1c-03ec-42ee-a217-040d43646d9d-workload-socket" pod "a2806f1c-03ec-42ee-a217-040d43646d9d" (UID: "a2806f1c-03ec-42ee-a217-040d43646d9d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:37:50.143432 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:50.143381 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2806f1c-03ec-42ee-a217-040d43646d9d-istio-data" pod "a2806f1c-03ec-42ee-a217-040d43646d9d" (UID: "a2806f1c-03ec-42ee-a217-040d43646d9d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:37:50.143769 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:50.143491 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2806f1c-03ec-42ee-a217-040d43646d9d-credential-socket" pod "a2806f1c-03ec-42ee-a217-040d43646d9d" (UID: "a2806f1c-03ec-42ee-a217-040d43646d9d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:37:50.143769 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:50.143639 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2806f1c-03ec-42ee-a217-040d43646d9d-istiod-ca-cert" pod "a2806f1c-03ec-42ee-a217-040d43646d9d" (UID: "a2806f1c-03ec-42ee-a217-040d43646d9d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:37:50.144132 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:50.144109 4908 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2806f1c-03ec-42ee-a217-040d43646d9d-workload-certs\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:37:50.144228 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:50.144135 4908 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2806f1c-03ec-42ee-a217-040d43646d9d-istio-data\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:37:50.144228 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:50.144150 4908 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2806f1c-03ec-42ee-a217-040d43646d9d-workload-socket\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:37:50.144228 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:50.144164 4908 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2806f1c-03ec-42ee-a217-040d43646d9d-credential-socket\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:37:50.144228 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:50.144177 4908 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2806f1c-03ec-42ee-a217-040d43646d9d-istiod-ca-cert\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:37:50.145345 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:50.145313 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a2806f1c-03ec-42ee-a217-040d43646d9d-istio-podinfo" pod "a2806f1c-03ec-42ee-a217-040d43646d9d" (UID: "a2806f1c-03ec-42ee-a217-040d43646d9d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 12:37:50.145445 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:50.145386 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2806f1c-03ec-42ee-a217-040d43646d9d-istio-envoy" pod "a2806f1c-03ec-42ee-a217-040d43646d9d" (UID: "a2806f1c-03ec-42ee-a217-040d43646d9d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:37:50.145538 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:50.145521 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2806f1c-03ec-42ee-a217-040d43646d9d-kube-api-access-5cvvg" pod "a2806f1c-03ec-42ee-a217-040d43646d9d" (UID: "a2806f1c-03ec-42ee-a217-040d43646d9d"). InnerVolumeSpecName "kube-api-access-5cvvg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:37:50.145785 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:50.145765 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2806f1c-03ec-42ee-a217-040d43646d9d-istio-token" pod "a2806f1c-03ec-42ee-a217-040d43646d9d" (UID: "a2806f1c-03ec-42ee-a217-040d43646d9d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:37:50.245746 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:50.245717 4908 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2806f1c-03ec-42ee-a217-040d43646d9d-istio-token\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:37:50.245746 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:50.245746 4908 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2806f1c-03ec-42ee-a217-040d43646d9d-istio-envoy\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:37:50.246048 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:50.245758 4908 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2806f1c-03ec-42ee-a217-040d43646d9d-istio-podinfo\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:37:50.246048 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:50.245770 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5cvvg\" (UniqueName: \"kubernetes.io/projected/a2806f1c-03ec-42ee-a217-040d43646d9d-kube-api-access-5cvvg\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:37:50.475733 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:50.475684 4908 generic.go:356] "Generic (PLEG): container finished" podID="a2806f1c-03ec-42ee-a217-040d43646d9d" containerID="89a39b6d77b062b4f7662480aab1f806c19ab69a85a13fdff103025375f3876e" exitCode=0 Oct 30 12:37:50.475967 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:50.475759 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-sn5mv" Oct 30 12:37:50.475967 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:50.475805 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-sn5mv" event={"ID":"a2806f1c-03ec-42ee-a217-040d43646d9d","Type":"ContainerDied","Data":"89a39b6d77b062b4f7662480aab1f806c19ab69a85a13fdff103025375f3876e"} Oct 30 12:37:50.475967 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:50.475888 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-sn5mv" event={"ID":"a2806f1c-03ec-42ee-a217-040d43646d9d","Type":"ContainerDied","Data":"1ba77a0ec9b6f5b4f99eecb54722df6c5b687f7c0b988c9f265efd879351abb5"} Oct 30 12:37:50.475967 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:50.475920 4908 scope.go:122] "RemoveContainer" containerID="89a39b6d77b062b4f7662480aab1f806c19ab69a85a13fdff103025375f3876e" Oct 30 12:37:50.491057 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:50.490758 4908 scope.go:122] "RemoveContainer" containerID="89a39b6d77b062b4f7662480aab1f806c19ab69a85a13fdff103025375f3876e" Oct 30 12:37:50.491304 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:37:50.491057 4908 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89a39b6d77b062b4f7662480aab1f806c19ab69a85a13fdff103025375f3876e\": container with ID starting with 89a39b6d77b062b4f7662480aab1f806c19ab69a85a13fdff103025375f3876e not found: ID does not exist" containerID="89a39b6d77b062b4f7662480aab1f806c19ab69a85a13fdff103025375f3876e" Oct 30 12:37:50.491304 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:50.491086 4908 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89a39b6d77b062b4f7662480aab1f806c19ab69a85a13fdff103025375f3876e"} err="failed to get container status \"89a39b6d77b062b4f7662480aab1f806c19ab69a85a13fdff103025375f3876e\": rpc error: code = NotFound desc = could not find container \"89a39b6d77b062b4f7662480aab1f806c19ab69a85a13fdff103025375f3876e\": container with ID starting with 89a39b6d77b062b4f7662480aab1f806c19ab69a85a13fdff103025375f3876e not found: ID does not exist" Oct 30 12:37:50.529980 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:50.529907 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-sn5mv"] Oct 30 12:37:50.539932 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:50.539908 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-sn5mv"] Oct 30 12:37:51.308378 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:51.308353 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-5d90d8-openshift-default-577df8c9d9-cjd27" Oct 30 12:37:51.357983 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:51.357957 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5526e66b-b972-4e0f-9a11-d9a9b8fdf0d1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5526e66b-b972-4e0f-9a11-d9a9b8fdf0d1-istio-podinfo\") pod \"5526e66b-b972-4e0f-9a11-d9a9b8fdf0d1\" (UID: \"5526e66b-b972-4e0f-9a11-d9a9b8fdf0d1\") " Oct 30 12:37:51.358119 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:51.358021 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5526e66b-b972-4e0f-9a11-d9a9b8fdf0d1-kube-api-access-k28xn\" (UniqueName: \"kubernetes.io/projected/5526e66b-b972-4e0f-9a11-d9a9b8fdf0d1-kube-api-access-k28xn\") pod \"5526e66b-b972-4e0f-9a11-d9a9b8fdf0d1\" (UID: \"5526e66b-b972-4e0f-9a11-d9a9b8fdf0d1\") " Oct 30 12:37:51.358164 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:51.358116 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5526e66b-b972-4e0f-9a11-d9a9b8fdf0d1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5526e66b-b972-4e0f-9a11-d9a9b8fdf0d1-istio-envoy\") pod \"5526e66b-b972-4e0f-9a11-d9a9b8fdf0d1\" (UID: \"5526e66b-b972-4e0f-9a11-d9a9b8fdf0d1\") " Oct 30 12:37:51.358211 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:51.358167 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5526e66b-b972-4e0f-9a11-d9a9b8fdf0d1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5526e66b-b972-4e0f-9a11-d9a9b8fdf0d1-istiod-ca-cert\") pod \"5526e66b-b972-4e0f-9a11-d9a9b8fdf0d1\" (UID: \"5526e66b-b972-4e0f-9a11-d9a9b8fdf0d1\") " Oct 30 12:37:51.358287 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:51.358268 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5526e66b-b972-4e0f-9a11-d9a9b8fdf0d1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5526e66b-b972-4e0f-9a11-d9a9b8fdf0d1-credential-socket\") pod \"5526e66b-b972-4e0f-9a11-d9a9b8fdf0d1\" (UID: \"5526e66b-b972-4e0f-9a11-d9a9b8fdf0d1\") " Oct 30 12:37:51.358374 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:51.358317 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5526e66b-b972-4e0f-9a11-d9a9b8fdf0d1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5526e66b-b972-4e0f-9a11-d9a9b8fdf0d1-workload-socket\") pod \"5526e66b-b972-4e0f-9a11-d9a9b8fdf0d1\" (UID: \"5526e66b-b972-4e0f-9a11-d9a9b8fdf0d1\") " Oct 30 12:37:51.358471 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:51.358453 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5526e66b-b972-4e0f-9a11-d9a9b8fdf0d1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5526e66b-b972-4e0f-9a11-d9a9b8fdf0d1-workload-certs\") pod \"5526e66b-b972-4e0f-9a11-d9a9b8fdf0d1\" (UID: \"5526e66b-b972-4e0f-9a11-d9a9b8fdf0d1\") " Oct 30 12:37:51.358561 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:51.358488 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5526e66b-b972-4e0f-9a11-d9a9b8fdf0d1-istio-token\" (UniqueName: \"kubernetes.io/projected/5526e66b-b972-4e0f-9a11-d9a9b8fdf0d1-istio-token\") pod \"5526e66b-b972-4e0f-9a11-d9a9b8fdf0d1\" (UID: \"5526e66b-b972-4e0f-9a11-d9a9b8fdf0d1\") " Oct 30 12:37:51.358561 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:51.358508 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5526e66b-b972-4e0f-9a11-d9a9b8fdf0d1-credential-socket" pod "5526e66b-b972-4e0f-9a11-d9a9b8fdf0d1" (UID: "5526e66b-b972-4e0f-9a11-d9a9b8fdf0d1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:37:51.358561 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:51.358521 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5526e66b-b972-4e0f-9a11-d9a9b8fdf0d1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5526e66b-b972-4e0f-9a11-d9a9b8fdf0d1-istio-data\") pod \"5526e66b-b972-4e0f-9a11-d9a9b8fdf0d1\" (UID: \"5526e66b-b972-4e0f-9a11-d9a9b8fdf0d1\") " Oct 30 12:37:51.358735 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:51.358578 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5526e66b-b972-4e0f-9a11-d9a9b8fdf0d1-istiod-ca-cert" pod "5526e66b-b972-4e0f-9a11-d9a9b8fdf0d1" (UID: "5526e66b-b972-4e0f-9a11-d9a9b8fdf0d1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:37:51.358735 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:51.358673 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5526e66b-b972-4e0f-9a11-d9a9b8fdf0d1-workload-certs" pod "5526e66b-b972-4e0f-9a11-d9a9b8fdf0d1" (UID: "5526e66b-b972-4e0f-9a11-d9a9b8fdf0d1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:37:51.358884 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:51.358815 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5526e66b-b972-4e0f-9a11-d9a9b8fdf0d1-workload-socket" pod "5526e66b-b972-4e0f-9a11-d9a9b8fdf0d1" (UID: "5526e66b-b972-4e0f-9a11-d9a9b8fdf0d1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:37:51.358967 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:51.358944 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5526e66b-b972-4e0f-9a11-d9a9b8fdf0d1-istio-data" pod "5526e66b-b972-4e0f-9a11-d9a9b8fdf0d1" (UID: "5526e66b-b972-4e0f-9a11-d9a9b8fdf0d1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:37:51.359305 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:51.359281 4908 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5526e66b-b972-4e0f-9a11-d9a9b8fdf0d1-istiod-ca-cert\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:37:51.359398 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:51.359311 4908 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5526e66b-b972-4e0f-9a11-d9a9b8fdf0d1-credential-socket\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:37:51.359398 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:51.359329 4908 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5526e66b-b972-4e0f-9a11-d9a9b8fdf0d1-workload-socket\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:37:51.359398 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:51.359344 4908 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5526e66b-b972-4e0f-9a11-d9a9b8fdf0d1-workload-certs\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:37:51.359398 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:51.359357 4908 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5526e66b-b972-4e0f-9a11-d9a9b8fdf0d1-istio-data\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:37:51.360565 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:51.360538 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5526e66b-b972-4e0f-9a11-d9a9b8fdf0d1-istio-podinfo" pod "5526e66b-b972-4e0f-9a11-d9a9b8fdf0d1" (UID: "5526e66b-b972-4e0f-9a11-d9a9b8fdf0d1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 12:37:51.360673 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:51.360657 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5526e66b-b972-4e0f-9a11-d9a9b8fdf0d1-kube-api-access-k28xn" pod "5526e66b-b972-4e0f-9a11-d9a9b8fdf0d1" (UID: "5526e66b-b972-4e0f-9a11-d9a9b8fdf0d1"). InnerVolumeSpecName "kube-api-access-k28xn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:37:51.361171 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:51.361149 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5526e66b-b972-4e0f-9a11-d9a9b8fdf0d1-istio-envoy" pod "5526e66b-b972-4e0f-9a11-d9a9b8fdf0d1" (UID: "5526e66b-b972-4e0f-9a11-d9a9b8fdf0d1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:37:51.361937 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:51.361642 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5526e66b-b972-4e0f-9a11-d9a9b8fdf0d1-istio-token" pod "5526e66b-b972-4e0f-9a11-d9a9b8fdf0d1" (UID: "5526e66b-b972-4e0f-9a11-d9a9b8fdf0d1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:37:51.460837 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:51.460792 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k28xn\" (UniqueName: \"kubernetes.io/projected/5526e66b-b972-4e0f-9a11-d9a9b8fdf0d1-kube-api-access-k28xn\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:37:51.460837 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:51.460822 4908 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5526e66b-b972-4e0f-9a11-d9a9b8fdf0d1-istio-envoy\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:37:51.461073 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:51.460849 4908 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5526e66b-b972-4e0f-9a11-d9a9b8fdf0d1-istio-token\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:37:51.461073 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:51.460857 4908 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5526e66b-b972-4e0f-9a11-d9a9b8fdf0d1-istio-podinfo\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:37:51.488090 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:51.487412 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w4-8d2170/auth-test-model-8d2170-kserve-7b6547c869-5q7r5"] Oct 30 12:37:51.490416 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:51.488283 4908 cpu_manager.go:414] "containerMap: removing container" podUID="5526e66b-b972-4e0f-9a11-d9a9b8fdf0d1" containerName="istio-proxy" Oct 30 12:37:51.490416 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:51.488308 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5526e66b-b972-4e0f-9a11-d9a9b8fdf0d1" containerName="istio-proxy" Oct 30 12:37:51.490416 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:51.488328 4908 cpu_manager.go:414] "containerMap: removing container" podUID="a2806f1c-03ec-42ee-a217-040d43646d9d" containerName="istio-proxy" Oct 30 12:37:51.490416 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:51.488336 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2806f1c-03ec-42ee-a217-040d43646d9d" containerName="istio-proxy" Oct 30 12:37:51.490416 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:51.488353 4908 cpu_manager.go:414] "containerMap: removing container" podUID="5526e66b-b972-4e0f-9a11-d9a9b8fdf0d1" containerName="istio-proxy" Oct 30 12:37:51.490416 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:51.488362 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5526e66b-b972-4e0f-9a11-d9a9b8fdf0d1" containerName="istio-proxy" Oct 30 12:37:51.490416 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:51.488393 4908 cpu_manager.go:414] "containerMap: removing container" podUID="34857a8b-c29b-43f0-8b2c-561cb869947c" containerName="main" Oct 30 12:37:51.490416 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:51.488399 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34857a8b-c29b-43f0-8b2c-561cb869947c" containerName="main" Oct 30 12:37:51.490416 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:51.488424 4908 cpu_manager.go:414] "containerMap: removing container" podUID="6f0ed460-7572-4518-af84-589e8ac4ad29" containerName="istio-proxy" Oct 30 12:37:51.490416 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:51.488431 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f0ed460-7572-4518-af84-589e8ac4ad29" containerName="istio-proxy" Oct 30 12:37:51.490416 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:51.488579 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="34857a8b-c29b-43f0-8b2c-561cb869947c" containerName="main" Oct 30 12:37:51.490416 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:51.488597 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f0ed460-7572-4518-af84-589e8ac4ad29" containerName="istio-proxy" Oct 30 12:37:51.490416 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:51.488612 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="5526e66b-b972-4e0f-9a11-d9a9b8fdf0d1" containerName="istio-proxy" Oct 30 12:37:51.490416 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:51.488621 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2806f1c-03ec-42ee-a217-040d43646d9d" containerName="istio-proxy" Oct 30 12:37:51.490416 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:51.488631 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="5526e66b-b972-4e0f-9a11-d9a9b8fdf0d1" containerName="istio-proxy" Oct 30 12:37:51.495960 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:51.493333 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-8d2170/auth-test-model-8d2170-kserve-7b6547c869-5q7r5" Oct 30 12:37:51.510165 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:51.510140 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-8d2170\"/\"openshift-service-ca.crt\"" Oct 30 12:37:51.510321 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:51.510300 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-8d2170\"/\"auth-test-model-8d2170-kserve-self-signed-certs\"" Oct 30 12:37:51.510545 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:51.510528 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-8d2170\"/\"default-dockercfg-7rhbf\"" Oct 30 12:37:51.510757 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:51.510742 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-8d2170\"/\"kube-root-ca.crt\"" Oct 30 12:37:51.512519 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:51.512498 4908 generic.go:356] "Generic (PLEG): container finished" podID="5526e66b-b972-4e0f-9a11-d9a9b8fdf0d1" containerID="8ef34bcb2159c7b9304fb55ce105e3d72d971d08763b9e9a6e7f00ef08275b93" exitCode=0 Oct 30 12:37:51.512665 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:51.512652 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-5d90d8-openshift-default-577df8c9d9-cjd27" Oct 30 12:37:51.514522 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:51.514222 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-5d90d8-openshift-default-577df8c9d9-cjd27" event={"ID":"5526e66b-b972-4e0f-9a11-d9a9b8fdf0d1","Type":"ContainerDied","Data":"8ef34bcb2159c7b9304fb55ce105e3d72d971d08763b9e9a6e7f00ef08275b93"} Oct 30 12:37:51.514522 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:51.514279 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-5d90d8-openshift-default-577df8c9d9-cjd27" event={"ID":"5526e66b-b972-4e0f-9a11-d9a9b8fdf0d1","Type":"ContainerDied","Data":"ef6c51bb7199166cf06ca14b83db4d5acfce9fad49d7a8dbe2d31346acce71bb"} Oct 30 12:37:51.514522 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:51.514308 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-8d2170/auth-test-model-8d2170-kserve-7b6547c869-5q7r5"] Oct 30 12:37:51.514522 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:51.514335 4908 scope.go:122] "RemoveContainer" containerID="8ef34bcb2159c7b9304fb55ce105e3d72d971d08763b9e9a6e7f00ef08275b93" Oct 30 12:37:51.529530 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:51.529511 4908 scope.go:122] "RemoveContainer" containerID="121bdcccd0dfbb5ad3e498b082c7ba5eeb2ee0f946a00128883ab8b2f3b68dc6" Oct 30 12:37:51.543495 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:51.543477 4908 scope.go:122] "RemoveContainer" containerID="8ef34bcb2159c7b9304fb55ce105e3d72d971d08763b9e9a6e7f00ef08275b93" Oct 30 12:37:51.543872 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:37:51.543823 4908 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ef34bcb2159c7b9304fb55ce105e3d72d971d08763b9e9a6e7f00ef08275b93\": container with ID starting with 8ef34bcb2159c7b9304fb55ce105e3d72d971d08763b9e9a6e7f00ef08275b93 not found: ID does not exist" containerID="8ef34bcb2159c7b9304fb55ce105e3d72d971d08763b9e9a6e7f00ef08275b93" Oct 30 12:37:51.543992 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:51.543877 4908 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ef34bcb2159c7b9304fb55ce105e3d72d971d08763b9e9a6e7f00ef08275b93"} err="failed to get container status \"8ef34bcb2159c7b9304fb55ce105e3d72d971d08763b9e9a6e7f00ef08275b93\": rpc error: code = NotFound desc = could not find container \"8ef34bcb2159c7b9304fb55ce105e3d72d971d08763b9e9a6e7f00ef08275b93\": container with ID starting with 8ef34bcb2159c7b9304fb55ce105e3d72d971d08763b9e9a6e7f00ef08275b93 not found: ID does not exist" Oct 30 12:37:51.543992 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:51.543896 4908 scope.go:122] "RemoveContainer" containerID="121bdcccd0dfbb5ad3e498b082c7ba5eeb2ee0f946a00128883ab8b2f3b68dc6" Oct 30 12:37:51.544199 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:37:51.544177 4908 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"121bdcccd0dfbb5ad3e498b082c7ba5eeb2ee0f946a00128883ab8b2f3b68dc6\": container with ID starting with 121bdcccd0dfbb5ad3e498b082c7ba5eeb2ee0f946a00128883ab8b2f3b68dc6 not found: ID does not exist" containerID="121bdcccd0dfbb5ad3e498b082c7ba5eeb2ee0f946a00128883ab8b2f3b68dc6" Oct 30 12:37:51.544266 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:51.544203 4908 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"121bdcccd0dfbb5ad3e498b082c7ba5eeb2ee0f946a00128883ab8b2f3b68dc6"} err="failed to get container status \"121bdcccd0dfbb5ad3e498b082c7ba5eeb2ee0f946a00128883ab8b2f3b68dc6\": rpc error: code = NotFound desc = could not find container \"121bdcccd0dfbb5ad3e498b082c7ba5eeb2ee0f946a00128883ab8b2f3b68dc6\": container with ID starting with 121bdcccd0dfbb5ad3e498b082c7ba5eeb2ee0f946a00128883ab8b2f3b68dc6 not found: ID does not exist" Oct 30 12:37:51.562325 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:51.562298 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff555264-0d24-4bc2-9d62-82006845bb77-model-cache\") pod \"auth-test-model-8d2170-kserve-7b6547c869-5q7r5\" (UID: \"ff555264-0d24-4bc2-9d62-82006845bb77\") " pod="ai-tenant-e2e-shared-b-w4-8d2170/auth-test-model-8d2170-kserve-7b6547c869-5q7r5" Oct 30 12:37:51.562480 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:51.562364 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff555264-0d24-4bc2-9d62-82006845bb77-tls-certs\") pod \"auth-test-model-8d2170-kserve-7b6547c869-5q7r5\" (UID: \"ff555264-0d24-4bc2-9d62-82006845bb77\") " pod="ai-tenant-e2e-shared-b-w4-8d2170/auth-test-model-8d2170-kserve-7b6547c869-5q7r5" Oct 30 12:37:51.562480 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:51.562461 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff555264-0d24-4bc2-9d62-82006845bb77-tmp-dir\") pod \"auth-test-model-8d2170-kserve-7b6547c869-5q7r5\" (UID: \"ff555264-0d24-4bc2-9d62-82006845bb77\") " pod="ai-tenant-e2e-shared-b-w4-8d2170/auth-test-model-8d2170-kserve-7b6547c869-5q7r5" Oct 30 12:37:51.562592 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:51.562536 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xdct\" (UniqueName: \"kubernetes.io/projected/ff555264-0d24-4bc2-9d62-82006845bb77-kube-api-access-5xdct\") pod \"auth-test-model-8d2170-kserve-7b6547c869-5q7r5\" (UID: \"ff555264-0d24-4bc2-9d62-82006845bb77\") " pod="ai-tenant-e2e-shared-b-w4-8d2170/auth-test-model-8d2170-kserve-7b6547c869-5q7r5" Oct 30 12:37:51.562649 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:51.562634 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff555264-0d24-4bc2-9d62-82006845bb77-dshm\") pod \"auth-test-model-8d2170-kserve-7b6547c869-5q7r5\" (UID: \"ff555264-0d24-4bc2-9d62-82006845bb77\") " pod="ai-tenant-e2e-shared-b-w4-8d2170/auth-test-model-8d2170-kserve-7b6547c869-5q7r5" Oct 30 12:37:51.562733 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:51.562720 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff555264-0d24-4bc2-9d62-82006845bb77-home\") pod \"auth-test-model-8d2170-kserve-7b6547c869-5q7r5\" (UID: \"ff555264-0d24-4bc2-9d62-82006845bb77\") " pod="ai-tenant-e2e-shared-b-w4-8d2170/auth-test-model-8d2170-kserve-7b6547c869-5q7r5" Oct 30 12:37:51.574484 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:51.574458 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-5d90d8-openshift-default-577df8c9d9-cjd27"] Oct 30 12:37:51.580859 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:51.580821 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-5d90d8-openshift-default-577df8c9d9-cjd27"] Oct 30 12:37:51.614718 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:51.614685 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5526e66b-b972-4e0f-9a11-d9a9b8fdf0d1" path="/var/lib/kubelet/pods/5526e66b-b972-4e0f-9a11-d9a9b8fdf0d1/volumes" Oct 30 12:37:51.615243 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:51.615221 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f0ed460-7572-4518-af84-589e8ac4ad29" path="/var/lib/kubelet/pods/6f0ed460-7572-4518-af84-589e8ac4ad29/volumes" Oct 30 12:37:51.615746 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:51.615727 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2806f1c-03ec-42ee-a217-040d43646d9d" path="/var/lib/kubelet/pods/a2806f1c-03ec-42ee-a217-040d43646d9d/volumes" Oct 30 12:37:51.663417 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:51.663384 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff555264-0d24-4bc2-9d62-82006845bb77-dshm\") pod \"auth-test-model-8d2170-kserve-7b6547c869-5q7r5\" (UID: \"ff555264-0d24-4bc2-9d62-82006845bb77\") " pod="ai-tenant-e2e-shared-b-w4-8d2170/auth-test-model-8d2170-kserve-7b6547c869-5q7r5" Oct 30 12:37:51.663564 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:51.663518 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff555264-0d24-4bc2-9d62-82006845bb77-home\") pod \"auth-test-model-8d2170-kserve-7b6547c869-5q7r5\" (UID: \"ff555264-0d24-4bc2-9d62-82006845bb77\") " pod="ai-tenant-e2e-shared-b-w4-8d2170/auth-test-model-8d2170-kserve-7b6547c869-5q7r5" Oct 30 12:37:51.663672 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:51.663655 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff555264-0d24-4bc2-9d62-82006845bb77-model-cache\") pod \"auth-test-model-8d2170-kserve-7b6547c869-5q7r5\" (UID: \"ff555264-0d24-4bc2-9d62-82006845bb77\") " pod="ai-tenant-e2e-shared-b-w4-8d2170/auth-test-model-8d2170-kserve-7b6547c869-5q7r5" Oct 30 12:37:51.663762 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:51.663744 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff555264-0d24-4bc2-9d62-82006845bb77-tls-certs\") pod \"auth-test-model-8d2170-kserve-7b6547c869-5q7r5\" (UID: \"ff555264-0d24-4bc2-9d62-82006845bb77\") " pod="ai-tenant-e2e-shared-b-w4-8d2170/auth-test-model-8d2170-kserve-7b6547c869-5q7r5" Oct 30 12:37:51.663865 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:51.663787 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff555264-0d24-4bc2-9d62-82006845bb77-tmp-dir\") pod \"auth-test-model-8d2170-kserve-7b6547c869-5q7r5\" (UID: \"ff555264-0d24-4bc2-9d62-82006845bb77\") " pod="ai-tenant-e2e-shared-b-w4-8d2170/auth-test-model-8d2170-kserve-7b6547c869-5q7r5" Oct 30 12:37:51.663959 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:51.663934 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xdct\" (UniqueName: \"kubernetes.io/projected/ff555264-0d24-4bc2-9d62-82006845bb77-kube-api-access-5xdct\") pod \"auth-test-model-8d2170-kserve-7b6547c869-5q7r5\" (UID: \"ff555264-0d24-4bc2-9d62-82006845bb77\") " pod="ai-tenant-e2e-shared-b-w4-8d2170/auth-test-model-8d2170-kserve-7b6547c869-5q7r5" Oct 30 12:37:51.664747 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:51.664724 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff555264-0d24-4bc2-9d62-82006845bb77-model-cache\") pod \"auth-test-model-8d2170-kserve-7b6547c869-5q7r5\" (UID: \"ff555264-0d24-4bc2-9d62-82006845bb77\") " pod="ai-tenant-e2e-shared-b-w4-8d2170/auth-test-model-8d2170-kserve-7b6547c869-5q7r5" Oct 30 12:37:51.664910 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:51.664770 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff555264-0d24-4bc2-9d62-82006845bb77-home\") pod \"auth-test-model-8d2170-kserve-7b6547c869-5q7r5\" (UID: \"ff555264-0d24-4bc2-9d62-82006845bb77\") " pod="ai-tenant-e2e-shared-b-w4-8d2170/auth-test-model-8d2170-kserve-7b6547c869-5q7r5" Oct 30 12:37:51.665757 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:51.665728 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff555264-0d24-4bc2-9d62-82006845bb77-tmp-dir\") pod \"auth-test-model-8d2170-kserve-7b6547c869-5q7r5\" (UID: \"ff555264-0d24-4bc2-9d62-82006845bb77\") " pod="ai-tenant-e2e-shared-b-w4-8d2170/auth-test-model-8d2170-kserve-7b6547c869-5q7r5" Oct 30 12:37:51.667284 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:51.667258 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff555264-0d24-4bc2-9d62-82006845bb77-dshm\") pod \"auth-test-model-8d2170-kserve-7b6547c869-5q7r5\" (UID: \"ff555264-0d24-4bc2-9d62-82006845bb77\") " pod="ai-tenant-e2e-shared-b-w4-8d2170/auth-test-model-8d2170-kserve-7b6547c869-5q7r5" Oct 30 12:37:51.669403 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:51.669380 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff555264-0d24-4bc2-9d62-82006845bb77-tls-certs\") pod \"auth-test-model-8d2170-kserve-7b6547c869-5q7r5\" (UID: \"ff555264-0d24-4bc2-9d62-82006845bb77\") " pod="ai-tenant-e2e-shared-b-w4-8d2170/auth-test-model-8d2170-kserve-7b6547c869-5q7r5" Oct 30 12:37:51.687020 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:51.686989 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xdct\" (UniqueName: \"kubernetes.io/projected/ff555264-0d24-4bc2-9d62-82006845bb77-kube-api-access-5xdct\") pod \"auth-test-model-8d2170-kserve-7b6547c869-5q7r5\" (UID: \"ff555264-0d24-4bc2-9d62-82006845bb77\") " pod="ai-tenant-e2e-shared-b-w4-8d2170/auth-test-model-8d2170-kserve-7b6547c869-5q7r5" Oct 30 12:37:51.830311 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:51.830157 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-8d2170/auth-test-model-8d2170-kserve-7b6547c869-5q7r5" Oct 30 12:37:52.193317 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:52.193288 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-8d2170/auth-test-model-8d2170-kserve-7b6547c869-5q7r5"] Oct 30 12:37:52.196356 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:37:52.196326 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podff555264_0d24_4bc2_9d62_82006845bb77.slice/crio-e3f7526dba7b7a4fa22ac06e8e4042cef28c7b164543400f4271aaee161ead94 WatchSource:0}: Error finding container e3f7526dba7b7a4fa22ac06e8e4042cef28c7b164543400f4271aaee161ead94: Status 404 returned error can't find the container with id e3f7526dba7b7a4fa22ac06e8e4042cef28c7b164543400f4271aaee161ead94 Oct 30 12:37:52.308853 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:52.306464 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9wwfc"] Oct 30 12:37:52.526610 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:52.526563 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-8d2170/auth-test-model-8d2170-kserve-7b6547c869-5q7r5" event={"ID":"ff555264-0d24-4bc2-9d62-82006845bb77","Type":"ContainerStarted","Data":"9da6929b078acf015a15b8646c7a7e0b4e32efaf60e554d7e3f278e8d1cf6518"} Oct 30 12:37:52.526809 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:52.526617 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-8d2170/auth-test-model-8d2170-kserve-7b6547c869-5q7r5" event={"ID":"ff555264-0d24-4bc2-9d62-82006845bb77","Type":"ContainerStarted","Data":"e3f7526dba7b7a4fa22ac06e8e4042cef28c7b164543400f4271aaee161ead94"} Oct 30 12:37:52.571682 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:52.571634 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w4-8d2170/auth-test-model-8d2170-kserve-7b6547c869-5q7r5" podStartSLOduration=1.5716198399999999 podStartE2EDuration="1.57161984s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:37:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:37:52.568482364 +0000 UTC m=+2629.082911111" watchObservedRunningTime="2026-10-30 12:37:52.57161984 +0000 UTC m=+2629.086048580" Oct 30 12:37:52.831054 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:52.830975 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-8d2170/auth-test-model-8d2170-kserve-7b6547c869-5q7r5" Oct 30 12:37:56.034706 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:56.034674 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec567b86-c9jqg"] Oct 30 12:37:56.045600 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:56.045570 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de431683-xg6lb"] Oct 30 12:37:56.049529 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:56.049503 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec567b86-c9jqg"] Oct 30 12:37:56.052597 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:56.052574 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de431683-xg6lb"] Oct 30 12:37:57.609978 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:57.609946 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2263d1e9-f4ce-4743-9b3a-51ad999798ed" path="/var/lib/kubelet/pods/2263d1e9-f4ce-4743-9b3a-51ad999798ed/volumes" Oct 30 12:37:57.610614 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:37:57.610585 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f00a2bcc-a63b-48a2-91a7-332f9fee2c68" path="/var/lib/kubelet/pods/f00a2bcc-a63b-48a2-91a7-332f9fee2c68/volumes" Oct 30 12:38:01.831056 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:38:01.831019 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w4-8d2170/auth-test-model-8d2170-kserve-7b6547c869-5q7r5" Oct 30 12:38:01.843409 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:38:01.843386 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w4-8d2170/auth-test-model-8d2170-kserve-7b6547c869-5q7r5" Oct 30 12:38:02.641189 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:38:02.641159 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-8d2170/auth-test-model-8d2170-kserve-7b6547c869-5q7r5" Oct 30 12:38:05.501949 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:38:05.501914 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-e1a321/noten-model-e1a321-kserve-7ffcbd8d69-c9pnn"] Oct 30 12:38:05.506271 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:38:05.506250 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-e1a321/noten-model-e1a321-kserve-7ffcbd8d69-c9pnn" Oct 30 12:38:05.524196 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:38:05.524172 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-e1a321/noten-model-e1a321-kserve-7ffcbd8d69-c9pnn"] Oct 30 12:38:05.530746 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:38:05.530720 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-e1a321\"/\"openshift-service-ca.crt\"" Oct 30 12:38:05.530940 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:38:05.530841 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-e1a321\"/\"kube-root-ca.crt\"" Oct 30 12:38:05.531029 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:38:05.531005 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-e1a321\"/\"default-dockercfg-rzh66\"" Oct 30 12:38:05.531110 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:38:05.531036 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-e1a321\"/\"noten-model-e1a321-kserve-self-signed-certs\"" Oct 30 12:38:05.557925 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:38:05.557899 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0caef4a9-3e1e-4af3-aeef-d25d5edd83ef-dshm\") pod \"noten-model-e1a321-kserve-7ffcbd8d69-c9pnn\" (UID: \"0caef4a9-3e1e-4af3-aeef-d25d5edd83ef\") " pod="ai-tenant-e2e-noten-e1a321/noten-model-e1a321-kserve-7ffcbd8d69-c9pnn" Oct 30 12:38:05.558076 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:38:05.558049 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0caef4a9-3e1e-4af3-aeef-d25d5edd83ef-model-cache\") pod \"noten-model-e1a321-kserve-7ffcbd8d69-c9pnn\" (UID: \"0caef4a9-3e1e-4af3-aeef-d25d5edd83ef\") " pod="ai-tenant-e2e-noten-e1a321/noten-model-e1a321-kserve-7ffcbd8d69-c9pnn" Oct 30 12:38:05.558144 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:38:05.558107 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0caef4a9-3e1e-4af3-aeef-d25d5edd83ef-tmp-dir\") pod \"noten-model-e1a321-kserve-7ffcbd8d69-c9pnn\" (UID: \"0caef4a9-3e1e-4af3-aeef-d25d5edd83ef\") " pod="ai-tenant-e2e-noten-e1a321/noten-model-e1a321-kserve-7ffcbd8d69-c9pnn" Oct 30 12:38:05.558216 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:38:05.558196 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hnsk\" (UniqueName: \"kubernetes.io/projected/0caef4a9-3e1e-4af3-aeef-d25d5edd83ef-kube-api-access-4hnsk\") pod \"noten-model-e1a321-kserve-7ffcbd8d69-c9pnn\" (UID: \"0caef4a9-3e1e-4af3-aeef-d25d5edd83ef\") " pod="ai-tenant-e2e-noten-e1a321/noten-model-e1a321-kserve-7ffcbd8d69-c9pnn" Oct 30 12:38:05.558281 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:38:05.558248 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0caef4a9-3e1e-4af3-aeef-d25d5edd83ef-tls-certs\") pod \"noten-model-e1a321-kserve-7ffcbd8d69-c9pnn\" (UID: \"0caef4a9-3e1e-4af3-aeef-d25d5edd83ef\") " pod="ai-tenant-e2e-noten-e1a321/noten-model-e1a321-kserve-7ffcbd8d69-c9pnn" Oct 30 12:38:05.558329 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:38:05.558308 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0caef4a9-3e1e-4af3-aeef-d25d5edd83ef-home\") pod \"noten-model-e1a321-kserve-7ffcbd8d69-c9pnn\" (UID: \"0caef4a9-3e1e-4af3-aeef-d25d5edd83ef\") " pod="ai-tenant-e2e-noten-e1a321/noten-model-e1a321-kserve-7ffcbd8d69-c9pnn" Oct 30 12:38:05.659633 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:38:05.659600 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0caef4a9-3e1e-4af3-aeef-d25d5edd83ef-home\") pod \"noten-model-e1a321-kserve-7ffcbd8d69-c9pnn\" (UID: \"0caef4a9-3e1e-4af3-aeef-d25d5edd83ef\") " pod="ai-tenant-e2e-noten-e1a321/noten-model-e1a321-kserve-7ffcbd8d69-c9pnn" Oct 30 12:38:05.659821 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:38:05.659703 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0caef4a9-3e1e-4af3-aeef-d25d5edd83ef-dshm\") pod \"noten-model-e1a321-kserve-7ffcbd8d69-c9pnn\" (UID: \"0caef4a9-3e1e-4af3-aeef-d25d5edd83ef\") " pod="ai-tenant-e2e-noten-e1a321/noten-model-e1a321-kserve-7ffcbd8d69-c9pnn" Oct 30 12:38:05.659821 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:38:05.659774 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0caef4a9-3e1e-4af3-aeef-d25d5edd83ef-model-cache\") pod \"noten-model-e1a321-kserve-7ffcbd8d69-c9pnn\" (UID: \"0caef4a9-3e1e-4af3-aeef-d25d5edd83ef\") " pod="ai-tenant-e2e-noten-e1a321/noten-model-e1a321-kserve-7ffcbd8d69-c9pnn" Oct 30 12:38:05.659821 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:38:05.659814 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0caef4a9-3e1e-4af3-aeef-d25d5edd83ef-tmp-dir\") pod \"noten-model-e1a321-kserve-7ffcbd8d69-c9pnn\" (UID: \"0caef4a9-3e1e-4af3-aeef-d25d5edd83ef\") " pod="ai-tenant-e2e-noten-e1a321/noten-model-e1a321-kserve-7ffcbd8d69-c9pnn" Oct 30 12:38:05.660066 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:38:05.659964 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hnsk\" (UniqueName: \"kubernetes.io/projected/0caef4a9-3e1e-4af3-aeef-d25d5edd83ef-kube-api-access-4hnsk\") pod \"noten-model-e1a321-kserve-7ffcbd8d69-c9pnn\" (UID: \"0caef4a9-3e1e-4af3-aeef-d25d5edd83ef\") " pod="ai-tenant-e2e-noten-e1a321/noten-model-e1a321-kserve-7ffcbd8d69-c9pnn" Oct 30 12:38:05.660066 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:38:05.660035 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0caef4a9-3e1e-4af3-aeef-d25d5edd83ef-tls-certs\") pod \"noten-model-e1a321-kserve-7ffcbd8d69-c9pnn\" (UID: \"0caef4a9-3e1e-4af3-aeef-d25d5edd83ef\") " pod="ai-tenant-e2e-noten-e1a321/noten-model-e1a321-kserve-7ffcbd8d69-c9pnn" Oct 30 12:38:05.660176 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:38:05.660080 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0caef4a9-3e1e-4af3-aeef-d25d5edd83ef-home\") pod \"noten-model-e1a321-kserve-7ffcbd8d69-c9pnn\" (UID: \"0caef4a9-3e1e-4af3-aeef-d25d5edd83ef\") " pod="ai-tenant-e2e-noten-e1a321/noten-model-e1a321-kserve-7ffcbd8d69-c9pnn" Oct 30 12:38:05.660246 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:38:05.660191 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0caef4a9-3e1e-4af3-aeef-d25d5edd83ef-model-cache\") pod \"noten-model-e1a321-kserve-7ffcbd8d69-c9pnn\" (UID: \"0caef4a9-3e1e-4af3-aeef-d25d5edd83ef\") " pod="ai-tenant-e2e-noten-e1a321/noten-model-e1a321-kserve-7ffcbd8d69-c9pnn" Oct 30 12:38:05.660424 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:38:05.660401 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0caef4a9-3e1e-4af3-aeef-d25d5edd83ef-tmp-dir\") pod \"noten-model-e1a321-kserve-7ffcbd8d69-c9pnn\" (UID: \"0caef4a9-3e1e-4af3-aeef-d25d5edd83ef\") " pod="ai-tenant-e2e-noten-e1a321/noten-model-e1a321-kserve-7ffcbd8d69-c9pnn" Oct 30 12:38:05.662475 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:38:05.662450 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0caef4a9-3e1e-4af3-aeef-d25d5edd83ef-dshm\") pod \"noten-model-e1a321-kserve-7ffcbd8d69-c9pnn\" (UID: \"0caef4a9-3e1e-4af3-aeef-d25d5edd83ef\") " pod="ai-tenant-e2e-noten-e1a321/noten-model-e1a321-kserve-7ffcbd8d69-c9pnn" Oct 30 12:38:05.662660 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:38:05.662645 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0caef4a9-3e1e-4af3-aeef-d25d5edd83ef-tls-certs\") pod \"noten-model-e1a321-kserve-7ffcbd8d69-c9pnn\" (UID: \"0caef4a9-3e1e-4af3-aeef-d25d5edd83ef\") " pod="ai-tenant-e2e-noten-e1a321/noten-model-e1a321-kserve-7ffcbd8d69-c9pnn" Oct 30 12:38:05.682653 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:38:05.682622 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hnsk\" (UniqueName: \"kubernetes.io/projected/0caef4a9-3e1e-4af3-aeef-d25d5edd83ef-kube-api-access-4hnsk\") pod \"noten-model-e1a321-kserve-7ffcbd8d69-c9pnn\" (UID: \"0caef4a9-3e1e-4af3-aeef-d25d5edd83ef\") " pod="ai-tenant-e2e-noten-e1a321/noten-model-e1a321-kserve-7ffcbd8d69-c9pnn" Oct 30 12:38:05.820175 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:38:05.820086 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-e1a321/noten-model-e1a321-kserve-7ffcbd8d69-c9pnn" Oct 30 12:38:06.187252 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:38:06.187220 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-e1a321/noten-model-e1a321-kserve-7ffcbd8d69-c9pnn"] Oct 30 12:38:06.190548 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:38:06.190524 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0caef4a9_3e1e_4af3_aeef_d25d5edd83ef.slice/crio-2d210bcc52a6c1cdc126b7b5ae79fe2507e8730ef2f06a21a1d4de4b34112d84 WatchSource:0}: Error finding container 2d210bcc52a6c1cdc126b7b5ae79fe2507e8730ef2f06a21a1d4de4b34112d84: Status 404 returned error can't find the container with id 2d210bcc52a6c1cdc126b7b5ae79fe2507e8730ef2f06a21a1d4de4b34112d84 Oct 30 12:38:06.424232 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:38:06.424199 4908 scope.go:122] "RemoveContainer" containerID="88ca2dd55da2cd140f2f3ddee213b4a8a72bc7df64ef4c716d3f96827795b0f7" Oct 30 12:38:06.434923 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:38:06.434903 4908 scope.go:122] "RemoveContainer" containerID="82417fef55ee8287db5d8f937e8fd624bd887a864a64b262e54cdecaa56d4bb9" Oct 30 12:38:06.444051 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:38:06.444028 4908 scope.go:122] "RemoveContainer" containerID="6bb846d7a23e9b48f6372682f8fd6188f3c99eb48a76bae6a24d88dbd609cec0" Oct 30 12:38:06.453617 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:38:06.453593 4908 scope.go:122] "RemoveContainer" containerID="1892285d1c5dd75fc039fb0e92de34f4befa7d7e4069137992706d256f3df3da" Oct 30 12:38:06.463289 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:38:06.463258 4908 scope.go:122] "RemoveContainer" containerID="882e5912e768ef799ce7a13998ce3fb9156654dd0c6ffe07f5a718d624cccc88" Oct 30 12:38:06.472651 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:38:06.472628 4908 scope.go:122] "RemoveContainer" containerID="c2891b6d1b422baf500e782d89e0dcf22fbee954bff97c950104fa74bc0093c7" Oct 30 12:38:06.483026 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:38:06.482996 4908 scope.go:122] "RemoveContainer" containerID="c816c843e168b8606377a972505948870e2aa37e0dd5e0f40f111834123f0ae7" Oct 30 12:38:06.492123 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:38:06.492101 4908 scope.go:122] "RemoveContainer" containerID="7dc80f4dccb74baa4941bcb7daa376d5406bdc1aef0c1d1c035bc8e138445f1d" Oct 30 12:38:06.503341 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:38:06.503317 4908 scope.go:122] "RemoveContainer" containerID="8558774ba3c8b13ceaca17d6c3a13abd2a48c16c9f2cc8de7c2b2c4fa9a37324" Oct 30 12:38:06.669787 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:38:06.669753 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-e1a321/noten-model-e1a321-kserve-7ffcbd8d69-c9pnn" event={"ID":"0caef4a9-3e1e-4af3-aeef-d25d5edd83ef","Type":"ContainerStarted","Data":"26237b73785ba8ce12916a80d9d903a507ff98a976b186e12dbe057f9ca49697"} Oct 30 12:38:06.669939 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:38:06.669803 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-e1a321/noten-model-e1a321-kserve-7ffcbd8d69-c9pnn" event={"ID":"0caef4a9-3e1e-4af3-aeef-d25d5edd83ef","Type":"ContainerStarted","Data":"2d210bcc52a6c1cdc126b7b5ae79fe2507e8730ef2f06a21a1d4de4b34112d84"} Oct 30 12:38:06.712520 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:38:06.712433 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-noten-e1a321/noten-model-e1a321-kserve-7ffcbd8d69-c9pnn" podStartSLOduration=1.712417946 podStartE2EDuration="1.712417946s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:38:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:38:06.70775898 +0000 UTC m=+2643.222187736" watchObservedRunningTime="2026-10-30 12:38:06.712417946 +0000 UTC m=+2643.226846692" Oct 30 12:38:06.825567 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:38:06.825505 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-e1a321/noten-model-e1a321-kserve-7ffcbd8d69-c9pnn" Oct 30 12:38:10.029774 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:38:10.029738 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c462d5af-5zf7l"] Oct 30 12:38:10.041703 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:38:10.041672 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c462d5af-5zf7l"] Oct 30 12:38:11.609037 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:38:11.609002 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46ba99ad-dda8-44f0-951a-63074f18fa27" path="/var/lib/kubelet/pods/46ba99ad-dda8-44f0-951a-63074f18fa27/volumes" Oct 30 12:38:15.821679 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:38:15.821643 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-e1a321/noten-model-e1a321-kserve-7ffcbd8d69-c9pnn" Oct 30 12:38:15.834485 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:38:15.834460 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-e1a321/noten-model-e1a321-kserve-7ffcbd8d69-c9pnn" Oct 30 12:38:16.756745 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:38:16.756713 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-e1a321/noten-model-e1a321-kserve-7ffcbd8d69-c9pnn" Oct 30 12:38:18.093952 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:38:18.091775 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-e1a321/noten-model-e1a321-kserve-7ffcbd8d69-c9pnn"] Oct 30 12:38:19.686211 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:38:19.686173 4908 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vn6vv" containerName="extractor" Oct 30 12:38:21.623191 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:38:21.623160 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9wwfc"] Oct 30 12:38:30.028394 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:38:30.028362 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c72dbbd-mvm9q"] Oct 30 12:38:30.039511 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:38:30.039478 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c72dbbd-mvm9q"] Oct 30 12:38:31.609870 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:38:31.609820 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ee71283-e8bf-474f-b557-49d37ad3e704" path="/var/lib/kubelet/pods/3ee71283-e8bf-474f-b557-49d37ad3e704/volumes" Oct 30 12:38:33.784638 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:38:33.784580 4908 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-noten-e1a321/noten-model-e1a321-kserve-7ffcbd8d69-c9pnn" podUID="0caef4a9-3e1e-4af3-aeef-d25d5edd83ef" containerName="main" containerID="cri-o://26237b73785ba8ce12916a80d9d903a507ff98a976b186e12dbe057f9ca49697" gracePeriod=45 Oct 30 12:38:33.925818 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:38:33.925790 4908 generic.go:356] "Generic (PLEG): container finished" podID="0caef4a9-3e1e-4af3-aeef-d25d5edd83ef" containerID="26237b73785ba8ce12916a80d9d903a507ff98a976b186e12dbe057f9ca49697" exitCode=0 Oct 30 12:38:33.925985 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:38:33.925865 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-e1a321/noten-model-e1a321-kserve-7ffcbd8d69-c9pnn" event={"ID":"0caef4a9-3e1e-4af3-aeef-d25d5edd83ef","Type":"ContainerDied","Data":"26237b73785ba8ce12916a80d9d903a507ff98a976b186e12dbe057f9ca49697"} Oct 30 12:38:34.069323 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:38:34.069294 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-e1a321/noten-model-e1a321-kserve-7ffcbd8d69-c9pnn" Oct 30 12:38:34.147780 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:38:34.147749 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0caef4a9-3e1e-4af3-aeef-d25d5edd83ef-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0caef4a9-3e1e-4af3-aeef-d25d5edd83ef-dshm\") pod \"0caef4a9-3e1e-4af3-aeef-d25d5edd83ef\" (UID: \"0caef4a9-3e1e-4af3-aeef-d25d5edd83ef\") " Oct 30 12:38:34.147992 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:38:34.147869 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0caef4a9-3e1e-4af3-aeef-d25d5edd83ef-home\" (UniqueName: \"kubernetes.io/empty-dir/0caef4a9-3e1e-4af3-aeef-d25d5edd83ef-home\") pod \"0caef4a9-3e1e-4af3-aeef-d25d5edd83ef\" (UID: \"0caef4a9-3e1e-4af3-aeef-d25d5edd83ef\") " Oct 30 12:38:34.147992 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:38:34.147949 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0caef4a9-3e1e-4af3-aeef-d25d5edd83ef-kube-api-access-4hnsk\" (UniqueName: \"kubernetes.io/projected/0caef4a9-3e1e-4af3-aeef-d25d5edd83ef-kube-api-access-4hnsk\") pod \"0caef4a9-3e1e-4af3-aeef-d25d5edd83ef\" (UID: \"0caef4a9-3e1e-4af3-aeef-d25d5edd83ef\") " Oct 30 12:38:34.148117 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:38:34.148033 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0caef4a9-3e1e-4af3-aeef-d25d5edd83ef-tls-certs\" (UniqueName: \"kubernetes.io/secret/0caef4a9-3e1e-4af3-aeef-d25d5edd83ef-tls-certs\") pod \"0caef4a9-3e1e-4af3-aeef-d25d5edd83ef\" (UID: \"0caef4a9-3e1e-4af3-aeef-d25d5edd83ef\") " Oct 30 12:38:34.148173 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:38:34.148163 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0caef4a9-3e1e-4af3-aeef-d25d5edd83ef-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0caef4a9-3e1e-4af3-aeef-d25d5edd83ef-model-cache\") pod \"0caef4a9-3e1e-4af3-aeef-d25d5edd83ef\" (UID: \"0caef4a9-3e1e-4af3-aeef-d25d5edd83ef\") " Oct 30 12:38:34.148230 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:38:34.148193 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0caef4a9-3e1e-4af3-aeef-d25d5edd83ef-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0caef4a9-3e1e-4af3-aeef-d25d5edd83ef-tmp-dir\") pod \"0caef4a9-3e1e-4af3-aeef-d25d5edd83ef\" (UID: \"0caef4a9-3e1e-4af3-aeef-d25d5edd83ef\") " Oct 30 12:38:34.148494 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:38:34.148468 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0caef4a9-3e1e-4af3-aeef-d25d5edd83ef-home" pod "0caef4a9-3e1e-4af3-aeef-d25d5edd83ef" (UID: "0caef4a9-3e1e-4af3-aeef-d25d5edd83ef"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:38:34.148734 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:38:34.148709 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0caef4a9-3e1e-4af3-aeef-d25d5edd83ef-tmp-dir" pod "0caef4a9-3e1e-4af3-aeef-d25d5edd83ef" (UID: "0caef4a9-3e1e-4af3-aeef-d25d5edd83ef"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:38:34.148865 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:38:34.148813 4908 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0caef4a9-3e1e-4af3-aeef-d25d5edd83ef-tmp-dir\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:38:34.148942 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:38:34.148876 4908 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0caef4a9-3e1e-4af3-aeef-d25d5edd83ef-home\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:38:34.148942 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:38:34.148913 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0caef4a9-3e1e-4af3-aeef-d25d5edd83ef-model-cache" pod "0caef4a9-3e1e-4af3-aeef-d25d5edd83ef" (UID: "0caef4a9-3e1e-4af3-aeef-d25d5edd83ef"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:38:34.150851 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:38:34.150808 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0caef4a9-3e1e-4af3-aeef-d25d5edd83ef-kube-api-access-4hnsk" pod "0caef4a9-3e1e-4af3-aeef-d25d5edd83ef" (UID: "0caef4a9-3e1e-4af3-aeef-d25d5edd83ef"). InnerVolumeSpecName "kube-api-access-4hnsk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:38:34.152044 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:38:34.152019 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0caef4a9-3e1e-4af3-aeef-d25d5edd83ef-tls-certs" pod "0caef4a9-3e1e-4af3-aeef-d25d5edd83ef" (UID: "0caef4a9-3e1e-4af3-aeef-d25d5edd83ef"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:38:34.154966 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:38:34.154942 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0caef4a9-3e1e-4af3-aeef-d25d5edd83ef-dshm" pod "0caef4a9-3e1e-4af3-aeef-d25d5edd83ef" (UID: "0caef4a9-3e1e-4af3-aeef-d25d5edd83ef"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:38:34.251345 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:38:34.251213 4908 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0caef4a9-3e1e-4af3-aeef-d25d5edd83ef-model-cache\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:38:34.251345 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:38:34.251270 4908 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0caef4a9-3e1e-4af3-aeef-d25d5edd83ef-dshm\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:38:34.251345 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:38:34.251294 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4hnsk\" (UniqueName: \"kubernetes.io/projected/0caef4a9-3e1e-4af3-aeef-d25d5edd83ef-kube-api-access-4hnsk\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:38:34.251345 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:38:34.251308 4908 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0caef4a9-3e1e-4af3-aeef-d25d5edd83ef-tls-certs\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:38:34.934969 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:38:34.934937 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-e1a321/noten-model-e1a321-kserve-7ffcbd8d69-c9pnn" Oct 30 12:38:34.935367 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:38:34.934975 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-e1a321/noten-model-e1a321-kserve-7ffcbd8d69-c9pnn" event={"ID":"0caef4a9-3e1e-4af3-aeef-d25d5edd83ef","Type":"ContainerDied","Data":"2d210bcc52a6c1cdc126b7b5ae79fe2507e8730ef2f06a21a1d4de4b34112d84"} Oct 30 12:38:34.935367 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:38:34.935029 4908 scope.go:122] "RemoveContainer" containerID="26237b73785ba8ce12916a80d9d903a507ff98a976b186e12dbe057f9ca49697" Oct 30 12:38:34.986187 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:38:34.986160 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-e1a321/noten-model-e1a321-kserve-7ffcbd8d69-c9pnn"] Oct 30 12:38:34.993466 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:38:34.993441 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-e1a321/noten-model-e1a321-kserve-7ffcbd8d69-c9pnn"] Oct 30 12:38:35.033447 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:38:35.033420 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-021452de-nv2nq"] Oct 30 12:38:35.038594 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:38:35.038565 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-021452de-nv2nq"] Oct 30 12:38:35.606659 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:38:35.606621 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0caef4a9-3e1e-4af3-aeef-d25d5edd83ef" path="/var/lib/kubelet/pods/0caef4a9-3e1e-4af3-aeef-d25d5edd83ef/volumes" Oct 30 12:38:35.607207 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:38:35.607185 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9a635aa-4e2d-4acd-920e-8bee665a8e91" path="/var/lib/kubelet/pods/d9a635aa-4e2d-4acd-920e-8bee665a8e91/volumes" Oct 30 12:38:49.686578 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:38:49.686534 4908 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vn6vv" containerName="extractor" Oct 30 12:38:50.028994 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:38:50.028963 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df6b0f62-lt5g8"] Oct 30 12:38:50.039914 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:38:50.039888 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df6b0f62-lt5g8"] Oct 30 12:38:51.029122 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:38:51.029088 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c983309c-82j7m"] Oct 30 12:38:51.039871 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:38:51.039846 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c983309c-82j7m"] Oct 30 12:38:51.609707 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:38:51.609666 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f29949d-d1fa-4455-91ed-f3fbdb3ddefd" path="/var/lib/kubelet/pods/5f29949d-d1fa-4455-91ed-f3fbdb3ddefd/volumes" Oct 30 12:38:51.610252 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:38:51.610226 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef65b9d1-cff6-4ada-af9e-1f21372a65bd" path="/var/lib/kubelet/pods/ef65b9d1-cff6-4ada-af9e-1f21372a65bd/volumes" Oct 30 12:38:57.705532 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:38:57.705501 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9wwfc"] Oct 30 12:39:01.432095 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:39:01.432057 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-11575-e07a4a"] Oct 30 12:39:01.432596 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:39:01.432581 4908 cpu_manager.go:414] "containerMap: removing container" podUID="0caef4a9-3e1e-4af3-aeef-d25d5edd83ef" containerName="main" Oct 30 12:39:01.432647 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:39:01.432596 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0caef4a9-3e1e-4af3-aeef-d25d5edd83ef" containerName="main" Oct 30 12:39:01.432712 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:39:01.432703 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="0caef4a9-3e1e-4af3-aeef-d25d5edd83ef" containerName="main" Oct 30 12:39:01.437748 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:39:01.437726 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11575-e07a4a" Oct 30 12:39:01.452789 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:39:01.452766 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11575-e07a4a"] Oct 30 12:39:01.458977 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:39:01.458938 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-ttfhc\"" Oct 30 12:39:01.535337 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:39:01.535303 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zscwv\" (UniqueName: \"kubernetes.io/projected/76dea331-6413-45d5-9603-dac5f95a8103-kube-api-access-zscwv\") pod \"test-curl-11575-e07a4a\" (UID: \"76dea331-6413-45d5-9603-dac5f95a8103\") " pod="opendatahub/test-curl-11575-e07a4a" Oct 30 12:39:01.636442 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:39:01.636407 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zscwv\" (UniqueName: \"kubernetes.io/projected/76dea331-6413-45d5-9603-dac5f95a8103-kube-api-access-zscwv\") pod \"test-curl-11575-e07a4a\" (UID: \"76dea331-6413-45d5-9603-dac5f95a8103\") " pod="opendatahub/test-curl-11575-e07a4a" Oct 30 12:39:01.658621 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:39:01.658591 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zscwv\" (UniqueName: \"kubernetes.io/projected/76dea331-6413-45d5-9603-dac5f95a8103-kube-api-access-zscwv\") pod \"test-curl-11575-e07a4a\" (UID: \"76dea331-6413-45d5-9603-dac5f95a8103\") " pod="opendatahub/test-curl-11575-e07a4a" Oct 30 12:39:01.750219 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:39:01.750193 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11575-e07a4a" Oct 30 12:39:02.056461 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:39:02.056437 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11575-e07a4a"] Oct 30 12:39:02.059737 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:39:02.059708 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod76dea331_6413_45d5_9603_dac5f95a8103.slice/crio-3f243bfc829793a2674d7da01707da25a28b060dd3e775bab16488f541f72a19 WatchSource:0}: Error finding container 3f243bfc829793a2674d7da01707da25a28b060dd3e775bab16488f541f72a19: Status 404 returned error can't find the container with id 3f243bfc829793a2674d7da01707da25a28b060dd3e775bab16488f541f72a19 Oct 30 12:39:02.210856 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:39:02.210229 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11575-e07a4a" event={"ID":"76dea331-6413-45d5-9603-dac5f95a8103","Type":"ContainerStarted","Data":"9ecc9f788af651876b282df3417dda700127d18e31f55d2fdb0f60fdde62f0ab"} Oct 30 12:39:02.210856 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:39:02.210282 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11575-e07a4a" event={"ID":"76dea331-6413-45d5-9603-dac5f95a8103","Type":"ContainerStarted","Data":"3f243bfc829793a2674d7da01707da25a28b060dd3e775bab16488f541f72a19"} Oct 30 12:39:02.248866 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:39:02.248782 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-11575-e07a4a" podStartSLOduration=1.248763077 podStartE2EDuration="1.248763077s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:39:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:39:02.247684685 +0000 UTC m=+2698.762113431" watchObservedRunningTime="2026-10-30 12:39:02.248763077 +0000 UTC m=+2698.763191823" Oct 30 12:39:03.147644 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:39:03.147602 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-11575-e07a4a"] Oct 30 12:39:03.152212 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:39:03.152176 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-11575-e07a4a"] Oct 30 12:39:03.306865 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:39:03.306816 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-11575-54de04"] Oct 30 12:39:03.307388 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:39:03.307372 4908 cpu_manager.go:414] "containerMap: removing container" podUID="76dea331-6413-45d5-9603-dac5f95a8103" containerName="test-curl-11575-e07a4a" Oct 30 12:39:03.307481 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:39:03.307388 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76dea331-6413-45d5-9603-dac5f95a8103" containerName="test-curl-11575-e07a4a" Oct 30 12:39:03.307553 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:39:03.307516 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="76dea331-6413-45d5-9603-dac5f95a8103" containerName="test-curl-11575-e07a4a" Oct 30 12:39:03.310863 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:39:03.310818 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11575-54de04" Oct 30 12:39:03.325644 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:39:03.325618 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11575-54de04"] Oct 30 12:39:03.456951 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:39:03.456874 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkgpk\" (UniqueName: \"kubernetes.io/projected/61d6dd36-b05b-45cb-8484-9ae0bb648b23-kube-api-access-gkgpk\") pod \"test-curl-11575-54de04\" (UID: \"61d6dd36-b05b-45cb-8484-9ae0bb648b23\") " pod="opendatahub/test-curl-11575-54de04" Oct 30 12:39:03.557517 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:39:03.557486 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkgpk\" (UniqueName: \"kubernetes.io/projected/61d6dd36-b05b-45cb-8484-9ae0bb648b23-kube-api-access-gkgpk\") pod \"test-curl-11575-54de04\" (UID: \"61d6dd36-b05b-45cb-8484-9ae0bb648b23\") " pod="opendatahub/test-curl-11575-54de04" Oct 30 12:39:03.579616 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:39:03.579587 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkgpk\" (UniqueName: \"kubernetes.io/projected/61d6dd36-b05b-45cb-8484-9ae0bb648b23-kube-api-access-gkgpk\") pod \"test-curl-11575-54de04\" (UID: \"61d6dd36-b05b-45cb-8484-9ae0bb648b23\") " pod="opendatahub/test-curl-11575-54de04" Oct 30 12:39:03.623624 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:39:03.623600 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11575-54de04" Oct 30 12:39:03.929486 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:39:03.929462 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11575-54de04"] Oct 30 12:39:03.931986 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:39:03.931958 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod61d6dd36_b05b_45cb_8484_9ae0bb648b23.slice/crio-af29e035a356595aecc55b8d77fc3d9b339a63a950faa32b5045f5068a56736c WatchSource:0}: Error finding container af29e035a356595aecc55b8d77fc3d9b339a63a950faa32b5045f5068a56736c: Status 404 returned error can't find the container with id af29e035a356595aecc55b8d77fc3d9b339a63a950faa32b5045f5068a56736c Oct 30 12:39:04.226470 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:39:04.226444 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11575-54de04" event={"ID":"61d6dd36-b05b-45cb-8484-9ae0bb648b23","Type":"ContainerStarted","Data":"47847b324a322cf928b6b48250720018ff59da50383b3e71c25619542e2b4d82"} Oct 30 12:39:04.226875 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:39:04.226488 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11575-54de04" event={"ID":"61d6dd36-b05b-45cb-8484-9ae0bb648b23","Type":"ContainerStarted","Data":"af29e035a356595aecc55b8d77fc3d9b339a63a950faa32b5045f5068a56736c"} Oct 30 12:39:04.226875 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:39:04.226696 4908 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-11575-e07a4a" podUID="76dea331-6413-45d5-9603-dac5f95a8103" containerName="test-curl-11575-e07a4a" containerID="cri-o://9ecc9f788af651876b282df3417dda700127d18e31f55d2fdb0f60fdde62f0ab" gracePeriod=2 Oct 30 12:39:04.262089 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:39:04.262042 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-11575-54de04" podStartSLOduration=1.262029771 podStartE2EDuration="1.262029771s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:39:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:39:04.261512929 +0000 UTC m=+2700.775941676" watchObservedRunningTime="2026-10-30 12:39:04.262029771 +0000 UTC m=+2700.776458517" Oct 30 12:39:05.194173 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:39:05.194140 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-11575-54de04"] Oct 30 12:39:05.198332 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:39:05.198303 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-11575-54de04"] Oct 30 12:39:06.029496 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:39:06.029466 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02be0005-mxjlm"] Oct 30 12:39:06.040108 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:39:06.040080 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02be0005-mxjlm"] Oct 30 12:39:06.074490 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:39:06.074460 4908 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-plczt_a70ee11c-c20e-4a4f-8476-ac5eb02816d0/console-operator/3.log" Oct 30 12:39:06.080948 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:39:06.080919 4908 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b8bxz_644e19b3-10ad-4145-9ed3-595d19ed9b07/ovn-acl-logging/0.log" Oct 30 12:39:06.090023 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:39:06.090002 4908 kubelet.go:1689] "Image garbage collection succeeded" Oct 30 12:39:06.247803 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:39:06.247742 4908 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-11575-54de04" podUID="61d6dd36-b05b-45cb-8484-9ae0bb648b23" containerName="test-curl-11575-54de04" containerID="cri-o://47847b324a322cf928b6b48250720018ff59da50383b3e71c25619542e2b4d82" gracePeriod=2 Oct 30 12:39:06.285079 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:39:06.282290 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-73bafc-openshift-default-64bfdb8459-rfkqt"] Oct 30 12:39:06.285079 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:39:06.283181 4908 cpu_manager.go:414] "containerMap: removing container" podUID="61d6dd36-b05b-45cb-8484-9ae0bb648b23" containerName="test-curl-11575-54de04" Oct 30 12:39:06.285079 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:39:06.283202 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61d6dd36-b05b-45cb-8484-9ae0bb648b23" containerName="test-curl-11575-54de04" Oct 30 12:39:06.285079 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:39:06.283366 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="61d6dd36-b05b-45cb-8484-9ae0bb648b23" containerName="test-curl-11575-54de04" Oct 30 12:39:06.306847 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:39:06.291946 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-73bafc-openshift-default-64bfdb8459-rfkqt" Oct 30 12:39:06.315559 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:39:06.315521 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-73bafc-openshift-default-dockercfg-bhzbh\"" Oct 30 12:39:06.332881 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:39:06.331838 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-73bafc-openshift-default-64bfdb8459-rfkqt"] Oct 30 12:39:06.400654 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:39:06.400617 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9c9d1f3-e29c-42d4-96e6-765da8de6ab9-istio-token\") pod \"e2e-shared-a-w0-73bafc-openshift-default-64bfdb8459-rfkqt\" (UID: \"c9c9d1f3-e29c-42d4-96e6-765da8de6ab9\") " pod="openshift-ingress/e2e-shared-a-w0-73bafc-openshift-default-64bfdb8459-rfkqt" Oct 30 12:39:06.400960 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:39:06.400938 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9c9d1f3-e29c-42d4-96e6-765da8de6ab9-workload-certs\") pod \"e2e-shared-a-w0-73bafc-openshift-default-64bfdb8459-rfkqt\" (UID: \"c9c9d1f3-e29c-42d4-96e6-765da8de6ab9\") " pod="openshift-ingress/e2e-shared-a-w0-73bafc-openshift-default-64bfdb8459-rfkqt" Oct 30 12:39:06.401154 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:39:06.401138 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9c9d1f3-e29c-42d4-96e6-765da8de6ab9-istio-envoy\") pod \"e2e-shared-a-w0-73bafc-openshift-default-64bfdb8459-rfkqt\" (UID: \"c9c9d1f3-e29c-42d4-96e6-765da8de6ab9\") " pod="openshift-ingress/e2e-shared-a-w0-73bafc-openshift-default-64bfdb8459-rfkqt" Oct 30 12:39:06.401319 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:39:06.401303 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f47fz\" (UniqueName: \"kubernetes.io/projected/c9c9d1f3-e29c-42d4-96e6-765da8de6ab9-kube-api-access-f47fz\") pod \"e2e-shared-a-w0-73bafc-openshift-default-64bfdb8459-rfkqt\" (UID: \"c9c9d1f3-e29c-42d4-96e6-765da8de6ab9\") " pod="openshift-ingress/e2e-shared-a-w0-73bafc-openshift-default-64bfdb8459-rfkqt" Oct 30 12:39:06.401486 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:39:06.401473 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9c9d1f3-e29c-42d4-96e6-765da8de6ab9-istio-data\") pod \"e2e-shared-a-w0-73bafc-openshift-default-64bfdb8459-rfkqt\" (UID: \"c9c9d1f3-e29c-42d4-96e6-765da8de6ab9\") " pod="openshift-ingress/e2e-shared-a-w0-73bafc-openshift-default-64bfdb8459-rfkqt" Oct 30 12:39:06.401600 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:39:06.401587 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9c9d1f3-e29c-42d4-96e6-765da8de6ab9-istio-podinfo\") pod \"e2e-shared-a-w0-73bafc-openshift-default-64bfdb8459-rfkqt\" (UID: \"c9c9d1f3-e29c-42d4-96e6-765da8de6ab9\") " pod="openshift-ingress/e2e-shared-a-w0-73bafc-openshift-default-64bfdb8459-rfkqt" Oct 30 12:39:06.401714 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:39:06.401702 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9c9d1f3-e29c-42d4-96e6-765da8de6ab9-workload-socket\") pod \"e2e-shared-a-w0-73bafc-openshift-default-64bfdb8459-rfkqt\" (UID: \"c9c9d1f3-e29c-42d4-96e6-765da8de6ab9\") " pod="openshift-ingress/e2e-shared-a-w0-73bafc-openshift-default-64bfdb8459-rfkqt" Oct 30 12:39:06.401973 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:39:06.401950 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9c9d1f3-e29c-42d4-96e6-765da8de6ab9-istiod-ca-cert\") pod \"e2e-shared-a-w0-73bafc-openshift-default-64bfdb8459-rfkqt\" (UID: \"c9c9d1f3-e29c-42d4-96e6-765da8de6ab9\") " pod="openshift-ingress/e2e-shared-a-w0-73bafc-openshift-default-64bfdb8459-rfkqt" Oct 30 12:39:06.402111 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:39:06.402097 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9c9d1f3-e29c-42d4-96e6-765da8de6ab9-credential-socket\") pod \"e2e-shared-a-w0-73bafc-openshift-default-64bfdb8459-rfkqt\" (UID: \"c9c9d1f3-e29c-42d4-96e6-765da8de6ab9\") " pod="openshift-ingress/e2e-shared-a-w0-73bafc-openshift-default-64bfdb8459-rfkqt" Oct 30 12:39:06.651903 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:39:06.650888 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9c9d1f3-e29c-42d4-96e6-765da8de6ab9-istiod-ca-cert\") pod \"e2e-shared-a-w0-73bafc-openshift-default-64bfdb8459-rfkqt\" (UID: \"c9c9d1f3-e29c-42d4-96e6-765da8de6ab9\") " pod="openshift-ingress/e2e-shared-a-w0-73bafc-openshift-default-64bfdb8459-rfkqt" Oct 30 12:39:06.651903 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:39:06.650940 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9c9d1f3-e29c-42d4-96e6-765da8de6ab9-credential-socket\") pod \"e2e-shared-a-w0-73bafc-openshift-default-64bfdb8459-rfkqt\" (UID: \"c9c9d1f3-e29c-42d4-96e6-765da8de6ab9\") " pod="openshift-ingress/e2e-shared-a-w0-73bafc-openshift-default-64bfdb8459-rfkqt" Oct 30 12:39:06.651903 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:39:06.651018 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9c9d1f3-e29c-42d4-96e6-765da8de6ab9-istio-token\") pod \"e2e-shared-a-w0-73bafc-openshift-default-64bfdb8459-rfkqt\" (UID: \"c9c9d1f3-e29c-42d4-96e6-765da8de6ab9\") " pod="openshift-ingress/e2e-shared-a-w0-73bafc-openshift-default-64bfdb8459-rfkqt" Oct 30 12:39:06.651903 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:39:06.651074 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9c9d1f3-e29c-42d4-96e6-765da8de6ab9-workload-certs\") pod \"e2e-shared-a-w0-73bafc-openshift-default-64bfdb8459-rfkqt\" (UID: \"c9c9d1f3-e29c-42d4-96e6-765da8de6ab9\") " pod="openshift-ingress/e2e-shared-a-w0-73bafc-openshift-default-64bfdb8459-rfkqt" Oct 30 12:39:06.651903 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:39:06.651128 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9c9d1f3-e29c-42d4-96e6-765da8de6ab9-istio-envoy\") pod \"e2e-shared-a-w0-73bafc-openshift-default-64bfdb8459-rfkqt\" (UID: \"c9c9d1f3-e29c-42d4-96e6-765da8de6ab9\") " pod="openshift-ingress/e2e-shared-a-w0-73bafc-openshift-default-64bfdb8459-rfkqt" Oct 30 12:39:06.651903 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:39:06.651188 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f47fz\" (UniqueName: \"kubernetes.io/projected/c9c9d1f3-e29c-42d4-96e6-765da8de6ab9-kube-api-access-f47fz\") pod \"e2e-shared-a-w0-73bafc-openshift-default-64bfdb8459-rfkqt\" (UID: \"c9c9d1f3-e29c-42d4-96e6-765da8de6ab9\") " pod="openshift-ingress/e2e-shared-a-w0-73bafc-openshift-default-64bfdb8459-rfkqt" Oct 30 12:39:06.651903 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:39:06.651271 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9c9d1f3-e29c-42d4-96e6-765da8de6ab9-istio-data\") pod \"e2e-shared-a-w0-73bafc-openshift-default-64bfdb8459-rfkqt\" (UID: \"c9c9d1f3-e29c-42d4-96e6-765da8de6ab9\") " pod="openshift-ingress/e2e-shared-a-w0-73bafc-openshift-default-64bfdb8459-rfkqt" Oct 30 12:39:06.651903 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:39:06.651311 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9c9d1f3-e29c-42d4-96e6-765da8de6ab9-istio-podinfo\") pod \"e2e-shared-a-w0-73bafc-openshift-default-64bfdb8459-rfkqt\" (UID: \"c9c9d1f3-e29c-42d4-96e6-765da8de6ab9\") " pod="openshift-ingress/e2e-shared-a-w0-73bafc-openshift-default-64bfdb8459-rfkqt" Oct 30 12:39:06.651903 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:39:06.651357 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9c9d1f3-e29c-42d4-96e6-765da8de6ab9-workload-socket\") pod \"e2e-shared-a-w0-73bafc-openshift-default-64bfdb8459-rfkqt\" (UID: \"c9c9d1f3-e29c-42d4-96e6-765da8de6ab9\") " pod="openshift-ingress/e2e-shared-a-w0-73bafc-openshift-default-64bfdb8459-rfkqt" Oct 30 12:39:06.663997 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:39:06.655880 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9c9d1f3-e29c-42d4-96e6-765da8de6ab9-workload-socket\") pod \"e2e-shared-a-w0-73bafc-openshift-default-64bfdb8459-rfkqt\" (UID: \"c9c9d1f3-e29c-42d4-96e6-765da8de6ab9\") " pod="openshift-ingress/e2e-shared-a-w0-73bafc-openshift-default-64bfdb8459-rfkqt" Oct 30 12:39:06.663997 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:39:06.656600 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9c9d1f3-e29c-42d4-96e6-765da8de6ab9-istiod-ca-cert\") pod \"e2e-shared-a-w0-73bafc-openshift-default-64bfdb8459-rfkqt\" (UID: \"c9c9d1f3-e29c-42d4-96e6-765da8de6ab9\") " pod="openshift-ingress/e2e-shared-a-w0-73bafc-openshift-default-64bfdb8459-rfkqt" Oct 30 12:39:06.663997 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:39:06.656886 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9c9d1f3-e29c-42d4-96e6-765da8de6ab9-credential-socket\") pod \"e2e-shared-a-w0-73bafc-openshift-default-64bfdb8459-rfkqt\" (UID: \"c9c9d1f3-e29c-42d4-96e6-765da8de6ab9\") " pod="openshift-ingress/e2e-shared-a-w0-73bafc-openshift-default-64bfdb8459-rfkqt" Oct 30 12:39:06.663997 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:39:06.657417 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9c9d1f3-e29c-42d4-96e6-765da8de6ab9-workload-certs\") pod \"e2e-shared-a-w0-73bafc-openshift-default-64bfdb8459-rfkqt\" (UID: \"c9c9d1f3-e29c-42d4-96e6-765da8de6ab9\") " pod="openshift-ingress/e2e-shared-a-w0-73bafc-openshift-default-64bfdb8459-rfkqt" Oct 30 12:39:06.664756 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:39:06.664690 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9c9d1f3-e29c-42d4-96e6-765da8de6ab9-istio-data\") pod \"e2e-shared-a-w0-73bafc-openshift-default-64bfdb8459-rfkqt\" (UID: \"c9c9d1f3-e29c-42d4-96e6-765da8de6ab9\") " pod="openshift-ingress/e2e-shared-a-w0-73bafc-openshift-default-64bfdb8459-rfkqt" Oct 30 12:39:06.710846 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:39:06.706243 4908 scope.go:122] "RemoveContainer" containerID="063f033d80f52e6559a06c7414d55b6bfbbb51177ba1cd5ef0c131dfdd1f86cd" Oct 30 12:39:06.758564 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:39:06.758460 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9c9d1f3-e29c-42d4-96e6-765da8de6ab9-istio-envoy\") pod \"e2e-shared-a-w0-73bafc-openshift-default-64bfdb8459-rfkqt\" (UID: \"c9c9d1f3-e29c-42d4-96e6-765da8de6ab9\") " pod="openshift-ingress/e2e-shared-a-w0-73bafc-openshift-default-64bfdb8459-rfkqt" Oct 30 12:39:06.761710 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:39:06.761633 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f47fz\" (UniqueName: \"kubernetes.io/projected/c9c9d1f3-e29c-42d4-96e6-765da8de6ab9-kube-api-access-f47fz\") pod \"e2e-shared-a-w0-73bafc-openshift-default-64bfdb8459-rfkqt\" (UID: \"c9c9d1f3-e29c-42d4-96e6-765da8de6ab9\") " pod="openshift-ingress/e2e-shared-a-w0-73bafc-openshift-default-64bfdb8459-rfkqt" Oct 30 12:39:06.764479 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:39:06.764444 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9c9d1f3-e29c-42d4-96e6-765da8de6ab9-istio-token\") pod \"e2e-shared-a-w0-73bafc-openshift-default-64bfdb8459-rfkqt\" (UID: \"c9c9d1f3-e29c-42d4-96e6-765da8de6ab9\") " pod="openshift-ingress/e2e-shared-a-w0-73bafc-openshift-default-64bfdb8459-rfkqt" Oct 30 12:39:06.773536 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:39:06.773505 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9c9d1f3-e29c-42d4-96e6-765da8de6ab9-istio-podinfo\") pod \"e2e-shared-a-w0-73bafc-openshift-default-64bfdb8459-rfkqt\" (UID: \"c9c9d1f3-e29c-42d4-96e6-765da8de6ab9\") " pod="openshift-ingress/e2e-shared-a-w0-73bafc-openshift-default-64bfdb8459-rfkqt" Oct 30 12:39:06.808075 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:39:06.807977 4908 scope.go:122] "RemoveContainer" containerID="151968bffad60e1c41cb6f0e393e6db13ac05c29e581582d4625309495a4f7ea" Oct 30 12:39:06.818888 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:39:06.818809 4908 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"063f033d80f52e6559a06c7414d55b6bfbbb51177ba1cd5ef0c131dfdd1f86cd\": container with ID starting with 063f033d80f52e6559a06c7414d55b6bfbbb51177ba1cd5ef0c131dfdd1f86cd not found: ID does not exist" containerID="063f033d80f52e6559a06c7414d55b6bfbbb51177ba1cd5ef0c131dfdd1f86cd" Oct 30 12:39:06.818888 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:39:06.818878 4908 image_gc_manager.go:223] "Failed to monitor images" err="get container status: runtime container status: rpc error: code = NotFound desc = could not find container \"063f033d80f52e6559a06c7414d55b6bfbbb51177ba1cd5ef0c131dfdd1f86cd\": container with ID starting with 063f033d80f52e6559a06c7414d55b6bfbbb51177ba1cd5ef0c131dfdd1f86cd not found: ID does not exist" Oct 30 12:39:06.837971 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:39:06.837949 4908 scope.go:122] "RemoveContainer" containerID="cdaf4c8a90744ae4aff020e781769e163ecda1e4eed3152cee854d2351b9b0c7" Oct 30 12:39:06.840015 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:39:06.839994 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11575-e07a4a" Oct 30 12:39:06.846885 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:39:06.846867 4908 scope.go:122] "RemoveContainer" containerID="3de7f024f719a3244919b92e0bef6983423b60207f3fe64a0f7e55d865409966" Oct 30 12:39:06.855777 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:39:06.855758 4908 scope.go:122] "RemoveContainer" containerID="9762fe3fa5c075c77601e330914209d4ac01b159c733d31a9437767c07fa41ca" Oct 30 12:39:06.867127 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:39:06.867106 4908 scope.go:122] "RemoveContainer" containerID="a905ebef08e66a7e5d0ea43808d2d913fd30c46433178de7e41ed403fac0af7f" Oct 30 12:39:06.879870 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:39:06.879853 4908 scope.go:122] "RemoveContainer" containerID="f607b93c16f0bfe4273ecba14cbd3db364579343512b15408d8ebb30a4daca2b" Oct 30 12:39:06.892207 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:39:06.892188 4908 scope.go:122] "RemoveContainer" containerID="1218b01a178df40f6763d11e286abe5be656f4b567567f834e5f624e6a547ba6" Oct 30 12:39:06.919595 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:39:06.919574 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-73bafc-openshift-default-64bfdb8459-rfkqt" Oct 30 12:39:06.977592 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:39:06.962693 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76dea331-6413-45d5-9603-dac5f95a8103-kube-api-access-zscwv\" (UniqueName: \"kubernetes.io/projected/76dea331-6413-45d5-9603-dac5f95a8103-kube-api-access-zscwv\") pod \"76dea331-6413-45d5-9603-dac5f95a8103\" (UID: \"76dea331-6413-45d5-9603-dac5f95a8103\") " Oct 30 12:39:06.977592 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:39:06.965153 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76dea331-6413-45d5-9603-dac5f95a8103-kube-api-access-zscwv" pod "76dea331-6413-45d5-9603-dac5f95a8103" (UID: "76dea331-6413-45d5-9603-dac5f95a8103"). InnerVolumeSpecName "kube-api-access-zscwv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:39:07.075960 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:39:07.065120 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zscwv\" (UniqueName: \"kubernetes.io/projected/76dea331-6413-45d5-9603-dac5f95a8103-kube-api-access-zscwv\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:39:07.263546 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:39:07.263511 4908 generic.go:356] "Generic (PLEG): container finished" podID="76dea331-6413-45d5-9603-dac5f95a8103" containerID="9ecc9f788af651876b282df3417dda700127d18e31f55d2fdb0f60fdde62f0ab" exitCode=137 Oct 30 12:39:07.263725 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:39:07.263566 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11575-e07a4a" Oct 30 12:39:07.263725 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:39:07.263597 4908 scope.go:122] "RemoveContainer" containerID="9ecc9f788af651876b282df3417dda700127d18e31f55d2fdb0f60fdde62f0ab" Oct 30 12:39:07.272471 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:39:07.272443 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-73bafc-openshift-default-64bfdb8459-rfkqt"] Oct 30 12:39:07.278309 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:39:07.278289 4908 scope.go:122] "RemoveContainer" containerID="9ecc9f788af651876b282df3417dda700127d18e31f55d2fdb0f60fdde62f0ab" Oct 30 12:39:07.278596 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:39:07.278570 4908 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ecc9f788af651876b282df3417dda700127d18e31f55d2fdb0f60fdde62f0ab\": container with ID starting with 9ecc9f788af651876b282df3417dda700127d18e31f55d2fdb0f60fdde62f0ab not found: ID does not exist" containerID="9ecc9f788af651876b282df3417dda700127d18e31f55d2fdb0f60fdde62f0ab" Oct 30 12:39:07.278688 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:39:07.278606 4908 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ecc9f788af651876b282df3417dda700127d18e31f55d2fdb0f60fdde62f0ab"} err="failed to get container status \"9ecc9f788af651876b282df3417dda700127d18e31f55d2fdb0f60fdde62f0ab\": rpc error: code = NotFound desc = could not find container \"9ecc9f788af651876b282df3417dda700127d18e31f55d2fdb0f60fdde62f0ab\": container with ID starting with 9ecc9f788af651876b282df3417dda700127d18e31f55d2fdb0f60fdde62f0ab not found: ID does not exist" Oct 30 12:39:07.280292 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:39:07.280266 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9c9d1f3_e29c_42d4_96e6_765da8de6ab9.slice/crio-b31c36a17ea35a625e9e6a669a21289b6baea6c208eb31a106384f0c541a1d69 WatchSource:0}: Error finding container b31c36a17ea35a625e9e6a669a21289b6baea6c208eb31a106384f0c541a1d69: Status 404 returned error can't find the container with id b31c36a17ea35a625e9e6a669a21289b6baea6c208eb31a106384f0c541a1d69 Oct 30 12:39:07.283335 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:39:07.283297 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:39:07.283466 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:39:07.283382 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:39:07.283466 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:39:07.283429 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:39:07.286967 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:39:07.286933 4908 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11575-e07a4a\" is forbidden: User \"system:node:ip-10-0-141-251.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-251.ec2.internal' and this object" podUID="76dea331-6413-45d5-9603-dac5f95a8103" pod="opendatahub/test-curl-11575-e07a4a" Oct 30 12:39:07.613771 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:39:07.613694 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="383e8645-c168-4e9c-859a-55a9781dbfb1" path="/var/lib/kubelet/pods/383e8645-c168-4e9c-859a-55a9781dbfb1/volumes" Oct 30 12:39:07.614273 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:39:07.614250 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76dea331-6413-45d5-9603-dac5f95a8103" path="/var/lib/kubelet/pods/76dea331-6413-45d5-9603-dac5f95a8103/volumes" Oct 30 12:39:08.276682 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:39:08.276653 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-73bafc-openshift-default-64bfdb8459-rfkqt" event={"ID":"c9c9d1f3-e29c-42d4-96e6-765da8de6ab9","Type":"ContainerStarted","Data":"b7fd1b50b654a93ce527f34005feaed728e662acddbe2b46b84e874364693d3a"} Oct 30 12:39:08.277045 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:39:08.276708 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-73bafc-openshift-default-64bfdb8459-rfkqt" event={"ID":"c9c9d1f3-e29c-42d4-96e6-765da8de6ab9","Type":"ContainerStarted","Data":"b31c36a17ea35a625e9e6a669a21289b6baea6c208eb31a106384f0c541a1d69"} Oct 30 12:39:08.331289 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:39:08.331239 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-73bafc-openshift-default-64bfdb8459-rfkqt" podStartSLOduration=2.331221981 podStartE2EDuration="2.331221981s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:39:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:39:08.329856546 +0000 UTC m=+2704.844285309" watchObservedRunningTime="2026-10-30 12:39:08.331221981 +0000 UTC m=+2704.845650728" Oct 30 12:39:08.518878 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:39:08.518774 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11575-54de04" Oct 30 12:39:08.604977 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:39:08.604941 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61d6dd36-b05b-45cb-8484-9ae0bb648b23-kube-api-access-gkgpk\" (UniqueName: \"kubernetes.io/projected/61d6dd36-b05b-45cb-8484-9ae0bb648b23-kube-api-access-gkgpk\") pod \"61d6dd36-b05b-45cb-8484-9ae0bb648b23\" (UID: \"61d6dd36-b05b-45cb-8484-9ae0bb648b23\") " Oct 30 12:39:08.607293 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:39:08.607257 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61d6dd36-b05b-45cb-8484-9ae0bb648b23-kube-api-access-gkgpk" pod "61d6dd36-b05b-45cb-8484-9ae0bb648b23" (UID: "61d6dd36-b05b-45cb-8484-9ae0bb648b23"). InnerVolumeSpecName "kube-api-access-gkgpk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:39:08.716237 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:39:08.716171 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gkgpk\" (UniqueName: \"kubernetes.io/projected/61d6dd36-b05b-45cb-8484-9ae0bb648b23-kube-api-access-gkgpk\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:39:08.920395 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:39:08.920311 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-73bafc-openshift-default-64bfdb8459-rfkqt" Oct 30 12:39:08.925922 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:39:08.925895 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-73bafc-openshift-default-64bfdb8459-rfkqt" Oct 30 12:39:09.296773 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:39:09.296737 4908 generic.go:356] "Generic (PLEG): container finished" podID="61d6dd36-b05b-45cb-8484-9ae0bb648b23" containerID="47847b324a322cf928b6b48250720018ff59da50383b3e71c25619542e2b4d82" exitCode=137 Oct 30 12:39:09.297221 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:39:09.296811 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11575-54de04" Oct 30 12:39:09.297221 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:39:09.296852 4908 scope.go:122] "RemoveContainer" containerID="47847b324a322cf928b6b48250720018ff59da50383b3e71c25619542e2b4d82" Oct 30 12:39:09.297362 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:39:09.297352 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-73bafc-openshift-default-64bfdb8459-rfkqt" Oct 30 12:39:09.298208 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:39:09.298092 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-73bafc-openshift-default-64bfdb8459-rfkqt" Oct 30 12:39:09.307609 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:39:09.307593 4908 scope.go:122] "RemoveContainer" containerID="47847b324a322cf928b6b48250720018ff59da50383b3e71c25619542e2b4d82" Oct 30 12:39:09.307873 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:39:09.307853 4908 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47847b324a322cf928b6b48250720018ff59da50383b3e71c25619542e2b4d82\": container with ID starting with 47847b324a322cf928b6b48250720018ff59da50383b3e71c25619542e2b4d82 not found: ID does not exist" containerID="47847b324a322cf928b6b48250720018ff59da50383b3e71c25619542e2b4d82" Oct 30 12:39:09.307953 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:39:09.307880 4908 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47847b324a322cf928b6b48250720018ff59da50383b3e71c25619542e2b4d82"} err="failed to get container status \"47847b324a322cf928b6b48250720018ff59da50383b3e71c25619542e2b4d82\": rpc error: code = NotFound desc = could not find container \"47847b324a322cf928b6b48250720018ff59da50383b3e71c25619542e2b4d82\": container with ID starting with 47847b324a322cf928b6b48250720018ff59da50383b3e71c25619542e2b4d82 not found: ID does not exist" Oct 30 12:39:09.352155 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:39:09.352114 4908 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11575-54de04\" is forbidden: User \"system:node:ip-10-0-141-251.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-251.ec2.internal' and this object" podUID="61d6dd36-b05b-45cb-8484-9ae0bb648b23" pod="opendatahub/test-curl-11575-54de04" Oct 30 12:39:09.607075 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:39:09.607005 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61d6dd36-b05b-45cb-8484-9ae0bb648b23" path="/var/lib/kubelet/pods/61d6dd36-b05b-45cb-8484-9ae0bb648b23/volumes" Oct 30 12:39:12.028242 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:39:12.028207 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bcafdbfc-ljf77"] Oct 30 12:39:12.041246 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:39:12.041219 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bcafdbfc-ljf77"] Oct 30 12:39:13.607849 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:39:13.607803 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe1dc6d4-2233-47d2-a637-f5d8531a8b4f" path="/var/lib/kubelet/pods/fe1dc6d4-2233-47d2-a637-f5d8531a8b4f/volumes" Oct 30 12:39:19.684472 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:39:19.684430 4908 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vn6vv" containerName="extractor" Oct 30 12:39:20.029140 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:39:20.029092 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8faf1aa4-lr8fq"] Oct 30 12:39:20.040205 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:39:20.040173 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8faf1aa4-lr8fq"] Oct 30 12:39:21.028094 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:39:21.028065 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-34ce52bf-fxf5g"] Oct 30 12:39:21.038512 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:39:21.038482 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-34ce52bf-fxf5g"] Oct 30 12:39:21.609331 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:39:21.609295 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36caef27-f0ce-486f-ab9c-ef667e8b56dc" path="/var/lib/kubelet/pods/36caef27-f0ce-486f-ab9c-ef667e8b56dc/volumes" Oct 30 12:39:21.609765 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:39:21.609739 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d84d4552-9bf6-427b-bd5b-78ee31a3f99d" path="/var/lib/kubelet/pods/d84d4552-9bf6-427b-bd5b-78ee31a3f99d/volumes" Oct 30 12:39:23.033574 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:39:23.033538 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0eee5157-k46dv"] Oct 30 12:39:23.043897 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:39:23.043871 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0eee5157-k46dv"] Oct 30 12:39:23.630108 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:39:23.630073 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d27421c-c8a1-41ef-85bd-9c18fd5d6aba" path="/var/lib/kubelet/pods/7d27421c-c8a1-41ef-85bd-9c18fd5d6aba/volumes" Oct 30 12:39:42.207051 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:39:42.207013 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9wwfc"] Oct 30 12:39:49.685002 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:39:49.684909 4908 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vn6vv" containerName="extractor" Oct 30 12:40:02.662651 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:02.662614 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-dd0628-openshift-default-54686d76fc-x98kl"] Oct 30 12:40:02.691122 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:02.691092 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-dd0628-openshift-default-54686d76fc-x98kl" Oct 30 12:40:02.695002 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:02.694973 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-dd0628-openshift-default-54686d76fc-x98kl"] Oct 30 12:40:02.707037 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:02.707012 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-dd0628-openshift-default-dockercfg-cv2d6\"" Oct 30 12:40:02.813333 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:02.813297 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d070106-c79c-44d5-a7a9-b766ee82b439-istio-token\") pod \"e2e-shared-b-w0-dd0628-openshift-default-54686d76fc-x98kl\" (UID: \"2d070106-c79c-44d5-a7a9-b766ee82b439\") " pod="openshift-ingress/e2e-shared-b-w0-dd0628-openshift-default-54686d76fc-x98kl" Oct 30 12:40:02.813508 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:02.813467 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d070106-c79c-44d5-a7a9-b766ee82b439-istio-podinfo\") pod \"e2e-shared-b-w0-dd0628-openshift-default-54686d76fc-x98kl\" (UID: \"2d070106-c79c-44d5-a7a9-b766ee82b439\") " pod="openshift-ingress/e2e-shared-b-w0-dd0628-openshift-default-54686d76fc-x98kl" Oct 30 12:40:02.813582 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:02.813528 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d070106-c79c-44d5-a7a9-b766ee82b439-istio-data\") pod \"e2e-shared-b-w0-dd0628-openshift-default-54686d76fc-x98kl\" (UID: \"2d070106-c79c-44d5-a7a9-b766ee82b439\") " pod="openshift-ingress/e2e-shared-b-w0-dd0628-openshift-default-54686d76fc-x98kl" Oct 30 12:40:02.813668 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:02.813650 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d070106-c79c-44d5-a7a9-b766ee82b439-istio-envoy\") pod \"e2e-shared-b-w0-dd0628-openshift-default-54686d76fc-x98kl\" (UID: \"2d070106-c79c-44d5-a7a9-b766ee82b439\") " pod="openshift-ingress/e2e-shared-b-w0-dd0628-openshift-default-54686d76fc-x98kl" Oct 30 12:40:02.813729 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:02.813703 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgchc\" (UniqueName: \"kubernetes.io/projected/2d070106-c79c-44d5-a7a9-b766ee82b439-kube-api-access-vgchc\") pod \"e2e-shared-b-w0-dd0628-openshift-default-54686d76fc-x98kl\" (UID: \"2d070106-c79c-44d5-a7a9-b766ee82b439\") " pod="openshift-ingress/e2e-shared-b-w0-dd0628-openshift-default-54686d76fc-x98kl" Oct 30 12:40:02.813868 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:02.813846 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d070106-c79c-44d5-a7a9-b766ee82b439-credential-socket\") pod \"e2e-shared-b-w0-dd0628-openshift-default-54686d76fc-x98kl\" (UID: \"2d070106-c79c-44d5-a7a9-b766ee82b439\") " pod="openshift-ingress/e2e-shared-b-w0-dd0628-openshift-default-54686d76fc-x98kl" Oct 30 12:40:02.813954 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:02.813880 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d070106-c79c-44d5-a7a9-b766ee82b439-istiod-ca-cert\") pod \"e2e-shared-b-w0-dd0628-openshift-default-54686d76fc-x98kl\" (UID: \"2d070106-c79c-44d5-a7a9-b766ee82b439\") " pod="openshift-ingress/e2e-shared-b-w0-dd0628-openshift-default-54686d76fc-x98kl" Oct 30 12:40:02.813990 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:02.813960 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d070106-c79c-44d5-a7a9-b766ee82b439-workload-socket\") pod \"e2e-shared-b-w0-dd0628-openshift-default-54686d76fc-x98kl\" (UID: \"2d070106-c79c-44d5-a7a9-b766ee82b439\") " pod="openshift-ingress/e2e-shared-b-w0-dd0628-openshift-default-54686d76fc-x98kl" Oct 30 12:40:02.814076 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:02.814053 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d070106-c79c-44d5-a7a9-b766ee82b439-workload-certs\") pod \"e2e-shared-b-w0-dd0628-openshift-default-54686d76fc-x98kl\" (UID: \"2d070106-c79c-44d5-a7a9-b766ee82b439\") " pod="openshift-ingress/e2e-shared-b-w0-dd0628-openshift-default-54686d76fc-x98kl" Oct 30 12:40:02.917013 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:02.916864 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d070106-c79c-44d5-a7a9-b766ee82b439-workload-certs\") pod \"e2e-shared-b-w0-dd0628-openshift-default-54686d76fc-x98kl\" (UID: \"2d070106-c79c-44d5-a7a9-b766ee82b439\") " pod="openshift-ingress/e2e-shared-b-w0-dd0628-openshift-default-54686d76fc-x98kl" Oct 30 12:40:02.917332 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:02.917273 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d070106-c79c-44d5-a7a9-b766ee82b439-istio-token\") pod \"e2e-shared-b-w0-dd0628-openshift-default-54686d76fc-x98kl\" (UID: \"2d070106-c79c-44d5-a7a9-b766ee82b439\") " pod="openshift-ingress/e2e-shared-b-w0-dd0628-openshift-default-54686d76fc-x98kl" Oct 30 12:40:02.921976 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:02.921944 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d070106-c79c-44d5-a7a9-b766ee82b439-istio-podinfo\") pod \"e2e-shared-b-w0-dd0628-openshift-default-54686d76fc-x98kl\" (UID: \"2d070106-c79c-44d5-a7a9-b766ee82b439\") " pod="openshift-ingress/e2e-shared-b-w0-dd0628-openshift-default-54686d76fc-x98kl" Oct 30 12:40:02.924684 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:02.924650 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d070106-c79c-44d5-a7a9-b766ee82b439-istio-data\") pod \"e2e-shared-b-w0-dd0628-openshift-default-54686d76fc-x98kl\" (UID: \"2d070106-c79c-44d5-a7a9-b766ee82b439\") " pod="openshift-ingress/e2e-shared-b-w0-dd0628-openshift-default-54686d76fc-x98kl" Oct 30 12:40:02.924989 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:02.924969 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d070106-c79c-44d5-a7a9-b766ee82b439-istio-envoy\") pod \"e2e-shared-b-w0-dd0628-openshift-default-54686d76fc-x98kl\" (UID: \"2d070106-c79c-44d5-a7a9-b766ee82b439\") " pod="openshift-ingress/e2e-shared-b-w0-dd0628-openshift-default-54686d76fc-x98kl" Oct 30 12:40:02.927449 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:02.921761 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d070106-c79c-44d5-a7a9-b766ee82b439-workload-certs\") pod \"e2e-shared-b-w0-dd0628-openshift-default-54686d76fc-x98kl\" (UID: \"2d070106-c79c-44d5-a7a9-b766ee82b439\") " pod="openshift-ingress/e2e-shared-b-w0-dd0628-openshift-default-54686d76fc-x98kl" Oct 30 12:40:02.927449 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:02.925158 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgchc\" (UniqueName: \"kubernetes.io/projected/2d070106-c79c-44d5-a7a9-b766ee82b439-kube-api-access-vgchc\") pod \"e2e-shared-b-w0-dd0628-openshift-default-54686d76fc-x98kl\" (UID: \"2d070106-c79c-44d5-a7a9-b766ee82b439\") " pod="openshift-ingress/e2e-shared-b-w0-dd0628-openshift-default-54686d76fc-x98kl" Oct 30 12:40:02.927449 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:02.925281 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d070106-c79c-44d5-a7a9-b766ee82b439-credential-socket\") pod \"e2e-shared-b-w0-dd0628-openshift-default-54686d76fc-x98kl\" (UID: \"2d070106-c79c-44d5-a7a9-b766ee82b439\") " pod="openshift-ingress/e2e-shared-b-w0-dd0628-openshift-default-54686d76fc-x98kl" Oct 30 12:40:02.927449 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:02.925309 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d070106-c79c-44d5-a7a9-b766ee82b439-istiod-ca-cert\") pod \"e2e-shared-b-w0-dd0628-openshift-default-54686d76fc-x98kl\" (UID: \"2d070106-c79c-44d5-a7a9-b766ee82b439\") " pod="openshift-ingress/e2e-shared-b-w0-dd0628-openshift-default-54686d76fc-x98kl" Oct 30 12:40:02.927449 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:02.925789 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d070106-c79c-44d5-a7a9-b766ee82b439-workload-socket\") pod \"e2e-shared-b-w0-dd0628-openshift-default-54686d76fc-x98kl\" (UID: \"2d070106-c79c-44d5-a7a9-b766ee82b439\") " pod="openshift-ingress/e2e-shared-b-w0-dd0628-openshift-default-54686d76fc-x98kl" Oct 30 12:40:02.927449 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:02.926616 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d070106-c79c-44d5-a7a9-b766ee82b439-istio-data\") pod \"e2e-shared-b-w0-dd0628-openshift-default-54686d76fc-x98kl\" (UID: \"2d070106-c79c-44d5-a7a9-b766ee82b439\") " pod="openshift-ingress/e2e-shared-b-w0-dd0628-openshift-default-54686d76fc-x98kl" Oct 30 12:40:02.927449 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:02.926882 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d070106-c79c-44d5-a7a9-b766ee82b439-credential-socket\") pod \"e2e-shared-b-w0-dd0628-openshift-default-54686d76fc-x98kl\" (UID: \"2d070106-c79c-44d5-a7a9-b766ee82b439\") " pod="openshift-ingress/e2e-shared-b-w0-dd0628-openshift-default-54686d76fc-x98kl" Oct 30 12:40:02.927449 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:02.927415 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d070106-c79c-44d5-a7a9-b766ee82b439-istiod-ca-cert\") pod \"e2e-shared-b-w0-dd0628-openshift-default-54686d76fc-x98kl\" (UID: \"2d070106-c79c-44d5-a7a9-b766ee82b439\") " pod="openshift-ingress/e2e-shared-b-w0-dd0628-openshift-default-54686d76fc-x98kl" Oct 30 12:40:02.933514 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:02.928711 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d070106-c79c-44d5-a7a9-b766ee82b439-workload-socket\") pod \"e2e-shared-b-w0-dd0628-openshift-default-54686d76fc-x98kl\" (UID: \"2d070106-c79c-44d5-a7a9-b766ee82b439\") " pod="openshift-ingress/e2e-shared-b-w0-dd0628-openshift-default-54686d76fc-x98kl" Oct 30 12:40:02.988926 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:02.968368 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d070106-c79c-44d5-a7a9-b766ee82b439-istio-token\") pod \"e2e-shared-b-w0-dd0628-openshift-default-54686d76fc-x98kl\" (UID: \"2d070106-c79c-44d5-a7a9-b766ee82b439\") " pod="openshift-ingress/e2e-shared-b-w0-dd0628-openshift-default-54686d76fc-x98kl" Oct 30 12:40:02.990208 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:02.990179 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d070106-c79c-44d5-a7a9-b766ee82b439-istio-podinfo\") pod \"e2e-shared-b-w0-dd0628-openshift-default-54686d76fc-x98kl\" (UID: \"2d070106-c79c-44d5-a7a9-b766ee82b439\") " pod="openshift-ingress/e2e-shared-b-w0-dd0628-openshift-default-54686d76fc-x98kl" Oct 30 12:40:03.002637 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:03.002566 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgchc\" (UniqueName: \"kubernetes.io/projected/2d070106-c79c-44d5-a7a9-b766ee82b439-kube-api-access-vgchc\") pod \"e2e-shared-b-w0-dd0628-openshift-default-54686d76fc-x98kl\" (UID: \"2d070106-c79c-44d5-a7a9-b766ee82b439\") " pod="openshift-ingress/e2e-shared-b-w0-dd0628-openshift-default-54686d76fc-x98kl" Oct 30 12:40:03.013881 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:03.013134 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d070106-c79c-44d5-a7a9-b766ee82b439-istio-envoy\") pod \"e2e-shared-b-w0-dd0628-openshift-default-54686d76fc-x98kl\" (UID: \"2d070106-c79c-44d5-a7a9-b766ee82b439\") " pod="openshift-ingress/e2e-shared-b-w0-dd0628-openshift-default-54686d76fc-x98kl" Oct 30 12:40:03.014479 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:03.014267 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-dd0628-openshift-default-54686d76fc-x98kl" Oct 30 12:40:03.408093 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:03.408063 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-dd0628-openshift-default-54686d76fc-x98kl"] Oct 30 12:40:03.420137 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:03.420092 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:40:03.420378 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:03.420355 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:40:03.420702 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:03.420670 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:40:03.925283 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:03.925249 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-dd0628-openshift-default-54686d76fc-x98kl" event={"ID":"2d070106-c79c-44d5-a7a9-b766ee82b439","Type":"ContainerStarted","Data":"43eb1ea14c35db3a88694c94fd544c2b1516431a0afe2c55170dceb70cbbd3ba"} Oct 30 12:40:03.925743 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:03.925306 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-dd0628-openshift-default-54686d76fc-x98kl" event={"ID":"2d070106-c79c-44d5-a7a9-b766ee82b439","Type":"ContainerStarted","Data":"090c66c5bb70b3ba6a07eb6de03272e4a5d079128aa493a23c85db202a843c00"} Oct 30 12:40:03.981866 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:03.981780 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-dd0628-openshift-default-54686d76fc-x98kl" podStartSLOduration=1.981762168 podStartE2EDuration="1.981762168s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:40:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:40:03.978408779 +0000 UTC m=+2760.492837538" watchObservedRunningTime="2026-10-30 12:40:03.981762168 +0000 UTC m=+2760.496190914" Oct 30 12:40:04.015096 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:04.015031 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-dd0628-openshift-default-54686d76fc-x98kl" Oct 30 12:40:04.022096 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:04.022070 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-dd0628-openshift-default-54686d76fc-x98kl" Oct 30 12:40:04.933062 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:04.933032 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-dd0628-openshift-default-54686d76fc-x98kl" Oct 30 12:40:04.933984 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:04.933962 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-dd0628-openshift-default-54686d76fc-x98kl" Oct 30 12:40:06.988036 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:06.987979 4908 scope.go:122] "RemoveContainer" containerID="1e8b2b85b4566566c62dee4f911343ccab3ba8caed248fcd20e4805506acd8af" Oct 30 12:40:07.009716 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:07.009686 4908 scope.go:122] "RemoveContainer" containerID="e9463f386879eb8f29856437a007fd02f7c48b25e6498e368970c8c04a2b2f1e" Oct 30 12:40:07.019288 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:07.019269 4908 scope.go:122] "RemoveContainer" containerID="1803ee80b5b4152f1cdcbbb792546378663092f01ba6dd3c93494fcc87ee07e3" Oct 30 12:40:07.027940 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:07.027922 4908 scope.go:122] "RemoveContainer" containerID="8b6a7259c94f59fb6e0fa357b6be5035db28a0f51158ab0790cbf48d010115e4" Oct 30 12:40:07.036173 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:07.036156 4908 scope.go:122] "RemoveContainer" containerID="c6f14a6e89ea764022f4ecb30fdb1926cc0b0c9f8ae5b51c6833bf7c350ae129" Oct 30 12:40:07.044310 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:07.044292 4908 scope.go:122] "RemoveContainer" containerID="d7ab541335fe03801bb86fbc1c59365d6b92b19ef428a61f307d57026731a213" Oct 30 12:40:07.052209 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:07.052190 4908 scope.go:122] "RemoveContainer" containerID="ad2ba82f46db36add7a67bb7a945c5037d0fcd6bd972a2d2a61dca693b5fba80" Oct 30 12:40:07.060495 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:07.060477 4908 scope.go:122] "RemoveContainer" containerID="52cb89440105cc8a315366335f4fb8101e95061ad0e6b461a0fc46c948735cd0" Oct 30 12:40:07.069963 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:07.069943 4908 scope.go:122] "RemoveContainer" containerID="fd840c81d5911f4fc6eee3c84fbfcb6cfd846b56198021948f75292d4fb42f89" Oct 30 12:40:14.057785 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:14.057750 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53111ea1-6dc7q"] Oct 30 12:40:14.068851 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:14.066982 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53111ea1-6dc7q" Oct 30 12:40:14.089943 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:14.089840 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s75hq\"" Oct 30 12:40:14.089943 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:14.089899 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 12:40:14.090546 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:14.090494 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 12:40:14.091614 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:14.091579 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 12:40:14.092315 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:14.092298 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53111ea1-6dc7q"] Oct 30 12:40:14.241578 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:14.241544 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5cfc\" (UniqueName: \"kubernetes.io/projected/aaccb644-e26d-4562-8cf1-bc344cd801e8-kube-api-access-m5cfc\") pod \"maas-api-revoke-sub-53111ea1-6dc7q\" (UID: \"aaccb644-e26d-4562-8cf1-bc344cd801e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53111ea1-6dc7q" Oct 30 12:40:14.241775 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:14.241695 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aaccb644-e26d-4562-8cf1-bc344cd801e8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-53111ea1-6dc7q\" (UID: \"aaccb644-e26d-4562-8cf1-bc344cd801e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53111ea1-6dc7q" Oct 30 12:40:14.241871 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:14.241820 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aaccb644-e26d-4562-8cf1-bc344cd801e8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-53111ea1-6dc7q\" (UID: \"aaccb644-e26d-4562-8cf1-bc344cd801e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53111ea1-6dc7q" Oct 30 12:40:14.343729 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:14.343648 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aaccb644-e26d-4562-8cf1-bc344cd801e8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-53111ea1-6dc7q\" (UID: \"aaccb644-e26d-4562-8cf1-bc344cd801e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53111ea1-6dc7q" Oct 30 12:40:14.343898 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:14.343762 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aaccb644-e26d-4562-8cf1-bc344cd801e8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-53111ea1-6dc7q\" (UID: \"aaccb644-e26d-4562-8cf1-bc344cd801e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53111ea1-6dc7q" Oct 30 12:40:14.343962 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:14.343943 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5cfc\" (UniqueName: \"kubernetes.io/projected/aaccb644-e26d-4562-8cf1-bc344cd801e8-kube-api-access-m5cfc\") pod \"maas-api-revoke-sub-53111ea1-6dc7q\" (UID: \"aaccb644-e26d-4562-8cf1-bc344cd801e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53111ea1-6dc7q" Oct 30 12:40:14.344541 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:14.344518 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aaccb644-e26d-4562-8cf1-bc344cd801e8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-53111ea1-6dc7q\" (UID: \"aaccb644-e26d-4562-8cf1-bc344cd801e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53111ea1-6dc7q" Oct 30 12:40:14.346177 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:14.346153 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aaccb644-e26d-4562-8cf1-bc344cd801e8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-53111ea1-6dc7q\" (UID: \"aaccb644-e26d-4562-8cf1-bc344cd801e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53111ea1-6dc7q" Oct 30 12:40:14.371580 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:14.371524 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5cfc\" (UniqueName: \"kubernetes.io/projected/aaccb644-e26d-4562-8cf1-bc344cd801e8-kube-api-access-m5cfc\") pod \"maas-api-revoke-sub-53111ea1-6dc7q\" (UID: \"aaccb644-e26d-4562-8cf1-bc344cd801e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53111ea1-6dc7q" Oct 30 12:40:14.411474 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:14.411445 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53111ea1-6dc7q" Oct 30 12:40:14.752439 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:14.750175 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-288d314b-qtpns"] Oct 30 12:40:14.799760 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:14.798415 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53111ea1-6dc7q"] Oct 30 12:40:14.799760 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:14.798464 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-288d314b-qtpns"] Oct 30 12:40:14.799760 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:14.798552 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-288d314b-qtpns" Oct 30 12:40:14.980691 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:14.980654 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s62v6\" (UniqueName: \"kubernetes.io/projected/359a81f9-7309-414e-924c-f82fbfa4037d-kube-api-access-s62v6\") pod \"maas-api-revoke-sub-288d314b-qtpns\" (UID: \"359a81f9-7309-414e-924c-f82fbfa4037d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-288d314b-qtpns" Oct 30 12:40:14.981070 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:14.981041 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/359a81f9-7309-414e-924c-f82fbfa4037d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-288d314b-qtpns\" (UID: \"359a81f9-7309-414e-924c-f82fbfa4037d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-288d314b-qtpns" Oct 30 12:40:14.981418 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:14.981395 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/359a81f9-7309-414e-924c-f82fbfa4037d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-288d314b-qtpns\" (UID: \"359a81f9-7309-414e-924c-f82fbfa4037d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-288d314b-qtpns" Oct 30 12:40:15.023386 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:15.023355 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53111ea1-6dc7q" event={"ID":"aaccb644-e26d-4562-8cf1-bc344cd801e8","Type":"ContainerStarted","Data":"85e78af667f5e208291eb9970eb472562388cfa1a76712413c6c97b53e53cf44"} Oct 30 12:40:15.086294 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:15.084709 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/359a81f9-7309-414e-924c-f82fbfa4037d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-288d314b-qtpns\" (UID: \"359a81f9-7309-414e-924c-f82fbfa4037d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-288d314b-qtpns" Oct 30 12:40:15.086294 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:15.084784 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s62v6\" (UniqueName: \"kubernetes.io/projected/359a81f9-7309-414e-924c-f82fbfa4037d-kube-api-access-s62v6\") pod \"maas-api-revoke-sub-288d314b-qtpns\" (UID: \"359a81f9-7309-414e-924c-f82fbfa4037d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-288d314b-qtpns" Oct 30 12:40:15.086294 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:15.085020 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/359a81f9-7309-414e-924c-f82fbfa4037d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-288d314b-qtpns\" (UID: \"359a81f9-7309-414e-924c-f82fbfa4037d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-288d314b-qtpns" Oct 30 12:40:15.086294 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:15.085926 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/359a81f9-7309-414e-924c-f82fbfa4037d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-288d314b-qtpns\" (UID: \"359a81f9-7309-414e-924c-f82fbfa4037d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-288d314b-qtpns" Oct 30 12:40:15.089311 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:15.089284 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/359a81f9-7309-414e-924c-f82fbfa4037d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-288d314b-qtpns\" (UID: \"359a81f9-7309-414e-924c-f82fbfa4037d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-288d314b-qtpns" Oct 30 12:40:15.106794 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:15.106772 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s62v6\" (UniqueName: \"kubernetes.io/projected/359a81f9-7309-414e-924c-f82fbfa4037d-kube-api-access-s62v6\") pod \"maas-api-revoke-sub-288d314b-qtpns\" (UID: \"359a81f9-7309-414e-924c-f82fbfa4037d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-288d314b-qtpns" Oct 30 12:40:15.119941 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:15.119922 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-288d314b-qtpns" Oct 30 12:40:15.507911 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:15.507881 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-288d314b-qtpns"] Oct 30 12:40:15.509753 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:40:15.509726 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod359a81f9_7309_414e_924c_f82fbfa4037d.slice/crio-05f3e13f562b891b18d0d90eae82c75bbb226369fa7a778be68f22d9407fc9a4 WatchSource:0}: Error finding container 05f3e13f562b891b18d0d90eae82c75bbb226369fa7a778be68f22d9407fc9a4: Status 404 returned error can't find the container with id 05f3e13f562b891b18d0d90eae82c75bbb226369fa7a778be68f22d9407fc9a4 Oct 30 12:40:16.031882 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:16.031838 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-288d314b-qtpns" event={"ID":"359a81f9-7309-414e-924c-f82fbfa4037d","Type":"ContainerStarted","Data":"5033b8b333aa8c77bc2396964e434e9f681e968a557dd9d91142cd168718b50a"} Oct 30 12:40:16.032030 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:16.031907 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-288d314b-qtpns" event={"ID":"359a81f9-7309-414e-924c-f82fbfa4037d","Type":"ContainerStarted","Data":"05f3e13f562b891b18d0d90eae82c75bbb226369fa7a778be68f22d9407fc9a4"} Oct 30 12:40:16.033520 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:16.033491 4908 generic.go:356] "Generic (PLEG): container finished" podID="aaccb644-e26d-4562-8cf1-bc344cd801e8" containerID="f5ef0ed8a525c358973d9cfabd5198a23117eaf13deb113916941bbea7be4a29" exitCode=0 Oct 30 12:40:16.033631 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:16.033558 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53111ea1-6dc7q" event={"ID":"aaccb644-e26d-4562-8cf1-bc344cd801e8","Type":"ContainerDied","Data":"f5ef0ed8a525c358973d9cfabd5198a23117eaf13deb113916941bbea7be4a29"} Oct 30 12:40:16.123685 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:16.123583 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-288d314b-qtpns" podStartSLOduration=2.123569698 podStartE2EDuration="2.123569698s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:40:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:40:16.081559097 +0000 UTC m=+2772.595987839" watchObservedRunningTime="2026-10-30 12:40:16.123569698 +0000 UTC m=+2772.637998442" Oct 30 12:40:17.048077 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:17.048033 4908 generic.go:356] "Generic (PLEG): container finished" podID="359a81f9-7309-414e-924c-f82fbfa4037d" containerID="5033b8b333aa8c77bc2396964e434e9f681e968a557dd9d91142cd168718b50a" exitCode=0 Oct 30 12:40:17.048291 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:17.048165 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-288d314b-qtpns" event={"ID":"359a81f9-7309-414e-924c-f82fbfa4037d","Type":"ContainerDied","Data":"5033b8b333aa8c77bc2396964e434e9f681e968a557dd9d91142cd168718b50a"} Oct 30 12:40:17.318735 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:17.318708 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53111ea1-6dc7q" Oct 30 12:40:17.446165 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:17.446131 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aaccb644-e26d-4562-8cf1-bc344cd801e8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aaccb644-e26d-4562-8cf1-bc344cd801e8-maas-api-openshift-ca\") pod \"aaccb644-e26d-4562-8cf1-bc344cd801e8\" (UID: \"aaccb644-e26d-4562-8cf1-bc344cd801e8\") " Oct 30 12:40:17.446402 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:17.446278 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aaccb644-e26d-4562-8cf1-bc344cd801e8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aaccb644-e26d-4562-8cf1-bc344cd801e8-maas-api-xks-ca\") pod \"aaccb644-e26d-4562-8cf1-bc344cd801e8\" (UID: \"aaccb644-e26d-4562-8cf1-bc344cd801e8\") " Oct 30 12:40:17.446497 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:17.446431 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aaccb644-e26d-4562-8cf1-bc344cd801e8-kube-api-access-m5cfc\" (UniqueName: \"kubernetes.io/projected/aaccb644-e26d-4562-8cf1-bc344cd801e8-kube-api-access-m5cfc\") pod \"aaccb644-e26d-4562-8cf1-bc344cd801e8\" (UID: \"aaccb644-e26d-4562-8cf1-bc344cd801e8\") " Oct 30 12:40:17.446611 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:17.446568 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aaccb644-e26d-4562-8cf1-bc344cd801e8-maas-api-openshift-ca" pod "aaccb644-e26d-4562-8cf1-bc344cd801e8" (UID: "aaccb644-e26d-4562-8cf1-bc344cd801e8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:40:17.446992 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:17.446963 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aaccb644-e26d-4562-8cf1-bc344cd801e8-maas-api-openshift-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:40:17.448675 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:17.448647 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aaccb644-e26d-4562-8cf1-bc344cd801e8-kube-api-access-m5cfc" pod "aaccb644-e26d-4562-8cf1-bc344cd801e8" (UID: "aaccb644-e26d-4562-8cf1-bc344cd801e8"). InnerVolumeSpecName "kube-api-access-m5cfc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:40:17.448775 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:17.448720 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aaccb644-e26d-4562-8cf1-bc344cd801e8-maas-api-xks-ca" pod "aaccb644-e26d-4562-8cf1-bc344cd801e8" (UID: "aaccb644-e26d-4562-8cf1-bc344cd801e8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:40:17.547904 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:17.547873 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aaccb644-e26d-4562-8cf1-bc344cd801e8-maas-api-xks-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:40:17.547904 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:17.547900 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5cfc\" (UniqueName: \"kubernetes.io/projected/aaccb644-e26d-4562-8cf1-bc344cd801e8-kube-api-access-m5cfc\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:40:18.057708 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:18.057680 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53111ea1-6dc7q" Oct 30 12:40:18.057954 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:18.057715 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53111ea1-6dc7q" event={"ID":"aaccb644-e26d-4562-8cf1-bc344cd801e8","Type":"ContainerDied","Data":"85e78af667f5e208291eb9970eb472562388cfa1a76712413c6c97b53e53cf44"} Oct 30 12:40:18.057954 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:18.057770 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="85e78af667f5e208291eb9970eb472562388cfa1a76712413c6c97b53e53cf44" Oct 30 12:40:18.281252 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:18.281226 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-288d314b-qtpns" Oct 30 12:40:18.366669 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:18.366584 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/359a81f9-7309-414e-924c-f82fbfa4037d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/359a81f9-7309-414e-924c-f82fbfa4037d-maas-api-openshift-ca\") pod \"359a81f9-7309-414e-924c-f82fbfa4037d\" (UID: \"359a81f9-7309-414e-924c-f82fbfa4037d\") " Oct 30 12:40:18.367245 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:18.367210 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/359a81f9-7309-414e-924c-f82fbfa4037d-kube-api-access-s62v6\" (UniqueName: \"kubernetes.io/projected/359a81f9-7309-414e-924c-f82fbfa4037d-kube-api-access-s62v6\") pod \"359a81f9-7309-414e-924c-f82fbfa4037d\" (UID: \"359a81f9-7309-414e-924c-f82fbfa4037d\") " Oct 30 12:40:18.367395 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:18.367377 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/359a81f9-7309-414e-924c-f82fbfa4037d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/359a81f9-7309-414e-924c-f82fbfa4037d-maas-api-xks-ca\") pod \"359a81f9-7309-414e-924c-f82fbfa4037d\" (UID: \"359a81f9-7309-414e-924c-f82fbfa4037d\") " Oct 30 12:40:18.367478 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:18.367011 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/359a81f9-7309-414e-924c-f82fbfa4037d-maas-api-openshift-ca" pod "359a81f9-7309-414e-924c-f82fbfa4037d" (UID: "359a81f9-7309-414e-924c-f82fbfa4037d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:40:18.368009 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:18.367983 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/359a81f9-7309-414e-924c-f82fbfa4037d-maas-api-openshift-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:40:18.370524 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:18.370444 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/359a81f9-7309-414e-924c-f82fbfa4037d-maas-api-xks-ca" pod "359a81f9-7309-414e-924c-f82fbfa4037d" (UID: "359a81f9-7309-414e-924c-f82fbfa4037d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:40:18.370524 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:18.370487 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/359a81f9-7309-414e-924c-f82fbfa4037d-kube-api-access-s62v6" pod "359a81f9-7309-414e-924c-f82fbfa4037d" (UID: "359a81f9-7309-414e-924c-f82fbfa4037d"). InnerVolumeSpecName "kube-api-access-s62v6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:40:18.469262 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:18.469232 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s62v6\" (UniqueName: \"kubernetes.io/projected/359a81f9-7309-414e-924c-f82fbfa4037d-kube-api-access-s62v6\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:40:18.469379 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:18.469266 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/359a81f9-7309-414e-924c-f82fbfa4037d-maas-api-xks-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:40:19.029642 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:19.029601 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bfc84a5-5vg6b"] Oct 30 12:40:19.039540 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:19.039505 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bfc84a5-5vg6b"] Oct 30 12:40:19.068973 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:19.068951 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-288d314b-qtpns" Oct 30 12:40:19.069121 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:19.069080 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-288d314b-qtpns" event={"ID":"359a81f9-7309-414e-924c-f82fbfa4037d","Type":"ContainerDied","Data":"05f3e13f562b891b18d0d90eae82c75bbb226369fa7a778be68f22d9407fc9a4"} Oct 30 12:40:19.069193 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:19.069137 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="05f3e13f562b891b18d0d90eae82c75bbb226369fa7a778be68f22d9407fc9a4" Oct 30 12:40:19.608308 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:19.608274 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2baaaf0-ce6b-48ee-93d3-5a34e266d5b1" path="/var/lib/kubelet/pods/f2baaaf0-ce6b-48ee-93d3-5a34e266d5b1/volumes" Oct 30 12:40:19.713781 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:40:19.713707 4908 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vn6vv" containerName="extractor" Oct 30 12:40:22.099917 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:22.099879 4908 generic.go:356] "Generic (PLEG): container finished" podID="c9c9d1f3-e29c-42d4-96e6-765da8de6ab9" containerID="b7fd1b50b654a93ce527f34005feaed728e662acddbe2b46b84e874364693d3a" exitCode=0 Oct 30 12:40:22.100472 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:22.099958 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-73bafc-openshift-default-64bfdb8459-rfkqt" event={"ID":"c9c9d1f3-e29c-42d4-96e6-765da8de6ab9","Type":"ContainerDied","Data":"b7fd1b50b654a93ce527f34005feaed728e662acddbe2b46b84e874364693d3a"} Oct 30 12:40:22.100472 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:22.100361 4908 scope.go:122] "RemoveContainer" containerID="b7fd1b50b654a93ce527f34005feaed728e662acddbe2b46b84e874364693d3a" Oct 30 12:40:22.102040 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:22.102011 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:40:22.102136 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:22.102080 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:40:22.102136 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:22.102109 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:40:22.828449 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:22.828412 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-9bpr4"] Oct 30 12:40:22.829085 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:22.829040 4908 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-9bpr4" podUID="434eb0c5-0505-4a8e-99c5-40eab19b2674" containerName="istio-proxy" containerID="cri-o://d7453a256dc0d4effd974255b973caf1cdcf397d33f3841cc20cecc576f9d8f4" gracePeriod=30 Oct 30 12:40:22.837886 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:22.837859 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-mcrfm"] Oct 30 12:40:22.838349 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:22.838295 4908 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-mcrfm" podUID="bcd6e5fa-dded-4c97-b694-a2a58766816e" containerName="istio-proxy" containerID="cri-o://811ec441fec961d76ef653554df13ecc16d64e0d186014187caf53628a2714a7" gracePeriod=30 Oct 30 12:40:23.109907 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:23.109865 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-73bafc-openshift-default-64bfdb8459-rfkqt" event={"ID":"c9c9d1f3-e29c-42d4-96e6-765da8de6ab9","Type":"ContainerStarted","Data":"19cf3f8257259bb14b18c415fd3fceb8beec31070c4c2a6fc2f61f373e2737a6"} Oct 30 12:40:23.919856 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:23.919797 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-73bafc-openshift-default-64bfdb8459-rfkqt" Oct 30 12:40:23.921164 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:23.921141 4908 patch_prober.go:29] interesting pod/e2e-shared-a-w0-73bafc-openshift-default-64bfdb8459-rfkqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" start-of-body= Oct 30 12:40:23.921296 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:23.921213 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-73bafc-openshift-default-64bfdb8459-rfkqt" podUID="c9c9d1f3-e29c-42d4-96e6-765da8de6ab9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" Oct 30 12:40:24.298114 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:24.298080 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-73bafc-openshift-default-64bfdb8459-rfkqt" Oct 30 12:40:24.396053 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:24.396023 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42836380-lr6fl"] Oct 30 12:40:24.396689 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:24.396667 4908 cpu_manager.go:414] "containerMap: removing container" podUID="359a81f9-7309-414e-924c-f82fbfa4037d" containerName="revoke-keys" Oct 30 12:40:24.396689 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:24.396690 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="359a81f9-7309-414e-924c-f82fbfa4037d" containerName="revoke-keys" Oct 30 12:40:24.396927 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:24.396710 4908 cpu_manager.go:414] "containerMap: removing container" podUID="aaccb644-e26d-4562-8cf1-bc344cd801e8" containerName="revoke-keys" Oct 30 12:40:24.396927 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:24.396718 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aaccb644-e26d-4562-8cf1-bc344cd801e8" containerName="revoke-keys" Oct 30 12:40:24.396927 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:24.396881 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="359a81f9-7309-414e-924c-f82fbfa4037d" containerName="revoke-keys" Oct 30 12:40:24.396927 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:24.396897 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="aaccb644-e26d-4562-8cf1-bc344cd801e8" containerName="revoke-keys" Oct 30 12:40:24.400470 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:24.400451 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42836380-lr6fl" Oct 30 12:40:24.415662 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:24.415641 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42836380-lr6fl"] Oct 30 12:40:24.426882 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:24.421620 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 12:40:24.426882 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:24.421688 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 12:40:24.426882 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:24.425096 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 12:40:24.426882 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:24.425099 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s75hq\"" Oct 30 12:40:24.591636 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:24.591540 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzp4x\" (UniqueName: \"kubernetes.io/projected/ec4c8aae-7586-4fdf-bddd-9ba01d54b071-kube-api-access-wzp4x\") pod \"maas-api-revoke-sub-42836380-lr6fl\" (UID: \"ec4c8aae-7586-4fdf-bddd-9ba01d54b071\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42836380-lr6fl" Oct 30 12:40:24.591861 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:24.591636 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec4c8aae-7586-4fdf-bddd-9ba01d54b071-maas-api-xks-ca\") pod \"maas-api-revoke-sub-42836380-lr6fl\" (UID: \"ec4c8aae-7586-4fdf-bddd-9ba01d54b071\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42836380-lr6fl" Oct 30 12:40:24.591861 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:24.591671 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec4c8aae-7586-4fdf-bddd-9ba01d54b071-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-42836380-lr6fl\" (UID: \"ec4c8aae-7586-4fdf-bddd-9ba01d54b071\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42836380-lr6fl" Oct 30 12:40:24.693474 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:24.693433 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzp4x\" (UniqueName: \"kubernetes.io/projected/ec4c8aae-7586-4fdf-bddd-9ba01d54b071-kube-api-access-wzp4x\") pod \"maas-api-revoke-sub-42836380-lr6fl\" (UID: \"ec4c8aae-7586-4fdf-bddd-9ba01d54b071\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42836380-lr6fl" Oct 30 12:40:24.693658 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:24.693561 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec4c8aae-7586-4fdf-bddd-9ba01d54b071-maas-api-xks-ca\") pod \"maas-api-revoke-sub-42836380-lr6fl\" (UID: \"ec4c8aae-7586-4fdf-bddd-9ba01d54b071\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42836380-lr6fl" Oct 30 12:40:24.693658 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:24.693612 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec4c8aae-7586-4fdf-bddd-9ba01d54b071-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-42836380-lr6fl\" (UID: \"ec4c8aae-7586-4fdf-bddd-9ba01d54b071\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42836380-lr6fl" Oct 30 12:40:24.694459 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:24.694432 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec4c8aae-7586-4fdf-bddd-9ba01d54b071-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-42836380-lr6fl\" (UID: \"ec4c8aae-7586-4fdf-bddd-9ba01d54b071\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42836380-lr6fl" Oct 30 12:40:24.696181 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:24.696159 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec4c8aae-7586-4fdf-bddd-9ba01d54b071-maas-api-xks-ca\") pod \"maas-api-revoke-sub-42836380-lr6fl\" (UID: \"ec4c8aae-7586-4fdf-bddd-9ba01d54b071\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42836380-lr6fl" Oct 30 12:40:24.715965 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:24.715937 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzp4x\" (UniqueName: \"kubernetes.io/projected/ec4c8aae-7586-4fdf-bddd-9ba01d54b071-kube-api-access-wzp4x\") pod \"maas-api-revoke-sub-42836380-lr6fl\" (UID: \"ec4c8aae-7586-4fdf-bddd-9ba01d54b071\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42836380-lr6fl" Oct 30 12:40:24.717722 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:24.717703 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42836380-lr6fl" Oct 30 12:40:24.921135 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:24.921058 4908 patch_prober.go:29] interesting pod/e2e-shared-a-w0-73bafc-openshift-default-64bfdb8459-rfkqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" start-of-body= Oct 30 12:40:24.921135 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:24.921104 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-73bafc-openshift-default-64bfdb8459-rfkqt" podUID="c9c9d1f3-e29c-42d4-96e6-765da8de6ab9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" Oct 30 12:40:25.085862 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:25.071161 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42836380-lr6fl"] Oct 30 12:40:25.111375 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:25.111350 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-94b0bf54-jxsz2"] Oct 30 12:40:25.116474 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:25.116455 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94b0bf54-jxsz2" Oct 30 12:40:25.131361 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:25.130789 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42836380-lr6fl" event={"ID":"ec4c8aae-7586-4fdf-bddd-9ba01d54b071","Type":"ContainerStarted","Data":"9b65cfd7848319398c044fda9e34e53401e80abc43428f421874c2162b5c79bf"} Oct 30 12:40:25.133778 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:25.133751 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-94b0bf54-jxsz2"] Oct 30 12:40:25.301515 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:25.300811 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhrvl\" (UniqueName: \"kubernetes.io/projected/985ef4ea-15fc-4ae1-94fd-c71c7dae96ca-kube-api-access-rhrvl\") pod \"maas-api-revoke-sub-94b0bf54-jxsz2\" (UID: \"985ef4ea-15fc-4ae1-94fd-c71c7dae96ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94b0bf54-jxsz2" Oct 30 12:40:25.301872 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:25.301622 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/985ef4ea-15fc-4ae1-94fd-c71c7dae96ca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-94b0bf54-jxsz2\" (UID: \"985ef4ea-15fc-4ae1-94fd-c71c7dae96ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94b0bf54-jxsz2" Oct 30 12:40:25.301872 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:25.301648 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/985ef4ea-15fc-4ae1-94fd-c71c7dae96ca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-94b0bf54-jxsz2\" (UID: \"985ef4ea-15fc-4ae1-94fd-c71c7dae96ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94b0bf54-jxsz2" Oct 30 12:40:25.403218 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:25.403159 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/985ef4ea-15fc-4ae1-94fd-c71c7dae96ca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-94b0bf54-jxsz2\" (UID: \"985ef4ea-15fc-4ae1-94fd-c71c7dae96ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94b0bf54-jxsz2" Oct 30 12:40:25.403402 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:25.403238 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/985ef4ea-15fc-4ae1-94fd-c71c7dae96ca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-94b0bf54-jxsz2\" (UID: \"985ef4ea-15fc-4ae1-94fd-c71c7dae96ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94b0bf54-jxsz2" Oct 30 12:40:25.403469 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:25.403420 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhrvl\" (UniqueName: \"kubernetes.io/projected/985ef4ea-15fc-4ae1-94fd-c71c7dae96ca-kube-api-access-rhrvl\") pod \"maas-api-revoke-sub-94b0bf54-jxsz2\" (UID: \"985ef4ea-15fc-4ae1-94fd-c71c7dae96ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94b0bf54-jxsz2" Oct 30 12:40:25.404391 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:25.404363 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/985ef4ea-15fc-4ae1-94fd-c71c7dae96ca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-94b0bf54-jxsz2\" (UID: \"985ef4ea-15fc-4ae1-94fd-c71c7dae96ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94b0bf54-jxsz2" Oct 30 12:40:25.406153 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:25.406129 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/985ef4ea-15fc-4ae1-94fd-c71c7dae96ca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-94b0bf54-jxsz2\" (UID: \"985ef4ea-15fc-4ae1-94fd-c71c7dae96ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94b0bf54-jxsz2" Oct 30 12:40:25.425926 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:25.425903 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhrvl\" (UniqueName: \"kubernetes.io/projected/985ef4ea-15fc-4ae1-94fd-c71c7dae96ca-kube-api-access-rhrvl\") pod \"maas-api-revoke-sub-94b0bf54-jxsz2\" (UID: \"985ef4ea-15fc-4ae1-94fd-c71c7dae96ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94b0bf54-jxsz2" Oct 30 12:40:25.451351 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:25.451319 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94b0bf54-jxsz2" Oct 30 12:40:25.795279 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:25.795247 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-94b0bf54-jxsz2"] Oct 30 12:40:25.797166 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:40:25.797104 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod985ef4ea_15fc_4ae1_94fd_c71c7dae96ca.slice/crio-34174d660cdf9cd8a24caffa30700fb5eea77d224b79b5e04cfc80982c9dcb16 WatchSource:0}: Error finding container 34174d660cdf9cd8a24caffa30700fb5eea77d224b79b5e04cfc80982c9dcb16: Status 404 returned error can't find the container with id 34174d660cdf9cd8a24caffa30700fb5eea77d224b79b5e04cfc80982c9dcb16 Oct 30 12:40:25.920635 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:25.920599 4908 patch_prober.go:29] interesting pod/e2e-shared-a-w0-73bafc-openshift-default-64bfdb8459-rfkqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" start-of-body= Oct 30 12:40:25.920782 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:25.920648 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-73bafc-openshift-default-64bfdb8459-rfkqt" podUID="c9c9d1f3-e29c-42d4-96e6-765da8de6ab9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" Oct 30 12:40:26.156427 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:26.156347 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94b0bf54-jxsz2" event={"ID":"985ef4ea-15fc-4ae1-94fd-c71c7dae96ca","Type":"ContainerStarted","Data":"34174d660cdf9cd8a24caffa30700fb5eea77d224b79b5e04cfc80982c9dcb16"} Oct 30 12:40:26.158191 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:26.158161 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42836380-lr6fl" event={"ID":"ec4c8aae-7586-4fdf-bddd-9ba01d54b071","Type":"ContainerStarted","Data":"2ccf30533e97c1fb76834692c8a0ffa8df453842fafa38ec22e91d70d9bb2c66"} Oct 30 12:40:26.920183 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:26.920153 4908 patch_prober.go:29] interesting pod/e2e-shared-a-w0-73bafc-openshift-default-64bfdb8459-rfkqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" start-of-body= Oct 30 12:40:26.920612 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:26.920209 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-73bafc-openshift-default-64bfdb8459-rfkqt" podUID="c9c9d1f3-e29c-42d4-96e6-765da8de6ab9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" Oct 30 12:40:27.197085 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:27.196979 4908 generic.go:356] "Generic (PLEG): container finished" podID="ec4c8aae-7586-4fdf-bddd-9ba01d54b071" containerID="2ccf30533e97c1fb76834692c8a0ffa8df453842fafa38ec22e91d70d9bb2c66" exitCode=0 Oct 30 12:40:27.197085 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:27.197059 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42836380-lr6fl" event={"ID":"ec4c8aae-7586-4fdf-bddd-9ba01d54b071","Type":"ContainerDied","Data":"2ccf30533e97c1fb76834692c8a0ffa8df453842fafa38ec22e91d70d9bb2c66"} Oct 30 12:40:27.199956 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:27.199926 4908 generic.go:356] "Generic (PLEG): container finished" podID="985ef4ea-15fc-4ae1-94fd-c71c7dae96ca" containerID="3e52c5ada6cc9bd63d869e6c56413e7d0c7cca4feef736264bb277d2823cadce" exitCode=0 Oct 30 12:40:27.200094 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:27.199971 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94b0bf54-jxsz2" event={"ID":"985ef4ea-15fc-4ae1-94fd-c71c7dae96ca","Type":"ContainerDied","Data":"3e52c5ada6cc9bd63d869e6c56413e7d0c7cca4feef736264bb277d2823cadce"} Oct 30 12:40:27.398239 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:27.398200 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42836380-lr6fl" Oct 30 12:40:27.436017 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:27.435983 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec4c8aae-7586-4fdf-bddd-9ba01d54b071-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec4c8aae-7586-4fdf-bddd-9ba01d54b071-maas-api-xks-ca\") pod \"ec4c8aae-7586-4fdf-bddd-9ba01d54b071\" (UID: \"ec4c8aae-7586-4fdf-bddd-9ba01d54b071\") " Oct 30 12:40:27.436169 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:27.436063 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec4c8aae-7586-4fdf-bddd-9ba01d54b071-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec4c8aae-7586-4fdf-bddd-9ba01d54b071-maas-api-openshift-ca\") pod \"ec4c8aae-7586-4fdf-bddd-9ba01d54b071\" (UID: \"ec4c8aae-7586-4fdf-bddd-9ba01d54b071\") " Oct 30 12:40:27.436169 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:27.436121 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec4c8aae-7586-4fdf-bddd-9ba01d54b071-kube-api-access-wzp4x\" (UniqueName: \"kubernetes.io/projected/ec4c8aae-7586-4fdf-bddd-9ba01d54b071-kube-api-access-wzp4x\") pod \"ec4c8aae-7586-4fdf-bddd-9ba01d54b071\" (UID: \"ec4c8aae-7586-4fdf-bddd-9ba01d54b071\") " Oct 30 12:40:27.436561 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:27.436448 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec4c8aae-7586-4fdf-bddd-9ba01d54b071-maas-api-openshift-ca" pod "ec4c8aae-7586-4fdf-bddd-9ba01d54b071" (UID: "ec4c8aae-7586-4fdf-bddd-9ba01d54b071"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:40:27.436846 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:27.436811 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec4c8aae-7586-4fdf-bddd-9ba01d54b071-maas-api-openshift-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:40:27.438796 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:27.438762 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec4c8aae-7586-4fdf-bddd-9ba01d54b071-maas-api-xks-ca" pod "ec4c8aae-7586-4fdf-bddd-9ba01d54b071" (UID: "ec4c8aae-7586-4fdf-bddd-9ba01d54b071"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:40:27.438933 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:27.438884 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec4c8aae-7586-4fdf-bddd-9ba01d54b071-kube-api-access-wzp4x" pod "ec4c8aae-7586-4fdf-bddd-9ba01d54b071" (UID: "ec4c8aae-7586-4fdf-bddd-9ba01d54b071"). InnerVolumeSpecName "kube-api-access-wzp4x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:40:27.537688 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:27.537651 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec4c8aae-7586-4fdf-bddd-9ba01d54b071-maas-api-xks-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:40:27.537688 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:27.537689 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzp4x\" (UniqueName: \"kubernetes.io/projected/ec4c8aae-7586-4fdf-bddd-9ba01d54b071-kube-api-access-wzp4x\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:40:27.920733 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:27.920698 4908 patch_prober.go:29] interesting pod/e2e-shared-a-w0-73bafc-openshift-default-64bfdb8459-rfkqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" start-of-body= Oct 30 12:40:27.921222 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:27.920756 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-73bafc-openshift-default-64bfdb8459-rfkqt" podUID="c9c9d1f3-e29c-42d4-96e6-765da8de6ab9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" Oct 30 12:40:28.212609 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:28.212570 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42836380-lr6fl" event={"ID":"ec4c8aae-7586-4fdf-bddd-9ba01d54b071","Type":"ContainerDied","Data":"9b65cfd7848319398c044fda9e34e53401e80abc43428f421874c2162b5c79bf"} Oct 30 12:40:28.212609 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:28.212601 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42836380-lr6fl" Oct 30 12:40:28.212930 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:28.212633 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9b65cfd7848319398c044fda9e34e53401e80abc43428f421874c2162b5c79bf" Oct 30 12:40:28.216394 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:28.216118 4908 generic.go:356] "Generic (PLEG): container finished" podID="bcd6e5fa-dded-4c97-b694-a2a58766816e" containerID="811ec441fec961d76ef653554df13ecc16d64e0d186014187caf53628a2714a7" exitCode=0 Oct 30 12:40:28.216394 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:28.216217 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-mcrfm" event={"ID":"bcd6e5fa-dded-4c97-b694-a2a58766816e","Type":"ContainerDied","Data":"811ec441fec961d76ef653554df13ecc16d64e0d186014187caf53628a2714a7"} Oct 30 12:40:28.218171 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:28.218150 4908 generic.go:356] "Generic (PLEG): container finished" podID="434eb0c5-0505-4a8e-99c5-40eab19b2674" containerID="d7453a256dc0d4effd974255b973caf1cdcf397d33f3841cc20cecc576f9d8f4" exitCode=0 Oct 30 12:40:28.218241 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:28.218204 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-9bpr4" event={"ID":"434eb0c5-0505-4a8e-99c5-40eab19b2674","Type":"ContainerDied","Data":"d7453a256dc0d4effd974255b973caf1cdcf397d33f3841cc20cecc576f9d8f4"} Oct 30 12:40:28.218286 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:28.218258 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-9bpr4" event={"ID":"434eb0c5-0505-4a8e-99c5-40eab19b2674","Type":"ContainerDied","Data":"8f4f96f3adfdf243df5628fbaaff2ecb496105e6909c34eaeb54acc0191584db"} Oct 30 12:40:28.218404 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:28.218300 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8f4f96f3adfdf243df5628fbaaff2ecb496105e6909c34eaeb54acc0191584db" Oct 30 12:40:28.221681 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:28.220581 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-9bpr4" Oct 30 12:40:28.248110 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:28.248082 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/434eb0c5-0505-4a8e-99c5-40eab19b2674-istio-token\" (UniqueName: \"kubernetes.io/projected/434eb0c5-0505-4a8e-99c5-40eab19b2674-istio-token\") pod \"434eb0c5-0505-4a8e-99c5-40eab19b2674\" (UID: \"434eb0c5-0505-4a8e-99c5-40eab19b2674\") " Oct 30 12:40:28.248244 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:28.248143 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/434eb0c5-0505-4a8e-99c5-40eab19b2674-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/434eb0c5-0505-4a8e-99c5-40eab19b2674-credential-socket\") pod \"434eb0c5-0505-4a8e-99c5-40eab19b2674\" (UID: \"434eb0c5-0505-4a8e-99c5-40eab19b2674\") " Oct 30 12:40:28.248244 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:28.248162 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/434eb0c5-0505-4a8e-99c5-40eab19b2674-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/434eb0c5-0505-4a8e-99c5-40eab19b2674-workload-certs\") pod \"434eb0c5-0505-4a8e-99c5-40eab19b2674\" (UID: \"434eb0c5-0505-4a8e-99c5-40eab19b2674\") " Oct 30 12:40:28.248244 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:28.248193 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/434eb0c5-0505-4a8e-99c5-40eab19b2674-kube-api-access-9lzms\" (UniqueName: \"kubernetes.io/projected/434eb0c5-0505-4a8e-99c5-40eab19b2674-kube-api-access-9lzms\") pod \"434eb0c5-0505-4a8e-99c5-40eab19b2674\" (UID: \"434eb0c5-0505-4a8e-99c5-40eab19b2674\") " Oct 30 12:40:28.248368 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:28.248308 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/434eb0c5-0505-4a8e-99c5-40eab19b2674-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/434eb0c5-0505-4a8e-99c5-40eab19b2674-istio-envoy\") pod \"434eb0c5-0505-4a8e-99c5-40eab19b2674\" (UID: \"434eb0c5-0505-4a8e-99c5-40eab19b2674\") " Oct 30 12:40:28.248368 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:28.248351 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/434eb0c5-0505-4a8e-99c5-40eab19b2674-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/434eb0c5-0505-4a8e-99c5-40eab19b2674-workload-socket\") pod \"434eb0c5-0505-4a8e-99c5-40eab19b2674\" (UID: \"434eb0c5-0505-4a8e-99c5-40eab19b2674\") " Oct 30 12:40:28.248433 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:28.248381 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/434eb0c5-0505-4a8e-99c5-40eab19b2674-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/434eb0c5-0505-4a8e-99c5-40eab19b2674-istiod-ca-cert\") pod \"434eb0c5-0505-4a8e-99c5-40eab19b2674\" (UID: \"434eb0c5-0505-4a8e-99c5-40eab19b2674\") " Oct 30 12:40:28.248499 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:28.248485 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/434eb0c5-0505-4a8e-99c5-40eab19b2674-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/434eb0c5-0505-4a8e-99c5-40eab19b2674-istio-podinfo\") pod \"434eb0c5-0505-4a8e-99c5-40eab19b2674\" (UID: \"434eb0c5-0505-4a8e-99c5-40eab19b2674\") " Oct 30 12:40:28.248540 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:28.248529 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/434eb0c5-0505-4a8e-99c5-40eab19b2674-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/434eb0c5-0505-4a8e-99c5-40eab19b2674-istio-data\") pod \"434eb0c5-0505-4a8e-99c5-40eab19b2674\" (UID: \"434eb0c5-0505-4a8e-99c5-40eab19b2674\") " Oct 30 12:40:28.248954 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:28.248926 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/434eb0c5-0505-4a8e-99c5-40eab19b2674-workload-socket" pod "434eb0c5-0505-4a8e-99c5-40eab19b2674" (UID: "434eb0c5-0505-4a8e-99c5-40eab19b2674"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:40:28.249068 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:28.249012 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/434eb0c5-0505-4a8e-99c5-40eab19b2674-credential-socket" pod "434eb0c5-0505-4a8e-99c5-40eab19b2674" (UID: "434eb0c5-0505-4a8e-99c5-40eab19b2674"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:40:28.249215 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:28.249165 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/434eb0c5-0505-4a8e-99c5-40eab19b2674-istio-data" pod "434eb0c5-0505-4a8e-99c5-40eab19b2674" (UID: "434eb0c5-0505-4a8e-99c5-40eab19b2674"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:40:28.249293 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:28.249177 4908 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/434eb0c5-0505-4a8e-99c5-40eab19b2674-credential-socket\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:40:28.249293 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:28.249248 4908 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/434eb0c5-0505-4a8e-99c5-40eab19b2674-workload-socket\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:40:28.249557 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:28.249530 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/434eb0c5-0505-4a8e-99c5-40eab19b2674-istiod-ca-cert" pod "434eb0c5-0505-4a8e-99c5-40eab19b2674" (UID: "434eb0c5-0505-4a8e-99c5-40eab19b2674"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:40:28.249635 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:28.249606 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/434eb0c5-0505-4a8e-99c5-40eab19b2674-workload-certs" pod "434eb0c5-0505-4a8e-99c5-40eab19b2674" (UID: "434eb0c5-0505-4a8e-99c5-40eab19b2674"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:40:28.251009 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:28.250980 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/434eb0c5-0505-4a8e-99c5-40eab19b2674-kube-api-access-9lzms" pod "434eb0c5-0505-4a8e-99c5-40eab19b2674" (UID: "434eb0c5-0505-4a8e-99c5-40eab19b2674"). InnerVolumeSpecName "kube-api-access-9lzms". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:40:28.251440 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:28.251421 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/434eb0c5-0505-4a8e-99c5-40eab19b2674-istio-podinfo" pod "434eb0c5-0505-4a8e-99c5-40eab19b2674" (UID: "434eb0c5-0505-4a8e-99c5-40eab19b2674"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 12:40:28.251502 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:28.251430 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/434eb0c5-0505-4a8e-99c5-40eab19b2674-istio-envoy" pod "434eb0c5-0505-4a8e-99c5-40eab19b2674" (UID: "434eb0c5-0505-4a8e-99c5-40eab19b2674"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:40:28.251565 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:28.251549 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/434eb0c5-0505-4a8e-99c5-40eab19b2674-istio-token" pod "434eb0c5-0505-4a8e-99c5-40eab19b2674" (UID: "434eb0c5-0505-4a8e-99c5-40eab19b2674"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:40:28.350443 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:28.350408 4908 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/434eb0c5-0505-4a8e-99c5-40eab19b2674-istio-envoy\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:40:28.350443 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:28.350434 4908 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/434eb0c5-0505-4a8e-99c5-40eab19b2674-istiod-ca-cert\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:40:28.350443 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:28.350442 4908 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/434eb0c5-0505-4a8e-99c5-40eab19b2674-istio-podinfo\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:40:28.350443 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:28.350451 4908 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/434eb0c5-0505-4a8e-99c5-40eab19b2674-istio-data\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:40:28.350753 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:28.350460 4908 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/434eb0c5-0505-4a8e-99c5-40eab19b2674-istio-token\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:40:28.350753 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:28.350469 4908 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/434eb0c5-0505-4a8e-99c5-40eab19b2674-workload-certs\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:40:28.350753 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:28.350477 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9lzms\" (UniqueName: \"kubernetes.io/projected/434eb0c5-0505-4a8e-99c5-40eab19b2674-kube-api-access-9lzms\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:40:28.466783 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:28.466748 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94b0bf54-jxsz2" Oct 30 12:40:28.552555 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:28.552526 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/985ef4ea-15fc-4ae1-94fd-c71c7dae96ca-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/985ef4ea-15fc-4ae1-94fd-c71c7dae96ca-maas-api-xks-ca\") pod \"985ef4ea-15fc-4ae1-94fd-c71c7dae96ca\" (UID: \"985ef4ea-15fc-4ae1-94fd-c71c7dae96ca\") " Oct 30 12:40:28.553054 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:28.553027 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/985ef4ea-15fc-4ae1-94fd-c71c7dae96ca-kube-api-access-rhrvl\" (UniqueName: \"kubernetes.io/projected/985ef4ea-15fc-4ae1-94fd-c71c7dae96ca-kube-api-access-rhrvl\") pod \"985ef4ea-15fc-4ae1-94fd-c71c7dae96ca\" (UID: \"985ef4ea-15fc-4ae1-94fd-c71c7dae96ca\") " Oct 30 12:40:28.553254 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:28.553068 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/985ef4ea-15fc-4ae1-94fd-c71c7dae96ca-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/985ef4ea-15fc-4ae1-94fd-c71c7dae96ca-maas-api-openshift-ca\") pod \"985ef4ea-15fc-4ae1-94fd-c71c7dae96ca\" (UID: \"985ef4ea-15fc-4ae1-94fd-c71c7dae96ca\") " Oct 30 12:40:28.553524 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:28.553495 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/985ef4ea-15fc-4ae1-94fd-c71c7dae96ca-maas-api-openshift-ca" pod "985ef4ea-15fc-4ae1-94fd-c71c7dae96ca" (UID: "985ef4ea-15fc-4ae1-94fd-c71c7dae96ca"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:40:28.555057 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:28.555032 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/985ef4ea-15fc-4ae1-94fd-c71c7dae96ca-maas-api-xks-ca" pod "985ef4ea-15fc-4ae1-94fd-c71c7dae96ca" (UID: "985ef4ea-15fc-4ae1-94fd-c71c7dae96ca"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:40:28.555651 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:28.555625 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/985ef4ea-15fc-4ae1-94fd-c71c7dae96ca-kube-api-access-rhrvl" pod "985ef4ea-15fc-4ae1-94fd-c71c7dae96ca" (UID: "985ef4ea-15fc-4ae1-94fd-c71c7dae96ca"). InnerVolumeSpecName "kube-api-access-rhrvl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:40:28.585386 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:28.585358 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-mcrfm" Oct 30 12:40:28.654891 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:28.654855 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bcd6e5fa-dded-4c97-b694-a2a58766816e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcd6e5fa-dded-4c97-b694-a2a58766816e-credential-socket\") pod \"bcd6e5fa-dded-4c97-b694-a2a58766816e\" (UID: \"bcd6e5fa-dded-4c97-b694-a2a58766816e\") " Oct 30 12:40:28.655081 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:28.654991 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bcd6e5fa-dded-4c97-b694-a2a58766816e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcd6e5fa-dded-4c97-b694-a2a58766816e-workload-socket\") pod \"bcd6e5fa-dded-4c97-b694-a2a58766816e\" (UID: \"bcd6e5fa-dded-4c97-b694-a2a58766816e\") " Oct 30 12:40:28.655081 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:28.655059 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bcd6e5fa-dded-4c97-b694-a2a58766816e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bcd6e5fa-dded-4c97-b694-a2a58766816e-istio-envoy\") pod \"bcd6e5fa-dded-4c97-b694-a2a58766816e\" (UID: \"bcd6e5fa-dded-4c97-b694-a2a58766816e\") " Oct 30 12:40:28.655208 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:28.655089 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bcd6e5fa-dded-4c97-b694-a2a58766816e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bcd6e5fa-dded-4c97-b694-a2a58766816e-istiod-ca-cert\") pod \"bcd6e5fa-dded-4c97-b694-a2a58766816e\" (UID: \"bcd6e5fa-dded-4c97-b694-a2a58766816e\") " Oct 30 12:40:28.655208 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:28.655115 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bcd6e5fa-dded-4c97-b694-a2a58766816e-kube-api-access-zmbwq\" (UniqueName: \"kubernetes.io/projected/bcd6e5fa-dded-4c97-b694-a2a58766816e-kube-api-access-zmbwq\") pod \"bcd6e5fa-dded-4c97-b694-a2a58766816e\" (UID: \"bcd6e5fa-dded-4c97-b694-a2a58766816e\") " Oct 30 12:40:28.655208 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:28.655172 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcd6e5fa-dded-4c97-b694-a2a58766816e-credential-socket" pod "bcd6e5fa-dded-4c97-b694-a2a58766816e" (UID: "bcd6e5fa-dded-4c97-b694-a2a58766816e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:40:28.655208 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:28.655198 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bcd6e5fa-dded-4c97-b694-a2a58766816e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bcd6e5fa-dded-4c97-b694-a2a58766816e-istio-data\") pod \"bcd6e5fa-dded-4c97-b694-a2a58766816e\" (UID: \"bcd6e5fa-dded-4c97-b694-a2a58766816e\") " Oct 30 12:40:28.655419 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:28.655267 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bcd6e5fa-dded-4c97-b694-a2a58766816e-istio-token\" (UniqueName: \"kubernetes.io/projected/bcd6e5fa-dded-4c97-b694-a2a58766816e-istio-token\") pod \"bcd6e5fa-dded-4c97-b694-a2a58766816e\" (UID: \"bcd6e5fa-dded-4c97-b694-a2a58766816e\") " Oct 30 12:40:28.655419 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:28.655311 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bcd6e5fa-dded-4c97-b694-a2a58766816e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bcd6e5fa-dded-4c97-b694-a2a58766816e-istio-podinfo\") pod \"bcd6e5fa-dded-4c97-b694-a2a58766816e\" (UID: \"bcd6e5fa-dded-4c97-b694-a2a58766816e\") " Oct 30 12:40:28.655419 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:28.655349 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bcd6e5fa-dded-4c97-b694-a2a58766816e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bcd6e5fa-dded-4c97-b694-a2a58766816e-workload-certs\") pod \"bcd6e5fa-dded-4c97-b694-a2a58766816e\" (UID: \"bcd6e5fa-dded-4c97-b694-a2a58766816e\") " Oct 30 12:40:28.656260 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:28.656017 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhrvl\" (UniqueName: \"kubernetes.io/projected/985ef4ea-15fc-4ae1-94fd-c71c7dae96ca-kube-api-access-rhrvl\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:40:28.656260 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:28.656042 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/985ef4ea-15fc-4ae1-94fd-c71c7dae96ca-maas-api-openshift-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:40:28.656260 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:28.656055 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/985ef4ea-15fc-4ae1-94fd-c71c7dae96ca-maas-api-xks-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:40:28.656260 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:28.656068 4908 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcd6e5fa-dded-4c97-b694-a2a58766816e-credential-socket\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:40:28.656260 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:28.656122 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcd6e5fa-dded-4c97-b694-a2a58766816e-workload-socket" pod "bcd6e5fa-dded-4c97-b694-a2a58766816e" (UID: "bcd6e5fa-dded-4c97-b694-a2a58766816e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:40:28.656619 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:28.656309 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcd6e5fa-dded-4c97-b694-a2a58766816e-istio-data" pod "bcd6e5fa-dded-4c97-b694-a2a58766816e" (UID: "bcd6e5fa-dded-4c97-b694-a2a58766816e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:40:28.656619 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:28.656338 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bcd6e5fa-dded-4c97-b694-a2a58766816e-istiod-ca-cert" pod "bcd6e5fa-dded-4c97-b694-a2a58766816e" (UID: "bcd6e5fa-dded-4c97-b694-a2a58766816e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:40:28.656731 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:28.656696 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcd6e5fa-dded-4c97-b694-a2a58766816e-workload-certs" pod "bcd6e5fa-dded-4c97-b694-a2a58766816e" (UID: "bcd6e5fa-dded-4c97-b694-a2a58766816e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:40:28.658881 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:28.658822 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcd6e5fa-dded-4c97-b694-a2a58766816e-istio-token" pod "bcd6e5fa-dded-4c97-b694-a2a58766816e" (UID: "bcd6e5fa-dded-4c97-b694-a2a58766816e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:40:28.659092 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:28.659058 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bcd6e5fa-dded-4c97-b694-a2a58766816e-istio-podinfo" pod "bcd6e5fa-dded-4c97-b694-a2a58766816e" (UID: "bcd6e5fa-dded-4c97-b694-a2a58766816e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 12:40:28.659179 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:28.659111 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcd6e5fa-dded-4c97-b694-a2a58766816e-istio-envoy" pod "bcd6e5fa-dded-4c97-b694-a2a58766816e" (UID: "bcd6e5fa-dded-4c97-b694-a2a58766816e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:40:28.662070 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:28.662040 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcd6e5fa-dded-4c97-b694-a2a58766816e-kube-api-access-zmbwq" pod "bcd6e5fa-dded-4c97-b694-a2a58766816e" (UID: "bcd6e5fa-dded-4c97-b694-a2a58766816e"). InnerVolumeSpecName "kube-api-access-zmbwq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:40:28.757776 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:28.757742 4908 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcd6e5fa-dded-4c97-b694-a2a58766816e-workload-socket\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:40:28.757776 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:28.757774 4908 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bcd6e5fa-dded-4c97-b694-a2a58766816e-istio-envoy\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:40:28.757776 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:28.757784 4908 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bcd6e5fa-dded-4c97-b694-a2a58766816e-istiod-ca-cert\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:40:28.758034 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:28.757793 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmbwq\" (UniqueName: \"kubernetes.io/projected/bcd6e5fa-dded-4c97-b694-a2a58766816e-kube-api-access-zmbwq\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:40:28.758034 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:28.757802 4908 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bcd6e5fa-dded-4c97-b694-a2a58766816e-istio-data\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:40:28.758034 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:28.757810 4908 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bcd6e5fa-dded-4c97-b694-a2a58766816e-istio-token\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:40:28.758034 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:28.757818 4908 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bcd6e5fa-dded-4c97-b694-a2a58766816e-istio-podinfo\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:40:28.758034 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:28.757839 4908 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bcd6e5fa-dded-4c97-b694-a2a58766816e-workload-certs\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:40:28.925228 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:28.925200 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-73bafc-openshift-default-64bfdb8459-rfkqt" Oct 30 12:40:28.926186 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:28.926164 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-73bafc-openshift-default-64bfdb8459-rfkqt" Oct 30 12:40:29.227170 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:29.227148 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94b0bf54-jxsz2" Oct 30 12:40:29.227338 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:29.227188 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94b0bf54-jxsz2" event={"ID":"985ef4ea-15fc-4ae1-94fd-c71c7dae96ca","Type":"ContainerDied","Data":"34174d660cdf9cd8a24caffa30700fb5eea77d224b79b5e04cfc80982c9dcb16"} Oct 30 12:40:29.227338 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:29.227244 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="34174d660cdf9cd8a24caffa30700fb5eea77d224b79b5e04cfc80982c9dcb16" Oct 30 12:40:29.228773 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:29.228750 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-mcrfm" event={"ID":"bcd6e5fa-dded-4c97-b694-a2a58766816e","Type":"ContainerDied","Data":"563e52b1a5ef1fe18006962e75ced2b6fb35f0715c6bc39e10389262628aa3d8"} Oct 30 12:40:29.228920 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:29.228802 4908 scope.go:122] "RemoveContainer" containerID="811ec441fec961d76ef653554df13ecc16d64e0d186014187caf53628a2714a7" Oct 30 12:40:29.228920 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:29.228862 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-mcrfm" Oct 30 12:40:29.229164 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:29.229144 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-9bpr4" Oct 30 12:40:29.316852 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:29.314353 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-mcrfm"] Oct 30 12:40:29.330390 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:29.328895 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-mcrfm"] Oct 30 12:40:29.364329 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:29.364285 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-9bpr4"] Oct 30 12:40:29.368061 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:29.368035 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-9bpr4"] Oct 30 12:40:29.608294 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:29.608223 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="434eb0c5-0505-4a8e-99c5-40eab19b2674" path="/var/lib/kubelet/pods/434eb0c5-0505-4a8e-99c5-40eab19b2674/volumes" Oct 30 12:40:29.608704 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:29.608682 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bcd6e5fa-dded-4c97-b694-a2a58766816e" path="/var/lib/kubelet/pods/bcd6e5fa-dded-4c97-b694-a2a58766816e/volumes" Oct 30 12:40:30.897821 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:30.897778 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9wwfc"] Oct 30 12:40:36.682053 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:36.682017 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-812990/auth-test-model-812990-kserve-849f9f7d57-w28mb"] Oct 30 12:40:42.688021 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:42.687987 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-2675c0/auth-test-model-2675c0-kserve-6bdcd4c665-qspzk"] Oct 30 12:40:49.686532 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:40:49.686489 4908 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vn6vv" containerName="extractor" Oct 30 12:40:51.723849 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:51.717010 4908 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w2-812990/auth-test-model-812990-kserve-849f9f7d57-w28mb" podUID="5d547033-e804-49e8-b329-b96c8ee1c2b1" containerName="main" containerID="cri-o://b0b9a92cec6f91aa0278818ee74303bdac7ccc339ba4136464ab4b46a6698505" gracePeriod=45 Oct 30 12:40:52.033566 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:52.033535 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-812990/auth-test-model-812990-kserve-849f9f7d57-w28mb" Oct 30 12:40:52.140890 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:52.140851 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d547033-e804-49e8-b329-b96c8ee1c2b1-home\" (UniqueName: \"kubernetes.io/empty-dir/5d547033-e804-49e8-b329-b96c8ee1c2b1-home\") pod \"5d547033-e804-49e8-b329-b96c8ee1c2b1\" (UID: \"5d547033-e804-49e8-b329-b96c8ee1c2b1\") " Oct 30 12:40:52.141068 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:52.140935 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5d547033-e804-49e8-b329-b96c8ee1c2b1-tls-certs\" (UniqueName: \"kubernetes.io/secret/5d547033-e804-49e8-b329-b96c8ee1c2b1-tls-certs\") pod \"5d547033-e804-49e8-b329-b96c8ee1c2b1\" (UID: \"5d547033-e804-49e8-b329-b96c8ee1c2b1\") " Oct 30 12:40:52.141068 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:52.140983 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d547033-e804-49e8-b329-b96c8ee1c2b1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d547033-e804-49e8-b329-b96c8ee1c2b1-tmp-dir\") pod \"5d547033-e804-49e8-b329-b96c8ee1c2b1\" (UID: \"5d547033-e804-49e8-b329-b96c8ee1c2b1\") " Oct 30 12:40:52.141068 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:52.141041 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d547033-e804-49e8-b329-b96c8ee1c2b1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d547033-e804-49e8-b329-b96c8ee1c2b1-model-cache\") pod \"5d547033-e804-49e8-b329-b96c8ee1c2b1\" (UID: \"5d547033-e804-49e8-b329-b96c8ee1c2b1\") " Oct 30 12:40:52.141263 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:52.141090 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d547033-e804-49e8-b329-b96c8ee1c2b1-home" pod "5d547033-e804-49e8-b329-b96c8ee1c2b1" (UID: "5d547033-e804-49e8-b329-b96c8ee1c2b1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:40:52.141263 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:52.141109 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d547033-e804-49e8-b329-b96c8ee1c2b1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d547033-e804-49e8-b329-b96c8ee1c2b1-dshm\") pod \"5d547033-e804-49e8-b329-b96c8ee1c2b1\" (UID: \"5d547033-e804-49e8-b329-b96c8ee1c2b1\") " Oct 30 12:40:52.141263 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:52.141151 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d547033-e804-49e8-b329-b96c8ee1c2b1-kube-api-access-wzllg\" (UniqueName: \"kubernetes.io/projected/5d547033-e804-49e8-b329-b96c8ee1c2b1-kube-api-access-wzllg\") pod \"5d547033-e804-49e8-b329-b96c8ee1c2b1\" (UID: \"5d547033-e804-49e8-b329-b96c8ee1c2b1\") " Oct 30 12:40:52.141435 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:52.141284 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d547033-e804-49e8-b329-b96c8ee1c2b1-tmp-dir" pod "5d547033-e804-49e8-b329-b96c8ee1c2b1" (UID: "5d547033-e804-49e8-b329-b96c8ee1c2b1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:40:52.141435 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:52.141379 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d547033-e804-49e8-b329-b96c8ee1c2b1-model-cache" pod "5d547033-e804-49e8-b329-b96c8ee1c2b1" (UID: "5d547033-e804-49e8-b329-b96c8ee1c2b1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:40:52.141732 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:52.141708 4908 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d547033-e804-49e8-b329-b96c8ee1c2b1-tmp-dir\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:40:52.141914 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:52.141738 4908 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d547033-e804-49e8-b329-b96c8ee1c2b1-model-cache\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:40:52.141914 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:52.141751 4908 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d547033-e804-49e8-b329-b96c8ee1c2b1-home\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:40:52.143392 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:52.143363 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d547033-e804-49e8-b329-b96c8ee1c2b1-dshm" pod "5d547033-e804-49e8-b329-b96c8ee1c2b1" (UID: "5d547033-e804-49e8-b329-b96c8ee1c2b1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:40:52.143491 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:52.143459 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d547033-e804-49e8-b329-b96c8ee1c2b1-tls-certs" pod "5d547033-e804-49e8-b329-b96c8ee1c2b1" (UID: "5d547033-e804-49e8-b329-b96c8ee1c2b1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:40:52.143491 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:52.143466 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d547033-e804-49e8-b329-b96c8ee1c2b1-kube-api-access-wzllg" pod "5d547033-e804-49e8-b329-b96c8ee1c2b1" (UID: "5d547033-e804-49e8-b329-b96c8ee1c2b1"). InnerVolumeSpecName "kube-api-access-wzllg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:40:52.252134 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:52.244922 4908 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d547033-e804-49e8-b329-b96c8ee1c2b1-dshm\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:40:52.252134 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:52.244978 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzllg\" (UniqueName: \"kubernetes.io/projected/5d547033-e804-49e8-b329-b96c8ee1c2b1-kube-api-access-wzllg\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:40:52.252134 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:52.244995 4908 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d547033-e804-49e8-b329-b96c8ee1c2b1-tls-certs\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:40:52.427660 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:52.427624 4908 generic.go:356] "Generic (PLEG): container finished" podID="5d547033-e804-49e8-b329-b96c8ee1c2b1" containerID="b0b9a92cec6f91aa0278818ee74303bdac7ccc339ba4136464ab4b46a6698505" exitCode=0 Oct 30 12:40:52.427852 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:52.427673 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-812990/auth-test-model-812990-kserve-849f9f7d57-w28mb" Oct 30 12:40:52.427852 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:52.427671 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-812990/auth-test-model-812990-kserve-849f9f7d57-w28mb" event={"ID":"5d547033-e804-49e8-b329-b96c8ee1c2b1","Type":"ContainerDied","Data":"b0b9a92cec6f91aa0278818ee74303bdac7ccc339ba4136464ab4b46a6698505"} Oct 30 12:40:52.427852 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:52.427798 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-812990/auth-test-model-812990-kserve-849f9f7d57-w28mb" event={"ID":"5d547033-e804-49e8-b329-b96c8ee1c2b1","Type":"ContainerDied","Data":"4f5399777e4899e06749446eea5bd45b97ae9615b9dcaae008d1949287bd28be"} Oct 30 12:40:52.427852 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:52.427822 4908 scope.go:122] "RemoveContainer" containerID="b0b9a92cec6f91aa0278818ee74303bdac7ccc339ba4136464ab4b46a6698505" Oct 30 12:40:52.440063 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:52.440045 4908 scope.go:122] "RemoveContainer" containerID="b0b9a92cec6f91aa0278818ee74303bdac7ccc339ba4136464ab4b46a6698505" Oct 30 12:40:52.440337 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:40:52.440311 4908 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0b9a92cec6f91aa0278818ee74303bdac7ccc339ba4136464ab4b46a6698505\": container with ID starting with b0b9a92cec6f91aa0278818ee74303bdac7ccc339ba4136464ab4b46a6698505 not found: ID does not exist" containerID="b0b9a92cec6f91aa0278818ee74303bdac7ccc339ba4136464ab4b46a6698505" Oct 30 12:40:52.440430 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:52.440347 4908 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0b9a92cec6f91aa0278818ee74303bdac7ccc339ba4136464ab4b46a6698505"} err="failed to get container status \"b0b9a92cec6f91aa0278818ee74303bdac7ccc339ba4136464ab4b46a6698505\": rpc error: code = NotFound desc = could not find container \"b0b9a92cec6f91aa0278818ee74303bdac7ccc339ba4136464ab4b46a6698505\": container with ID starting with b0b9a92cec6f91aa0278818ee74303bdac7ccc339ba4136464ab4b46a6698505 not found: ID does not exist" Oct 30 12:40:52.482553 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:52.482529 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-812990/auth-test-model-812990-kserve-849f9f7d57-w28mb"] Oct 30 12:40:52.493459 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:52.493433 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w2-812990/auth-test-model-812990-kserve-849f9f7d57-w28mb"] Oct 30 12:40:52.716927 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:52.716856 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-11581-9ddde6"] Oct 30 12:40:52.717410 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:52.717396 4908 cpu_manager.go:414] "containerMap: removing container" podUID="bcd6e5fa-dded-4c97-b694-a2a58766816e" containerName="istio-proxy" Oct 30 12:40:52.717459 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:52.717412 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bcd6e5fa-dded-4c97-b694-a2a58766816e" containerName="istio-proxy" Oct 30 12:40:52.717459 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:52.717441 4908 cpu_manager.go:414] "containerMap: removing container" podUID="434eb0c5-0505-4a8e-99c5-40eab19b2674" containerName="istio-proxy" Oct 30 12:40:52.717459 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:52.717446 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="434eb0c5-0505-4a8e-99c5-40eab19b2674" containerName="istio-proxy" Oct 30 12:40:52.717459 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:52.717454 4908 cpu_manager.go:414] "containerMap: removing container" podUID="5d547033-e804-49e8-b329-b96c8ee1c2b1" containerName="main" Oct 30 12:40:52.717579 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:52.717462 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d547033-e804-49e8-b329-b96c8ee1c2b1" containerName="main" Oct 30 12:40:52.717579 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:52.717475 4908 cpu_manager.go:414] "containerMap: removing container" podUID="985ef4ea-15fc-4ae1-94fd-c71c7dae96ca" containerName="revoke-keys" Oct 30 12:40:52.717579 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:52.717480 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="985ef4ea-15fc-4ae1-94fd-c71c7dae96ca" containerName="revoke-keys" Oct 30 12:40:52.717579 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:52.717494 4908 cpu_manager.go:414] "containerMap: removing container" podUID="ec4c8aae-7586-4fdf-bddd-9ba01d54b071" containerName="revoke-keys" Oct 30 12:40:52.717579 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:52.717499 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec4c8aae-7586-4fdf-bddd-9ba01d54b071" containerName="revoke-keys" Oct 30 12:40:52.717744 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:52.717582 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="434eb0c5-0505-4a8e-99c5-40eab19b2674" containerName="istio-proxy" Oct 30 12:40:52.717744 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:52.717589 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d547033-e804-49e8-b329-b96c8ee1c2b1" containerName="main" Oct 30 12:40:52.717744 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:52.717596 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="985ef4ea-15fc-4ae1-94fd-c71c7dae96ca" containerName="revoke-keys" Oct 30 12:40:52.717744 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:52.717604 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec4c8aae-7586-4fdf-bddd-9ba01d54b071" containerName="revoke-keys" Oct 30 12:40:52.717744 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:52.717612 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="bcd6e5fa-dded-4c97-b694-a2a58766816e" containerName="istio-proxy" Oct 30 12:40:52.722507 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:52.722487 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11581-9ddde6" Oct 30 12:40:52.749045 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:52.738545 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11581-9ddde6"] Oct 30 12:40:52.749045 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:52.738803 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-ttfhc\"" Oct 30 12:40:52.874857 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:52.868556 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjnww\" (UniqueName: \"kubernetes.io/projected/8bd49bca-092f-40a1-b4a1-6b836952e44d-kube-api-access-zjnww\") pod \"test-curl-11581-9ddde6\" (UID: \"8bd49bca-092f-40a1-b4a1-6b836952e44d\") " pod="opendatahub/test-curl-11581-9ddde6" Oct 30 12:40:53.008851 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:52.992274 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjnww\" (UniqueName: \"kubernetes.io/projected/8bd49bca-092f-40a1-b4a1-6b836952e44d-kube-api-access-zjnww\") pod \"test-curl-11581-9ddde6\" (UID: \"8bd49bca-092f-40a1-b4a1-6b836952e44d\") " pod="opendatahub/test-curl-11581-9ddde6" Oct 30 12:40:53.061894 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:53.058928 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjnww\" (UniqueName: \"kubernetes.io/projected/8bd49bca-092f-40a1-b4a1-6b836952e44d-kube-api-access-zjnww\") pod \"test-curl-11581-9ddde6\" (UID: \"8bd49bca-092f-40a1-b4a1-6b836952e44d\") " pod="opendatahub/test-curl-11581-9ddde6" Oct 30 12:40:53.345137 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:53.345060 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11581-9ddde6" Oct 30 12:40:53.607910 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:53.607811 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d547033-e804-49e8-b329-b96c8ee1c2b1" path="/var/lib/kubelet/pods/5d547033-e804-49e8-b329-b96c8ee1c2b1/volumes" Oct 30 12:40:53.643287 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:53.643204 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11581-9ddde6"] Oct 30 12:40:53.646090 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:40:53.646062 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8bd49bca_092f_40a1_b4a1_6b836952e44d.slice/crio-e7eddb0c224e6b1bb6e5e1f63dafe40b360a0afbca6a63a1eebef16a31b9cbcf WatchSource:0}: Error finding container e7eddb0c224e6b1bb6e5e1f63dafe40b360a0afbca6a63a1eebef16a31b9cbcf: Status 404 returned error can't find the container with id e7eddb0c224e6b1bb6e5e1f63dafe40b360a0afbca6a63a1eebef16a31b9cbcf Oct 30 12:40:54.447146 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:54.447101 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11581-9ddde6" event={"ID":"8bd49bca-092f-40a1-b4a1-6b836952e44d","Type":"ContainerStarted","Data":"831bdac6fb833d6e1330e0d628ffbfdfa91b4b08b74672b233a6d4da690c99e7"} Oct 30 12:40:54.447146 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:54.447150 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11581-9ddde6" event={"ID":"8bd49bca-092f-40a1-b4a1-6b836952e44d","Type":"ContainerStarted","Data":"e7eddb0c224e6b1bb6e5e1f63dafe40b360a0afbca6a63a1eebef16a31b9cbcf"} Oct 30 12:40:54.508735 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:54.508679 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-11581-9ddde6" podStartSLOduration=2.508665482 podStartE2EDuration="2.508665482s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:40:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:40:54.505139078 +0000 UTC m=+2811.019567821" watchObservedRunningTime="2026-10-30 12:40:54.508665482 +0000 UTC m=+2811.023094206" Oct 30 12:40:55.193110 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:55.193078 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-gk4rg"] Oct 30 12:40:55.193339 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:55.193282 4908 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-gk4rg" podUID="9ac0ce00-fe15-408d-90eb-9633ea3e67d0" containerName="istio-proxy" containerID="cri-o://763181d0f06ab314d19a69c97efef3ce3e5f17327808faa7187c9d8d99668eb6" gracePeriod=30 Oct 30 12:40:55.493711 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:55.493679 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-11581-9ddde6"] Oct 30 12:40:55.499224 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:55.499188 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-11581-9ddde6"] Oct 30 12:40:55.657733 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:55.657696 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-11581-2e9061"] Oct 30 12:40:55.658547 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:55.658523 4908 cpu_manager.go:414] "containerMap: removing container" podUID="8bd49bca-092f-40a1-b4a1-6b836952e44d" containerName="test-curl-11581-9ddde6" Oct 30 12:40:55.658547 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:55.658545 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8bd49bca-092f-40a1-b4a1-6b836952e44d" containerName="test-curl-11581-9ddde6" Oct 30 12:40:55.658742 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:55.658685 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="8bd49bca-092f-40a1-b4a1-6b836952e44d" containerName="test-curl-11581-9ddde6" Oct 30 12:40:55.677068 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:55.677041 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11581-2e9061" Oct 30 12:40:55.696208 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:55.691264 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11581-2e9061"] Oct 30 12:40:55.838532 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:55.838439 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcrrq\" (UniqueName: \"kubernetes.io/projected/f7800ec1-a617-413f-a262-769762631d30-kube-api-access-pcrrq\") pod \"test-curl-11581-2e9061\" (UID: \"f7800ec1-a617-413f-a262-769762631d30\") " pod="opendatahub/test-curl-11581-2e9061" Oct 30 12:40:55.940882 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:55.939993 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcrrq\" (UniqueName: \"kubernetes.io/projected/f7800ec1-a617-413f-a262-769762631d30-kube-api-access-pcrrq\") pod \"test-curl-11581-2e9061\" (UID: \"f7800ec1-a617-413f-a262-769762631d30\") " pod="opendatahub/test-curl-11581-2e9061" Oct 30 12:40:55.992521 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:55.992486 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcrrq\" (UniqueName: \"kubernetes.io/projected/f7800ec1-a617-413f-a262-769762631d30-kube-api-access-pcrrq\") pod \"test-curl-11581-2e9061\" (UID: \"f7800ec1-a617-413f-a262-769762631d30\") " pod="opendatahub/test-curl-11581-2e9061" Oct 30 12:40:56.040936 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:56.040857 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11581-2e9061" Oct 30 12:40:56.395734 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:56.395701 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11581-2e9061"] Oct 30 12:40:56.397634 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:40:56.397609 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf7800ec1_a617_413f_a262_769762631d30.slice/crio-e05448280a6ec95c8c42cf5827d680fed181e567414e1c0a3cdf5de16ba847b5 WatchSource:0}: Error finding container e05448280a6ec95c8c42cf5827d680fed181e567414e1c0a3cdf5de16ba847b5: Status 404 returned error can't find the container with id e05448280a6ec95c8c42cf5827d680fed181e567414e1c0a3cdf5de16ba847b5 Oct 30 12:40:56.474051 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:56.474020 4908 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-11581-9ddde6" podUID="8bd49bca-092f-40a1-b4a1-6b836952e44d" containerName="test-curl-11581-9ddde6" containerID="cri-o://831bdac6fb833d6e1330e0d628ffbfdfa91b4b08b74672b233a6d4da690c99e7" gracePeriod=2 Oct 30 12:40:56.478999 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:56.474003 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11581-2e9061" event={"ID":"f7800ec1-a617-413f-a262-769762631d30","Type":"ContainerStarted","Data":"e05448280a6ec95c8c42cf5827d680fed181e567414e1c0a3cdf5de16ba847b5"} Oct 30 12:40:57.494118 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:57.494079 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11581-2e9061" event={"ID":"f7800ec1-a617-413f-a262-769762631d30","Type":"ContainerStarted","Data":"d09f8f0e70f897b24c1b7b718ae20602dc7ed8ba792e8576eb427d6938a2c6d2"} Oct 30 12:40:57.533308 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:57.533257 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-11581-2e9061" podStartSLOduration=2.533242858 podStartE2EDuration="2.533242858s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:40:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:40:57.530411839 +0000 UTC m=+2814.044840584" watchObservedRunningTime="2026-10-30 12:40:57.533242858 +0000 UTC m=+2814.047671603" Oct 30 12:40:57.738265 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:57.736986 4908 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-2675c0/auth-test-model-2675c0-kserve-6bdcd4c665-qspzk" podUID="8595d8d1-88b3-412e-9b02-bed8ed21f57b" containerName="main" containerID="cri-o://cf1eedddf7102b0cc04a9b3dfd986b12770f4083d02a70c039a39b0965907318" gracePeriod=45 Oct 30 12:40:57.919596 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:57.919530 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4afe31ef-bd98f"] Oct 30 12:40:57.925561 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:57.925537 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4afe31ef-bd98f" Oct 30 12:40:57.933113 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:57.933084 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4afe31ef-bd98f"] Oct 30 12:40:57.943781 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:57.943441 4908 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w3-2675c0/auth-test-model-2675c0-kserve-6bdcd4c665-qspzk" podUID="8595d8d1-88b3-412e-9b02-bed8ed21f57b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.204:8000/ready\": dial tcp 10.132.0.204:8000: connect: connection refused" Oct 30 12:40:57.951171 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:57.950731 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s75hq\"" Oct 30 12:40:57.951171 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:57.950794 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 12:40:57.951171 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:57.950946 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 12:40:57.951171 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:57.951042 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 12:40:57.995986 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:57.995822 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ce4096e-12a9-4293-9c47-6585f6ead95f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4afe31ef-bd98f\" (UID: \"7ce4096e-12a9-4293-9c47-6585f6ead95f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4afe31ef-bd98f" Oct 30 12:40:57.996142 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:57.996085 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgt66\" (UniqueName: \"kubernetes.io/projected/7ce4096e-12a9-4293-9c47-6585f6ead95f-kube-api-access-qgt66\") pod \"maas-api-revoke-sub-4afe31ef-bd98f\" (UID: \"7ce4096e-12a9-4293-9c47-6585f6ead95f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4afe31ef-bd98f" Oct 30 12:40:57.996218 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:57.996169 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ce4096e-12a9-4293-9c47-6585f6ead95f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4afe31ef-bd98f\" (UID: \"7ce4096e-12a9-4293-9c47-6585f6ead95f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4afe31ef-bd98f" Oct 30 12:40:58.097787 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:58.097718 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ce4096e-12a9-4293-9c47-6585f6ead95f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4afe31ef-bd98f\" (UID: \"7ce4096e-12a9-4293-9c47-6585f6ead95f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4afe31ef-bd98f" Oct 30 12:40:58.099747 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:58.099718 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgt66\" (UniqueName: \"kubernetes.io/projected/7ce4096e-12a9-4293-9c47-6585f6ead95f-kube-api-access-qgt66\") pod \"maas-api-revoke-sub-4afe31ef-bd98f\" (UID: \"7ce4096e-12a9-4293-9c47-6585f6ead95f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4afe31ef-bd98f" Oct 30 12:40:58.100044 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:58.100021 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ce4096e-12a9-4293-9c47-6585f6ead95f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4afe31ef-bd98f\" (UID: \"7ce4096e-12a9-4293-9c47-6585f6ead95f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4afe31ef-bd98f" Oct 30 12:40:58.100633 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:58.099066 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ce4096e-12a9-4293-9c47-6585f6ead95f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4afe31ef-bd98f\" (UID: \"7ce4096e-12a9-4293-9c47-6585f6ead95f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4afe31ef-bd98f" Oct 30 12:40:58.102916 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:58.102898 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ce4096e-12a9-4293-9c47-6585f6ead95f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4afe31ef-bd98f\" (UID: \"7ce4096e-12a9-4293-9c47-6585f6ead95f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4afe31ef-bd98f" Oct 30 12:40:58.122708 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:58.122684 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgt66\" (UniqueName: \"kubernetes.io/projected/7ce4096e-12a9-4293-9c47-6585f6ead95f-kube-api-access-qgt66\") pod \"maas-api-revoke-sub-4afe31ef-bd98f\" (UID: \"7ce4096e-12a9-4293-9c47-6585f6ead95f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4afe31ef-bd98f" Oct 30 12:40:58.149317 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:58.149296 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-2675c0/auth-test-model-2675c0-kserve-6bdcd4c665-qspzk" Oct 30 12:40:58.202936 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:58.200966 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8595d8d1-88b3-412e-9b02-bed8ed21f57b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8595d8d1-88b3-412e-9b02-bed8ed21f57b-tmp-dir\") pod \"8595d8d1-88b3-412e-9b02-bed8ed21f57b\" (UID: \"8595d8d1-88b3-412e-9b02-bed8ed21f57b\") " Oct 30 12:40:58.202936 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:58.201006 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8595d8d1-88b3-412e-9b02-bed8ed21f57b-home\" (UniqueName: \"kubernetes.io/empty-dir/8595d8d1-88b3-412e-9b02-bed8ed21f57b-home\") pod \"8595d8d1-88b3-412e-9b02-bed8ed21f57b\" (UID: \"8595d8d1-88b3-412e-9b02-bed8ed21f57b\") " Oct 30 12:40:58.202936 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:58.201039 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8595d8d1-88b3-412e-9b02-bed8ed21f57b-tls-certs\" (UniqueName: \"kubernetes.io/secret/8595d8d1-88b3-412e-9b02-bed8ed21f57b-tls-certs\") pod \"8595d8d1-88b3-412e-9b02-bed8ed21f57b\" (UID: \"8595d8d1-88b3-412e-9b02-bed8ed21f57b\") " Oct 30 12:40:58.202936 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:58.201086 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8595d8d1-88b3-412e-9b02-bed8ed21f57b-kube-api-access-h96c8\" (UniqueName: \"kubernetes.io/projected/8595d8d1-88b3-412e-9b02-bed8ed21f57b-kube-api-access-h96c8\") pod \"8595d8d1-88b3-412e-9b02-bed8ed21f57b\" (UID: \"8595d8d1-88b3-412e-9b02-bed8ed21f57b\") " Oct 30 12:40:58.202936 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:58.201178 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8595d8d1-88b3-412e-9b02-bed8ed21f57b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8595d8d1-88b3-412e-9b02-bed8ed21f57b-model-cache\") pod \"8595d8d1-88b3-412e-9b02-bed8ed21f57b\" (UID: \"8595d8d1-88b3-412e-9b02-bed8ed21f57b\") " Oct 30 12:40:58.202936 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:58.201215 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8595d8d1-88b3-412e-9b02-bed8ed21f57b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8595d8d1-88b3-412e-9b02-bed8ed21f57b-dshm\") pod \"8595d8d1-88b3-412e-9b02-bed8ed21f57b\" (UID: \"8595d8d1-88b3-412e-9b02-bed8ed21f57b\") " Oct 30 12:40:58.202936 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:58.201303 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8595d8d1-88b3-412e-9b02-bed8ed21f57b-tmp-dir" pod "8595d8d1-88b3-412e-9b02-bed8ed21f57b" (UID: "8595d8d1-88b3-412e-9b02-bed8ed21f57b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:40:58.202936 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:58.201939 4908 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8595d8d1-88b3-412e-9b02-bed8ed21f57b-tmp-dir\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:40:58.202936 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:58.202057 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8595d8d1-88b3-412e-9b02-bed8ed21f57b-home" pod "8595d8d1-88b3-412e-9b02-bed8ed21f57b" (UID: "8595d8d1-88b3-412e-9b02-bed8ed21f57b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:40:58.202936 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:58.202220 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8595d8d1-88b3-412e-9b02-bed8ed21f57b-model-cache" pod "8595d8d1-88b3-412e-9b02-bed8ed21f57b" (UID: "8595d8d1-88b3-412e-9b02-bed8ed21f57b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:40:58.204857 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:58.204798 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8595d8d1-88b3-412e-9b02-bed8ed21f57b-kube-api-access-h96c8" pod "8595d8d1-88b3-412e-9b02-bed8ed21f57b" (UID: "8595d8d1-88b3-412e-9b02-bed8ed21f57b"). InnerVolumeSpecName "kube-api-access-h96c8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:40:58.205681 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:58.205656 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8595d8d1-88b3-412e-9b02-bed8ed21f57b-tls-certs" pod "8595d8d1-88b3-412e-9b02-bed8ed21f57b" (UID: "8595d8d1-88b3-412e-9b02-bed8ed21f57b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:40:58.205806 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:58.205778 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8595d8d1-88b3-412e-9b02-bed8ed21f57b-dshm" pod "8595d8d1-88b3-412e-9b02-bed8ed21f57b" (UID: "8595d8d1-88b3-412e-9b02-bed8ed21f57b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:40:58.245214 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:58.245183 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4afe31ef-bd98f" Oct 30 12:40:58.303471 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:58.303368 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h96c8\" (UniqueName: \"kubernetes.io/projected/8595d8d1-88b3-412e-9b02-bed8ed21f57b-kube-api-access-h96c8\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:40:58.303471 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:58.303401 4908 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8595d8d1-88b3-412e-9b02-bed8ed21f57b-model-cache\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:40:58.303471 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:58.303417 4908 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8595d8d1-88b3-412e-9b02-bed8ed21f57b-dshm\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:40:58.303471 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:58.303429 4908 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8595d8d1-88b3-412e-9b02-bed8ed21f57b-home\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:40:58.303471 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:58.303444 4908 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8595d8d1-88b3-412e-9b02-bed8ed21f57b-tls-certs\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:40:58.509157 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:58.509128 4908 generic.go:356] "Generic (PLEG): container finished" podID="8595d8d1-88b3-412e-9b02-bed8ed21f57b" containerID="cf1eedddf7102b0cc04a9b3dfd986b12770f4083d02a70c039a39b0965907318" exitCode=0 Oct 30 12:40:58.509713 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:58.509696 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-2675c0/auth-test-model-2675c0-kserve-6bdcd4c665-qspzk" Oct 30 12:40:58.510625 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:58.510599 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-2675c0/auth-test-model-2675c0-kserve-6bdcd4c665-qspzk" event={"ID":"8595d8d1-88b3-412e-9b02-bed8ed21f57b","Type":"ContainerDied","Data":"cf1eedddf7102b0cc04a9b3dfd986b12770f4083d02a70c039a39b0965907318"} Oct 30 12:40:58.510675 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:58.510648 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-2675c0/auth-test-model-2675c0-kserve-6bdcd4c665-qspzk" event={"ID":"8595d8d1-88b3-412e-9b02-bed8ed21f57b","Type":"ContainerDied","Data":"e6159d5e6ac341d6bfcbac109d13152dae04ab219a14ca62b308c4e4ca36d5be"} Oct 30 12:40:58.510715 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:58.510671 4908 scope.go:122] "RemoveContainer" containerID="cf1eedddf7102b0cc04a9b3dfd986b12770f4083d02a70c039a39b0965907318" Oct 30 12:40:58.520728 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:58.519733 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-11581-2e9061"] Oct 30 12:40:58.526892 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:58.526871 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-11581-2e9061"] Oct 30 12:40:58.552145 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:58.552124 4908 scope.go:122] "RemoveContainer" containerID="cf1eedddf7102b0cc04a9b3dfd986b12770f4083d02a70c039a39b0965907318" Oct 30 12:40:58.552457 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:40:58.552425 4908 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf1eedddf7102b0cc04a9b3dfd986b12770f4083d02a70c039a39b0965907318\": container with ID starting with cf1eedddf7102b0cc04a9b3dfd986b12770f4083d02a70c039a39b0965907318 not found: ID does not exist" containerID="cf1eedddf7102b0cc04a9b3dfd986b12770f4083d02a70c039a39b0965907318" Oct 30 12:40:58.552538 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:58.552469 4908 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf1eedddf7102b0cc04a9b3dfd986b12770f4083d02a70c039a39b0965907318"} err="failed to get container status \"cf1eedddf7102b0cc04a9b3dfd986b12770f4083d02a70c039a39b0965907318\": rpc error: code = NotFound desc = could not find container \"cf1eedddf7102b0cc04a9b3dfd986b12770f4083d02a70c039a39b0965907318\": container with ID starting with cf1eedddf7102b0cc04a9b3dfd986b12770f4083d02a70c039a39b0965907318 not found: ID does not exist" Oct 30 12:40:58.599986 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:58.597997 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-2675c0/auth-test-model-2675c0-kserve-6bdcd4c665-qspzk"] Oct 30 12:40:58.602318 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:40:58.602270 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ce4096e_12a9_4293_9c47_6585f6ead95f.slice/crio-c7a17c8e94dc2d70c293467d41def795b879b15bc1227f06e02c1b8dbda33a37 WatchSource:0}: Error finding container c7a17c8e94dc2d70c293467d41def795b879b15bc1227f06e02c1b8dbda33a37: Status 404 returned error can't find the container with id c7a17c8e94dc2d70c293467d41def795b879b15bc1227f06e02c1b8dbda33a37 Oct 30 12:40:58.603113 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:58.603045 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4afe31ef-bd98f"] Oct 30 12:40:58.608685 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:58.608051 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-2675c0/auth-test-model-2675c0-kserve-6bdcd4c665-qspzk"] Oct 30 12:40:58.722368 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:58.721436 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bvpc2"] Oct 30 12:40:58.722368 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:58.721654 4908 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bvpc2" podUID="79ee2692-b905-4301-9389-ec07c74cc426" containerName="istio-proxy" containerID="cri-o://3639b0a7ab88cc99f973404e1dcbc8c51205dac754ff2a91721109e2e0bec131" gracePeriod=30 Oct 30 12:40:58.854421 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:58.854397 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11581-9ddde6" Oct 30 12:40:58.920106 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:58.920072 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8bd49bca-092f-40a1-b4a1-6b836952e44d-kube-api-access-zjnww\" (UniqueName: \"kubernetes.io/projected/8bd49bca-092f-40a1-b4a1-6b836952e44d-kube-api-access-zjnww\") pod \"8bd49bca-092f-40a1-b4a1-6b836952e44d\" (UID: \"8bd49bca-092f-40a1-b4a1-6b836952e44d\") " Oct 30 12:40:58.970021 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:58.969979 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bd49bca-092f-40a1-b4a1-6b836952e44d-kube-api-access-zjnww" pod "8bd49bca-092f-40a1-b4a1-6b836952e44d" (UID: "8bd49bca-092f-40a1-b4a1-6b836952e44d"). InnerVolumeSpecName "kube-api-access-zjnww". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:40:59.021821 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:59.021784 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjnww\" (UniqueName: \"kubernetes.io/projected/8bd49bca-092f-40a1-b4a1-6b836952e44d-kube-api-access-zjnww\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:40:59.520787 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:59.520752 4908 generic.go:356] "Generic (PLEG): container finished" podID="8bd49bca-092f-40a1-b4a1-6b836952e44d" containerID="831bdac6fb833d6e1330e0d628ffbfdfa91b4b08b74672b233a6d4da690c99e7" exitCode=137 Oct 30 12:40:59.521374 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:59.520841 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11581-9ddde6" Oct 30 12:40:59.521374 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:59.520886 4908 scope.go:122] "RemoveContainer" containerID="831bdac6fb833d6e1330e0d628ffbfdfa91b4b08b74672b233a6d4da690c99e7" Oct 30 12:40:59.524631 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:59.524596 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4afe31ef-bd98f" event={"ID":"7ce4096e-12a9-4293-9c47-6585f6ead95f","Type":"ContainerStarted","Data":"37797f2216adf38d654723ffe76577870ba8031b30f35dc3ed0281fbafd3835a"} Oct 30 12:40:59.524770 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:59.524648 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4afe31ef-bd98f" event={"ID":"7ce4096e-12a9-4293-9c47-6585f6ead95f","Type":"ContainerStarted","Data":"c7a17c8e94dc2d70c293467d41def795b879b15bc1227f06e02c1b8dbda33a37"} Oct 30 12:40:59.524770 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:59.524715 4908 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-11581-2e9061" podUID="f7800ec1-a617-413f-a262-769762631d30" containerName="test-curl-11581-2e9061" containerID="cri-o://d09f8f0e70f897b24c1b7b718ae20602dc7ed8ba792e8576eb427d6938a2c6d2" gracePeriod=2 Oct 30 12:40:59.539506 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:59.539462 4908 scope.go:122] "RemoveContainer" containerID="831bdac6fb833d6e1330e0d628ffbfdfa91b4b08b74672b233a6d4da690c99e7" Oct 30 12:40:59.539971 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:40:59.539923 4908 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"831bdac6fb833d6e1330e0d628ffbfdfa91b4b08b74672b233a6d4da690c99e7\": container with ID starting with 831bdac6fb833d6e1330e0d628ffbfdfa91b4b08b74672b233a6d4da690c99e7 not found: ID does not exist" containerID="831bdac6fb833d6e1330e0d628ffbfdfa91b4b08b74672b233a6d4da690c99e7" Oct 30 12:40:59.540055 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:59.539986 4908 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"831bdac6fb833d6e1330e0d628ffbfdfa91b4b08b74672b233a6d4da690c99e7"} err="failed to get container status \"831bdac6fb833d6e1330e0d628ffbfdfa91b4b08b74672b233a6d4da690c99e7\": rpc error: code = NotFound desc = could not find container \"831bdac6fb833d6e1330e0d628ffbfdfa91b4b08b74672b233a6d4da690c99e7\": container with ID starting with 831bdac6fb833d6e1330e0d628ffbfdfa91b4b08b74672b233a6d4da690c99e7 not found: ID does not exist" Oct 30 12:40:59.566840 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:59.566777 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4afe31ef-bd98f" podStartSLOduration=2.566760523 podStartE2EDuration="2.566760523s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:40:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:40:59.562727785 +0000 UTC m=+2816.077156529" watchObservedRunningTime="2026-10-30 12:40:59.566760523 +0000 UTC m=+2816.081189270" Oct 30 12:40:59.575078 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:40:59.575050 4908 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11581-2e9061\" is forbidden: User \"system:node:ip-10-0-141-251.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-251.ec2.internal' and this object" podUID="f7800ec1-a617-413f-a262-769762631d30" pod="opendatahub/test-curl-11581-2e9061" Oct 30 12:40:59.587460 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:40:59.587429 4908 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11581-9ddde6\" is forbidden: User \"system:node:ip-10-0-141-251.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-251.ec2.internal' and this object" podUID="8bd49bca-092f-40a1-b4a1-6b836952e44d" pod="opendatahub/test-curl-11581-9ddde6" Oct 30 12:40:59.599649 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:40:59.599623 4908 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11581-2e9061\" is forbidden: User \"system:node:ip-10-0-141-251.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-251.ec2.internal' and this object" podUID="f7800ec1-a617-413f-a262-769762631d30" pod="opendatahub/test-curl-11581-2e9061" Oct 30 12:40:59.610955 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:59.610930 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8595d8d1-88b3-412e-9b02-bed8ed21f57b" path="/var/lib/kubelet/pods/8595d8d1-88b3-412e-9b02-bed8ed21f57b/volumes" Oct 30 12:40:59.611435 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:40:59.611420 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8bd49bca-092f-40a1-b4a1-6b836952e44d" path="/var/lib/kubelet/pods/8bd49bca-092f-40a1-b4a1-6b836952e44d/volumes" Oct 30 12:41:00.514076 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:00.514044 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-gk4rg" Oct 30 12:41:00.536756 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:00.536716 4908 generic.go:356] "Generic (PLEG): container finished" podID="7ce4096e-12a9-4293-9c47-6585f6ead95f" containerID="37797f2216adf38d654723ffe76577870ba8031b30f35dc3ed0281fbafd3835a" exitCode=0 Oct 30 12:41:00.537212 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:00.536805 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4afe31ef-bd98f" event={"ID":"7ce4096e-12a9-4293-9c47-6585f6ead95f","Type":"ContainerDied","Data":"37797f2216adf38d654723ffe76577870ba8031b30f35dc3ed0281fbafd3835a"} Oct 30 12:41:00.538548 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:00.538524 4908 generic.go:356] "Generic (PLEG): container finished" podID="9ac0ce00-fe15-408d-90eb-9633ea3e67d0" containerID="763181d0f06ab314d19a69c97efef3ce3e5f17327808faa7187c9d8d99668eb6" exitCode=0 Oct 30 12:41:00.538663 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:00.538571 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-gk4rg" Oct 30 12:41:00.538663 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:00.538609 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-gk4rg" event={"ID":"9ac0ce00-fe15-408d-90eb-9633ea3e67d0","Type":"ContainerDied","Data":"763181d0f06ab314d19a69c97efef3ce3e5f17327808faa7187c9d8d99668eb6"} Oct 30 12:41:00.538759 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:00.538663 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-gk4rg" event={"ID":"9ac0ce00-fe15-408d-90eb-9633ea3e67d0","Type":"ContainerDied","Data":"f22db1aea7ee9ed274a2deae674830cc32fadea3fc21d20484d4ef0bd6bbbc4d"} Oct 30 12:41:00.538759 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:00.538695 4908 scope.go:122] "RemoveContainer" containerID="763181d0f06ab314d19a69c97efef3ce3e5f17327808faa7187c9d8d99668eb6" Oct 30 12:41:00.563225 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:00.554562 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ac0ce00-fe15-408d-90eb-9633ea3e67d0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ac0ce00-fe15-408d-90eb-9633ea3e67d0-istiod-ca-cert\") pod \"9ac0ce00-fe15-408d-90eb-9633ea3e67d0\" (UID: \"9ac0ce00-fe15-408d-90eb-9633ea3e67d0\") " Oct 30 12:41:00.563225 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:00.554721 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ac0ce00-fe15-408d-90eb-9633ea3e67d0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ac0ce00-fe15-408d-90eb-9633ea3e67d0-istio-envoy\") pod \"9ac0ce00-fe15-408d-90eb-9633ea3e67d0\" (UID: \"9ac0ce00-fe15-408d-90eb-9633ea3e67d0\") " Oct 30 12:41:00.563225 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:00.554768 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ac0ce00-fe15-408d-90eb-9633ea3e67d0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ac0ce00-fe15-408d-90eb-9633ea3e67d0-credential-socket\") pod \"9ac0ce00-fe15-408d-90eb-9633ea3e67d0\" (UID: \"9ac0ce00-fe15-408d-90eb-9633ea3e67d0\") " Oct 30 12:41:00.563225 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:00.554842 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ac0ce00-fe15-408d-90eb-9633ea3e67d0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ac0ce00-fe15-408d-90eb-9633ea3e67d0-workload-certs\") pod \"9ac0ce00-fe15-408d-90eb-9633ea3e67d0\" (UID: \"9ac0ce00-fe15-408d-90eb-9633ea3e67d0\") " Oct 30 12:41:00.563225 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:00.554937 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ac0ce00-fe15-408d-90eb-9633ea3e67d0-kube-api-access-fzcf5\" (UniqueName: \"kubernetes.io/projected/9ac0ce00-fe15-408d-90eb-9633ea3e67d0-kube-api-access-fzcf5\") pod \"9ac0ce00-fe15-408d-90eb-9633ea3e67d0\" (UID: \"9ac0ce00-fe15-408d-90eb-9633ea3e67d0\") " Oct 30 12:41:00.563225 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:00.554982 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ac0ce00-fe15-408d-90eb-9633ea3e67d0-istio-token\" (UniqueName: \"kubernetes.io/projected/9ac0ce00-fe15-408d-90eb-9633ea3e67d0-istio-token\") pod \"9ac0ce00-fe15-408d-90eb-9633ea3e67d0\" (UID: \"9ac0ce00-fe15-408d-90eb-9633ea3e67d0\") " Oct 30 12:41:00.563225 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:00.555052 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ac0ce00-fe15-408d-90eb-9633ea3e67d0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ac0ce00-fe15-408d-90eb-9633ea3e67d0-workload-socket\") pod \"9ac0ce00-fe15-408d-90eb-9633ea3e67d0\" (UID: \"9ac0ce00-fe15-408d-90eb-9633ea3e67d0\") " Oct 30 12:41:00.563225 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:00.555111 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ac0ce00-fe15-408d-90eb-9633ea3e67d0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ac0ce00-fe15-408d-90eb-9633ea3e67d0-istio-data\") pod \"9ac0ce00-fe15-408d-90eb-9633ea3e67d0\" (UID: \"9ac0ce00-fe15-408d-90eb-9633ea3e67d0\") " Oct 30 12:41:00.563225 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:00.555188 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9ac0ce00-fe15-408d-90eb-9633ea3e67d0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ac0ce00-fe15-408d-90eb-9633ea3e67d0-istio-podinfo\") pod \"9ac0ce00-fe15-408d-90eb-9633ea3e67d0\" (UID: \"9ac0ce00-fe15-408d-90eb-9633ea3e67d0\") " Oct 30 12:41:00.563225 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:00.561715 4908 scope.go:122] "RemoveContainer" containerID="763181d0f06ab314d19a69c97efef3ce3e5f17327808faa7187c9d8d99668eb6" Oct 30 12:41:00.563225 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:00.562053 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ac0ce00-fe15-408d-90eb-9633ea3e67d0-workload-certs" pod "9ac0ce00-fe15-408d-90eb-9633ea3e67d0" (UID: "9ac0ce00-fe15-408d-90eb-9633ea3e67d0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:41:00.563225 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:00.562334 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ac0ce00-fe15-408d-90eb-9633ea3e67d0-istiod-ca-cert" pod "9ac0ce00-fe15-408d-90eb-9633ea3e67d0" (UID: "9ac0ce00-fe15-408d-90eb-9633ea3e67d0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:41:00.563902 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:41:00.563733 4908 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"763181d0f06ab314d19a69c97efef3ce3e5f17327808faa7187c9d8d99668eb6\": container with ID starting with 763181d0f06ab314d19a69c97efef3ce3e5f17327808faa7187c9d8d99668eb6 not found: ID does not exist" containerID="763181d0f06ab314d19a69c97efef3ce3e5f17327808faa7187c9d8d99668eb6" Oct 30 12:41:00.563902 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:00.563772 4908 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"763181d0f06ab314d19a69c97efef3ce3e5f17327808faa7187c9d8d99668eb6"} err="failed to get container status \"763181d0f06ab314d19a69c97efef3ce3e5f17327808faa7187c9d8d99668eb6\": rpc error: code = NotFound desc = could not find container \"763181d0f06ab314d19a69c97efef3ce3e5f17327808faa7187c9d8d99668eb6\": container with ID starting with 763181d0f06ab314d19a69c97efef3ce3e5f17327808faa7187c9d8d99668eb6 not found: ID does not exist" Oct 30 12:41:00.567188 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:00.567157 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9ac0ce00-fe15-408d-90eb-9633ea3e67d0-istio-podinfo" pod "9ac0ce00-fe15-408d-90eb-9633ea3e67d0" (UID: "9ac0ce00-fe15-408d-90eb-9633ea3e67d0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 12:41:00.567533 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:00.567506 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ac0ce00-fe15-408d-90eb-9633ea3e67d0-workload-socket" pod "9ac0ce00-fe15-408d-90eb-9633ea3e67d0" (UID: "9ac0ce00-fe15-408d-90eb-9633ea3e67d0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:41:00.567669 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:00.567641 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ac0ce00-fe15-408d-90eb-9633ea3e67d0-istio-envoy" pod "9ac0ce00-fe15-408d-90eb-9633ea3e67d0" (UID: "9ac0ce00-fe15-408d-90eb-9633ea3e67d0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:41:00.567857 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:00.567655 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ac0ce00-fe15-408d-90eb-9633ea3e67d0-istio-data" pod "9ac0ce00-fe15-408d-90eb-9633ea3e67d0" (UID: "9ac0ce00-fe15-408d-90eb-9633ea3e67d0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:41:00.567968 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:00.567749 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ac0ce00-fe15-408d-90eb-9633ea3e67d0-credential-socket" pod "9ac0ce00-fe15-408d-90eb-9633ea3e67d0" (UID: "9ac0ce00-fe15-408d-90eb-9633ea3e67d0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:41:00.572780 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:00.572756 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ac0ce00-fe15-408d-90eb-9633ea3e67d0-istio-token" pod "9ac0ce00-fe15-408d-90eb-9633ea3e67d0" (UID: "9ac0ce00-fe15-408d-90eb-9633ea3e67d0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:41:00.576259 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:00.576226 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ac0ce00-fe15-408d-90eb-9633ea3e67d0-kube-api-access-fzcf5" pod "9ac0ce00-fe15-408d-90eb-9633ea3e67d0" (UID: "9ac0ce00-fe15-408d-90eb-9633ea3e67d0"). InnerVolumeSpecName "kube-api-access-fzcf5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:41:00.627848 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:00.627247 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-a786af-openshift-default-f89f5ffc7-b62z2"] Oct 30 12:41:00.629257 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:00.628050 4908 cpu_manager.go:414] "containerMap: removing container" podUID="f7800ec1-a617-413f-a262-769762631d30" containerName="test-curl-11581-2e9061" Oct 30 12:41:00.629257 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:00.628076 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7800ec1-a617-413f-a262-769762631d30" containerName="test-curl-11581-2e9061" Oct 30 12:41:00.629257 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:00.628109 4908 cpu_manager.go:414] "containerMap: removing container" podUID="8595d8d1-88b3-412e-9b02-bed8ed21f57b" containerName="main" Oct 30 12:41:00.629257 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:00.628117 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8595d8d1-88b3-412e-9b02-bed8ed21f57b" containerName="main" Oct 30 12:41:00.629257 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:00.628146 4908 cpu_manager.go:414] "containerMap: removing container" podUID="9ac0ce00-fe15-408d-90eb-9633ea3e67d0" containerName="istio-proxy" Oct 30 12:41:00.629257 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:00.628153 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ac0ce00-fe15-408d-90eb-9633ea3e67d0" containerName="istio-proxy" Oct 30 12:41:00.629257 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:00.628295 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ac0ce00-fe15-408d-90eb-9633ea3e67d0" containerName="istio-proxy" Oct 30 12:41:00.629257 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:00.628316 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7800ec1-a617-413f-a262-769762631d30" containerName="test-curl-11581-2e9061" Oct 30 12:41:00.629257 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:00.628329 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="8595d8d1-88b3-412e-9b02-bed8ed21f57b" containerName="main" Oct 30 12:41:00.649403 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:00.649363 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-a786af-openshift-default-f89f5ffc7-b62z2"] Oct 30 12:41:00.649558 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:00.649483 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-a786af-openshift-default-f89f5ffc7-b62z2" Oct 30 12:41:00.659852 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:00.658753 4908 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ac0ce00-fe15-408d-90eb-9633ea3e67d0-istio-token\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:41:00.659852 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:00.658782 4908 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ac0ce00-fe15-408d-90eb-9633ea3e67d0-workload-socket\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:41:00.659852 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:00.658796 4908 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ac0ce00-fe15-408d-90eb-9633ea3e67d0-istio-data\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:41:00.659852 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:00.659760 4908 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ac0ce00-fe15-408d-90eb-9633ea3e67d0-istio-podinfo\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:41:00.660284 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:00.659970 4908 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ac0ce00-fe15-408d-90eb-9633ea3e67d0-istiod-ca-cert\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:41:00.660284 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:00.660015 4908 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ac0ce00-fe15-408d-90eb-9633ea3e67d0-istio-envoy\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:41:00.660284 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:00.660042 4908 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ac0ce00-fe15-408d-90eb-9633ea3e67d0-credential-socket\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:41:00.660284 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:00.660056 4908 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ac0ce00-fe15-408d-90eb-9633ea3e67d0-workload-certs\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:41:00.660284 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:00.660070 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzcf5\" (UniqueName: \"kubernetes.io/projected/9ac0ce00-fe15-408d-90eb-9633ea3e67d0-kube-api-access-fzcf5\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:41:00.666105 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:00.666073 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-a786af-openshift-default-dockercfg-m5p6b\"" Oct 30 12:41:00.765850 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:00.763886 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20af951f-19a7-49e6-ab1d-3bdee930084d-istio-envoy\") pod \"e2e-extmod-a-a786af-openshift-default-f89f5ffc7-b62z2\" (UID: \"20af951f-19a7-49e6-ab1d-3bdee930084d\") " pod="openshift-ingress/e2e-extmod-a-a786af-openshift-default-f89f5ffc7-b62z2" Oct 30 12:41:00.765850 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:00.763956 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20af951f-19a7-49e6-ab1d-3bdee930084d-istiod-ca-cert\") pod \"e2e-extmod-a-a786af-openshift-default-f89f5ffc7-b62z2\" (UID: \"20af951f-19a7-49e6-ab1d-3bdee930084d\") " pod="openshift-ingress/e2e-extmod-a-a786af-openshift-default-f89f5ffc7-b62z2" Oct 30 12:41:00.765850 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:00.763992 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zq55s\" (UniqueName: \"kubernetes.io/projected/20af951f-19a7-49e6-ab1d-3bdee930084d-kube-api-access-zq55s\") pod \"e2e-extmod-a-a786af-openshift-default-f89f5ffc7-b62z2\" (UID: \"20af951f-19a7-49e6-ab1d-3bdee930084d\") " pod="openshift-ingress/e2e-extmod-a-a786af-openshift-default-f89f5ffc7-b62z2" Oct 30 12:41:00.765850 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:00.764052 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20af951f-19a7-49e6-ab1d-3bdee930084d-istio-data\") pod \"e2e-extmod-a-a786af-openshift-default-f89f5ffc7-b62z2\" (UID: \"20af951f-19a7-49e6-ab1d-3bdee930084d\") " pod="openshift-ingress/e2e-extmod-a-a786af-openshift-default-f89f5ffc7-b62z2" Oct 30 12:41:00.765850 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:00.764076 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20af951f-19a7-49e6-ab1d-3bdee930084d-credential-socket\") pod \"e2e-extmod-a-a786af-openshift-default-f89f5ffc7-b62z2\" (UID: \"20af951f-19a7-49e6-ab1d-3bdee930084d\") " pod="openshift-ingress/e2e-extmod-a-a786af-openshift-default-f89f5ffc7-b62z2" Oct 30 12:41:00.765850 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:00.764139 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20af951f-19a7-49e6-ab1d-3bdee930084d-istio-token\") pod \"e2e-extmod-a-a786af-openshift-default-f89f5ffc7-b62z2\" (UID: \"20af951f-19a7-49e6-ab1d-3bdee930084d\") " pod="openshift-ingress/e2e-extmod-a-a786af-openshift-default-f89f5ffc7-b62z2" Oct 30 12:41:00.765850 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:00.764213 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20af951f-19a7-49e6-ab1d-3bdee930084d-workload-certs\") pod \"e2e-extmod-a-a786af-openshift-default-f89f5ffc7-b62z2\" (UID: \"20af951f-19a7-49e6-ab1d-3bdee930084d\") " pod="openshift-ingress/e2e-extmod-a-a786af-openshift-default-f89f5ffc7-b62z2" Oct 30 12:41:00.765850 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:00.764242 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20af951f-19a7-49e6-ab1d-3bdee930084d-workload-socket\") pod \"e2e-extmod-a-a786af-openshift-default-f89f5ffc7-b62z2\" (UID: \"20af951f-19a7-49e6-ab1d-3bdee930084d\") " pod="openshift-ingress/e2e-extmod-a-a786af-openshift-default-f89f5ffc7-b62z2" Oct 30 12:41:00.765850 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:00.764297 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20af951f-19a7-49e6-ab1d-3bdee930084d-istio-podinfo\") pod \"e2e-extmod-a-a786af-openshift-default-f89f5ffc7-b62z2\" (UID: \"20af951f-19a7-49e6-ab1d-3bdee930084d\") " pod="openshift-ingress/e2e-extmod-a-a786af-openshift-default-f89f5ffc7-b62z2" Oct 30 12:41:00.871066 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:00.869237 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20af951f-19a7-49e6-ab1d-3bdee930084d-istio-token\") pod \"e2e-extmod-a-a786af-openshift-default-f89f5ffc7-b62z2\" (UID: \"20af951f-19a7-49e6-ab1d-3bdee930084d\") " pod="openshift-ingress/e2e-extmod-a-a786af-openshift-default-f89f5ffc7-b62z2" Oct 30 12:41:00.871066 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:00.869334 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20af951f-19a7-49e6-ab1d-3bdee930084d-workload-certs\") pod \"e2e-extmod-a-a786af-openshift-default-f89f5ffc7-b62z2\" (UID: \"20af951f-19a7-49e6-ab1d-3bdee930084d\") " pod="openshift-ingress/e2e-extmod-a-a786af-openshift-default-f89f5ffc7-b62z2" Oct 30 12:41:00.871066 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:00.869365 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20af951f-19a7-49e6-ab1d-3bdee930084d-workload-socket\") pod \"e2e-extmod-a-a786af-openshift-default-f89f5ffc7-b62z2\" (UID: \"20af951f-19a7-49e6-ab1d-3bdee930084d\") " pod="openshift-ingress/e2e-extmod-a-a786af-openshift-default-f89f5ffc7-b62z2" Oct 30 12:41:00.871066 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:00.869419 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20af951f-19a7-49e6-ab1d-3bdee930084d-istio-podinfo\") pod \"e2e-extmod-a-a786af-openshift-default-f89f5ffc7-b62z2\" (UID: \"20af951f-19a7-49e6-ab1d-3bdee930084d\") " pod="openshift-ingress/e2e-extmod-a-a786af-openshift-default-f89f5ffc7-b62z2" Oct 30 12:41:00.871066 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:00.869515 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20af951f-19a7-49e6-ab1d-3bdee930084d-istio-envoy\") pod \"e2e-extmod-a-a786af-openshift-default-f89f5ffc7-b62z2\" (UID: \"20af951f-19a7-49e6-ab1d-3bdee930084d\") " pod="openshift-ingress/e2e-extmod-a-a786af-openshift-default-f89f5ffc7-b62z2" Oct 30 12:41:00.871066 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:00.869564 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20af951f-19a7-49e6-ab1d-3bdee930084d-istiod-ca-cert\") pod \"e2e-extmod-a-a786af-openshift-default-f89f5ffc7-b62z2\" (UID: \"20af951f-19a7-49e6-ab1d-3bdee930084d\") " pod="openshift-ingress/e2e-extmod-a-a786af-openshift-default-f89f5ffc7-b62z2" Oct 30 12:41:00.871066 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:00.869592 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zq55s\" (UniqueName: \"kubernetes.io/projected/20af951f-19a7-49e6-ab1d-3bdee930084d-kube-api-access-zq55s\") pod \"e2e-extmod-a-a786af-openshift-default-f89f5ffc7-b62z2\" (UID: \"20af951f-19a7-49e6-ab1d-3bdee930084d\") " pod="openshift-ingress/e2e-extmod-a-a786af-openshift-default-f89f5ffc7-b62z2" Oct 30 12:41:00.871066 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:00.869655 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20af951f-19a7-49e6-ab1d-3bdee930084d-istio-data\") pod \"e2e-extmod-a-a786af-openshift-default-f89f5ffc7-b62z2\" (UID: \"20af951f-19a7-49e6-ab1d-3bdee930084d\") " pod="openshift-ingress/e2e-extmod-a-a786af-openshift-default-f89f5ffc7-b62z2" Oct 30 12:41:00.871066 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:00.869681 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20af951f-19a7-49e6-ab1d-3bdee930084d-credential-socket\") pod \"e2e-extmod-a-a786af-openshift-default-f89f5ffc7-b62z2\" (UID: \"20af951f-19a7-49e6-ab1d-3bdee930084d\") " pod="openshift-ingress/e2e-extmod-a-a786af-openshift-default-f89f5ffc7-b62z2" Oct 30 12:41:00.871066 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:00.870124 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20af951f-19a7-49e6-ab1d-3bdee930084d-credential-socket\") pod \"e2e-extmod-a-a786af-openshift-default-f89f5ffc7-b62z2\" (UID: \"20af951f-19a7-49e6-ab1d-3bdee930084d\") " pod="openshift-ingress/e2e-extmod-a-a786af-openshift-default-f89f5ffc7-b62z2" Oct 30 12:41:00.871066 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:00.870612 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20af951f-19a7-49e6-ab1d-3bdee930084d-workload-certs\") pod \"e2e-extmod-a-a786af-openshift-default-f89f5ffc7-b62z2\" (UID: \"20af951f-19a7-49e6-ab1d-3bdee930084d\") " pod="openshift-ingress/e2e-extmod-a-a786af-openshift-default-f89f5ffc7-b62z2" Oct 30 12:41:00.872283 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:00.871863 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20af951f-19a7-49e6-ab1d-3bdee930084d-workload-socket\") pod \"e2e-extmod-a-a786af-openshift-default-f89f5ffc7-b62z2\" (UID: \"20af951f-19a7-49e6-ab1d-3bdee930084d\") " pod="openshift-ingress/e2e-extmod-a-a786af-openshift-default-f89f5ffc7-b62z2" Oct 30 12:41:00.876675 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:00.876652 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20af951f-19a7-49e6-ab1d-3bdee930084d-istiod-ca-cert\") pod \"e2e-extmod-a-a786af-openshift-default-f89f5ffc7-b62z2\" (UID: \"20af951f-19a7-49e6-ab1d-3bdee930084d\") " pod="openshift-ingress/e2e-extmod-a-a786af-openshift-default-f89f5ffc7-b62z2" Oct 30 12:41:00.877743 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:00.877411 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20af951f-19a7-49e6-ab1d-3bdee930084d-istio-data\") pod \"e2e-extmod-a-a786af-openshift-default-f89f5ffc7-b62z2\" (UID: \"20af951f-19a7-49e6-ab1d-3bdee930084d\") " pod="openshift-ingress/e2e-extmod-a-a786af-openshift-default-f89f5ffc7-b62z2" Oct 30 12:41:00.885945 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:00.885919 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20af951f-19a7-49e6-ab1d-3bdee930084d-istio-envoy\") pod \"e2e-extmod-a-a786af-openshift-default-f89f5ffc7-b62z2\" (UID: \"20af951f-19a7-49e6-ab1d-3bdee930084d\") " pod="openshift-ingress/e2e-extmod-a-a786af-openshift-default-f89f5ffc7-b62z2" Oct 30 12:41:00.886578 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:00.886537 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20af951f-19a7-49e6-ab1d-3bdee930084d-istio-podinfo\") pod \"e2e-extmod-a-a786af-openshift-default-f89f5ffc7-b62z2\" (UID: \"20af951f-19a7-49e6-ab1d-3bdee930084d\") " pod="openshift-ingress/e2e-extmod-a-a786af-openshift-default-f89f5ffc7-b62z2" Oct 30 12:41:00.900412 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:00.900350 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20af951f-19a7-49e6-ab1d-3bdee930084d-istio-token\") pod \"e2e-extmod-a-a786af-openshift-default-f89f5ffc7-b62z2\" (UID: \"20af951f-19a7-49e6-ab1d-3bdee930084d\") " pod="openshift-ingress/e2e-extmod-a-a786af-openshift-default-f89f5ffc7-b62z2" Oct 30 12:41:00.924189 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:00.924155 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zq55s\" (UniqueName: \"kubernetes.io/projected/20af951f-19a7-49e6-ab1d-3bdee930084d-kube-api-access-zq55s\") pod \"e2e-extmod-a-a786af-openshift-default-f89f5ffc7-b62z2\" (UID: \"20af951f-19a7-49e6-ab1d-3bdee930084d\") " pod="openshift-ingress/e2e-extmod-a-a786af-openshift-default-f89f5ffc7-b62z2" Oct 30 12:41:00.929632 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:00.929571 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-gk4rg"] Oct 30 12:41:00.933660 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:00.933588 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-gk4rg"] Oct 30 12:41:00.971821 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:00.971771 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-a786af-openshift-default-f89f5ffc7-b62z2" Oct 30 12:41:01.332791 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:01.332712 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-a786af-openshift-default-f89f5ffc7-b62z2"] Oct 30 12:41:01.336862 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:41:01.336802 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20af951f_19a7_49e6_ab1d_3bdee930084d.slice/crio-15c8456823cdf8e59d0b9b3a7c9a2aa85b704de19d4224e9dd6ee8507df047e1 WatchSource:0}: Error finding container 15c8456823cdf8e59d0b9b3a7c9a2aa85b704de19d4224e9dd6ee8507df047e1: Status 404 returned error can't find the container with id 15c8456823cdf8e59d0b9b3a7c9a2aa85b704de19d4224e9dd6ee8507df047e1 Oct 30 12:41:01.340341 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:01.340307 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:41:01.340447 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:01.340389 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:41:01.340447 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:01.340441 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:41:01.475565 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:01.475540 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5021d04-xfvtm"] Oct 30 12:41:01.480193 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:01.480167 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5021d04-xfvtm" Oct 30 12:41:01.505644 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:01.503385 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5021d04-xfvtm"] Oct 30 12:41:01.575869 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:01.566317 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a786af-openshift-default-f89f5ffc7-b62z2" event={"ID":"20af951f-19a7-49e6-ab1d-3bdee930084d","Type":"ContainerStarted","Data":"134480ee395452bba7c2b489dee184e5ec1afcfd6618812b3430bd3479a010d8"} Oct 30 12:41:01.575869 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:01.566385 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a786af-openshift-default-f89f5ffc7-b62z2" event={"ID":"20af951f-19a7-49e6-ab1d-3bdee930084d","Type":"ContainerStarted","Data":"15c8456823cdf8e59d0b9b3a7c9a2aa85b704de19d4224e9dd6ee8507df047e1"} Oct 30 12:41:01.608767 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:01.608384 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fdkc\" (UniqueName: \"kubernetes.io/projected/a437fd08-0b7b-47c2-a47a-36ce86fc62ff-kube-api-access-9fdkc\") pod \"maas-api-revoke-sub-f5021d04-xfvtm\" (UID: \"a437fd08-0b7b-47c2-a47a-36ce86fc62ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5021d04-xfvtm" Oct 30 12:41:01.608767 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:01.608464 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a437fd08-0b7b-47c2-a47a-36ce86fc62ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f5021d04-xfvtm\" (UID: \"a437fd08-0b7b-47c2-a47a-36ce86fc62ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5021d04-xfvtm" Oct 30 12:41:01.608767 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:01.608562 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a437fd08-0b7b-47c2-a47a-36ce86fc62ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f5021d04-xfvtm\" (UID: \"a437fd08-0b7b-47c2-a47a-36ce86fc62ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5021d04-xfvtm" Oct 30 12:41:01.625775 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:01.625708 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-a786af-openshift-default-f89f5ffc7-b62z2" podStartSLOduration=1.625695948 podStartE2EDuration="1.625695948s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:41:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:41:01.609106669 +0000 UTC m=+2818.123535458" watchObservedRunningTime="2026-10-30 12:41:01.625695948 +0000 UTC m=+2818.140124692" Oct 30 12:41:01.637496 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:01.636669 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ac0ce00-fe15-408d-90eb-9633ea3e67d0" path="/var/lib/kubelet/pods/9ac0ce00-fe15-408d-90eb-9633ea3e67d0/volumes" Oct 30 12:41:01.719099 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:01.713498 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a437fd08-0b7b-47c2-a47a-36ce86fc62ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f5021d04-xfvtm\" (UID: \"a437fd08-0b7b-47c2-a47a-36ce86fc62ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5021d04-xfvtm" Oct 30 12:41:01.719099 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:01.713804 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fdkc\" (UniqueName: \"kubernetes.io/projected/a437fd08-0b7b-47c2-a47a-36ce86fc62ff-kube-api-access-9fdkc\") pod \"maas-api-revoke-sub-f5021d04-xfvtm\" (UID: \"a437fd08-0b7b-47c2-a47a-36ce86fc62ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5021d04-xfvtm" Oct 30 12:41:01.719099 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:01.713901 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a437fd08-0b7b-47c2-a47a-36ce86fc62ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f5021d04-xfvtm\" (UID: \"a437fd08-0b7b-47c2-a47a-36ce86fc62ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5021d04-xfvtm" Oct 30 12:41:01.719099 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:01.715345 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a437fd08-0b7b-47c2-a47a-36ce86fc62ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f5021d04-xfvtm\" (UID: \"a437fd08-0b7b-47c2-a47a-36ce86fc62ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5021d04-xfvtm" Oct 30 12:41:01.795099 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:01.790328 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a437fd08-0b7b-47c2-a47a-36ce86fc62ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f5021d04-xfvtm\" (UID: \"a437fd08-0b7b-47c2-a47a-36ce86fc62ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5021d04-xfvtm" Oct 30 12:41:01.795099 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:01.793220 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fdkc\" (UniqueName: \"kubernetes.io/projected/a437fd08-0b7b-47c2-a47a-36ce86fc62ff-kube-api-access-9fdkc\") pod \"maas-api-revoke-sub-f5021d04-xfvtm\" (UID: \"a437fd08-0b7b-47c2-a47a-36ce86fc62ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5021d04-xfvtm" Oct 30 12:41:01.800185 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:01.799942 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5021d04-xfvtm" Oct 30 12:41:01.878925 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:01.878519 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4afe31ef-bd98f" Oct 30 12:41:01.941059 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:01.940884 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11581-2e9061" Oct 30 12:41:01.973948 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:01.973913 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-a786af-openshift-default-f89f5ffc7-b62z2" Oct 30 12:41:02.028653 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:02.028430 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7800ec1-a617-413f-a262-769762631d30-kube-api-access-pcrrq\" (UniqueName: \"kubernetes.io/projected/f7800ec1-a617-413f-a262-769762631d30-kube-api-access-pcrrq\") pod \"f7800ec1-a617-413f-a262-769762631d30\" (UID: \"f7800ec1-a617-413f-a262-769762631d30\") " Oct 30 12:41:02.028653 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:02.028480 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7ce4096e-12a9-4293-9c47-6585f6ead95f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ce4096e-12a9-4293-9c47-6585f6ead95f-maas-api-xks-ca\") pod \"7ce4096e-12a9-4293-9c47-6585f6ead95f\" (UID: \"7ce4096e-12a9-4293-9c47-6585f6ead95f\") " Oct 30 12:41:02.028653 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:02.028643 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ce4096e-12a9-4293-9c47-6585f6ead95f-kube-api-access-qgt66\" (UniqueName: \"kubernetes.io/projected/7ce4096e-12a9-4293-9c47-6585f6ead95f-kube-api-access-qgt66\") pod \"7ce4096e-12a9-4293-9c47-6585f6ead95f\" (UID: \"7ce4096e-12a9-4293-9c47-6585f6ead95f\") " Oct 30 12:41:02.029011 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:02.028684 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ce4096e-12a9-4293-9c47-6585f6ead95f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ce4096e-12a9-4293-9c47-6585f6ead95f-maas-api-openshift-ca\") pod \"7ce4096e-12a9-4293-9c47-6585f6ead95f\" (UID: \"7ce4096e-12a9-4293-9c47-6585f6ead95f\") " Oct 30 12:41:02.029552 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:02.029500 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ce4096e-12a9-4293-9c47-6585f6ead95f-maas-api-openshift-ca" pod "7ce4096e-12a9-4293-9c47-6585f6ead95f" (UID: "7ce4096e-12a9-4293-9c47-6585f6ead95f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:41:02.031579 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:02.031557 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7800ec1-a617-413f-a262-769762631d30-kube-api-access-pcrrq" pod "f7800ec1-a617-413f-a262-769762631d30" (UID: "f7800ec1-a617-413f-a262-769762631d30"). InnerVolumeSpecName "kube-api-access-pcrrq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:41:02.031693 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:02.031619 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ce4096e-12a9-4293-9c47-6585f6ead95f-kube-api-access-qgt66" pod "7ce4096e-12a9-4293-9c47-6585f6ead95f" (UID: "7ce4096e-12a9-4293-9c47-6585f6ead95f"). InnerVolumeSpecName "kube-api-access-qgt66". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:41:02.031986 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:02.031968 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ce4096e-12a9-4293-9c47-6585f6ead95f-maas-api-xks-ca" pod "7ce4096e-12a9-4293-9c47-6585f6ead95f" (UID: "7ce4096e-12a9-4293-9c47-6585f6ead95f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:41:02.130911 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:02.130816 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qgt66\" (UniqueName: \"kubernetes.io/projected/7ce4096e-12a9-4293-9c47-6585f6ead95f-kube-api-access-qgt66\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:41:02.130911 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:02.130861 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ce4096e-12a9-4293-9c47-6585f6ead95f-maas-api-openshift-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:41:02.130911 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:02.130871 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pcrrq\" (UniqueName: \"kubernetes.io/projected/f7800ec1-a617-413f-a262-769762631d30-kube-api-access-pcrrq\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:41:02.130911 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:02.130880 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ce4096e-12a9-4293-9c47-6585f6ead95f-maas-api-xks-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:41:02.248276 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:02.248249 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5021d04-xfvtm"] Oct 30 12:41:02.250467 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:41:02.250432 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda437fd08_0b7b_47c2_a47a_36ce86fc62ff.slice/crio-ddc7e73d09b6ad3d17d5e242db97e1e669665bffe6c79a4af9283cfddafc2eca WatchSource:0}: Error finding container ddc7e73d09b6ad3d17d5e242db97e1e669665bffe6c79a4af9283cfddafc2eca: Status 404 returned error can't find the container with id ddc7e73d09b6ad3d17d5e242db97e1e669665bffe6c79a4af9283cfddafc2eca Oct 30 12:41:02.608106 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:02.608073 4908 generic.go:356] "Generic (PLEG): container finished" podID="f7800ec1-a617-413f-a262-769762631d30" containerID="d09f8f0e70f897b24c1b7b718ae20602dc7ed8ba792e8576eb427d6938a2c6d2" exitCode=137 Oct 30 12:41:02.608702 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:02.608239 4908 scope.go:122] "RemoveContainer" containerID="d09f8f0e70f897b24c1b7b718ae20602dc7ed8ba792e8576eb427d6938a2c6d2" Oct 30 12:41:02.608785 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:02.608302 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11581-2e9061" Oct 30 12:41:02.612491 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:02.612467 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5021d04-xfvtm" event={"ID":"a437fd08-0b7b-47c2-a47a-36ce86fc62ff","Type":"ContainerStarted","Data":"ddc7e73d09b6ad3d17d5e242db97e1e669665bffe6c79a4af9283cfddafc2eca"} Oct 30 12:41:02.617309 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:02.617223 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4afe31ef-bd98f" Oct 30 12:41:02.617309 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:02.617251 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4afe31ef-bd98f" event={"ID":"7ce4096e-12a9-4293-9c47-6585f6ead95f","Type":"ContainerDied","Data":"c7a17c8e94dc2d70c293467d41def795b879b15bc1227f06e02c1b8dbda33a37"} Oct 30 12:41:02.617309 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:02.617285 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c7a17c8e94dc2d70c293467d41def795b879b15bc1227f06e02c1b8dbda33a37" Oct 30 12:41:02.629352 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:02.629327 4908 scope.go:122] "RemoveContainer" containerID="d09f8f0e70f897b24c1b7b718ae20602dc7ed8ba792e8576eb427d6938a2c6d2" Oct 30 12:41:02.629620 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:41:02.629602 4908 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d09f8f0e70f897b24c1b7b718ae20602dc7ed8ba792e8576eb427d6938a2c6d2\": container with ID starting with d09f8f0e70f897b24c1b7b718ae20602dc7ed8ba792e8576eb427d6938a2c6d2 not found: ID does not exist" containerID="d09f8f0e70f897b24c1b7b718ae20602dc7ed8ba792e8576eb427d6938a2c6d2" Oct 30 12:41:02.629682 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:02.629634 4908 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d09f8f0e70f897b24c1b7b718ae20602dc7ed8ba792e8576eb427d6938a2c6d2"} err="failed to get container status \"d09f8f0e70f897b24c1b7b718ae20602dc7ed8ba792e8576eb427d6938a2c6d2\": rpc error: code = NotFound desc = could not find container \"d09f8f0e70f897b24c1b7b718ae20602dc7ed8ba792e8576eb427d6938a2c6d2\": container with ID starting with d09f8f0e70f897b24c1b7b718ae20602dc7ed8ba792e8576eb427d6938a2c6d2 not found: ID does not exist" Oct 30 12:41:02.640091 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:41:02.640063 4908 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11581-2e9061\" is forbidden: User \"system:node:ip-10-0-141-251.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-251.ec2.internal' and this object" podUID="f7800ec1-a617-413f-a262-769762631d30" pod="opendatahub/test-curl-11581-2e9061" Oct 30 12:41:02.666034 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:41:02.665990 4908 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11581-2e9061\" is forbidden: User \"system:node:ip-10-0-141-251.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-251.ec2.internal' and this object" podUID="f7800ec1-a617-413f-a262-769762631d30" pod="opendatahub/test-curl-11581-2e9061" Oct 30 12:41:02.978505 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:02.978467 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-a786af-openshift-default-f89f5ffc7-b62z2" Oct 30 12:41:02.990927 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:41:02.990798 4908 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11581-2e9061\" is forbidden: User \"system:node:ip-10-0-141-251.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-251.ec2.internal' and this object" podUID="f7800ec1-a617-413f-a262-769762631d30" pod="opendatahub/test-curl-11581-2e9061" Oct 30 12:41:03.610279 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:03.610240 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7800ec1-a617-413f-a262-769762631d30" path="/var/lib/kubelet/pods/f7800ec1-a617-413f-a262-769762631d30/volumes" Oct 30 12:41:03.616623 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:41:03.616582 4908 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11581-2e9061\" is forbidden: User \"system:node:ip-10-0-141-251.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-251.ec2.internal' and this object" podUID="f7800ec1-a617-413f-a262-769762631d30" pod="opendatahub/test-curl-11581-2e9061" Oct 30 12:41:03.626150 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:03.626120 4908 generic.go:356] "Generic (PLEG): container finished" podID="a437fd08-0b7b-47c2-a47a-36ce86fc62ff" containerID="3e9498d5c01d69e185e4252e6022d7ae3633c42042aea138817c6e520bd17a45" exitCode=0 Oct 30 12:41:03.626267 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:03.626190 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5021d04-xfvtm" event={"ID":"a437fd08-0b7b-47c2-a47a-36ce86fc62ff","Type":"ContainerDied","Data":"3e9498d5c01d69e185e4252e6022d7ae3633c42042aea138817c6e520bd17a45"} Oct 30 12:41:03.627510 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:03.627487 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-a786af-openshift-default-f89f5ffc7-b62z2" Oct 30 12:41:03.628468 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:03.628450 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-a786af-openshift-default-f89f5ffc7-b62z2" Oct 30 12:41:04.016263 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:04.016235 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bvpc2" Oct 30 12:41:04.078322 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:04.075244 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/79ee2692-b905-4301-9389-ec07c74cc426-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79ee2692-b905-4301-9389-ec07c74cc426-istio-podinfo\") pod \"79ee2692-b905-4301-9389-ec07c74cc426\" (UID: \"79ee2692-b905-4301-9389-ec07c74cc426\") " Oct 30 12:41:04.078322 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:04.075337 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79ee2692-b905-4301-9389-ec07c74cc426-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79ee2692-b905-4301-9389-ec07c74cc426-istiod-ca-cert\") pod \"79ee2692-b905-4301-9389-ec07c74cc426\" (UID: \"79ee2692-b905-4301-9389-ec07c74cc426\") " Oct 30 12:41:04.078322 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:04.075398 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79ee2692-b905-4301-9389-ec07c74cc426-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79ee2692-b905-4301-9389-ec07c74cc426-credential-socket\") pod \"79ee2692-b905-4301-9389-ec07c74cc426\" (UID: \"79ee2692-b905-4301-9389-ec07c74cc426\") " Oct 30 12:41:04.078322 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:04.075439 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79ee2692-b905-4301-9389-ec07c74cc426-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79ee2692-b905-4301-9389-ec07c74cc426-istio-envoy\") pod \"79ee2692-b905-4301-9389-ec07c74cc426\" (UID: \"79ee2692-b905-4301-9389-ec07c74cc426\") " Oct 30 12:41:04.078322 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:04.075486 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79ee2692-b905-4301-9389-ec07c74cc426-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79ee2692-b905-4301-9389-ec07c74cc426-istio-data\") pod \"79ee2692-b905-4301-9389-ec07c74cc426\" (UID: \"79ee2692-b905-4301-9389-ec07c74cc426\") " Oct 30 12:41:04.078322 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:04.075526 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79ee2692-b905-4301-9389-ec07c74cc426-istio-token\" (UniqueName: \"kubernetes.io/projected/79ee2692-b905-4301-9389-ec07c74cc426-istio-token\") pod \"79ee2692-b905-4301-9389-ec07c74cc426\" (UID: \"79ee2692-b905-4301-9389-ec07c74cc426\") " Oct 30 12:41:04.078322 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:04.075565 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79ee2692-b905-4301-9389-ec07c74cc426-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79ee2692-b905-4301-9389-ec07c74cc426-workload-socket\") pod \"79ee2692-b905-4301-9389-ec07c74cc426\" (UID: \"79ee2692-b905-4301-9389-ec07c74cc426\") " Oct 30 12:41:04.078322 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:04.075615 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79ee2692-b905-4301-9389-ec07c74cc426-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79ee2692-b905-4301-9389-ec07c74cc426-workload-certs\") pod \"79ee2692-b905-4301-9389-ec07c74cc426\" (UID: \"79ee2692-b905-4301-9389-ec07c74cc426\") " Oct 30 12:41:04.078322 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:04.075687 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79ee2692-b905-4301-9389-ec07c74cc426-kube-api-access-wkpth\" (UniqueName: \"kubernetes.io/projected/79ee2692-b905-4301-9389-ec07c74cc426-kube-api-access-wkpth\") pod \"79ee2692-b905-4301-9389-ec07c74cc426\" (UID: \"79ee2692-b905-4301-9389-ec07c74cc426\") " Oct 30 12:41:04.078322 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:04.075986 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79ee2692-b905-4301-9389-ec07c74cc426-istio-data" pod "79ee2692-b905-4301-9389-ec07c74cc426" (UID: "79ee2692-b905-4301-9389-ec07c74cc426"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:41:04.078322 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:04.076372 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79ee2692-b905-4301-9389-ec07c74cc426-workload-certs" pod "79ee2692-b905-4301-9389-ec07c74cc426" (UID: "79ee2692-b905-4301-9389-ec07c74cc426"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:41:04.078322 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:04.076699 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79ee2692-b905-4301-9389-ec07c74cc426-istiod-ca-cert" pod "79ee2692-b905-4301-9389-ec07c74cc426" (UID: "79ee2692-b905-4301-9389-ec07c74cc426"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:41:04.079701 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:04.079611 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79ee2692-b905-4301-9389-ec07c74cc426-workload-socket" pod "79ee2692-b905-4301-9389-ec07c74cc426" (UID: "79ee2692-b905-4301-9389-ec07c74cc426"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:41:04.079701 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:04.079676 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79ee2692-b905-4301-9389-ec07c74cc426-credential-socket" pod "79ee2692-b905-4301-9389-ec07c74cc426" (UID: "79ee2692-b905-4301-9389-ec07c74cc426"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:41:04.080359 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:04.080337 4908 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79ee2692-b905-4301-9389-ec07c74cc426-istiod-ca-cert\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:41:04.080450 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:04.080364 4908 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79ee2692-b905-4301-9389-ec07c74cc426-credential-socket\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:41:04.080450 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:04.080377 4908 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79ee2692-b905-4301-9389-ec07c74cc426-istio-data\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:41:04.080450 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:04.080389 4908 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79ee2692-b905-4301-9389-ec07c74cc426-workload-socket\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:41:04.080450 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:04.080400 4908 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79ee2692-b905-4301-9389-ec07c74cc426-workload-certs\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:41:04.081164 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:04.081138 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79ee2692-b905-4301-9389-ec07c74cc426-kube-api-access-wkpth" pod "79ee2692-b905-4301-9389-ec07c74cc426" (UID: "79ee2692-b905-4301-9389-ec07c74cc426"). InnerVolumeSpecName "kube-api-access-wkpth". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:41:04.098868 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:04.098818 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/79ee2692-b905-4301-9389-ec07c74cc426-istio-podinfo" pod "79ee2692-b905-4301-9389-ec07c74cc426" (UID: "79ee2692-b905-4301-9389-ec07c74cc426"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 12:41:04.099013 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:04.098968 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79ee2692-b905-4301-9389-ec07c74cc426-istio-envoy" pod "79ee2692-b905-4301-9389-ec07c74cc426" (UID: "79ee2692-b905-4301-9389-ec07c74cc426"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:41:04.100344 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:04.100316 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79ee2692-b905-4301-9389-ec07c74cc426-istio-token" pod "79ee2692-b905-4301-9389-ec07c74cc426" (UID: "79ee2692-b905-4301-9389-ec07c74cc426"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:41:04.181988 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:04.181915 4908 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79ee2692-b905-4301-9389-ec07c74cc426-istio-podinfo\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:41:04.181988 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:04.181943 4908 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79ee2692-b905-4301-9389-ec07c74cc426-istio-envoy\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:41:04.181988 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:04.181956 4908 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79ee2692-b905-4301-9389-ec07c74cc426-istio-token\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:41:04.181988 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:04.181969 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wkpth\" (UniqueName: \"kubernetes.io/projected/79ee2692-b905-4301-9389-ec07c74cc426-kube-api-access-wkpth\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:41:04.635769 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:04.635737 4908 generic.go:356] "Generic (PLEG): container finished" podID="2d070106-c79c-44d5-a7a9-b766ee82b439" containerID="43eb1ea14c35db3a88694c94fd544c2b1516431a0afe2c55170dceb70cbbd3ba" exitCode=0 Oct 30 12:41:04.636221 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:04.635807 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-dd0628-openshift-default-54686d76fc-x98kl" event={"ID":"2d070106-c79c-44d5-a7a9-b766ee82b439","Type":"ContainerDied","Data":"43eb1ea14c35db3a88694c94fd544c2b1516431a0afe2c55170dceb70cbbd3ba"} Oct 30 12:41:04.636221 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:04.636185 4908 scope.go:122] "RemoveContainer" containerID="43eb1ea14c35db3a88694c94fd544c2b1516431a0afe2c55170dceb70cbbd3ba" Oct 30 12:41:04.637420 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:04.637394 4908 generic.go:356] "Generic (PLEG): container finished" podID="79ee2692-b905-4301-9389-ec07c74cc426" containerID="3639b0a7ab88cc99f973404e1dcbc8c51205dac754ff2a91721109e2e0bec131" exitCode=0 Oct 30 12:41:04.637538 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:04.637433 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bvpc2" Oct 30 12:41:04.637538 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:04.637471 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bvpc2" event={"ID":"79ee2692-b905-4301-9389-ec07c74cc426","Type":"ContainerDied","Data":"3639b0a7ab88cc99f973404e1dcbc8c51205dac754ff2a91721109e2e0bec131"} Oct 30 12:41:04.637538 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:04.637519 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bvpc2" event={"ID":"79ee2692-b905-4301-9389-ec07c74cc426","Type":"ContainerDied","Data":"69ba7d0a2926683cb0f55aa46f1075d701d0a02c41224a1af0d7d4d02838db03"} Oct 30 12:41:04.637706 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:04.637545 4908 scope.go:122] "RemoveContainer" containerID="3639b0a7ab88cc99f973404e1dcbc8c51205dac754ff2a91721109e2e0bec131" Oct 30 12:41:04.638006 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:04.637956 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:41:04.638078 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:04.638033 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:41:04.638134 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:04.638078 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:41:04.656642 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:04.656597 4908 scope.go:122] "RemoveContainer" containerID="3639b0a7ab88cc99f973404e1dcbc8c51205dac754ff2a91721109e2e0bec131" Oct 30 12:41:04.656898 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:41:04.656880 4908 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3639b0a7ab88cc99f973404e1dcbc8c51205dac754ff2a91721109e2e0bec131\": container with ID starting with 3639b0a7ab88cc99f973404e1dcbc8c51205dac754ff2a91721109e2e0bec131 not found: ID does not exist" containerID="3639b0a7ab88cc99f973404e1dcbc8c51205dac754ff2a91721109e2e0bec131" Oct 30 12:41:04.656978 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:04.656912 4908 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3639b0a7ab88cc99f973404e1dcbc8c51205dac754ff2a91721109e2e0bec131"} err="failed to get container status \"3639b0a7ab88cc99f973404e1dcbc8c51205dac754ff2a91721109e2e0bec131\": rpc error: code = NotFound desc = could not find container \"3639b0a7ab88cc99f973404e1dcbc8c51205dac754ff2a91721109e2e0bec131\": container with ID starting with 3639b0a7ab88cc99f973404e1dcbc8c51205dac754ff2a91721109e2e0bec131 not found: ID does not exist" Oct 30 12:41:04.737745 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:04.737713 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bvpc2"] Oct 30 12:41:04.741913 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:04.741852 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bvpc2"] Oct 30 12:41:04.882432 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:04.882401 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5021d04-xfvtm" Oct 30 12:41:04.935893 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:04.933784 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-dd0628-openshift-default-54686d76fc-x98kl" Oct 30 12:41:05.004852 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:05.003894 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a437fd08-0b7b-47c2-a47a-36ce86fc62ff-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a437fd08-0b7b-47c2-a47a-36ce86fc62ff-maas-api-xks-ca\") pod \"a437fd08-0b7b-47c2-a47a-36ce86fc62ff\" (UID: \"a437fd08-0b7b-47c2-a47a-36ce86fc62ff\") " Oct 30 12:41:05.004852 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:05.004081 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a437fd08-0b7b-47c2-a47a-36ce86fc62ff-kube-api-access-9fdkc\" (UniqueName: \"kubernetes.io/projected/a437fd08-0b7b-47c2-a47a-36ce86fc62ff-kube-api-access-9fdkc\") pod \"a437fd08-0b7b-47c2-a47a-36ce86fc62ff\" (UID: \"a437fd08-0b7b-47c2-a47a-36ce86fc62ff\") " Oct 30 12:41:05.004852 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:05.004197 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a437fd08-0b7b-47c2-a47a-36ce86fc62ff-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a437fd08-0b7b-47c2-a47a-36ce86fc62ff-maas-api-openshift-ca\") pod \"a437fd08-0b7b-47c2-a47a-36ce86fc62ff\" (UID: \"a437fd08-0b7b-47c2-a47a-36ce86fc62ff\") " Oct 30 12:41:05.006523 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:05.005523 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a437fd08-0b7b-47c2-a47a-36ce86fc62ff-maas-api-openshift-ca" pod "a437fd08-0b7b-47c2-a47a-36ce86fc62ff" (UID: "a437fd08-0b7b-47c2-a47a-36ce86fc62ff"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:41:05.006523 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:05.006485 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a437fd08-0b7b-47c2-a47a-36ce86fc62ff-maas-api-openshift-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:41:05.009499 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:05.007914 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-2545c7-openshift-default-69cf958465-vb4sg"] Oct 30 12:41:05.009499 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:05.008635 4908 cpu_manager.go:414] "containerMap: removing container" podUID="79ee2692-b905-4301-9389-ec07c74cc426" containerName="istio-proxy" Oct 30 12:41:05.009499 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:05.008652 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79ee2692-b905-4301-9389-ec07c74cc426" containerName="istio-proxy" Oct 30 12:41:05.009499 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:05.008667 4908 cpu_manager.go:414] "containerMap: removing container" podUID="7ce4096e-12a9-4293-9c47-6585f6ead95f" containerName="revoke-keys" Oct 30 12:41:05.009499 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:05.008675 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ce4096e-12a9-4293-9c47-6585f6ead95f" containerName="revoke-keys" Oct 30 12:41:05.009499 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:05.008696 4908 cpu_manager.go:414] "containerMap: removing container" podUID="a437fd08-0b7b-47c2-a47a-36ce86fc62ff" containerName="revoke-keys" Oct 30 12:41:05.009499 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:05.008704 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a437fd08-0b7b-47c2-a47a-36ce86fc62ff" containerName="revoke-keys" Oct 30 12:41:05.009499 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:05.008908 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="79ee2692-b905-4301-9389-ec07c74cc426" containerName="istio-proxy" Oct 30 12:41:05.009499 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:05.008919 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ce4096e-12a9-4293-9c47-6585f6ead95f" containerName="revoke-keys" Oct 30 12:41:05.009499 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:05.008930 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="a437fd08-0b7b-47c2-a47a-36ce86fc62ff" containerName="revoke-keys" Oct 30 12:41:05.009499 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:05.009416 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a437fd08-0b7b-47c2-a47a-36ce86fc62ff-maas-api-xks-ca" pod "a437fd08-0b7b-47c2-a47a-36ce86fc62ff" (UID: "a437fd08-0b7b-47c2-a47a-36ce86fc62ff"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:41:05.019095 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:05.019065 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a437fd08-0b7b-47c2-a47a-36ce86fc62ff-kube-api-access-9fdkc" pod "a437fd08-0b7b-47c2-a47a-36ce86fc62ff" (UID: "a437fd08-0b7b-47c2-a47a-36ce86fc62ff"). InnerVolumeSpecName "kube-api-access-9fdkc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:41:05.020146 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:05.020122 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-2545c7-openshift-default-69cf958465-vb4sg" Oct 30 12:41:05.037757 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:05.037729 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-2545c7-openshift-default-69cf958465-vb4sg"] Oct 30 12:41:05.042483 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:05.042457 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-2545c7-openshift-default-dockercfg-46k4s\"" Oct 30 12:41:05.109611 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:05.109567 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/136fdf42-e3f2-4e4a-83de-a2482e084e5b-istiod-ca-cert\") pod \"e2e-extmod-a-2545c7-openshift-default-69cf958465-vb4sg\" (UID: \"136fdf42-e3f2-4e4a-83de-a2482e084e5b\") " pod="openshift-ingress/e2e-extmod-a-2545c7-openshift-default-69cf958465-vb4sg" Oct 30 12:41:05.109972 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:05.109950 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqftf\" (UniqueName: \"kubernetes.io/projected/136fdf42-e3f2-4e4a-83de-a2482e084e5b-kube-api-access-qqftf\") pod \"e2e-extmod-a-2545c7-openshift-default-69cf958465-vb4sg\" (UID: \"136fdf42-e3f2-4e4a-83de-a2482e084e5b\") " pod="openshift-ingress/e2e-extmod-a-2545c7-openshift-default-69cf958465-vb4sg" Oct 30 12:41:05.110205 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:05.110187 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/136fdf42-e3f2-4e4a-83de-a2482e084e5b-istio-data\") pod \"e2e-extmod-a-2545c7-openshift-default-69cf958465-vb4sg\" (UID: \"136fdf42-e3f2-4e4a-83de-a2482e084e5b\") " pod="openshift-ingress/e2e-extmod-a-2545c7-openshift-default-69cf958465-vb4sg" Oct 30 12:41:05.110432 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:05.110417 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/136fdf42-e3f2-4e4a-83de-a2482e084e5b-istio-podinfo\") pod \"e2e-extmod-a-2545c7-openshift-default-69cf958465-vb4sg\" (UID: \"136fdf42-e3f2-4e4a-83de-a2482e084e5b\") " pod="openshift-ingress/e2e-extmod-a-2545c7-openshift-default-69cf958465-vb4sg" Oct 30 12:41:05.110655 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:05.110637 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/136fdf42-e3f2-4e4a-83de-a2482e084e5b-workload-certs\") pod \"e2e-extmod-a-2545c7-openshift-default-69cf958465-vb4sg\" (UID: \"136fdf42-e3f2-4e4a-83de-a2482e084e5b\") " pod="openshift-ingress/e2e-extmod-a-2545c7-openshift-default-69cf958465-vb4sg" Oct 30 12:41:05.110870 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:05.110855 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/136fdf42-e3f2-4e4a-83de-a2482e084e5b-istio-token\") pod \"e2e-extmod-a-2545c7-openshift-default-69cf958465-vb4sg\" (UID: \"136fdf42-e3f2-4e4a-83de-a2482e084e5b\") " pod="openshift-ingress/e2e-extmod-a-2545c7-openshift-default-69cf958465-vb4sg" Oct 30 12:41:05.111088 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:05.111026 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/136fdf42-e3f2-4e4a-83de-a2482e084e5b-credential-socket\") pod \"e2e-extmod-a-2545c7-openshift-default-69cf958465-vb4sg\" (UID: \"136fdf42-e3f2-4e4a-83de-a2482e084e5b\") " pod="openshift-ingress/e2e-extmod-a-2545c7-openshift-default-69cf958465-vb4sg" Oct 30 12:41:05.111245 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:05.111231 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/136fdf42-e3f2-4e4a-83de-a2482e084e5b-istio-envoy\") pod \"e2e-extmod-a-2545c7-openshift-default-69cf958465-vb4sg\" (UID: \"136fdf42-e3f2-4e4a-83de-a2482e084e5b\") " pod="openshift-ingress/e2e-extmod-a-2545c7-openshift-default-69cf958465-vb4sg" Oct 30 12:41:05.111447 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:05.111433 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/136fdf42-e3f2-4e4a-83de-a2482e084e5b-workload-socket\") pod \"e2e-extmod-a-2545c7-openshift-default-69cf958465-vb4sg\" (UID: \"136fdf42-e3f2-4e4a-83de-a2482e084e5b\") " pod="openshift-ingress/e2e-extmod-a-2545c7-openshift-default-69cf958465-vb4sg" Oct 30 12:41:05.111661 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:05.111647 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9fdkc\" (UniqueName: \"kubernetes.io/projected/a437fd08-0b7b-47c2-a47a-36ce86fc62ff-kube-api-access-9fdkc\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:41:05.111785 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:05.111774 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a437fd08-0b7b-47c2-a47a-36ce86fc62ff-maas-api-xks-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:41:05.215664 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:05.215574 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/136fdf42-e3f2-4e4a-83de-a2482e084e5b-istio-token\") pod \"e2e-extmod-a-2545c7-openshift-default-69cf958465-vb4sg\" (UID: \"136fdf42-e3f2-4e4a-83de-a2482e084e5b\") " pod="openshift-ingress/e2e-extmod-a-2545c7-openshift-default-69cf958465-vb4sg" Oct 30 12:41:05.215664 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:05.215634 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/136fdf42-e3f2-4e4a-83de-a2482e084e5b-credential-socket\") pod \"e2e-extmod-a-2545c7-openshift-default-69cf958465-vb4sg\" (UID: \"136fdf42-e3f2-4e4a-83de-a2482e084e5b\") " pod="openshift-ingress/e2e-extmod-a-2545c7-openshift-default-69cf958465-vb4sg" Oct 30 12:41:05.216112 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:05.215681 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/136fdf42-e3f2-4e4a-83de-a2482e084e5b-istio-envoy\") pod \"e2e-extmod-a-2545c7-openshift-default-69cf958465-vb4sg\" (UID: \"136fdf42-e3f2-4e4a-83de-a2482e084e5b\") " pod="openshift-ingress/e2e-extmod-a-2545c7-openshift-default-69cf958465-vb4sg" Oct 30 12:41:05.216112 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:05.215741 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/136fdf42-e3f2-4e4a-83de-a2482e084e5b-workload-socket\") pod \"e2e-extmod-a-2545c7-openshift-default-69cf958465-vb4sg\" (UID: \"136fdf42-e3f2-4e4a-83de-a2482e084e5b\") " pod="openshift-ingress/e2e-extmod-a-2545c7-openshift-default-69cf958465-vb4sg" Oct 30 12:41:05.216112 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:05.215794 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/136fdf42-e3f2-4e4a-83de-a2482e084e5b-istiod-ca-cert\") pod \"e2e-extmod-a-2545c7-openshift-default-69cf958465-vb4sg\" (UID: \"136fdf42-e3f2-4e4a-83de-a2482e084e5b\") " pod="openshift-ingress/e2e-extmod-a-2545c7-openshift-default-69cf958465-vb4sg" Oct 30 12:41:05.216112 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:05.215937 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqftf\" (UniqueName: \"kubernetes.io/projected/136fdf42-e3f2-4e4a-83de-a2482e084e5b-kube-api-access-qqftf\") pod \"e2e-extmod-a-2545c7-openshift-default-69cf958465-vb4sg\" (UID: \"136fdf42-e3f2-4e4a-83de-a2482e084e5b\") " pod="openshift-ingress/e2e-extmod-a-2545c7-openshift-default-69cf958465-vb4sg" Oct 30 12:41:05.216112 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:05.215967 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/136fdf42-e3f2-4e4a-83de-a2482e084e5b-istio-data\") pod \"e2e-extmod-a-2545c7-openshift-default-69cf958465-vb4sg\" (UID: \"136fdf42-e3f2-4e4a-83de-a2482e084e5b\") " pod="openshift-ingress/e2e-extmod-a-2545c7-openshift-default-69cf958465-vb4sg" Oct 30 12:41:05.216112 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:05.216058 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/136fdf42-e3f2-4e4a-83de-a2482e084e5b-istio-podinfo\") pod \"e2e-extmod-a-2545c7-openshift-default-69cf958465-vb4sg\" (UID: \"136fdf42-e3f2-4e4a-83de-a2482e084e5b\") " pod="openshift-ingress/e2e-extmod-a-2545c7-openshift-default-69cf958465-vb4sg" Oct 30 12:41:05.216440 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:05.216118 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/136fdf42-e3f2-4e4a-83de-a2482e084e5b-workload-certs\") pod \"e2e-extmod-a-2545c7-openshift-default-69cf958465-vb4sg\" (UID: \"136fdf42-e3f2-4e4a-83de-a2482e084e5b\") " pod="openshift-ingress/e2e-extmod-a-2545c7-openshift-default-69cf958465-vb4sg" Oct 30 12:41:05.216561 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:05.216536 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/136fdf42-e3f2-4e4a-83de-a2482e084e5b-workload-certs\") pod \"e2e-extmod-a-2545c7-openshift-default-69cf958465-vb4sg\" (UID: \"136fdf42-e3f2-4e4a-83de-a2482e084e5b\") " pod="openshift-ingress/e2e-extmod-a-2545c7-openshift-default-69cf958465-vb4sg" Oct 30 12:41:05.222302 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:05.217099 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/136fdf42-e3f2-4e4a-83de-a2482e084e5b-workload-socket\") pod \"e2e-extmod-a-2545c7-openshift-default-69cf958465-vb4sg\" (UID: \"136fdf42-e3f2-4e4a-83de-a2482e084e5b\") " pod="openshift-ingress/e2e-extmod-a-2545c7-openshift-default-69cf958465-vb4sg" Oct 30 12:41:05.222580 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:05.217588 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/136fdf42-e3f2-4e4a-83de-a2482e084e5b-credential-socket\") pod \"e2e-extmod-a-2545c7-openshift-default-69cf958465-vb4sg\" (UID: \"136fdf42-e3f2-4e4a-83de-a2482e084e5b\") " pod="openshift-ingress/e2e-extmod-a-2545c7-openshift-default-69cf958465-vb4sg" Oct 30 12:41:05.225429 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:05.217916 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/136fdf42-e3f2-4e4a-83de-a2482e084e5b-istiod-ca-cert\") pod \"e2e-extmod-a-2545c7-openshift-default-69cf958465-vb4sg\" (UID: \"136fdf42-e3f2-4e4a-83de-a2482e084e5b\") " pod="openshift-ingress/e2e-extmod-a-2545c7-openshift-default-69cf958465-vb4sg" Oct 30 12:41:05.225596 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:05.218269 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/136fdf42-e3f2-4e4a-83de-a2482e084e5b-istio-data\") pod \"e2e-extmod-a-2545c7-openshift-default-69cf958465-vb4sg\" (UID: \"136fdf42-e3f2-4e4a-83de-a2482e084e5b\") " pod="openshift-ingress/e2e-extmod-a-2545c7-openshift-default-69cf958465-vb4sg" Oct 30 12:41:05.233873 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:05.231193 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/136fdf42-e3f2-4e4a-83de-a2482e084e5b-istio-podinfo\") pod \"e2e-extmod-a-2545c7-openshift-default-69cf958465-vb4sg\" (UID: \"136fdf42-e3f2-4e4a-83de-a2482e084e5b\") " pod="openshift-ingress/e2e-extmod-a-2545c7-openshift-default-69cf958465-vb4sg" Oct 30 12:41:05.270870 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:05.268117 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/136fdf42-e3f2-4e4a-83de-a2482e084e5b-istio-envoy\") pod \"e2e-extmod-a-2545c7-openshift-default-69cf958465-vb4sg\" (UID: \"136fdf42-e3f2-4e4a-83de-a2482e084e5b\") " pod="openshift-ingress/e2e-extmod-a-2545c7-openshift-default-69cf958465-vb4sg" Oct 30 12:41:05.270870 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:05.268901 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqftf\" (UniqueName: \"kubernetes.io/projected/136fdf42-e3f2-4e4a-83de-a2482e084e5b-kube-api-access-qqftf\") pod \"e2e-extmod-a-2545c7-openshift-default-69cf958465-vb4sg\" (UID: \"136fdf42-e3f2-4e4a-83de-a2482e084e5b\") " pod="openshift-ingress/e2e-extmod-a-2545c7-openshift-default-69cf958465-vb4sg" Oct 30 12:41:05.274848 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:05.272248 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/136fdf42-e3f2-4e4a-83de-a2482e084e5b-istio-token\") pod \"e2e-extmod-a-2545c7-openshift-default-69cf958465-vb4sg\" (UID: \"136fdf42-e3f2-4e4a-83de-a2482e084e5b\") " pod="openshift-ingress/e2e-extmod-a-2545c7-openshift-default-69cf958465-vb4sg" Oct 30 12:41:05.373851 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:05.369377 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-2545c7-openshift-default-69cf958465-vb4sg" Oct 30 12:41:05.853360 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:05.853317 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79ee2692-b905-4301-9389-ec07c74cc426" path="/var/lib/kubelet/pods/79ee2692-b905-4301-9389-ec07c74cc426/volumes" Oct 30 12:41:06.034859 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:06.028048 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5021d04-xfvtm" Oct 30 12:41:06.034859 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:06.027932 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5021d04-xfvtm" event={"ID":"a437fd08-0b7b-47c2-a47a-36ce86fc62ff","Type":"ContainerDied","Data":"ddc7e73d09b6ad3d17d5e242db97e1e669665bffe6c79a4af9283cfddafc2eca"} Oct 30 12:41:06.034859 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:06.028382 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ddc7e73d09b6ad3d17d5e242db97e1e669665bffe6c79a4af9283cfddafc2eca" Oct 30 12:41:06.047101 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:06.047066 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-dd0628-openshift-default-54686d76fc-x98kl" event={"ID":"2d070106-c79c-44d5-a7a9-b766ee82b439","Type":"ContainerStarted","Data":"5c86db12fa0656d65f0487fd1defd3ebacc6d436d4eab07243d282cfcbb509b4"} Oct 30 12:41:06.107084 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:06.107035 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-2545c7-openshift-default-69cf958465-vb4sg"] Oct 30 12:41:06.207870 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:41:06.199620 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod136fdf42_e3f2_4e4a_83de_a2482e084e5b.slice/crio-7c91310107e7843b1ad04b84f339e4e434ae7e717b36d675f064b0a218f8705b WatchSource:0}: Error finding container 7c91310107e7843b1ad04b84f339e4e434ae7e717b36d675f064b0a218f8705b: Status 404 returned error can't find the container with id 7c91310107e7843b1ad04b84f339e4e434ae7e717b36d675f064b0a218f8705b Oct 30 12:41:06.207870 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:06.207313 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:41:06.207870 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:06.207398 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:41:06.207870 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:06.207441 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:41:06.616171 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:06.615318 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-7z8r7"] Oct 30 12:41:06.616171 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:06.615631 4908 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-7z8r7" podUID="6b875062-a976-42cd-8b43-85dc197f3f4c" containerName="istio-proxy" containerID="cri-o://b6b5da560cca29c612e9861dea20b806ee92c440fc8f39dc6f5ed876e79024e2" gracePeriod=30 Oct 30 12:41:07.015243 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:07.015204 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-dd0628-openshift-default-54686d76fc-x98kl" Oct 30 12:41:07.016372 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:07.016344 4908 patch_prober.go:29] interesting pod/e2e-shared-b-w0-dd0628-openshift-default-54686d76fc-x98kl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" start-of-body= Oct 30 12:41:07.016514 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:07.016401 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-dd0628-openshift-default-54686d76fc-x98kl" podUID="2d070106-c79c-44d5-a7a9-b766ee82b439" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" Oct 30 12:41:07.078802 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:07.078702 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-2545c7-openshift-default-69cf958465-vb4sg" event={"ID":"136fdf42-e3f2-4e4a-83de-a2482e084e5b","Type":"ContainerStarted","Data":"0885278d558403846e921e2da33c6a94cc6ac2b4ae2d0710882eaa03c1a72b2c"} Oct 30 12:41:07.078802 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:07.078763 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-2545c7-openshift-default-69cf958465-vb4sg" event={"ID":"136fdf42-e3f2-4e4a-83de-a2482e084e5b","Type":"ContainerStarted","Data":"7c91310107e7843b1ad04b84f339e4e434ae7e717b36d675f064b0a218f8705b"} Oct 30 12:41:07.130421 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:07.130372 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-2545c7-openshift-default-69cf958465-vb4sg" podStartSLOduration=3.130359817 podStartE2EDuration="3.130359817s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:41:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:41:07.126921324 +0000 UTC m=+2823.641350068" watchObservedRunningTime="2026-10-30 12:41:07.130359817 +0000 UTC m=+2823.644788561" Oct 30 12:41:07.155601 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:07.155566 4908 scope.go:122] "RemoveContainer" containerID="59e678f3dd2c69efaf7de237c0915a73554ee8f8dce52c612aa1f025b424326b" Oct 30 12:41:07.210400 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:07.210015 4908 scope.go:122] "RemoveContainer" containerID="70a9909ffcbfff07b707e392236fb1c69b410da8c98ee9b99b74499e83b908da" Oct 30 12:41:07.223218 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:07.223187 4908 scope.go:122] "RemoveContainer" containerID="da1e8c404a3b9dad86ffc2d42241694e2d9447ed9a58343cab19b55e1c918b34" Oct 30 12:41:07.232166 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:07.232149 4908 scope.go:122] "RemoveContainer" containerID="6f92403d6f4fd8df4b95daf93fcbbdef2b920c12c4fd17cd088688ab45f1ae31" Oct 30 12:41:07.245535 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:07.245178 4908 scope.go:122] "RemoveContainer" containerID="d7453a256dc0d4effd974255b973caf1cdcf397d33f3841cc20cecc576f9d8f4" Oct 30 12:41:07.333850 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:07.331007 4908 scope.go:122] "RemoveContainer" containerID="83e1584662a5ce905bf88e8c81ec8c3490697b4bf544fde13f9b1c2c697329e7" Oct 30 12:41:07.358061 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:07.358040 4908 scope.go:122] "RemoveContainer" containerID="8e78d324ec999e4eab99acf45aa7f25e8679f7ba42377aac33ffff04ca120681" Oct 30 12:41:07.386849 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:07.382385 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-2545c7-openshift-default-69cf958465-vb4sg" Oct 30 12:41:07.407941 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:07.394226 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-2545c7-openshift-default-69cf958465-vb4sg" Oct 30 12:41:08.015360 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:08.015328 4908 patch_prober.go:29] interesting pod/e2e-shared-b-w0-dd0628-openshift-default-54686d76fc-x98kl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" start-of-body= Oct 30 12:41:08.015883 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:08.015388 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-dd0628-openshift-default-54686d76fc-x98kl" podUID="2d070106-c79c-44d5-a7a9-b766ee82b439" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" Oct 30 12:41:08.095854 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:08.094208 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-2545c7-openshift-default-69cf958465-vb4sg" Oct 30 12:41:08.105851 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:08.103039 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-2545c7-openshift-default-69cf958465-vb4sg" Oct 30 12:41:08.320898 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:08.317928 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f84120f4-vm445"] Oct 30 12:41:08.354953 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:08.349895 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f84120f4-vm445"] Oct 30 12:41:08.354953 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:08.350011 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f84120f4-vm445" Oct 30 12:41:08.376849 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:08.365658 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 12:41:08.376849 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:08.365776 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s75hq\"" Oct 30 12:41:08.376849 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:08.365922 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 12:41:08.376849 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:08.366009 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 12:41:08.442458 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:08.442423 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37837576-7f1a-4341-b1b3-71a4e74afa81-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f84120f4-vm445\" (UID: \"37837576-7f1a-4341-b1b3-71a4e74afa81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f84120f4-vm445" Oct 30 12:41:08.442658 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:08.442466 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37837576-7f1a-4341-b1b3-71a4e74afa81-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f84120f4-vm445\" (UID: \"37837576-7f1a-4341-b1b3-71a4e74afa81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f84120f4-vm445" Oct 30 12:41:08.442658 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:08.442504 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qq6b\" (UniqueName: \"kubernetes.io/projected/37837576-7f1a-4341-b1b3-71a4e74afa81-kube-api-access-8qq6b\") pod \"maas-api-revoke-sub-f84120f4-vm445\" (UID: \"37837576-7f1a-4341-b1b3-71a4e74afa81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f84120f4-vm445" Oct 30 12:41:08.543436 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:08.543393 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37837576-7f1a-4341-b1b3-71a4e74afa81-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f84120f4-vm445\" (UID: \"37837576-7f1a-4341-b1b3-71a4e74afa81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f84120f4-vm445" Oct 30 12:41:08.543436 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:08.543433 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37837576-7f1a-4341-b1b3-71a4e74afa81-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f84120f4-vm445\" (UID: \"37837576-7f1a-4341-b1b3-71a4e74afa81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f84120f4-vm445" Oct 30 12:41:08.543693 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:08.543481 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qq6b\" (UniqueName: \"kubernetes.io/projected/37837576-7f1a-4341-b1b3-71a4e74afa81-kube-api-access-8qq6b\") pod \"maas-api-revoke-sub-f84120f4-vm445\" (UID: \"37837576-7f1a-4341-b1b3-71a4e74afa81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f84120f4-vm445" Oct 30 12:41:08.544348 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:08.544317 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37837576-7f1a-4341-b1b3-71a4e74afa81-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f84120f4-vm445\" (UID: \"37837576-7f1a-4341-b1b3-71a4e74afa81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f84120f4-vm445" Oct 30 12:41:08.545865 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:08.545821 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37837576-7f1a-4341-b1b3-71a4e74afa81-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f84120f4-vm445\" (UID: \"37837576-7f1a-4341-b1b3-71a4e74afa81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f84120f4-vm445" Oct 30 12:41:08.600901 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:08.591881 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qq6b\" (UniqueName: \"kubernetes.io/projected/37837576-7f1a-4341-b1b3-71a4e74afa81-kube-api-access-8qq6b\") pod \"maas-api-revoke-sub-f84120f4-vm445\" (UID: \"37837576-7f1a-4341-b1b3-71a4e74afa81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f84120f4-vm445" Oct 30 12:41:08.711180 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:08.711143 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f84120f4-vm445" Oct 30 12:41:09.015861 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:09.015308 4908 patch_prober.go:29] interesting pod/e2e-shared-b-w0-dd0628-openshift-default-54686d76fc-x98kl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" start-of-body= Oct 30 12:41:09.015861 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:09.015361 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-dd0628-openshift-default-54686d76fc-x98kl" podUID="2d070106-c79c-44d5-a7a9-b766ee82b439" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" Oct 30 12:41:09.088573 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:09.076888 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f84120f4-vm445"] Oct 30 12:41:09.143719 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:41:09.143685 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37837576_7f1a_4341_b1b3_71a4e74afa81.slice/crio-b7202ed834f29123ad16890871b63f5e0ab145417c400ca4e997071ab95f8711 WatchSource:0}: Error finding container b7202ed834f29123ad16890871b63f5e0ab145417c400ca4e997071ab95f8711: Status 404 returned error can't find the container with id b7202ed834f29123ad16890871b63f5e0ab145417c400ca4e997071ab95f8711 Oct 30 12:41:10.015346 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:10.015308 4908 patch_prober.go:29] interesting pod/e2e-shared-b-w0-dd0628-openshift-default-54686d76fc-x98kl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" start-of-body= Oct 30 12:41:10.015538 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:10.015363 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-dd0628-openshift-default-54686d76fc-x98kl" podUID="2d070106-c79c-44d5-a7a9-b766ee82b439" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" Oct 30 12:41:10.191169 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:10.191132 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f84120f4-vm445" event={"ID":"37837576-7f1a-4341-b1b3-71a4e74afa81","Type":"ContainerStarted","Data":"f4ba6a1670388f2dd2358a126cf01869973cf3a763392ac738e89ea6b95d7d6d"} Oct 30 12:41:10.191554 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:10.191195 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f84120f4-vm445" event={"ID":"37837576-7f1a-4341-b1b3-71a4e74afa81","Type":"ContainerStarted","Data":"b7202ed834f29123ad16890871b63f5e0ab145417c400ca4e997071ab95f8711"} Oct 30 12:41:10.231937 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:10.231881 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f84120f4-vm445" podStartSLOduration=2.231866485 podStartE2EDuration="2.231866485s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:41:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:41:10.228414081 +0000 UTC m=+2826.742842827" watchObservedRunningTime="2026-10-30 12:41:10.231866485 +0000 UTC m=+2826.746295224" Oct 30 12:41:10.818195 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:10.818152 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-nntkd"] Oct 30 12:41:10.823036 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:10.823016 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-nntkd" Oct 30 12:41:10.839206 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:10.839185 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-dbe123-openshift-default-dockercfg-xkks6\"" Oct 30 12:41:10.858772 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:10.858740 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-nntkd"] Oct 30 12:41:11.018549 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:11.016529 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6810256-6b25-44b9-a7f1-fb0cb023a5b9-credential-socket\") pod \"e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-nntkd\" (UID: \"e6810256-6b25-44b9-a7f1-fb0cb023a5b9\") " pod="openshift-ingress/e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-nntkd" Oct 30 12:41:11.018549 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:11.016627 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6810256-6b25-44b9-a7f1-fb0cb023a5b9-istio-token\") pod \"e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-nntkd\" (UID: \"e6810256-6b25-44b9-a7f1-fb0cb023a5b9\") " pod="openshift-ingress/e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-nntkd" Oct 30 12:41:11.018549 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:11.016661 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6810256-6b25-44b9-a7f1-fb0cb023a5b9-istio-podinfo\") pod \"e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-nntkd\" (UID: \"e6810256-6b25-44b9-a7f1-fb0cb023a5b9\") " pod="openshift-ingress/e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-nntkd" Oct 30 12:41:11.018549 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:11.016773 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6810256-6b25-44b9-a7f1-fb0cb023a5b9-istiod-ca-cert\") pod \"e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-nntkd\" (UID: \"e6810256-6b25-44b9-a7f1-fb0cb023a5b9\") " pod="openshift-ingress/e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-nntkd" Oct 30 12:41:11.018549 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:11.016816 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6810256-6b25-44b9-a7f1-fb0cb023a5b9-istio-envoy\") pod \"e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-nntkd\" (UID: \"e6810256-6b25-44b9-a7f1-fb0cb023a5b9\") " pod="openshift-ingress/e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-nntkd" Oct 30 12:41:11.018549 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:11.016878 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6810256-6b25-44b9-a7f1-fb0cb023a5b9-istio-data\") pod \"e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-nntkd\" (UID: \"e6810256-6b25-44b9-a7f1-fb0cb023a5b9\") " pod="openshift-ingress/e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-nntkd" Oct 30 12:41:11.018549 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:11.016914 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6810256-6b25-44b9-a7f1-fb0cb023a5b9-workload-socket\") pod \"e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-nntkd\" (UID: \"e6810256-6b25-44b9-a7f1-fb0cb023a5b9\") " pod="openshift-ingress/e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-nntkd" Oct 30 12:41:11.018549 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:11.016958 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6810256-6b25-44b9-a7f1-fb0cb023a5b9-workload-certs\") pod \"e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-nntkd\" (UID: \"e6810256-6b25-44b9-a7f1-fb0cb023a5b9\") " pod="openshift-ingress/e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-nntkd" Oct 30 12:41:11.018549 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:11.017014 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wv2v6\" (UniqueName: \"kubernetes.io/projected/e6810256-6b25-44b9-a7f1-fb0cb023a5b9-kube-api-access-wv2v6\") pod \"e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-nntkd\" (UID: \"e6810256-6b25-44b9-a7f1-fb0cb023a5b9\") " pod="openshift-ingress/e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-nntkd" Oct 30 12:41:11.042146 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:11.042111 4908 patch_prober.go:29] interesting pod/e2e-shared-b-w0-dd0628-openshift-default-54686d76fc-x98kl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" start-of-body= Oct 30 12:41:11.042553 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:11.042524 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-dd0628-openshift-default-54686d76fc-x98kl" podUID="2d070106-c79c-44d5-a7a9-b766ee82b439" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" Oct 30 12:41:11.118957 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:11.118866 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6810256-6b25-44b9-a7f1-fb0cb023a5b9-istio-data\") pod \"e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-nntkd\" (UID: \"e6810256-6b25-44b9-a7f1-fb0cb023a5b9\") " pod="openshift-ingress/e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-nntkd" Oct 30 12:41:11.119245 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:11.119218 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6810256-6b25-44b9-a7f1-fb0cb023a5b9-workload-socket\") pod \"e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-nntkd\" (UID: \"e6810256-6b25-44b9-a7f1-fb0cb023a5b9\") " pod="openshift-ingress/e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-nntkd" Oct 30 12:41:11.119717 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:11.119696 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6810256-6b25-44b9-a7f1-fb0cb023a5b9-workload-certs\") pod \"e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-nntkd\" (UID: \"e6810256-6b25-44b9-a7f1-fb0cb023a5b9\") " pod="openshift-ingress/e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-nntkd" Oct 30 12:41:11.120128 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:11.120112 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wv2v6\" (UniqueName: \"kubernetes.io/projected/e6810256-6b25-44b9-a7f1-fb0cb023a5b9-kube-api-access-wv2v6\") pod \"e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-nntkd\" (UID: \"e6810256-6b25-44b9-a7f1-fb0cb023a5b9\") " pod="openshift-ingress/e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-nntkd" Oct 30 12:41:11.120260 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:11.120247 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6810256-6b25-44b9-a7f1-fb0cb023a5b9-credential-socket\") pod \"e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-nntkd\" (UID: \"e6810256-6b25-44b9-a7f1-fb0cb023a5b9\") " pod="openshift-ingress/e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-nntkd" Oct 30 12:41:11.120730 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:11.120698 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6810256-6b25-44b9-a7f1-fb0cb023a5b9-istio-token\") pod \"e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-nntkd\" (UID: \"e6810256-6b25-44b9-a7f1-fb0cb023a5b9\") " pod="openshift-ingress/e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-nntkd" Oct 30 12:41:11.120903 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:11.120889 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6810256-6b25-44b9-a7f1-fb0cb023a5b9-istio-podinfo\") pod \"e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-nntkd\" (UID: \"e6810256-6b25-44b9-a7f1-fb0cb023a5b9\") " pod="openshift-ingress/e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-nntkd" Oct 30 12:41:11.121523 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:11.121504 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6810256-6b25-44b9-a7f1-fb0cb023a5b9-istiod-ca-cert\") pod \"e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-nntkd\" (UID: \"e6810256-6b25-44b9-a7f1-fb0cb023a5b9\") " pod="openshift-ingress/e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-nntkd" Oct 30 12:41:11.121691 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:11.121677 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6810256-6b25-44b9-a7f1-fb0cb023a5b9-istio-envoy\") pod \"e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-nntkd\" (UID: \"e6810256-6b25-44b9-a7f1-fb0cb023a5b9\") " pod="openshift-ingress/e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-nntkd" Oct 30 12:41:11.122202 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:11.119497 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6810256-6b25-44b9-a7f1-fb0cb023a5b9-istio-data\") pod \"e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-nntkd\" (UID: \"e6810256-6b25-44b9-a7f1-fb0cb023a5b9\") " pod="openshift-ingress/e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-nntkd" Oct 30 12:41:11.122373 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:11.119631 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6810256-6b25-44b9-a7f1-fb0cb023a5b9-workload-socket\") pod \"e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-nntkd\" (UID: \"e6810256-6b25-44b9-a7f1-fb0cb023a5b9\") " pod="openshift-ingress/e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-nntkd" Oct 30 12:41:11.122482 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:11.120044 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6810256-6b25-44b9-a7f1-fb0cb023a5b9-workload-certs\") pod \"e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-nntkd\" (UID: \"e6810256-6b25-44b9-a7f1-fb0cb023a5b9\") " pod="openshift-ingress/e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-nntkd" Oct 30 12:41:11.122574 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:11.120743 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6810256-6b25-44b9-a7f1-fb0cb023a5b9-credential-socket\") pod \"e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-nntkd\" (UID: \"e6810256-6b25-44b9-a7f1-fb0cb023a5b9\") " pod="openshift-ingress/e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-nntkd" Oct 30 12:41:11.122656 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:11.122288 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6810256-6b25-44b9-a7f1-fb0cb023a5b9-istiod-ca-cert\") pod \"e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-nntkd\" (UID: \"e6810256-6b25-44b9-a7f1-fb0cb023a5b9\") " pod="openshift-ingress/e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-nntkd" Oct 30 12:41:11.124794 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:11.124753 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6810256-6b25-44b9-a7f1-fb0cb023a5b9-istio-envoy\") pod \"e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-nntkd\" (UID: \"e6810256-6b25-44b9-a7f1-fb0cb023a5b9\") " pod="openshift-ingress/e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-nntkd" Oct 30 12:41:11.133354 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:11.133334 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6810256-6b25-44b9-a7f1-fb0cb023a5b9-istio-podinfo\") pod \"e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-nntkd\" (UID: \"e6810256-6b25-44b9-a7f1-fb0cb023a5b9\") " pod="openshift-ingress/e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-nntkd" Oct 30 12:41:11.146869 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:11.143589 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wv2v6\" (UniqueName: \"kubernetes.io/projected/e6810256-6b25-44b9-a7f1-fb0cb023a5b9-kube-api-access-wv2v6\") pod \"e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-nntkd\" (UID: \"e6810256-6b25-44b9-a7f1-fb0cb023a5b9\") " pod="openshift-ingress/e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-nntkd" Oct 30 12:41:11.146869 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:11.145259 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6810256-6b25-44b9-a7f1-fb0cb023a5b9-istio-token\") pod \"e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-nntkd\" (UID: \"e6810256-6b25-44b9-a7f1-fb0cb023a5b9\") " pod="openshift-ingress/e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-nntkd" Oct 30 12:41:11.200894 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:11.200479 4908 generic.go:356] "Generic (PLEG): container finished" podID="37837576-7f1a-4341-b1b3-71a4e74afa81" containerID="f4ba6a1670388f2dd2358a126cf01869973cf3a763392ac738e89ea6b95d7d6d" exitCode=0 Oct 30 12:41:11.200894 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:11.200572 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f84120f4-vm445" event={"ID":"37837576-7f1a-4341-b1b3-71a4e74afa81","Type":"ContainerDied","Data":"f4ba6a1670388f2dd2358a126cf01869973cf3a763392ac738e89ea6b95d7d6d"} Oct 30 12:41:11.436229 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:11.436138 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-nntkd" Oct 30 12:41:11.784227 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:11.784201 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-nntkd"] Oct 30 12:41:11.785491 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:41:11.785463 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6810256_6b25_44b9_a7f1_fb0cb023a5b9.slice/crio-bdaf6dec313f3748acecbf10ec608f8dc6ee578209416d5bfea52900d0a6aa46 WatchSource:0}: Error finding container bdaf6dec313f3748acecbf10ec608f8dc6ee578209416d5bfea52900d0a6aa46: Status 404 returned error can't find the container with id bdaf6dec313f3748acecbf10ec608f8dc6ee578209416d5bfea52900d0a6aa46 Oct 30 12:41:11.788298 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:11.788263 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:41:11.788403 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:11.788328 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:41:11.788403 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:11.788355 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:41:11.882325 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:11.882295 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-751c30fd-lrtv2"] Oct 30 12:41:11.886762 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:11.886602 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-751c30fd-lrtv2" Oct 30 12:41:11.909508 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:11.909481 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-751c30fd-lrtv2"] Oct 30 12:41:11.913992 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:11.913970 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-7z8r7" Oct 30 12:41:12.024625 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:12.024523 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-dd0628-openshift-default-54686d76fc-x98kl" Oct 30 12:41:12.025107 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:12.025084 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-dd0628-openshift-default-54686d76fc-x98kl" Oct 30 12:41:12.026098 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:12.026077 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-dd0628-openshift-default-54686d76fc-x98kl" Oct 30 12:41:12.049316 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:12.049290 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b875062-a976-42cd-8b43-85dc197f3f4c-istio-token\" (UniqueName: \"kubernetes.io/projected/6b875062-a976-42cd-8b43-85dc197f3f4c-istio-token\") pod \"6b875062-a976-42cd-8b43-85dc197f3f4c\" (UID: \"6b875062-a976-42cd-8b43-85dc197f3f4c\") " Oct 30 12:41:12.049422 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:12.049405 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6b875062-a976-42cd-8b43-85dc197f3f4c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b875062-a976-42cd-8b43-85dc197f3f4c-istio-podinfo\") pod \"6b875062-a976-42cd-8b43-85dc197f3f4c\" (UID: \"6b875062-a976-42cd-8b43-85dc197f3f4c\") " Oct 30 12:41:12.049478 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:12.049454 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b875062-a976-42cd-8b43-85dc197f3f4c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b875062-a976-42cd-8b43-85dc197f3f4c-istio-envoy\") pod \"6b875062-a976-42cd-8b43-85dc197f3f4c\" (UID: \"6b875062-a976-42cd-8b43-85dc197f3f4c\") " Oct 30 12:41:12.049523 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:12.049493 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b875062-a976-42cd-8b43-85dc197f3f4c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b875062-a976-42cd-8b43-85dc197f3f4c-istio-data\") pod \"6b875062-a976-42cd-8b43-85dc197f3f4c\" (UID: \"6b875062-a976-42cd-8b43-85dc197f3f4c\") " Oct 30 12:41:12.049571 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:12.049559 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b875062-a976-42cd-8b43-85dc197f3f4c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b875062-a976-42cd-8b43-85dc197f3f4c-workload-certs\") pod \"6b875062-a976-42cd-8b43-85dc197f3f4c\" (UID: \"6b875062-a976-42cd-8b43-85dc197f3f4c\") " Oct 30 12:41:12.049620 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:12.049595 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b875062-a976-42cd-8b43-85dc197f3f4c-kube-api-access-sv9rt\" (UniqueName: \"kubernetes.io/projected/6b875062-a976-42cd-8b43-85dc197f3f4c-kube-api-access-sv9rt\") pod \"6b875062-a976-42cd-8b43-85dc197f3f4c\" (UID: \"6b875062-a976-42cd-8b43-85dc197f3f4c\") " Oct 30 12:41:12.049675 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:12.049637 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b875062-a976-42cd-8b43-85dc197f3f4c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b875062-a976-42cd-8b43-85dc197f3f4c-workload-socket\") pod \"6b875062-a976-42cd-8b43-85dc197f3f4c\" (UID: \"6b875062-a976-42cd-8b43-85dc197f3f4c\") " Oct 30 12:41:12.049728 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:12.049684 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b875062-a976-42cd-8b43-85dc197f3f4c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b875062-a976-42cd-8b43-85dc197f3f4c-credential-socket\") pod \"6b875062-a976-42cd-8b43-85dc197f3f4c\" (UID: \"6b875062-a976-42cd-8b43-85dc197f3f4c\") " Oct 30 12:41:12.049784 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:12.049766 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b875062-a976-42cd-8b43-85dc197f3f4c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b875062-a976-42cd-8b43-85dc197f3f4c-istiod-ca-cert\") pod \"6b875062-a976-42cd-8b43-85dc197f3f4c\" (UID: \"6b875062-a976-42cd-8b43-85dc197f3f4c\") " Oct 30 12:41:12.050124 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:12.050088 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b875062-a976-42cd-8b43-85dc197f3f4c-workload-certs" pod "6b875062-a976-42cd-8b43-85dc197f3f4c" (UID: "6b875062-a976-42cd-8b43-85dc197f3f4c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:41:12.050223 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:12.050132 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5dhl\" (UniqueName: \"kubernetes.io/projected/98e23727-b5b0-4c90-844a-d732b3e001de-kube-api-access-s5dhl\") pod \"maas-api-revoke-sub-751c30fd-lrtv2\" (UID: \"98e23727-b5b0-4c90-844a-d732b3e001de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-751c30fd-lrtv2" Oct 30 12:41:12.050223 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:12.050210 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b875062-a976-42cd-8b43-85dc197f3f4c-workload-socket" pod "6b875062-a976-42cd-8b43-85dc197f3f4c" (UID: "6b875062-a976-42cd-8b43-85dc197f3f4c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:41:12.050342 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:12.050220 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98e23727-b5b0-4c90-844a-d732b3e001de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-751c30fd-lrtv2\" (UID: \"98e23727-b5b0-4c90-844a-d732b3e001de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-751c30fd-lrtv2" Oct 30 12:41:12.050342 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:12.050329 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98e23727-b5b0-4c90-844a-d732b3e001de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-751c30fd-lrtv2\" (UID: \"98e23727-b5b0-4c90-844a-d732b3e001de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-751c30fd-lrtv2" Oct 30 12:41:12.050456 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:12.050330 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b875062-a976-42cd-8b43-85dc197f3f4c-credential-socket" pod "6b875062-a976-42cd-8b43-85dc197f3f4c" (UID: "6b875062-a976-42cd-8b43-85dc197f3f4c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:41:12.050456 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:12.050412 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b875062-a976-42cd-8b43-85dc197f3f4c-istio-data" pod "6b875062-a976-42cd-8b43-85dc197f3f4c" (UID: "6b875062-a976-42cd-8b43-85dc197f3f4c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:41:12.050633 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:12.050568 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b875062-a976-42cd-8b43-85dc197f3f4c-istiod-ca-cert" pod "6b875062-a976-42cd-8b43-85dc197f3f4c" (UID: "6b875062-a976-42cd-8b43-85dc197f3f4c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:41:12.050633 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:12.050580 4908 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b875062-a976-42cd-8b43-85dc197f3f4c-credential-socket\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:41:12.050633 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:12.050603 4908 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b875062-a976-42cd-8b43-85dc197f3f4c-istio-data\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:41:12.050791 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:12.050617 4908 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b875062-a976-42cd-8b43-85dc197f3f4c-workload-certs\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:41:12.050791 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:12.050654 4908 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b875062-a976-42cd-8b43-85dc197f3f4c-workload-socket\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:41:12.052093 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:12.052061 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b875062-a976-42cd-8b43-85dc197f3f4c-istio-token" pod "6b875062-a976-42cd-8b43-85dc197f3f4c" (UID: "6b875062-a976-42cd-8b43-85dc197f3f4c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:41:12.052093 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:12.052076 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b875062-a976-42cd-8b43-85dc197f3f4c-istio-envoy" pod "6b875062-a976-42cd-8b43-85dc197f3f4c" (UID: "6b875062-a976-42cd-8b43-85dc197f3f4c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:41:12.052407 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:12.052388 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b875062-a976-42cd-8b43-85dc197f3f4c-kube-api-access-sv9rt" pod "6b875062-a976-42cd-8b43-85dc197f3f4c" (UID: "6b875062-a976-42cd-8b43-85dc197f3f4c"). InnerVolumeSpecName "kube-api-access-sv9rt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:41:12.052590 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:12.052565 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6b875062-a976-42cd-8b43-85dc197f3f4c-istio-podinfo" pod "6b875062-a976-42cd-8b43-85dc197f3f4c" (UID: "6b875062-a976-42cd-8b43-85dc197f3f4c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 12:41:12.152357 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:12.152324 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5dhl\" (UniqueName: \"kubernetes.io/projected/98e23727-b5b0-4c90-844a-d732b3e001de-kube-api-access-s5dhl\") pod \"maas-api-revoke-sub-751c30fd-lrtv2\" (UID: \"98e23727-b5b0-4c90-844a-d732b3e001de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-751c30fd-lrtv2" Oct 30 12:41:12.152616 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:12.152600 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98e23727-b5b0-4c90-844a-d732b3e001de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-751c30fd-lrtv2\" (UID: \"98e23727-b5b0-4c90-844a-d732b3e001de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-751c30fd-lrtv2" Oct 30 12:41:12.152741 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:12.152728 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98e23727-b5b0-4c90-844a-d732b3e001de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-751c30fd-lrtv2\" (UID: \"98e23727-b5b0-4c90-844a-d732b3e001de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-751c30fd-lrtv2" Oct 30 12:41:12.152994 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:12.152981 4908 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b875062-a976-42cd-8b43-85dc197f3f4c-istio-token\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:41:12.153081 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:12.153072 4908 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b875062-a976-42cd-8b43-85dc197f3f4c-istio-podinfo\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:41:12.153163 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:12.153153 4908 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b875062-a976-42cd-8b43-85dc197f3f4c-istio-envoy\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:41:12.153236 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:12.153226 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sv9rt\" (UniqueName: \"kubernetes.io/projected/6b875062-a976-42cd-8b43-85dc197f3f4c-kube-api-access-sv9rt\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:41:12.153322 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:12.153312 4908 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b875062-a976-42cd-8b43-85dc197f3f4c-istiod-ca-cert\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:41:12.154313 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:12.154286 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98e23727-b5b0-4c90-844a-d732b3e001de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-751c30fd-lrtv2\" (UID: \"98e23727-b5b0-4c90-844a-d732b3e001de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-751c30fd-lrtv2" Oct 30 12:41:12.158559 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:12.158535 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98e23727-b5b0-4c90-844a-d732b3e001de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-751c30fd-lrtv2\" (UID: \"98e23727-b5b0-4c90-844a-d732b3e001de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-751c30fd-lrtv2" Oct 30 12:41:12.182984 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:12.182953 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5dhl\" (UniqueName: \"kubernetes.io/projected/98e23727-b5b0-4c90-844a-d732b3e001de-kube-api-access-s5dhl\") pod \"maas-api-revoke-sub-751c30fd-lrtv2\" (UID: \"98e23727-b5b0-4c90-844a-d732b3e001de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-751c30fd-lrtv2" Oct 30 12:41:12.204241 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:12.204215 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-751c30fd-lrtv2" Oct 30 12:41:12.211319 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:12.211285 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-nntkd" event={"ID":"e6810256-6b25-44b9-a7f1-fb0cb023a5b9","Type":"ContainerStarted","Data":"61e639a16ad650e85438962111c2da99430d9e83ce51902b2bed8c1b8d716516"} Oct 30 12:41:12.211453 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:12.211381 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-nntkd" event={"ID":"e6810256-6b25-44b9-a7f1-fb0cb023a5b9","Type":"ContainerStarted","Data":"bdaf6dec313f3748acecbf10ec608f8dc6ee578209416d5bfea52900d0a6aa46"} Oct 30 12:41:12.213094 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:12.213068 4908 generic.go:356] "Generic (PLEG): container finished" podID="6b875062-a976-42cd-8b43-85dc197f3f4c" containerID="b6b5da560cca29c612e9861dea20b806ee92c440fc8f39dc6f5ed876e79024e2" exitCode=0 Oct 30 12:41:12.213202 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:12.213115 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-7z8r7" Oct 30 12:41:12.213202 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:12.213155 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-7z8r7" event={"ID":"6b875062-a976-42cd-8b43-85dc197f3f4c","Type":"ContainerDied","Data":"b6b5da560cca29c612e9861dea20b806ee92c440fc8f39dc6f5ed876e79024e2"} Oct 30 12:41:12.213312 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:12.213219 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-7z8r7" event={"ID":"6b875062-a976-42cd-8b43-85dc197f3f4c","Type":"ContainerDied","Data":"7ea80cb364e812c832d652a4bd459ce97def3eb6f964efe348734849c08b55d7"} Oct 30 12:41:12.213312 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:12.213254 4908 scope.go:122] "RemoveContainer" containerID="b6b5da560cca29c612e9861dea20b806ee92c440fc8f39dc6f5ed876e79024e2" Oct 30 12:41:12.270605 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:12.267238 4908 scope.go:122] "RemoveContainer" containerID="b6b5da560cca29c612e9861dea20b806ee92c440fc8f39dc6f5ed876e79024e2" Oct 30 12:41:12.270605 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:12.268858 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-nntkd" podStartSLOduration=2.268822447 podStartE2EDuration="2.268822447s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:41:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:41:12.260967033 +0000 UTC m=+2828.775395796" watchObservedRunningTime="2026-10-30 12:41:12.268822447 +0000 UTC m=+2828.783251192" Oct 30 12:41:12.270605 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:41:12.269451 4908 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6b5da560cca29c612e9861dea20b806ee92c440fc8f39dc6f5ed876e79024e2\": container with ID starting with b6b5da560cca29c612e9861dea20b806ee92c440fc8f39dc6f5ed876e79024e2 not found: ID does not exist" containerID="b6b5da560cca29c612e9861dea20b806ee92c440fc8f39dc6f5ed876e79024e2" Oct 30 12:41:12.270605 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:12.269482 4908 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6b5da560cca29c612e9861dea20b806ee92c440fc8f39dc6f5ed876e79024e2"} err="failed to get container status \"b6b5da560cca29c612e9861dea20b806ee92c440fc8f39dc6f5ed876e79024e2\": rpc error: code = NotFound desc = could not find container \"b6b5da560cca29c612e9861dea20b806ee92c440fc8f39dc6f5ed876e79024e2\": container with ID starting with b6b5da560cca29c612e9861dea20b806ee92c440fc8f39dc6f5ed876e79024e2 not found: ID does not exist" Oct 30 12:41:12.317894 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:12.317866 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-7z8r7"] Oct 30 12:41:12.330890 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:12.330622 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-7z8r7"] Oct 30 12:41:12.436711 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:12.436654 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-nntkd" Oct 30 12:41:12.439029 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:12.438706 4908 patch_prober.go:29] interesting pod/e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-nntkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" start-of-body= Oct 30 12:41:12.439029 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:12.438780 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-nntkd" podUID="e6810256-6b25-44b9-a7f1-fb0cb023a5b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" Oct 30 12:41:12.497899 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:12.496174 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f84120f4-vm445" Oct 30 12:41:12.563902 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:12.558368 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-751c30fd-lrtv2"] Oct 30 12:41:12.662410 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:12.661322 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37837576-7f1a-4341-b1b3-71a4e74afa81-kube-api-access-8qq6b\" (UniqueName: \"kubernetes.io/projected/37837576-7f1a-4341-b1b3-71a4e74afa81-kube-api-access-8qq6b\") pod \"37837576-7f1a-4341-b1b3-71a4e74afa81\" (UID: \"37837576-7f1a-4341-b1b3-71a4e74afa81\") " Oct 30 12:41:12.662410 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:12.661427 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/37837576-7f1a-4341-b1b3-71a4e74afa81-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37837576-7f1a-4341-b1b3-71a4e74afa81-maas-api-xks-ca\") pod \"37837576-7f1a-4341-b1b3-71a4e74afa81\" (UID: \"37837576-7f1a-4341-b1b3-71a4e74afa81\") " Oct 30 12:41:12.662410 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:12.661462 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37837576-7f1a-4341-b1b3-71a4e74afa81-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37837576-7f1a-4341-b1b3-71a4e74afa81-maas-api-openshift-ca\") pod \"37837576-7f1a-4341-b1b3-71a4e74afa81\" (UID: \"37837576-7f1a-4341-b1b3-71a4e74afa81\") " Oct 30 12:41:12.663896 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:12.663851 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37837576-7f1a-4341-b1b3-71a4e74afa81-maas-api-openshift-ca" pod "37837576-7f1a-4341-b1b3-71a4e74afa81" (UID: "37837576-7f1a-4341-b1b3-71a4e74afa81"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:41:12.816927 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:12.780947 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37837576-7f1a-4341-b1b3-71a4e74afa81-maas-api-openshift-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:41:12.901662 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:12.899280 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37837576-7f1a-4341-b1b3-71a4e74afa81-kube-api-access-8qq6b" pod "37837576-7f1a-4341-b1b3-71a4e74afa81" (UID: "37837576-7f1a-4341-b1b3-71a4e74afa81"). InnerVolumeSpecName "kube-api-access-8qq6b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:41:12.901662 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:12.899996 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37837576-7f1a-4341-b1b3-71a4e74afa81-maas-api-xks-ca" pod "37837576-7f1a-4341-b1b3-71a4e74afa81" (UID: "37837576-7f1a-4341-b1b3-71a4e74afa81"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:41:12.919306 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:12.919267 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qq6b\" (UniqueName: \"kubernetes.io/projected/37837576-7f1a-4341-b1b3-71a4e74afa81-kube-api-access-8qq6b\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:41:12.919306 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:12.919300 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37837576-7f1a-4341-b1b3-71a4e74afa81-maas-api-xks-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:41:13.227101 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:13.227066 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f84120f4-vm445" event={"ID":"37837576-7f1a-4341-b1b3-71a4e74afa81","Type":"ContainerDied","Data":"b7202ed834f29123ad16890871b63f5e0ab145417c400ca4e997071ab95f8711"} Oct 30 12:41:13.227613 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:13.227119 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b7202ed834f29123ad16890871b63f5e0ab145417c400ca4e997071ab95f8711" Oct 30 12:41:13.227613 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:13.227170 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f84120f4-vm445" Oct 30 12:41:13.232786 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:13.232169 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-751c30fd-lrtv2" event={"ID":"98e23727-b5b0-4c90-844a-d732b3e001de","Type":"ContainerStarted","Data":"5ceb1dad7ed7b749047b80f8d1655056a55a3fbb61602151a46fc110298e5982"} Oct 30 12:41:13.437307 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:13.437231 4908 patch_prober.go:29] interesting pod/e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-nntkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" start-of-body= Oct 30 12:41:13.437307 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:13.437286 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-nntkd" podUID="e6810256-6b25-44b9-a7f1-fb0cb023a5b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" Oct 30 12:41:13.607431 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:13.607397 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b875062-a976-42cd-8b43-85dc197f3f4c" path="/var/lib/kubelet/pods/6b875062-a976-42cd-8b43-85dc197f3f4c/volumes" Oct 30 12:41:14.240260 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:14.240225 4908 generic.go:356] "Generic (PLEG): container finished" podID="98e23727-b5b0-4c90-844a-d732b3e001de" containerID="e8cdb41e9670645c5c4439d943a2ed9183b0d209b61a6840c8c6eeab1f32e59f" exitCode=0 Oct 30 12:41:14.240621 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:14.240293 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-751c30fd-lrtv2" event={"ID":"98e23727-b5b0-4c90-844a-d732b3e001de","Type":"ContainerDied","Data":"e8cdb41e9670645c5c4439d943a2ed9183b0d209b61a6840c8c6eeab1f32e59f"} Oct 30 12:41:14.436669 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:14.436571 4908 patch_prober.go:29] interesting pod/e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-nntkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" start-of-body= Oct 30 12:41:14.436669 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:14.436648 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-nntkd" podUID="e6810256-6b25-44b9-a7f1-fb0cb023a5b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" Oct 30 12:41:15.437113 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:15.437087 4908 patch_prober.go:29] interesting pod/e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-nntkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" start-of-body= Oct 30 12:41:15.437469 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:15.437141 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-nntkd" podUID="e6810256-6b25-44b9-a7f1-fb0cb023a5b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" Oct 30 12:41:15.455359 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:15.455333 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-751c30fd-lrtv2" Oct 30 12:41:15.554863 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:15.554768 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98e23727-b5b0-4c90-844a-d732b3e001de-kube-api-access-s5dhl\" (UniqueName: \"kubernetes.io/projected/98e23727-b5b0-4c90-844a-d732b3e001de-kube-api-access-s5dhl\") pod \"98e23727-b5b0-4c90-844a-d732b3e001de\" (UID: \"98e23727-b5b0-4c90-844a-d732b3e001de\") " Oct 30 12:41:15.555048 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:15.555016 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/98e23727-b5b0-4c90-844a-d732b3e001de-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98e23727-b5b0-4c90-844a-d732b3e001de-maas-api-xks-ca\") pod \"98e23727-b5b0-4c90-844a-d732b3e001de\" (UID: \"98e23727-b5b0-4c90-844a-d732b3e001de\") " Oct 30 12:41:15.555120 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:15.555104 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98e23727-b5b0-4c90-844a-d732b3e001de-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98e23727-b5b0-4c90-844a-d732b3e001de-maas-api-openshift-ca\") pod \"98e23727-b5b0-4c90-844a-d732b3e001de\" (UID: \"98e23727-b5b0-4c90-844a-d732b3e001de\") " Oct 30 12:41:15.555463 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:15.555443 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98e23727-b5b0-4c90-844a-d732b3e001de-maas-api-openshift-ca" pod "98e23727-b5b0-4c90-844a-d732b3e001de" (UID: "98e23727-b5b0-4c90-844a-d732b3e001de"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:41:15.555723 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:15.555699 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98e23727-b5b0-4c90-844a-d732b3e001de-maas-api-openshift-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:41:15.557366 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:15.557339 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98e23727-b5b0-4c90-844a-d732b3e001de-kube-api-access-s5dhl" pod "98e23727-b5b0-4c90-844a-d732b3e001de" (UID: "98e23727-b5b0-4c90-844a-d732b3e001de"). InnerVolumeSpecName "kube-api-access-s5dhl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:41:15.557496 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:15.557370 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98e23727-b5b0-4c90-844a-d732b3e001de-maas-api-xks-ca" pod "98e23727-b5b0-4c90-844a-d732b3e001de" (UID: "98e23727-b5b0-4c90-844a-d732b3e001de"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:41:15.656532 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:15.656501 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98e23727-b5b0-4c90-844a-d732b3e001de-maas-api-xks-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:41:15.656532 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:15.656531 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5dhl\" (UniqueName: \"kubernetes.io/projected/98e23727-b5b0-4c90-844a-d732b3e001de-kube-api-access-s5dhl\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:41:16.258084 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:16.258049 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-751c30fd-lrtv2" event={"ID":"98e23727-b5b0-4c90-844a-d732b3e001de","Type":"ContainerDied","Data":"5ceb1dad7ed7b749047b80f8d1655056a55a3fbb61602151a46fc110298e5982"} Oct 30 12:41:16.258084 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:16.258084 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-751c30fd-lrtv2" Oct 30 12:41:16.258354 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:16.258095 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5ceb1dad7ed7b749047b80f8d1655056a55a3fbb61602151a46fc110298e5982" Oct 30 12:41:16.437081 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:16.437043 4908 patch_prober.go:29] interesting pod/e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-nntkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" start-of-body= Oct 30 12:41:16.437255 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:16.437095 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-nntkd" podUID="e6810256-6b25-44b9-a7f1-fb0cb023a5b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" Oct 30 12:41:17.437062 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:17.437020 4908 patch_prober.go:29] interesting pod/e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-nntkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" start-of-body= Oct 30 12:41:17.437246 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:17.437088 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-nntkd" podUID="e6810256-6b25-44b9-a7f1-fb0cb023a5b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" Oct 30 12:41:18.441590 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:18.441561 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-nntkd" Oct 30 12:41:18.442053 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:18.442031 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-nntkd" Oct 30 12:41:18.442725 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:18.442703 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-nntkd" Oct 30 12:41:19.688186 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:41:19.688150 4908 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vn6vv" containerName="extractor" Oct 30 12:41:21.849394 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:21.849360 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-0bda67/auth-test-model-0bda67-kserve-686fd886d5-kngvr"] Oct 30 12:41:23.783174 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:23.783144 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9wwfc"] Oct 30 12:41:34.451617 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:34.451576 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-327e31/auth-test-model-327e31-kserve-587f6fcd8f-r8vm4"] Oct 30 12:41:36.879999 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:36.879946 4908 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w1-0bda67/auth-test-model-0bda67-kserve-686fd886d5-kngvr" podUID="3577ee1a-5102-41e3-99b2-03dc9618ae0b" containerName="main" containerID="cri-o://dddddc5235b7fc4ff7f05bd97ed2adcd2c67a24af57a8df200c45abb9f8d855b" gracePeriod=45 Oct 30 12:41:37.176623 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:37.176599 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-0bda67/auth-test-model-0bda67-kserve-686fd886d5-kngvr" Oct 30 12:41:37.251502 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:37.251474 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3577ee1a-5102-41e3-99b2-03dc9618ae0b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3577ee1a-5102-41e3-99b2-03dc9618ae0b-tmp-dir\") pod \"3577ee1a-5102-41e3-99b2-03dc9618ae0b\" (UID: \"3577ee1a-5102-41e3-99b2-03dc9618ae0b\") " Oct 30 12:41:37.251692 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:37.251510 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3577ee1a-5102-41e3-99b2-03dc9618ae0b-home\" (UniqueName: \"kubernetes.io/empty-dir/3577ee1a-5102-41e3-99b2-03dc9618ae0b-home\") pod \"3577ee1a-5102-41e3-99b2-03dc9618ae0b\" (UID: \"3577ee1a-5102-41e3-99b2-03dc9618ae0b\") " Oct 30 12:41:37.251692 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:37.251605 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3577ee1a-5102-41e3-99b2-03dc9618ae0b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3577ee1a-5102-41e3-99b2-03dc9618ae0b-dshm\") pod \"3577ee1a-5102-41e3-99b2-03dc9618ae0b\" (UID: \"3577ee1a-5102-41e3-99b2-03dc9618ae0b\") " Oct 30 12:41:37.251692 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:37.251666 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3577ee1a-5102-41e3-99b2-03dc9618ae0b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3577ee1a-5102-41e3-99b2-03dc9618ae0b-model-cache\") pod \"3577ee1a-5102-41e3-99b2-03dc9618ae0b\" (UID: \"3577ee1a-5102-41e3-99b2-03dc9618ae0b\") " Oct 30 12:41:37.251913 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:37.251735 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3577ee1a-5102-41e3-99b2-03dc9618ae0b-kube-api-access-dkg2f\" (UniqueName: \"kubernetes.io/projected/3577ee1a-5102-41e3-99b2-03dc9618ae0b-kube-api-access-dkg2f\") pod \"3577ee1a-5102-41e3-99b2-03dc9618ae0b\" (UID: \"3577ee1a-5102-41e3-99b2-03dc9618ae0b\") " Oct 30 12:41:37.251913 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:37.251783 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3577ee1a-5102-41e3-99b2-03dc9618ae0b-tls-certs\" (UniqueName: \"kubernetes.io/secret/3577ee1a-5102-41e3-99b2-03dc9618ae0b-tls-certs\") pod \"3577ee1a-5102-41e3-99b2-03dc9618ae0b\" (UID: \"3577ee1a-5102-41e3-99b2-03dc9618ae0b\") " Oct 30 12:41:37.251913 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:37.251786 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3577ee1a-5102-41e3-99b2-03dc9618ae0b-tmp-dir" pod "3577ee1a-5102-41e3-99b2-03dc9618ae0b" (UID: "3577ee1a-5102-41e3-99b2-03dc9618ae0b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:41:37.252071 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:37.252028 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3577ee1a-5102-41e3-99b2-03dc9618ae0b-model-cache" pod "3577ee1a-5102-41e3-99b2-03dc9618ae0b" (UID: "3577ee1a-5102-41e3-99b2-03dc9618ae0b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:41:37.252172 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:37.252149 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3577ee1a-5102-41e3-99b2-03dc9618ae0b-home" pod "3577ee1a-5102-41e3-99b2-03dc9618ae0b" (UID: "3577ee1a-5102-41e3-99b2-03dc9618ae0b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:41:37.252395 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:37.252378 4908 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3577ee1a-5102-41e3-99b2-03dc9618ae0b-tmp-dir\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:41:37.252395 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:37.252397 4908 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3577ee1a-5102-41e3-99b2-03dc9618ae0b-home\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:41:37.252555 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:37.252405 4908 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3577ee1a-5102-41e3-99b2-03dc9618ae0b-model-cache\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:41:37.257945 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:37.257923 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3577ee1a-5102-41e3-99b2-03dc9618ae0b-dshm" pod "3577ee1a-5102-41e3-99b2-03dc9618ae0b" (UID: "3577ee1a-5102-41e3-99b2-03dc9618ae0b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:41:37.263761 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:37.263736 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3577ee1a-5102-41e3-99b2-03dc9618ae0b-tls-certs" pod "3577ee1a-5102-41e3-99b2-03dc9618ae0b" (UID: "3577ee1a-5102-41e3-99b2-03dc9618ae0b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:41:37.265160 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:37.265138 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3577ee1a-5102-41e3-99b2-03dc9618ae0b-kube-api-access-dkg2f" pod "3577ee1a-5102-41e3-99b2-03dc9618ae0b" (UID: "3577ee1a-5102-41e3-99b2-03dc9618ae0b"). InnerVolumeSpecName "kube-api-access-dkg2f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:41:37.353328 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:37.353300 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dkg2f\" (UniqueName: \"kubernetes.io/projected/3577ee1a-5102-41e3-99b2-03dc9618ae0b-kube-api-access-dkg2f\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:41:37.353328 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:37.353326 4908 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3577ee1a-5102-41e3-99b2-03dc9618ae0b-tls-certs\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:41:37.353514 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:37.353336 4908 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3577ee1a-5102-41e3-99b2-03dc9618ae0b-dshm\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:41:37.484951 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:37.484923 4908 generic.go:356] "Generic (PLEG): container finished" podID="3577ee1a-5102-41e3-99b2-03dc9618ae0b" containerID="dddddc5235b7fc4ff7f05bd97ed2adcd2c67a24af57a8df200c45abb9f8d855b" exitCode=0 Oct 30 12:41:37.485101 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:37.484985 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-0bda67/auth-test-model-0bda67-kserve-686fd886d5-kngvr" Oct 30 12:41:37.485101 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:37.485016 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-0bda67/auth-test-model-0bda67-kserve-686fd886d5-kngvr" event={"ID":"3577ee1a-5102-41e3-99b2-03dc9618ae0b","Type":"ContainerDied","Data":"dddddc5235b7fc4ff7f05bd97ed2adcd2c67a24af57a8df200c45abb9f8d855b"} Oct 30 12:41:37.485101 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:37.485066 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-0bda67/auth-test-model-0bda67-kserve-686fd886d5-kngvr" event={"ID":"3577ee1a-5102-41e3-99b2-03dc9618ae0b","Type":"ContainerDied","Data":"c5fbde98ba89424aeee3171677e41ffee1bfdd090e88c505d17e40cdd247b90d"} Oct 30 12:41:37.485101 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:37.485092 4908 scope.go:122] "RemoveContainer" containerID="dddddc5235b7fc4ff7f05bd97ed2adcd2c67a24af57a8df200c45abb9f8d855b" Oct 30 12:41:37.496315 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:37.496156 4908 scope.go:122] "RemoveContainer" containerID="dddddc5235b7fc4ff7f05bd97ed2adcd2c67a24af57a8df200c45abb9f8d855b" Oct 30 12:41:37.496484 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:41:37.496461 4908 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dddddc5235b7fc4ff7f05bd97ed2adcd2c67a24af57a8df200c45abb9f8d855b\": container with ID starting with dddddc5235b7fc4ff7f05bd97ed2adcd2c67a24af57a8df200c45abb9f8d855b not found: ID does not exist" containerID="dddddc5235b7fc4ff7f05bd97ed2adcd2c67a24af57a8df200c45abb9f8d855b" Oct 30 12:41:37.496560 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:37.496497 4908 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dddddc5235b7fc4ff7f05bd97ed2adcd2c67a24af57a8df200c45abb9f8d855b"} err="failed to get container status \"dddddc5235b7fc4ff7f05bd97ed2adcd2c67a24af57a8df200c45abb9f8d855b\": rpc error: code = NotFound desc = could not find container \"dddddc5235b7fc4ff7f05bd97ed2adcd2c67a24af57a8df200c45abb9f8d855b\": container with ID starting with dddddc5235b7fc4ff7f05bd97ed2adcd2c67a24af57a8df200c45abb9f8d855b not found: ID does not exist" Oct 30 12:41:37.538506 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:37.538481 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-0bda67/auth-test-model-0bda67-kserve-686fd886d5-kngvr"] Oct 30 12:41:37.544063 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:37.544041 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w1-0bda67/auth-test-model-0bda67-kserve-686fd886d5-kngvr"] Oct 30 12:41:37.608267 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:37.608237 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3577ee1a-5102-41e3-99b2-03dc9618ae0b" path="/var/lib/kubelet/pods/3577ee1a-5102-41e3-99b2-03dc9618ae0b/volumes" Oct 30 12:41:40.315512 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:40.310542 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-8d2170/auth-test-model-8d2170-kserve-7b6547c869-5q7r5"] Oct 30 12:41:41.631626 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:41.631595 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-ac674b-openshift-default-798564ccc7-2nk9v"] Oct 30 12:41:41.632267 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:41.632245 4908 cpu_manager.go:414] "containerMap: removing container" podUID="37837576-7f1a-4341-b1b3-71a4e74afa81" containerName="revoke-keys" Oct 30 12:41:41.632363 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:41.632268 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37837576-7f1a-4341-b1b3-71a4e74afa81" containerName="revoke-keys" Oct 30 12:41:41.632363 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:41.632325 4908 cpu_manager.go:414] "containerMap: removing container" podUID="6b875062-a976-42cd-8b43-85dc197f3f4c" containerName="istio-proxy" Oct 30 12:41:41.632363 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:41.632334 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b875062-a976-42cd-8b43-85dc197f3f4c" containerName="istio-proxy" Oct 30 12:41:41.632363 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:41.632347 4908 cpu_manager.go:414] "containerMap: removing container" podUID="3577ee1a-5102-41e3-99b2-03dc9618ae0b" containerName="main" Oct 30 12:41:41.632363 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:41.632355 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3577ee1a-5102-41e3-99b2-03dc9618ae0b" containerName="main" Oct 30 12:41:41.632620 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:41.632386 4908 cpu_manager.go:414] "containerMap: removing container" podUID="98e23727-b5b0-4c90-844a-d732b3e001de" containerName="revoke-keys" Oct 30 12:41:41.632620 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:41.632394 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98e23727-b5b0-4c90-844a-d732b3e001de" containerName="revoke-keys" Oct 30 12:41:41.632620 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:41.632520 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="98e23727-b5b0-4c90-844a-d732b3e001de" containerName="revoke-keys" Oct 30 12:41:41.632620 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:41.632536 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="37837576-7f1a-4341-b1b3-71a4e74afa81" containerName="revoke-keys" Oct 30 12:41:41.632620 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:41.632552 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b875062-a976-42cd-8b43-85dc197f3f4c" containerName="istio-proxy" Oct 30 12:41:41.632620 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:41.632562 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="3577ee1a-5102-41e3-99b2-03dc9618ae0b" containerName="main" Oct 30 12:41:41.642686 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:41.642655 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-ac674b-openshift-default-798564ccc7-2nk9v"] Oct 30 12:41:41.644440 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:41.644420 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-ac674b-openshift-default-798564ccc7-2nk9v" Oct 30 12:41:41.659483 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:41.659446 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-ac674b-openshift-default-dockercfg-tr7jm\"" Oct 30 12:41:41.763853 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:41.750635 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a63f6b6-6cb7-4ed6-af39-c9fe090ebc87-workload-certs\") pod \"e2e-autores-ac674b-openshift-default-798564ccc7-2nk9v\" (UID: \"4a63f6b6-6cb7-4ed6-af39-c9fe090ebc87\") " pod="openshift-ingress/e2e-autores-ac674b-openshift-default-798564ccc7-2nk9v" Oct 30 12:41:41.763853 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:41.750689 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a63f6b6-6cb7-4ed6-af39-c9fe090ebc87-workload-socket\") pod \"e2e-autores-ac674b-openshift-default-798564ccc7-2nk9v\" (UID: \"4a63f6b6-6cb7-4ed6-af39-c9fe090ebc87\") " pod="openshift-ingress/e2e-autores-ac674b-openshift-default-798564ccc7-2nk9v" Oct 30 12:41:41.763853 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:41.750731 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a63f6b6-6cb7-4ed6-af39-c9fe090ebc87-credential-socket\") pod \"e2e-autores-ac674b-openshift-default-798564ccc7-2nk9v\" (UID: \"4a63f6b6-6cb7-4ed6-af39-c9fe090ebc87\") " pod="openshift-ingress/e2e-autores-ac674b-openshift-default-798564ccc7-2nk9v" Oct 30 12:41:41.763853 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:41.750816 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a63f6b6-6cb7-4ed6-af39-c9fe090ebc87-istio-data\") pod \"e2e-autores-ac674b-openshift-default-798564ccc7-2nk9v\" (UID: \"4a63f6b6-6cb7-4ed6-af39-c9fe090ebc87\") " pod="openshift-ingress/e2e-autores-ac674b-openshift-default-798564ccc7-2nk9v" Oct 30 12:41:41.763853 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:41.750897 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a63f6b6-6cb7-4ed6-af39-c9fe090ebc87-istio-token\") pod \"e2e-autores-ac674b-openshift-default-798564ccc7-2nk9v\" (UID: \"4a63f6b6-6cb7-4ed6-af39-c9fe090ebc87\") " pod="openshift-ingress/e2e-autores-ac674b-openshift-default-798564ccc7-2nk9v" Oct 30 12:41:41.763853 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:41.750920 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a63f6b6-6cb7-4ed6-af39-c9fe090ebc87-istiod-ca-cert\") pod \"e2e-autores-ac674b-openshift-default-798564ccc7-2nk9v\" (UID: \"4a63f6b6-6cb7-4ed6-af39-c9fe090ebc87\") " pod="openshift-ingress/e2e-autores-ac674b-openshift-default-798564ccc7-2nk9v" Oct 30 12:41:41.763853 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:41.750990 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a63f6b6-6cb7-4ed6-af39-c9fe090ebc87-istio-envoy\") pod \"e2e-autores-ac674b-openshift-default-798564ccc7-2nk9v\" (UID: \"4a63f6b6-6cb7-4ed6-af39-c9fe090ebc87\") " pod="openshift-ingress/e2e-autores-ac674b-openshift-default-798564ccc7-2nk9v" Oct 30 12:41:41.763853 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:41.751043 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a63f6b6-6cb7-4ed6-af39-c9fe090ebc87-istio-podinfo\") pod \"e2e-autores-ac674b-openshift-default-798564ccc7-2nk9v\" (UID: \"4a63f6b6-6cb7-4ed6-af39-c9fe090ebc87\") " pod="openshift-ingress/e2e-autores-ac674b-openshift-default-798564ccc7-2nk9v" Oct 30 12:41:41.763853 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:41.751155 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64gvm\" (UniqueName: \"kubernetes.io/projected/4a63f6b6-6cb7-4ed6-af39-c9fe090ebc87-kube-api-access-64gvm\") pod \"e2e-autores-ac674b-openshift-default-798564ccc7-2nk9v\" (UID: \"4a63f6b6-6cb7-4ed6-af39-c9fe090ebc87\") " pod="openshift-ingress/e2e-autores-ac674b-openshift-default-798564ccc7-2nk9v" Oct 30 12:41:41.858855 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:41.858171 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64gvm\" (UniqueName: \"kubernetes.io/projected/4a63f6b6-6cb7-4ed6-af39-c9fe090ebc87-kube-api-access-64gvm\") pod \"e2e-autores-ac674b-openshift-default-798564ccc7-2nk9v\" (UID: \"4a63f6b6-6cb7-4ed6-af39-c9fe090ebc87\") " pod="openshift-ingress/e2e-autores-ac674b-openshift-default-798564ccc7-2nk9v" Oct 30 12:41:41.858855 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:41.858230 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a63f6b6-6cb7-4ed6-af39-c9fe090ebc87-workload-certs\") pod \"e2e-autores-ac674b-openshift-default-798564ccc7-2nk9v\" (UID: \"4a63f6b6-6cb7-4ed6-af39-c9fe090ebc87\") " pod="openshift-ingress/e2e-autores-ac674b-openshift-default-798564ccc7-2nk9v" Oct 30 12:41:41.858855 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:41.858254 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a63f6b6-6cb7-4ed6-af39-c9fe090ebc87-workload-socket\") pod \"e2e-autores-ac674b-openshift-default-798564ccc7-2nk9v\" (UID: \"4a63f6b6-6cb7-4ed6-af39-c9fe090ebc87\") " pod="openshift-ingress/e2e-autores-ac674b-openshift-default-798564ccc7-2nk9v" Oct 30 12:41:41.858855 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:41.858292 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a63f6b6-6cb7-4ed6-af39-c9fe090ebc87-credential-socket\") pod \"e2e-autores-ac674b-openshift-default-798564ccc7-2nk9v\" (UID: \"4a63f6b6-6cb7-4ed6-af39-c9fe090ebc87\") " pod="openshift-ingress/e2e-autores-ac674b-openshift-default-798564ccc7-2nk9v" Oct 30 12:41:41.858855 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:41.858365 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a63f6b6-6cb7-4ed6-af39-c9fe090ebc87-istio-data\") pod \"e2e-autores-ac674b-openshift-default-798564ccc7-2nk9v\" (UID: \"4a63f6b6-6cb7-4ed6-af39-c9fe090ebc87\") " pod="openshift-ingress/e2e-autores-ac674b-openshift-default-798564ccc7-2nk9v" Oct 30 12:41:41.858855 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:41.858417 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a63f6b6-6cb7-4ed6-af39-c9fe090ebc87-istio-token\") pod \"e2e-autores-ac674b-openshift-default-798564ccc7-2nk9v\" (UID: \"4a63f6b6-6cb7-4ed6-af39-c9fe090ebc87\") " pod="openshift-ingress/e2e-autores-ac674b-openshift-default-798564ccc7-2nk9v" Oct 30 12:41:41.858855 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:41.858438 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a63f6b6-6cb7-4ed6-af39-c9fe090ebc87-istiod-ca-cert\") pod \"e2e-autores-ac674b-openshift-default-798564ccc7-2nk9v\" (UID: \"4a63f6b6-6cb7-4ed6-af39-c9fe090ebc87\") " pod="openshift-ingress/e2e-autores-ac674b-openshift-default-798564ccc7-2nk9v" Oct 30 12:41:41.858855 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:41.858501 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a63f6b6-6cb7-4ed6-af39-c9fe090ebc87-istio-envoy\") pod \"e2e-autores-ac674b-openshift-default-798564ccc7-2nk9v\" (UID: \"4a63f6b6-6cb7-4ed6-af39-c9fe090ebc87\") " pod="openshift-ingress/e2e-autores-ac674b-openshift-default-798564ccc7-2nk9v" Oct 30 12:41:41.858855 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:41.858547 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a63f6b6-6cb7-4ed6-af39-c9fe090ebc87-istio-podinfo\") pod \"e2e-autores-ac674b-openshift-default-798564ccc7-2nk9v\" (UID: \"4a63f6b6-6cb7-4ed6-af39-c9fe090ebc87\") " pod="openshift-ingress/e2e-autores-ac674b-openshift-default-798564ccc7-2nk9v" Oct 30 12:41:41.859572 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:41.859494 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a63f6b6-6cb7-4ed6-af39-c9fe090ebc87-workload-certs\") pod \"e2e-autores-ac674b-openshift-default-798564ccc7-2nk9v\" (UID: \"4a63f6b6-6cb7-4ed6-af39-c9fe090ebc87\") " pod="openshift-ingress/e2e-autores-ac674b-openshift-default-798564ccc7-2nk9v" Oct 30 12:41:41.859749 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:41.859724 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a63f6b6-6cb7-4ed6-af39-c9fe090ebc87-workload-socket\") pod \"e2e-autores-ac674b-openshift-default-798564ccc7-2nk9v\" (UID: \"4a63f6b6-6cb7-4ed6-af39-c9fe090ebc87\") " pod="openshift-ingress/e2e-autores-ac674b-openshift-default-798564ccc7-2nk9v" Oct 30 12:41:41.880912 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:41.860251 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a63f6b6-6cb7-4ed6-af39-c9fe090ebc87-credential-socket\") pod \"e2e-autores-ac674b-openshift-default-798564ccc7-2nk9v\" (UID: \"4a63f6b6-6cb7-4ed6-af39-c9fe090ebc87\") " pod="openshift-ingress/e2e-autores-ac674b-openshift-default-798564ccc7-2nk9v" Oct 30 12:41:41.881128 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:41.860469 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a63f6b6-6cb7-4ed6-af39-c9fe090ebc87-istio-data\") pod \"e2e-autores-ac674b-openshift-default-798564ccc7-2nk9v\" (UID: \"4a63f6b6-6cb7-4ed6-af39-c9fe090ebc87\") " pod="openshift-ingress/e2e-autores-ac674b-openshift-default-798564ccc7-2nk9v" Oct 30 12:41:41.881227 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:41.861476 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a63f6b6-6cb7-4ed6-af39-c9fe090ebc87-istiod-ca-cert\") pod \"e2e-autores-ac674b-openshift-default-798564ccc7-2nk9v\" (UID: \"4a63f6b6-6cb7-4ed6-af39-c9fe090ebc87\") " pod="openshift-ingress/e2e-autores-ac674b-openshift-default-798564ccc7-2nk9v" Oct 30 12:41:41.885909 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:41.884946 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a63f6b6-6cb7-4ed6-af39-c9fe090ebc87-istio-envoy\") pod \"e2e-autores-ac674b-openshift-default-798564ccc7-2nk9v\" (UID: \"4a63f6b6-6cb7-4ed6-af39-c9fe090ebc87\") " pod="openshift-ingress/e2e-autores-ac674b-openshift-default-798564ccc7-2nk9v" Oct 30 12:41:41.892430 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:41.892401 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a63f6b6-6cb7-4ed6-af39-c9fe090ebc87-istio-podinfo\") pod \"e2e-autores-ac674b-openshift-default-798564ccc7-2nk9v\" (UID: \"4a63f6b6-6cb7-4ed6-af39-c9fe090ebc87\") " pod="openshift-ingress/e2e-autores-ac674b-openshift-default-798564ccc7-2nk9v" Oct 30 12:41:41.892599 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:41.892549 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64gvm\" (UniqueName: \"kubernetes.io/projected/4a63f6b6-6cb7-4ed6-af39-c9fe090ebc87-kube-api-access-64gvm\") pod \"e2e-autores-ac674b-openshift-default-798564ccc7-2nk9v\" (UID: \"4a63f6b6-6cb7-4ed6-af39-c9fe090ebc87\") " pod="openshift-ingress/e2e-autores-ac674b-openshift-default-798564ccc7-2nk9v" Oct 30 12:41:41.894255 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:41.894230 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a63f6b6-6cb7-4ed6-af39-c9fe090ebc87-istio-token\") pod \"e2e-autores-ac674b-openshift-default-798564ccc7-2nk9v\" (UID: \"4a63f6b6-6cb7-4ed6-af39-c9fe090ebc87\") " pod="openshift-ingress/e2e-autores-ac674b-openshift-default-798564ccc7-2nk9v" Oct 30 12:41:41.973530 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:41.973498 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-ac674b-openshift-default-798564ccc7-2nk9v" Oct 30 12:41:42.304710 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:42.304677 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-ac674b-openshift-default-798564ccc7-2nk9v"] Oct 30 12:41:42.311627 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:42.311590 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:41:42.311742 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:42.311660 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:41:42.311742 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:42.311688 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:41:42.536863 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:42.536260 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-ac674b-openshift-default-798564ccc7-2nk9v" event={"ID":"4a63f6b6-6cb7-4ed6-af39-c9fe090ebc87","Type":"ContainerStarted","Data":"9ce960f38f4c4100401ee91fbb26ddb7fea096e58551a875564449b6d515b345"} Oct 30 12:41:42.536863 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:42.536324 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-ac674b-openshift-default-798564ccc7-2nk9v" event={"ID":"4a63f6b6-6cb7-4ed6-af39-c9fe090ebc87","Type":"ContainerStarted","Data":"5b58a286df337aabc133243b1fd729691ea5849889a530aa4340e8be9a8a2c6c"} Oct 30 12:41:42.582184 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:42.582075 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-ac674b-openshift-default-798564ccc7-2nk9v" podStartSLOduration=1.582058658 podStartE2EDuration="1.582058658s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:41:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:41:42.579378611 +0000 UTC m=+2859.093807358" watchObservedRunningTime="2026-10-30 12:41:42.582058658 +0000 UTC m=+2859.096487403" Oct 30 12:41:42.974563 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:42.974486 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-ac674b-openshift-default-798564ccc7-2nk9v" Oct 30 12:41:43.979903 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:43.979869 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-ac674b-openshift-default-798564ccc7-2nk9v" Oct 30 12:41:44.551166 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:44.551136 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-ac674b-openshift-default-798564ccc7-2nk9v" Oct 30 12:41:44.552091 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:44.552070 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-ac674b-openshift-default-798564ccc7-2nk9v" Oct 30 12:41:49.488219 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:49.488169 4908 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w4-327e31/auth-test-model-327e31-kserve-587f6fcd8f-r8vm4" podUID="f69baca9-270e-4b25-a96d-a771ffc429a8" containerName="main" containerID="cri-o://8c11f5fadeed0c1a30749fb2a7b31b36f774caf860cdfbefc2770c765061d2f8" gracePeriod=45 Oct 30 12:41:49.708351 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:41:49.708279 4908 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vn6vv" containerName="extractor" Oct 30 12:41:49.785145 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:49.785119 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-327e31/auth-test-model-327e31-kserve-587f6fcd8f-r8vm4" Oct 30 12:41:49.907675 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:49.907643 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f69baca9-270e-4b25-a96d-a771ffc429a8-tls-certs\" (UniqueName: \"kubernetes.io/secret/f69baca9-270e-4b25-a96d-a771ffc429a8-tls-certs\") pod \"f69baca9-270e-4b25-a96d-a771ffc429a8\" (UID: \"f69baca9-270e-4b25-a96d-a771ffc429a8\") " Oct 30 12:41:49.907675 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:49.907686 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f69baca9-270e-4b25-a96d-a771ffc429a8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f69baca9-270e-4b25-a96d-a771ffc429a8-dshm\") pod \"f69baca9-270e-4b25-a96d-a771ffc429a8\" (UID: \"f69baca9-270e-4b25-a96d-a771ffc429a8\") " Oct 30 12:41:49.907985 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:49.907745 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f69baca9-270e-4b25-a96d-a771ffc429a8-home\" (UniqueName: \"kubernetes.io/empty-dir/f69baca9-270e-4b25-a96d-a771ffc429a8-home\") pod \"f69baca9-270e-4b25-a96d-a771ffc429a8\" (UID: \"f69baca9-270e-4b25-a96d-a771ffc429a8\") " Oct 30 12:41:49.907985 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:49.907914 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f69baca9-270e-4b25-a96d-a771ffc429a8-kube-api-access-zc8g2\" (UniqueName: \"kubernetes.io/projected/f69baca9-270e-4b25-a96d-a771ffc429a8-kube-api-access-zc8g2\") pod \"f69baca9-270e-4b25-a96d-a771ffc429a8\" (UID: \"f69baca9-270e-4b25-a96d-a771ffc429a8\") " Oct 30 12:41:49.908113 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:49.907981 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f69baca9-270e-4b25-a96d-a771ffc429a8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f69baca9-270e-4b25-a96d-a771ffc429a8-model-cache\") pod \"f69baca9-270e-4b25-a96d-a771ffc429a8\" (UID: \"f69baca9-270e-4b25-a96d-a771ffc429a8\") " Oct 30 12:41:49.908113 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:49.908039 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f69baca9-270e-4b25-a96d-a771ffc429a8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f69baca9-270e-4b25-a96d-a771ffc429a8-tmp-dir\") pod \"f69baca9-270e-4b25-a96d-a771ffc429a8\" (UID: \"f69baca9-270e-4b25-a96d-a771ffc429a8\") " Oct 30 12:41:49.908113 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:49.908047 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f69baca9-270e-4b25-a96d-a771ffc429a8-home" pod "f69baca9-270e-4b25-a96d-a771ffc429a8" (UID: "f69baca9-270e-4b25-a96d-a771ffc429a8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:41:49.908253 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:49.908234 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f69baca9-270e-4b25-a96d-a771ffc429a8-model-cache" pod "f69baca9-270e-4b25-a96d-a771ffc429a8" (UID: "f69baca9-270e-4b25-a96d-a771ffc429a8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:41:49.908371 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:49.908340 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f69baca9-270e-4b25-a96d-a771ffc429a8-tmp-dir" pod "f69baca9-270e-4b25-a96d-a771ffc429a8" (UID: "f69baca9-270e-4b25-a96d-a771ffc429a8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:41:49.908578 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:49.908560 4908 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f69baca9-270e-4b25-a96d-a771ffc429a8-model-cache\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:41:49.908651 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:49.908582 4908 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f69baca9-270e-4b25-a96d-a771ffc429a8-tmp-dir\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:41:49.908651 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:49.908596 4908 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f69baca9-270e-4b25-a96d-a771ffc429a8-home\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:41:49.910050 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:49.910025 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f69baca9-270e-4b25-a96d-a771ffc429a8-tls-certs" pod "f69baca9-270e-4b25-a96d-a771ffc429a8" (UID: "f69baca9-270e-4b25-a96d-a771ffc429a8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:41:49.910317 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:49.910301 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f69baca9-270e-4b25-a96d-a771ffc429a8-dshm" pod "f69baca9-270e-4b25-a96d-a771ffc429a8" (UID: "f69baca9-270e-4b25-a96d-a771ffc429a8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:41:49.910399 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:49.910333 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f69baca9-270e-4b25-a96d-a771ffc429a8-kube-api-access-zc8g2" pod "f69baca9-270e-4b25-a96d-a771ffc429a8" (UID: "f69baca9-270e-4b25-a96d-a771ffc429a8"). InnerVolumeSpecName "kube-api-access-zc8g2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:41:50.009417 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:50.009354 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zc8g2\" (UniqueName: \"kubernetes.io/projected/f69baca9-270e-4b25-a96d-a771ffc429a8-kube-api-access-zc8g2\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:41:50.009417 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:50.009379 4908 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f69baca9-270e-4b25-a96d-a771ffc429a8-tls-certs\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:41:50.009417 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:50.009387 4908 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f69baca9-270e-4b25-a96d-a771ffc429a8-dshm\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:41:50.595948 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:50.595910 4908 generic.go:356] "Generic (PLEG): container finished" podID="f69baca9-270e-4b25-a96d-a771ffc429a8" containerID="8c11f5fadeed0c1a30749fb2a7b31b36f774caf860cdfbefc2770c765061d2f8" exitCode=0 Oct 30 12:41:50.596448 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:50.595968 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-327e31/auth-test-model-327e31-kserve-587f6fcd8f-r8vm4" Oct 30 12:41:50.596448 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:50.595986 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-327e31/auth-test-model-327e31-kserve-587f6fcd8f-r8vm4" event={"ID":"f69baca9-270e-4b25-a96d-a771ffc429a8","Type":"ContainerDied","Data":"8c11f5fadeed0c1a30749fb2a7b31b36f774caf860cdfbefc2770c765061d2f8"} Oct 30 12:41:50.596448 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:50.596035 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-327e31/auth-test-model-327e31-kserve-587f6fcd8f-r8vm4" event={"ID":"f69baca9-270e-4b25-a96d-a771ffc429a8","Type":"ContainerDied","Data":"89ba5dbb380d8c684f342f2ad44455204f5d54f519b72c48ed74a1ac2bbf135f"} Oct 30 12:41:50.596448 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:50.596057 4908 scope.go:122] "RemoveContainer" containerID="8c11f5fadeed0c1a30749fb2a7b31b36f774caf860cdfbefc2770c765061d2f8" Oct 30 12:41:50.607305 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:50.607287 4908 scope.go:122] "RemoveContainer" containerID="8c11f5fadeed0c1a30749fb2a7b31b36f774caf860cdfbefc2770c765061d2f8" Oct 30 12:41:50.607642 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:41:50.607618 4908 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c11f5fadeed0c1a30749fb2a7b31b36f774caf860cdfbefc2770c765061d2f8\": container with ID starting with 8c11f5fadeed0c1a30749fb2a7b31b36f774caf860cdfbefc2770c765061d2f8 not found: ID does not exist" containerID="8c11f5fadeed0c1a30749fb2a7b31b36f774caf860cdfbefc2770c765061d2f8" Oct 30 12:41:50.607727 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:50.607649 4908 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c11f5fadeed0c1a30749fb2a7b31b36f774caf860cdfbefc2770c765061d2f8"} err="failed to get container status \"8c11f5fadeed0c1a30749fb2a7b31b36f774caf860cdfbefc2770c765061d2f8\": rpc error: code = NotFound desc = could not find container \"8c11f5fadeed0c1a30749fb2a7b31b36f774caf860cdfbefc2770c765061d2f8\": container with ID starting with 8c11f5fadeed0c1a30749fb2a7b31b36f774caf860cdfbefc2770c765061d2f8 not found: ID does not exist" Oct 30 12:41:50.647704 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:50.647681 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-327e31/auth-test-model-327e31-kserve-587f6fcd8f-r8vm4"] Oct 30 12:41:50.658396 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:50.658376 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w4-327e31/auth-test-model-327e31-kserve-587f6fcd8f-r8vm4"] Oct 30 12:41:51.618129 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:51.618094 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f69baca9-270e-4b25-a96d-a771ffc429a8" path="/var/lib/kubelet/pods/f69baca9-270e-4b25-a96d-a771ffc429a8/volumes" Oct 30 12:41:55.377709 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:55.377673 4908 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w4-8d2170/auth-test-model-8d2170-kserve-7b6547c869-5q7r5" podUID="ff555264-0d24-4bc2-9d62-82006845bb77" containerName="main" containerID="cri-o://9da6929b078acf015a15b8646c7a7e0b4e32efaf60e554d7e3f278e8d1cf6518" gracePeriod=45 Oct 30 12:41:55.651117 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:55.651087 4908 generic.go:356] "Generic (PLEG): container finished" podID="ff555264-0d24-4bc2-9d62-82006845bb77" containerID="9da6929b078acf015a15b8646c7a7e0b4e32efaf60e554d7e3f278e8d1cf6518" exitCode=0 Oct 30 12:41:55.651256 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:55.651157 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-8d2170/auth-test-model-8d2170-kserve-7b6547c869-5q7r5" event={"ID":"ff555264-0d24-4bc2-9d62-82006845bb77","Type":"ContainerDied","Data":"9da6929b078acf015a15b8646c7a7e0b4e32efaf60e554d7e3f278e8d1cf6518"} Oct 30 12:41:55.651256 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:55.651196 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-8d2170/auth-test-model-8d2170-kserve-7b6547c869-5q7r5" event={"ID":"ff555264-0d24-4bc2-9d62-82006845bb77","Type":"ContainerDied","Data":"e3f7526dba7b7a4fa22ac06e8e4042cef28c7b164543400f4271aaee161ead94"} Oct 30 12:41:55.651256 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:55.651207 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e3f7526dba7b7a4fa22ac06e8e4042cef28c7b164543400f4271aaee161ead94" Oct 30 12:41:55.656299 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:55.656277 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-8d2170/auth-test-model-8d2170-kserve-7b6547c869-5q7r5" Oct 30 12:41:55.774201 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:55.774173 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff555264-0d24-4bc2-9d62-82006845bb77-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff555264-0d24-4bc2-9d62-82006845bb77-model-cache\") pod \"ff555264-0d24-4bc2-9d62-82006845bb77\" (UID: \"ff555264-0d24-4bc2-9d62-82006845bb77\") " Oct 30 12:41:55.774383 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:55.774217 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff555264-0d24-4bc2-9d62-82006845bb77-home\" (UniqueName: \"kubernetes.io/empty-dir/ff555264-0d24-4bc2-9d62-82006845bb77-home\") pod \"ff555264-0d24-4bc2-9d62-82006845bb77\" (UID: \"ff555264-0d24-4bc2-9d62-82006845bb77\") " Oct 30 12:41:55.774383 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:55.774246 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff555264-0d24-4bc2-9d62-82006845bb77-kube-api-access-5xdct\" (UniqueName: \"kubernetes.io/projected/ff555264-0d24-4bc2-9d62-82006845bb77-kube-api-access-5xdct\") pod \"ff555264-0d24-4bc2-9d62-82006845bb77\" (UID: \"ff555264-0d24-4bc2-9d62-82006845bb77\") " Oct 30 12:41:55.774383 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:55.774286 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff555264-0d24-4bc2-9d62-82006845bb77-tls-certs\" (UniqueName: \"kubernetes.io/secret/ff555264-0d24-4bc2-9d62-82006845bb77-tls-certs\") pod \"ff555264-0d24-4bc2-9d62-82006845bb77\" (UID: \"ff555264-0d24-4bc2-9d62-82006845bb77\") " Oct 30 12:41:55.774574 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:55.774452 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff555264-0d24-4bc2-9d62-82006845bb77-model-cache" pod "ff555264-0d24-4bc2-9d62-82006845bb77" (UID: "ff555264-0d24-4bc2-9d62-82006845bb77"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:41:55.774574 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:55.774471 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff555264-0d24-4bc2-9d62-82006845bb77-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff555264-0d24-4bc2-9d62-82006845bb77-dshm\") pod \"ff555264-0d24-4bc2-9d62-82006845bb77\" (UID: \"ff555264-0d24-4bc2-9d62-82006845bb77\") " Oct 30 12:41:55.774574 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:55.774479 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff555264-0d24-4bc2-9d62-82006845bb77-home" pod "ff555264-0d24-4bc2-9d62-82006845bb77" (UID: "ff555264-0d24-4bc2-9d62-82006845bb77"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:41:55.774574 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:55.774506 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff555264-0d24-4bc2-9d62-82006845bb77-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff555264-0d24-4bc2-9d62-82006845bb77-tmp-dir\") pod \"ff555264-0d24-4bc2-9d62-82006845bb77\" (UID: \"ff555264-0d24-4bc2-9d62-82006845bb77\") " Oct 30 12:41:55.774785 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:55.774743 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff555264-0d24-4bc2-9d62-82006845bb77-tmp-dir" pod "ff555264-0d24-4bc2-9d62-82006845bb77" (UID: "ff555264-0d24-4bc2-9d62-82006845bb77"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:41:55.775147 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:55.775126 4908 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff555264-0d24-4bc2-9d62-82006845bb77-tmp-dir\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:41:55.775147 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:55.775145 4908 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff555264-0d24-4bc2-9d62-82006845bb77-model-cache\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:41:55.775325 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:55.775156 4908 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff555264-0d24-4bc2-9d62-82006845bb77-home\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:41:55.776644 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:55.776615 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff555264-0d24-4bc2-9d62-82006845bb77-kube-api-access-5xdct" pod "ff555264-0d24-4bc2-9d62-82006845bb77" (UID: "ff555264-0d24-4bc2-9d62-82006845bb77"). InnerVolumeSpecName "kube-api-access-5xdct". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:41:55.776789 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:55.776771 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff555264-0d24-4bc2-9d62-82006845bb77-tls-certs" pod "ff555264-0d24-4bc2-9d62-82006845bb77" (UID: "ff555264-0d24-4bc2-9d62-82006845bb77"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:41:55.776869 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:55.776855 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff555264-0d24-4bc2-9d62-82006845bb77-dshm" pod "ff555264-0d24-4bc2-9d62-82006845bb77" (UID: "ff555264-0d24-4bc2-9d62-82006845bb77"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:41:55.876360 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:55.876333 4908 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff555264-0d24-4bc2-9d62-82006845bb77-dshm\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:41:55.876360 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:55.876358 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xdct\" (UniqueName: \"kubernetes.io/projected/ff555264-0d24-4bc2-9d62-82006845bb77-kube-api-access-5xdct\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:41:55.876502 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:55.876369 4908 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff555264-0d24-4bc2-9d62-82006845bb77-tls-certs\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:41:56.630688 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:56.630648 4908 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-8d2170/auth-test-model-8d2170-kserve-7b6547c869-5q7r5" podUID="ff555264-0d24-4bc2-9d62-82006845bb77" containerName="main" probeResult="failure" output="Get \"https://10.132.0.213:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:41:56.658683 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:56.658659 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-8d2170/auth-test-model-8d2170-kserve-7b6547c869-5q7r5" Oct 30 12:41:56.708013 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:56.707985 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-8d2170/auth-test-model-8d2170-kserve-7b6547c869-5q7r5"] Oct 30 12:41:56.718428 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:56.718402 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w4-8d2170/auth-test-model-8d2170-kserve-7b6547c869-5q7r5"] Oct 30 12:41:57.608971 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:41:57.608941 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff555264-0d24-4bc2-9d62-82006845bb77" path="/var/lib/kubelet/pods/ff555264-0d24-4bc2-9d62-82006845bb77/volumes" Oct 30 12:42:06.080524 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:06.080482 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9wwfc"] Oct 30 12:42:07.548390 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:07.548356 4908 scope.go:122] "RemoveContainer" containerID="9c80293cd6952df5a1087f76b7ae664bb4011a5677d69af423338e81237f4d29" Oct 30 12:42:12.319248 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:12.319198 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-0f7d14-openshift-default-cf79d4dcc-mcrqq"] Oct 30 12:42:12.320637 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:12.320608 4908 cpu_manager.go:414] "containerMap: removing container" podUID="f69baca9-270e-4b25-a96d-a771ffc429a8" containerName="main" Oct 30 12:42:12.320800 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:12.320784 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f69baca9-270e-4b25-a96d-a771ffc429a8" containerName="main" Oct 30 12:42:12.320973 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:12.320960 4908 cpu_manager.go:414] "containerMap: removing container" podUID="ff555264-0d24-4bc2-9d62-82006845bb77" containerName="main" Oct 30 12:42:12.321069 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:12.321057 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff555264-0d24-4bc2-9d62-82006845bb77" containerName="main" Oct 30 12:42:12.321379 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:12.321361 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff555264-0d24-4bc2-9d62-82006845bb77" containerName="main" Oct 30 12:42:12.321503 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:12.321492 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="f69baca9-270e-4b25-a96d-a771ffc429a8" containerName="main" Oct 30 12:42:12.331849 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:12.331685 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-0f7d14-openshift-default-cf79d4dcc-mcrqq" Oct 30 12:42:12.346599 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:12.346049 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-0f7d14-openshift-default-dockercfg-wfqsh\"" Oct 30 12:42:12.424850 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:12.385041 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3590e8ff-89a3-482b-b022-064bb84fe09c-istio-podinfo\") pod \"e2e-extmod-b-0f7d14-openshift-default-cf79d4dcc-mcrqq\" (UID: \"3590e8ff-89a3-482b-b022-064bb84fe09c\") " pod="openshift-ingress/e2e-extmod-b-0f7d14-openshift-default-cf79d4dcc-mcrqq" Oct 30 12:42:12.441036 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:12.440997 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3590e8ff-89a3-482b-b022-064bb84fe09c-istiod-ca-cert\") pod \"e2e-extmod-b-0f7d14-openshift-default-cf79d4dcc-mcrqq\" (UID: \"3590e8ff-89a3-482b-b022-064bb84fe09c\") " pod="openshift-ingress/e2e-extmod-b-0f7d14-openshift-default-cf79d4dcc-mcrqq" Oct 30 12:42:12.441340 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:12.441320 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3590e8ff-89a3-482b-b022-064bb84fe09c-credential-socket\") pod \"e2e-extmod-b-0f7d14-openshift-default-cf79d4dcc-mcrqq\" (UID: \"3590e8ff-89a3-482b-b022-064bb84fe09c\") " pod="openshift-ingress/e2e-extmod-b-0f7d14-openshift-default-cf79d4dcc-mcrqq" Oct 30 12:42:12.441493 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:12.441478 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3590e8ff-89a3-482b-b022-064bb84fe09c-istio-token\") pod \"e2e-extmod-b-0f7d14-openshift-default-cf79d4dcc-mcrqq\" (UID: \"3590e8ff-89a3-482b-b022-064bb84fe09c\") " pod="openshift-ingress/e2e-extmod-b-0f7d14-openshift-default-cf79d4dcc-mcrqq" Oct 30 12:42:12.441652 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:12.441638 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xd8k\" (UniqueName: \"kubernetes.io/projected/3590e8ff-89a3-482b-b022-064bb84fe09c-kube-api-access-6xd8k\") pod \"e2e-extmod-b-0f7d14-openshift-default-cf79d4dcc-mcrqq\" (UID: \"3590e8ff-89a3-482b-b022-064bb84fe09c\") " pod="openshift-ingress/e2e-extmod-b-0f7d14-openshift-default-cf79d4dcc-mcrqq" Oct 30 12:42:12.441773 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:12.441761 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3590e8ff-89a3-482b-b022-064bb84fe09c-istio-envoy\") pod \"e2e-extmod-b-0f7d14-openshift-default-cf79d4dcc-mcrqq\" (UID: \"3590e8ff-89a3-482b-b022-064bb84fe09c\") " pod="openshift-ingress/e2e-extmod-b-0f7d14-openshift-default-cf79d4dcc-mcrqq" Oct 30 12:42:12.441966 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:12.441950 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3590e8ff-89a3-482b-b022-064bb84fe09c-istio-data\") pod \"e2e-extmod-b-0f7d14-openshift-default-cf79d4dcc-mcrqq\" (UID: \"3590e8ff-89a3-482b-b022-064bb84fe09c\") " pod="openshift-ingress/e2e-extmod-b-0f7d14-openshift-default-cf79d4dcc-mcrqq" Oct 30 12:42:12.442114 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:12.442097 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3590e8ff-89a3-482b-b022-064bb84fe09c-workload-certs\") pod \"e2e-extmod-b-0f7d14-openshift-default-cf79d4dcc-mcrqq\" (UID: \"3590e8ff-89a3-482b-b022-064bb84fe09c\") " pod="openshift-ingress/e2e-extmod-b-0f7d14-openshift-default-cf79d4dcc-mcrqq" Oct 30 12:42:12.442242 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:12.442229 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3590e8ff-89a3-482b-b022-064bb84fe09c-workload-socket\") pod \"e2e-extmod-b-0f7d14-openshift-default-cf79d4dcc-mcrqq\" (UID: \"3590e8ff-89a3-482b-b022-064bb84fe09c\") " pod="openshift-ingress/e2e-extmod-b-0f7d14-openshift-default-cf79d4dcc-mcrqq" Oct 30 12:42:12.442517 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:12.442501 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-0f7d14-openshift-default-cf79d4dcc-mcrqq"] Oct 30 12:42:12.549324 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:12.549243 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3590e8ff-89a3-482b-b022-064bb84fe09c-workload-certs\") pod \"e2e-extmod-b-0f7d14-openshift-default-cf79d4dcc-mcrqq\" (UID: \"3590e8ff-89a3-482b-b022-064bb84fe09c\") " pod="openshift-ingress/e2e-extmod-b-0f7d14-openshift-default-cf79d4dcc-mcrqq" Oct 30 12:42:12.557482 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:12.550944 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3590e8ff-89a3-482b-b022-064bb84fe09c-workload-socket\") pod \"e2e-extmod-b-0f7d14-openshift-default-cf79d4dcc-mcrqq\" (UID: \"3590e8ff-89a3-482b-b022-064bb84fe09c\") " pod="openshift-ingress/e2e-extmod-b-0f7d14-openshift-default-cf79d4dcc-mcrqq" Oct 30 12:42:12.558145 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:12.558121 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3590e8ff-89a3-482b-b022-064bb84fe09c-istio-podinfo\") pod \"e2e-extmod-b-0f7d14-openshift-default-cf79d4dcc-mcrqq\" (UID: \"3590e8ff-89a3-482b-b022-064bb84fe09c\") " pod="openshift-ingress/e2e-extmod-b-0f7d14-openshift-default-cf79d4dcc-mcrqq" Oct 30 12:42:12.558387 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:12.558371 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3590e8ff-89a3-482b-b022-064bb84fe09c-istiod-ca-cert\") pod \"e2e-extmod-b-0f7d14-openshift-default-cf79d4dcc-mcrqq\" (UID: \"3590e8ff-89a3-482b-b022-064bb84fe09c\") " pod="openshift-ingress/e2e-extmod-b-0f7d14-openshift-default-cf79d4dcc-mcrqq" Oct 30 12:42:12.572897 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:12.558545 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3590e8ff-89a3-482b-b022-064bb84fe09c-credential-socket\") pod \"e2e-extmod-b-0f7d14-openshift-default-cf79d4dcc-mcrqq\" (UID: \"3590e8ff-89a3-482b-b022-064bb84fe09c\") " pod="openshift-ingress/e2e-extmod-b-0f7d14-openshift-default-cf79d4dcc-mcrqq" Oct 30 12:42:12.574550 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:12.574524 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3590e8ff-89a3-482b-b022-064bb84fe09c-istio-token\") pod \"e2e-extmod-b-0f7d14-openshift-default-cf79d4dcc-mcrqq\" (UID: \"3590e8ff-89a3-482b-b022-064bb84fe09c\") " pod="openshift-ingress/e2e-extmod-b-0f7d14-openshift-default-cf79d4dcc-mcrqq" Oct 30 12:42:12.574800 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:12.574782 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xd8k\" (UniqueName: \"kubernetes.io/projected/3590e8ff-89a3-482b-b022-064bb84fe09c-kube-api-access-6xd8k\") pod \"e2e-extmod-b-0f7d14-openshift-default-cf79d4dcc-mcrqq\" (UID: \"3590e8ff-89a3-482b-b022-064bb84fe09c\") " pod="openshift-ingress/e2e-extmod-b-0f7d14-openshift-default-cf79d4dcc-mcrqq" Oct 30 12:42:12.574978 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:12.574961 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3590e8ff-89a3-482b-b022-064bb84fe09c-istio-envoy\") pod \"e2e-extmod-b-0f7d14-openshift-default-cf79d4dcc-mcrqq\" (UID: \"3590e8ff-89a3-482b-b022-064bb84fe09c\") " pod="openshift-ingress/e2e-extmod-b-0f7d14-openshift-default-cf79d4dcc-mcrqq" Oct 30 12:42:12.575142 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:12.575126 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3590e8ff-89a3-482b-b022-064bb84fe09c-istio-data\") pod \"e2e-extmod-b-0f7d14-openshift-default-cf79d4dcc-mcrqq\" (UID: \"3590e8ff-89a3-482b-b022-064bb84fe09c\") " pod="openshift-ingress/e2e-extmod-b-0f7d14-openshift-default-cf79d4dcc-mcrqq" Oct 30 12:42:12.595436 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:12.587251 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3590e8ff-89a3-482b-b022-064bb84fe09c-istio-data\") pod \"e2e-extmod-b-0f7d14-openshift-default-cf79d4dcc-mcrqq\" (UID: \"3590e8ff-89a3-482b-b022-064bb84fe09c\") " pod="openshift-ingress/e2e-extmod-b-0f7d14-openshift-default-cf79d4dcc-mcrqq" Oct 30 12:42:12.595641 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:12.558053 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3590e8ff-89a3-482b-b022-064bb84fe09c-workload-socket\") pod \"e2e-extmod-b-0f7d14-openshift-default-cf79d4dcc-mcrqq\" (UID: \"3590e8ff-89a3-482b-b022-064bb84fe09c\") " pod="openshift-ingress/e2e-extmod-b-0f7d14-openshift-default-cf79d4dcc-mcrqq" Oct 30 12:42:12.595755 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:12.574440 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3590e8ff-89a3-482b-b022-064bb84fe09c-istiod-ca-cert\") pod \"e2e-extmod-b-0f7d14-openshift-default-cf79d4dcc-mcrqq\" (UID: \"3590e8ff-89a3-482b-b022-064bb84fe09c\") " pod="openshift-ingress/e2e-extmod-b-0f7d14-openshift-default-cf79d4dcc-mcrqq" Oct 30 12:42:12.595888 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:12.573396 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3590e8ff-89a3-482b-b022-064bb84fe09c-credential-socket\") pod \"e2e-extmod-b-0f7d14-openshift-default-cf79d4dcc-mcrqq\" (UID: \"3590e8ff-89a3-482b-b022-064bb84fe09c\") " pod="openshift-ingress/e2e-extmod-b-0f7d14-openshift-default-cf79d4dcc-mcrqq" Oct 30 12:42:12.596000 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:12.550006 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3590e8ff-89a3-482b-b022-064bb84fe09c-workload-certs\") pod \"e2e-extmod-b-0f7d14-openshift-default-cf79d4dcc-mcrqq\" (UID: \"3590e8ff-89a3-482b-b022-064bb84fe09c\") " pod="openshift-ingress/e2e-extmod-b-0f7d14-openshift-default-cf79d4dcc-mcrqq" Oct 30 12:42:12.628519 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:12.628098 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3590e8ff-89a3-482b-b022-064bb84fe09c-istio-envoy\") pod \"e2e-extmod-b-0f7d14-openshift-default-cf79d4dcc-mcrqq\" (UID: \"3590e8ff-89a3-482b-b022-064bb84fe09c\") " pod="openshift-ingress/e2e-extmod-b-0f7d14-openshift-default-cf79d4dcc-mcrqq" Oct 30 12:42:12.632644 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:12.632611 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3590e8ff-89a3-482b-b022-064bb84fe09c-istio-podinfo\") pod \"e2e-extmod-b-0f7d14-openshift-default-cf79d4dcc-mcrqq\" (UID: \"3590e8ff-89a3-482b-b022-064bb84fe09c\") " pod="openshift-ingress/e2e-extmod-b-0f7d14-openshift-default-cf79d4dcc-mcrqq" Oct 30 12:42:12.634874 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:12.634808 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xd8k\" (UniqueName: \"kubernetes.io/projected/3590e8ff-89a3-482b-b022-064bb84fe09c-kube-api-access-6xd8k\") pod \"e2e-extmod-b-0f7d14-openshift-default-cf79d4dcc-mcrqq\" (UID: \"3590e8ff-89a3-482b-b022-064bb84fe09c\") " pod="openshift-ingress/e2e-extmod-b-0f7d14-openshift-default-cf79d4dcc-mcrqq" Oct 30 12:42:12.635406 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:12.635384 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3590e8ff-89a3-482b-b022-064bb84fe09c-istio-token\") pod \"e2e-extmod-b-0f7d14-openshift-default-cf79d4dcc-mcrqq\" (UID: \"3590e8ff-89a3-482b-b022-064bb84fe09c\") " pod="openshift-ingress/e2e-extmod-b-0f7d14-openshift-default-cf79d4dcc-mcrqq" Oct 30 12:42:12.650252 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:12.650222 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-0f7d14-openshift-default-cf79d4dcc-mcrqq" Oct 30 12:42:13.247893 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:42:13.247861 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3590e8ff_89a3_482b_b022_064bb84fe09c.slice/crio-7049a1b39cdfedca988893c5706732cfbefa6743380d37bce0de7c0734546040 WatchSource:0}: Error finding container 7049a1b39cdfedca988893c5706732cfbefa6743380d37bce0de7c0734546040: Status 404 returned error can't find the container with id 7049a1b39cdfedca988893c5706732cfbefa6743380d37bce0de7c0734546040 Oct 30 12:42:13.248842 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:13.248800 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-0f7d14-openshift-default-cf79d4dcc-mcrqq"] Oct 30 12:42:13.250672 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:13.250608 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:42:13.250792 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:13.250678 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:42:13.250792 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:13.250720 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:42:13.894029 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:13.893985 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-0f7d14-openshift-default-cf79d4dcc-mcrqq" event={"ID":"3590e8ff-89a3-482b-b022-064bb84fe09c","Type":"ContainerStarted","Data":"e9795be9cfa47e1d277b621a66ccf1c4f0749ec3581d6f56ac16ca0e760718f7"} Oct 30 12:42:13.894489 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:13.894048 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-0f7d14-openshift-default-cf79d4dcc-mcrqq" event={"ID":"3590e8ff-89a3-482b-b022-064bb84fe09c","Type":"ContainerStarted","Data":"7049a1b39cdfedca988893c5706732cfbefa6743380d37bce0de7c0734546040"} Oct 30 12:42:13.954719 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:13.954662 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-0f7d14-openshift-default-cf79d4dcc-mcrqq" podStartSLOduration=1.954643213 podStartE2EDuration="1.954643213s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:42:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:42:13.947544695 +0000 UTC m=+2890.461973441" watchObservedRunningTime="2026-10-30 12:42:13.954643213 +0000 UTC m=+2890.469071959" Oct 30 12:42:14.651034 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:14.651002 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-0f7d14-openshift-default-cf79d4dcc-mcrqq" Oct 30 12:42:14.658255 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:14.658229 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-0f7d14-openshift-default-cf79d4dcc-mcrqq" Oct 30 12:42:14.903026 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:14.902621 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-0f7d14-openshift-default-cf79d4dcc-mcrqq" Oct 30 12:42:14.903594 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:14.903575 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-0f7d14-openshift-default-cf79d4dcc-mcrqq" Oct 30 12:42:17.930091 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:17.930015 4908 generic.go:356] "Generic (PLEG): container finished" podID="136fdf42-e3f2-4e4a-83de-a2482e084e5b" containerID="0885278d558403846e921e2da33c6a94cc6ac2b4ae2d0710882eaa03c1a72b2c" exitCode=0 Oct 30 12:42:17.930521 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:17.930083 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-2545c7-openshift-default-69cf958465-vb4sg" event={"ID":"136fdf42-e3f2-4e4a-83de-a2482e084e5b","Type":"ContainerDied","Data":"0885278d558403846e921e2da33c6a94cc6ac2b4ae2d0710882eaa03c1a72b2c"} Oct 30 12:42:17.930521 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:17.930443 4908 scope.go:122] "RemoveContainer" containerID="0885278d558403846e921e2da33c6a94cc6ac2b4ae2d0710882eaa03c1a72b2c" Oct 30 12:42:17.932075 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:17.932052 4908 generic.go:356] "Generic (PLEG): container finished" podID="20af951f-19a7-49e6-ab1d-3bdee930084d" containerID="134480ee395452bba7c2b489dee184e5ec1afcfd6618812b3430bd3479a010d8" exitCode=0 Oct 30 12:42:17.932185 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:17.932095 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a786af-openshift-default-f89f5ffc7-b62z2" event={"ID":"20af951f-19a7-49e6-ab1d-3bdee930084d","Type":"ContainerDied","Data":"134480ee395452bba7c2b489dee184e5ec1afcfd6618812b3430bd3479a010d8"} Oct 30 12:42:17.932315 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:17.932284 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:42:17.932381 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:17.932368 4908 scope.go:122] "RemoveContainer" containerID="134480ee395452bba7c2b489dee184e5ec1afcfd6618812b3430bd3479a010d8" Oct 30 12:42:17.932420 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:17.932375 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:42:17.932453 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:17.932420 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:42:17.933741 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:17.933711 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:42:17.933845 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:17.933777 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:42:17.933845 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:17.933817 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:42:18.627592 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:18.627557 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-a786af-openshift-default-f89f5ffc7-b62z2" Oct 30 12:42:18.946101 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:18.945995 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a786af-openshift-default-f89f5ffc7-b62z2" event={"ID":"20af951f-19a7-49e6-ab1d-3bdee930084d","Type":"ContainerStarted","Data":"1944c701fba206b3f327f0931743a5746733758cced662ffe3dc1b778b05a403"} Oct 30 12:42:18.956452 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:18.955886 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-2545c7-openshift-default-69cf958465-vb4sg" event={"ID":"136fdf42-e3f2-4e4a-83de-a2482e084e5b","Type":"ContainerStarted","Data":"6f0bc4f496d801fc0d5151f7869ad45663818f6e21061d36165dfe95e4a3636e"} Oct 30 12:42:18.982203 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:18.982171 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-a786af-openshift-default-f89f5ffc7-b62z2" Oct 30 12:42:19.375848 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:19.374933 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-2545c7-openshift-default-69cf958465-vb4sg" Oct 30 12:42:19.375848 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:19.375064 4908 patch_prober.go:29] interesting pod/e2e-extmod-a-2545c7-openshift-default-69cf958465-vb4sg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" start-of-body= Oct 30 12:42:19.375848 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:19.375106 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-2545c7-openshift-default-69cf958465-vb4sg" podUID="136fdf42-e3f2-4e4a-83de-a2482e084e5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" Oct 30 12:42:19.757802 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:42:19.757764 4908 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vn6vv" containerName="extractor" Oct 30 12:42:19.979850 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:19.975593 4908 patch_prober.go:29] interesting pod/e2e-extmod-a-a786af-openshift-default-f89f5ffc7-b62z2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" start-of-body= Oct 30 12:42:19.979850 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:19.975647 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a786af-openshift-default-f89f5ffc7-b62z2" podUID="20af951f-19a7-49e6-ab1d-3bdee930084d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" Oct 30 12:42:20.384850 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:20.382627 4908 patch_prober.go:29] interesting pod/e2e-extmod-a-2545c7-openshift-default-69cf958465-vb4sg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" start-of-body= Oct 30 12:42:20.384850 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:20.382712 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-2545c7-openshift-default-69cf958465-vb4sg" podUID="136fdf42-e3f2-4e4a-83de-a2482e084e5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" Oct 30 12:42:20.999853 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:20.989605 4908 patch_prober.go:29] interesting pod/e2e-extmod-a-a786af-openshift-default-f89f5ffc7-b62z2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" start-of-body= Oct 30 12:42:20.999853 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:20.989660 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a786af-openshift-default-f89f5ffc7-b62z2" podUID="20af951f-19a7-49e6-ab1d-3bdee930084d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" Oct 30 12:42:21.377918 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:21.377210 4908 patch_prober.go:29] interesting pod/e2e-extmod-a-2545c7-openshift-default-69cf958465-vb4sg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" start-of-body= Oct 30 12:42:21.377918 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:21.377272 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-2545c7-openshift-default-69cf958465-vb4sg" podUID="136fdf42-e3f2-4e4a-83de-a2482e084e5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" Oct 30 12:42:21.977853 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:21.975114 4908 patch_prober.go:29] interesting pod/e2e-extmod-a-a786af-openshift-default-f89f5ffc7-b62z2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" start-of-body= Oct 30 12:42:21.977853 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:21.975172 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a786af-openshift-default-f89f5ffc7-b62z2" podUID="20af951f-19a7-49e6-ab1d-3bdee930084d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" Oct 30 12:42:22.376902 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:22.376791 4908 patch_prober.go:29] interesting pod/e2e-extmod-a-2545c7-openshift-default-69cf958465-vb4sg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" start-of-body= Oct 30 12:42:22.377472 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:22.377423 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-2545c7-openshift-default-69cf958465-vb4sg" podUID="136fdf42-e3f2-4e4a-83de-a2482e084e5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" Oct 30 12:42:22.508848 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:22.492457 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-0de4b9-openshift-default-5d57d77697-g7ct6"] Oct 30 12:42:22.590851 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:22.588072 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-0de4b9-openshift-default-5d57d77697-g7ct6"] Oct 30 12:42:22.590851 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:22.588225 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-0de4b9-openshift-default-5d57d77697-g7ct6" Oct 30 12:42:22.654571 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:22.619557 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-0de4b9-openshift-default-dockercfg-59pps\"" Oct 30 12:42:22.784856 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:22.784791 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfd62d0e-d1b7-4695-8673-63c1d68f4f40-istio-token\") pod \"e2e-extmod-b-0de4b9-openshift-default-5d57d77697-g7ct6\" (UID: \"cfd62d0e-d1b7-4695-8673-63c1d68f4f40\") " pod="openshift-ingress/e2e-extmod-b-0de4b9-openshift-default-5d57d77697-g7ct6" Oct 30 12:42:22.785048 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:22.784871 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfd62d0e-d1b7-4695-8673-63c1d68f4f40-istiod-ca-cert\") pod \"e2e-extmod-b-0de4b9-openshift-default-5d57d77697-g7ct6\" (UID: \"cfd62d0e-d1b7-4695-8673-63c1d68f4f40\") " pod="openshift-ingress/e2e-extmod-b-0de4b9-openshift-default-5d57d77697-g7ct6" Oct 30 12:42:22.785120 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:22.785069 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65qjb\" (UniqueName: \"kubernetes.io/projected/cfd62d0e-d1b7-4695-8673-63c1d68f4f40-kube-api-access-65qjb\") pod \"e2e-extmod-b-0de4b9-openshift-default-5d57d77697-g7ct6\" (UID: \"cfd62d0e-d1b7-4695-8673-63c1d68f4f40\") " pod="openshift-ingress/e2e-extmod-b-0de4b9-openshift-default-5d57d77697-g7ct6" Oct 30 12:42:22.805849 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:22.785223 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfd62d0e-d1b7-4695-8673-63c1d68f4f40-istio-data\") pod \"e2e-extmod-b-0de4b9-openshift-default-5d57d77697-g7ct6\" (UID: \"cfd62d0e-d1b7-4695-8673-63c1d68f4f40\") " pod="openshift-ingress/e2e-extmod-b-0de4b9-openshift-default-5d57d77697-g7ct6" Oct 30 12:42:22.805849 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:22.785272 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfd62d0e-d1b7-4695-8673-63c1d68f4f40-istio-podinfo\") pod \"e2e-extmod-b-0de4b9-openshift-default-5d57d77697-g7ct6\" (UID: \"cfd62d0e-d1b7-4695-8673-63c1d68f4f40\") " pod="openshift-ingress/e2e-extmod-b-0de4b9-openshift-default-5d57d77697-g7ct6" Oct 30 12:42:22.805849 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:22.785461 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfd62d0e-d1b7-4695-8673-63c1d68f4f40-credential-socket\") pod \"e2e-extmod-b-0de4b9-openshift-default-5d57d77697-g7ct6\" (UID: \"cfd62d0e-d1b7-4695-8673-63c1d68f4f40\") " pod="openshift-ingress/e2e-extmod-b-0de4b9-openshift-default-5d57d77697-g7ct6" Oct 30 12:42:22.805849 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:22.785500 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfd62d0e-d1b7-4695-8673-63c1d68f4f40-workload-socket\") pod \"e2e-extmod-b-0de4b9-openshift-default-5d57d77697-g7ct6\" (UID: \"cfd62d0e-d1b7-4695-8673-63c1d68f4f40\") " pod="openshift-ingress/e2e-extmod-b-0de4b9-openshift-default-5d57d77697-g7ct6" Oct 30 12:42:22.805849 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:22.785531 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfd62d0e-d1b7-4695-8673-63c1d68f4f40-workload-certs\") pod \"e2e-extmod-b-0de4b9-openshift-default-5d57d77697-g7ct6\" (UID: \"cfd62d0e-d1b7-4695-8673-63c1d68f4f40\") " pod="openshift-ingress/e2e-extmod-b-0de4b9-openshift-default-5d57d77697-g7ct6" Oct 30 12:42:22.805849 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:22.785637 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfd62d0e-d1b7-4695-8673-63c1d68f4f40-istio-envoy\") pod \"e2e-extmod-b-0de4b9-openshift-default-5d57d77697-g7ct6\" (UID: \"cfd62d0e-d1b7-4695-8673-63c1d68f4f40\") " pod="openshift-ingress/e2e-extmod-b-0de4b9-openshift-default-5d57d77697-g7ct6" Oct 30 12:42:22.969896 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:22.934943 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfd62d0e-d1b7-4695-8673-63c1d68f4f40-istio-envoy\") pod \"e2e-extmod-b-0de4b9-openshift-default-5d57d77697-g7ct6\" (UID: \"cfd62d0e-d1b7-4695-8673-63c1d68f4f40\") " pod="openshift-ingress/e2e-extmod-b-0de4b9-openshift-default-5d57d77697-g7ct6" Oct 30 12:42:22.970741 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:22.970708 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfd62d0e-d1b7-4695-8673-63c1d68f4f40-istio-token\") pod \"e2e-extmod-b-0de4b9-openshift-default-5d57d77697-g7ct6\" (UID: \"cfd62d0e-d1b7-4695-8673-63c1d68f4f40\") " pod="openshift-ingress/e2e-extmod-b-0de4b9-openshift-default-5d57d77697-g7ct6" Oct 30 12:42:22.970973 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:22.970953 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfd62d0e-d1b7-4695-8673-63c1d68f4f40-istiod-ca-cert\") pod \"e2e-extmod-b-0de4b9-openshift-default-5d57d77697-g7ct6\" (UID: \"cfd62d0e-d1b7-4695-8673-63c1d68f4f40\") " pod="openshift-ingress/e2e-extmod-b-0de4b9-openshift-default-5d57d77697-g7ct6" Oct 30 12:42:22.971158 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:22.971131 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65qjb\" (UniqueName: \"kubernetes.io/projected/cfd62d0e-d1b7-4695-8673-63c1d68f4f40-kube-api-access-65qjb\") pod \"e2e-extmod-b-0de4b9-openshift-default-5d57d77697-g7ct6\" (UID: \"cfd62d0e-d1b7-4695-8673-63c1d68f4f40\") " pod="openshift-ingress/e2e-extmod-b-0de4b9-openshift-default-5d57d77697-g7ct6" Oct 30 12:42:22.971413 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:22.971396 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfd62d0e-d1b7-4695-8673-63c1d68f4f40-istio-data\") pod \"e2e-extmod-b-0de4b9-openshift-default-5d57d77697-g7ct6\" (UID: \"cfd62d0e-d1b7-4695-8673-63c1d68f4f40\") " pod="openshift-ingress/e2e-extmod-b-0de4b9-openshift-default-5d57d77697-g7ct6" Oct 30 12:42:22.971553 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:22.971534 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfd62d0e-d1b7-4695-8673-63c1d68f4f40-istio-podinfo\") pod \"e2e-extmod-b-0de4b9-openshift-default-5d57d77697-g7ct6\" (UID: \"cfd62d0e-d1b7-4695-8673-63c1d68f4f40\") " pod="openshift-ingress/e2e-extmod-b-0de4b9-openshift-default-5d57d77697-g7ct6" Oct 30 12:42:22.993303 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:22.972416 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfd62d0e-d1b7-4695-8673-63c1d68f4f40-istiod-ca-cert\") pod \"e2e-extmod-b-0de4b9-openshift-default-5d57d77697-g7ct6\" (UID: \"cfd62d0e-d1b7-4695-8673-63c1d68f4f40\") " pod="openshift-ingress/e2e-extmod-b-0de4b9-openshift-default-5d57d77697-g7ct6" Oct 30 12:42:22.993470 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:22.973293 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfd62d0e-d1b7-4695-8673-63c1d68f4f40-istio-data\") pod \"e2e-extmod-b-0de4b9-openshift-default-5d57d77697-g7ct6\" (UID: \"cfd62d0e-d1b7-4695-8673-63c1d68f4f40\") " pod="openshift-ingress/e2e-extmod-b-0de4b9-openshift-default-5d57d77697-g7ct6" Oct 30 12:42:22.993470 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:22.976128 4908 patch_prober.go:29] interesting pod/e2e-extmod-a-a786af-openshift-default-f89f5ffc7-b62z2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" start-of-body= Oct 30 12:42:22.993470 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:22.993410 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a786af-openshift-default-f89f5ffc7-b62z2" podUID="20af951f-19a7-49e6-ab1d-3bdee930084d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" Oct 30 12:42:22.993470 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:22.992279 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfd62d0e-d1b7-4695-8673-63c1d68f4f40-istio-envoy\") pod \"e2e-extmod-b-0de4b9-openshift-default-5d57d77697-g7ct6\" (UID: \"cfd62d0e-d1b7-4695-8673-63c1d68f4f40\") " pod="openshift-ingress/e2e-extmod-b-0de4b9-openshift-default-5d57d77697-g7ct6" Oct 30 12:42:22.993470 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:22.993279 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfd62d0e-d1b7-4695-8673-63c1d68f4f40-credential-socket\") pod \"e2e-extmod-b-0de4b9-openshift-default-5d57d77697-g7ct6\" (UID: \"cfd62d0e-d1b7-4695-8673-63c1d68f4f40\") " pod="openshift-ingress/e2e-extmod-b-0de4b9-openshift-default-5d57d77697-g7ct6" Oct 30 12:42:22.993739 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:22.993514 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfd62d0e-d1b7-4695-8673-63c1d68f4f40-workload-socket\") pod \"e2e-extmod-b-0de4b9-openshift-default-5d57d77697-g7ct6\" (UID: \"cfd62d0e-d1b7-4695-8673-63c1d68f4f40\") " pod="openshift-ingress/e2e-extmod-b-0de4b9-openshift-default-5d57d77697-g7ct6" Oct 30 12:42:22.993739 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:22.993541 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfd62d0e-d1b7-4695-8673-63c1d68f4f40-workload-certs\") pod \"e2e-extmod-b-0de4b9-openshift-default-5d57d77697-g7ct6\" (UID: \"cfd62d0e-d1b7-4695-8673-63c1d68f4f40\") " pod="openshift-ingress/e2e-extmod-b-0de4b9-openshift-default-5d57d77697-g7ct6" Oct 30 12:42:23.025851 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:22.993986 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfd62d0e-d1b7-4695-8673-63c1d68f4f40-workload-certs\") pod \"e2e-extmod-b-0de4b9-openshift-default-5d57d77697-g7ct6\" (UID: \"cfd62d0e-d1b7-4695-8673-63c1d68f4f40\") " pod="openshift-ingress/e2e-extmod-b-0de4b9-openshift-default-5d57d77697-g7ct6" Oct 30 12:42:23.025851 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:22.992116 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfd62d0e-d1b7-4695-8673-63c1d68f4f40-istio-podinfo\") pod \"e2e-extmod-b-0de4b9-openshift-default-5d57d77697-g7ct6\" (UID: \"cfd62d0e-d1b7-4695-8673-63c1d68f4f40\") " pod="openshift-ingress/e2e-extmod-b-0de4b9-openshift-default-5d57d77697-g7ct6" Oct 30 12:42:23.025851 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:22.994295 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfd62d0e-d1b7-4695-8673-63c1d68f4f40-workload-socket\") pod \"e2e-extmod-b-0de4b9-openshift-default-5d57d77697-g7ct6\" (UID: \"cfd62d0e-d1b7-4695-8673-63c1d68f4f40\") " pod="openshift-ingress/e2e-extmod-b-0de4b9-openshift-default-5d57d77697-g7ct6" Oct 30 12:42:23.025851 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:22.995016 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfd62d0e-d1b7-4695-8673-63c1d68f4f40-credential-socket\") pod \"e2e-extmod-b-0de4b9-openshift-default-5d57d77697-g7ct6\" (UID: \"cfd62d0e-d1b7-4695-8673-63c1d68f4f40\") " pod="openshift-ingress/e2e-extmod-b-0de4b9-openshift-default-5d57d77697-g7ct6" Oct 30 12:42:23.025851 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:22.997659 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65qjb\" (UniqueName: \"kubernetes.io/projected/cfd62d0e-d1b7-4695-8673-63c1d68f4f40-kube-api-access-65qjb\") pod \"e2e-extmod-b-0de4b9-openshift-default-5d57d77697-g7ct6\" (UID: \"cfd62d0e-d1b7-4695-8673-63c1d68f4f40\") " pod="openshift-ingress/e2e-extmod-b-0de4b9-openshift-default-5d57d77697-g7ct6" Oct 30 12:42:23.086856 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:23.076817 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfd62d0e-d1b7-4695-8673-63c1d68f4f40-istio-token\") pod \"e2e-extmod-b-0de4b9-openshift-default-5d57d77697-g7ct6\" (UID: \"cfd62d0e-d1b7-4695-8673-63c1d68f4f40\") " pod="openshift-ingress/e2e-extmod-b-0de4b9-openshift-default-5d57d77697-g7ct6" Oct 30 12:42:23.097861 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:23.094528 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-2545c7-openshift-default-69cf958465-vb4sg" Oct 30 12:42:23.208852 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:23.207191 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-0de4b9-openshift-default-5d57d77697-g7ct6" Oct 30 12:42:23.376874 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:23.373566 4908 patch_prober.go:29] interesting pod/e2e-extmod-a-2545c7-openshift-default-69cf958465-vb4sg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" start-of-body= Oct 30 12:42:23.376874 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:23.373639 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-2545c7-openshift-default-69cf958465-vb4sg" podUID="136fdf42-e3f2-4e4a-83de-a2482e084e5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" Oct 30 12:42:23.406242 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:23.406207 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-12fd33-openshift-default-846b7dbb9-pt6kj"] Oct 30 12:42:23.480808 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:23.480781 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-12fd33-openshift-default-846b7dbb9-pt6kj" Oct 30 12:42:23.499300 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:23.499270 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-12fd33-openshift-default-dockercfg-jkq9w\"" Oct 30 12:42:23.654551 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:23.654459 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/396046e4-b596-4fed-943d-a5f42fea4f77-istio-envoy\") pod \"e2e-extmod-b-12fd33-openshift-default-846b7dbb9-pt6kj\" (UID: \"396046e4-b596-4fed-943d-a5f42fea4f77\") " pod="openshift-ingress/e2e-extmod-b-12fd33-openshift-default-846b7dbb9-pt6kj" Oct 30 12:42:23.654817 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:23.654786 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/396046e4-b596-4fed-943d-a5f42fea4f77-istiod-ca-cert\") pod \"e2e-extmod-b-12fd33-openshift-default-846b7dbb9-pt6kj\" (UID: \"396046e4-b596-4fed-943d-a5f42fea4f77\") " pod="openshift-ingress/e2e-extmod-b-12fd33-openshift-default-846b7dbb9-pt6kj" Oct 30 12:42:23.654957 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:23.654936 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/396046e4-b596-4fed-943d-a5f42fea4f77-istio-data\") pod \"e2e-extmod-b-12fd33-openshift-default-846b7dbb9-pt6kj\" (UID: \"396046e4-b596-4fed-943d-a5f42fea4f77\") " pod="openshift-ingress/e2e-extmod-b-12fd33-openshift-default-846b7dbb9-pt6kj" Oct 30 12:42:23.655111 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:23.655095 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69z66\" (UniqueName: \"kubernetes.io/projected/396046e4-b596-4fed-943d-a5f42fea4f77-kube-api-access-69z66\") pod \"e2e-extmod-b-12fd33-openshift-default-846b7dbb9-pt6kj\" (UID: \"396046e4-b596-4fed-943d-a5f42fea4f77\") " pod="openshift-ingress/e2e-extmod-b-12fd33-openshift-default-846b7dbb9-pt6kj" Oct 30 12:42:23.655316 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:23.655294 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/396046e4-b596-4fed-943d-a5f42fea4f77-workload-socket\") pod \"e2e-extmod-b-12fd33-openshift-default-846b7dbb9-pt6kj\" (UID: \"396046e4-b596-4fed-943d-a5f42fea4f77\") " pod="openshift-ingress/e2e-extmod-b-12fd33-openshift-default-846b7dbb9-pt6kj" Oct 30 12:42:23.655452 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:23.655432 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/396046e4-b596-4fed-943d-a5f42fea4f77-istio-podinfo\") pod \"e2e-extmod-b-12fd33-openshift-default-846b7dbb9-pt6kj\" (UID: \"396046e4-b596-4fed-943d-a5f42fea4f77\") " pod="openshift-ingress/e2e-extmod-b-12fd33-openshift-default-846b7dbb9-pt6kj" Oct 30 12:42:23.655585 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:23.655562 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/396046e4-b596-4fed-943d-a5f42fea4f77-workload-certs\") pod \"e2e-extmod-b-12fd33-openshift-default-846b7dbb9-pt6kj\" (UID: \"396046e4-b596-4fed-943d-a5f42fea4f77\") " pod="openshift-ingress/e2e-extmod-b-12fd33-openshift-default-846b7dbb9-pt6kj" Oct 30 12:42:23.655691 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:23.655677 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/396046e4-b596-4fed-943d-a5f42fea4f77-credential-socket\") pod \"e2e-extmod-b-12fd33-openshift-default-846b7dbb9-pt6kj\" (UID: \"396046e4-b596-4fed-943d-a5f42fea4f77\") " pod="openshift-ingress/e2e-extmod-b-12fd33-openshift-default-846b7dbb9-pt6kj" Oct 30 12:42:23.655846 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:23.655810 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/396046e4-b596-4fed-943d-a5f42fea4f77-istio-token\") pod \"e2e-extmod-b-12fd33-openshift-default-846b7dbb9-pt6kj\" (UID: \"396046e4-b596-4fed-943d-a5f42fea4f77\") " pod="openshift-ingress/e2e-extmod-b-12fd33-openshift-default-846b7dbb9-pt6kj" Oct 30 12:42:23.706873 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:23.697245 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-12fd33-openshift-default-846b7dbb9-pt6kj"] Oct 30 12:42:23.757494 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:23.757463 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/396046e4-b596-4fed-943d-a5f42fea4f77-istio-envoy\") pod \"e2e-extmod-b-12fd33-openshift-default-846b7dbb9-pt6kj\" (UID: \"396046e4-b596-4fed-943d-a5f42fea4f77\") " pod="openshift-ingress/e2e-extmod-b-12fd33-openshift-default-846b7dbb9-pt6kj" Oct 30 12:42:23.757708 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:23.757695 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/396046e4-b596-4fed-943d-a5f42fea4f77-istiod-ca-cert\") pod \"e2e-extmod-b-12fd33-openshift-default-846b7dbb9-pt6kj\" (UID: \"396046e4-b596-4fed-943d-a5f42fea4f77\") " pod="openshift-ingress/e2e-extmod-b-12fd33-openshift-default-846b7dbb9-pt6kj" Oct 30 12:42:23.757782 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:23.757773 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/396046e4-b596-4fed-943d-a5f42fea4f77-istio-data\") pod \"e2e-extmod-b-12fd33-openshift-default-846b7dbb9-pt6kj\" (UID: \"396046e4-b596-4fed-943d-a5f42fea4f77\") " pod="openshift-ingress/e2e-extmod-b-12fd33-openshift-default-846b7dbb9-pt6kj" Oct 30 12:42:23.757870 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:23.757859 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69z66\" (UniqueName: \"kubernetes.io/projected/396046e4-b596-4fed-943d-a5f42fea4f77-kube-api-access-69z66\") pod \"e2e-extmod-b-12fd33-openshift-default-846b7dbb9-pt6kj\" (UID: \"396046e4-b596-4fed-943d-a5f42fea4f77\") " pod="openshift-ingress/e2e-extmod-b-12fd33-openshift-default-846b7dbb9-pt6kj" Oct 30 12:42:23.758049 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:23.758038 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/396046e4-b596-4fed-943d-a5f42fea4f77-workload-socket\") pod \"e2e-extmod-b-12fd33-openshift-default-846b7dbb9-pt6kj\" (UID: \"396046e4-b596-4fed-943d-a5f42fea4f77\") " pod="openshift-ingress/e2e-extmod-b-12fd33-openshift-default-846b7dbb9-pt6kj" Oct 30 12:42:23.758122 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:23.758113 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/396046e4-b596-4fed-943d-a5f42fea4f77-istio-podinfo\") pod \"e2e-extmod-b-12fd33-openshift-default-846b7dbb9-pt6kj\" (UID: \"396046e4-b596-4fed-943d-a5f42fea4f77\") " pod="openshift-ingress/e2e-extmod-b-12fd33-openshift-default-846b7dbb9-pt6kj" Oct 30 12:42:23.758186 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:23.758178 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/396046e4-b596-4fed-943d-a5f42fea4f77-workload-certs\") pod \"e2e-extmod-b-12fd33-openshift-default-846b7dbb9-pt6kj\" (UID: \"396046e4-b596-4fed-943d-a5f42fea4f77\") " pod="openshift-ingress/e2e-extmod-b-12fd33-openshift-default-846b7dbb9-pt6kj" Oct 30 12:42:23.758245 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:23.758236 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/396046e4-b596-4fed-943d-a5f42fea4f77-credential-socket\") pod \"e2e-extmod-b-12fd33-openshift-default-846b7dbb9-pt6kj\" (UID: \"396046e4-b596-4fed-943d-a5f42fea4f77\") " pod="openshift-ingress/e2e-extmod-b-12fd33-openshift-default-846b7dbb9-pt6kj" Oct 30 12:42:23.758305 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:23.758296 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/396046e4-b596-4fed-943d-a5f42fea4f77-istio-token\") pod \"e2e-extmod-b-12fd33-openshift-default-846b7dbb9-pt6kj\" (UID: \"396046e4-b596-4fed-943d-a5f42fea4f77\") " pod="openshift-ingress/e2e-extmod-b-12fd33-openshift-default-846b7dbb9-pt6kj" Oct 30 12:42:23.761885 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:23.759243 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/396046e4-b596-4fed-943d-a5f42fea4f77-workload-socket\") pod \"e2e-extmod-b-12fd33-openshift-default-846b7dbb9-pt6kj\" (UID: \"396046e4-b596-4fed-943d-a5f42fea4f77\") " pod="openshift-ingress/e2e-extmod-b-12fd33-openshift-default-846b7dbb9-pt6kj" Oct 30 12:42:23.761885 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:23.759765 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/396046e4-b596-4fed-943d-a5f42fea4f77-workload-certs\") pod \"e2e-extmod-b-12fd33-openshift-default-846b7dbb9-pt6kj\" (UID: \"396046e4-b596-4fed-943d-a5f42fea4f77\") " pod="openshift-ingress/e2e-extmod-b-12fd33-openshift-default-846b7dbb9-pt6kj" Oct 30 12:42:23.761885 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:23.760155 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/396046e4-b596-4fed-943d-a5f42fea4f77-istio-data\") pod \"e2e-extmod-b-12fd33-openshift-default-846b7dbb9-pt6kj\" (UID: \"396046e4-b596-4fed-943d-a5f42fea4f77\") " pod="openshift-ingress/e2e-extmod-b-12fd33-openshift-default-846b7dbb9-pt6kj" Oct 30 12:42:23.761885 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:23.760633 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/396046e4-b596-4fed-943d-a5f42fea4f77-credential-socket\") pod \"e2e-extmod-b-12fd33-openshift-default-846b7dbb9-pt6kj\" (UID: \"396046e4-b596-4fed-943d-a5f42fea4f77\") " pod="openshift-ingress/e2e-extmod-b-12fd33-openshift-default-846b7dbb9-pt6kj" Oct 30 12:42:23.761885 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:23.760661 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/396046e4-b596-4fed-943d-a5f42fea4f77-istiod-ca-cert\") pod \"e2e-extmod-b-12fd33-openshift-default-846b7dbb9-pt6kj\" (UID: \"396046e4-b596-4fed-943d-a5f42fea4f77\") " pod="openshift-ingress/e2e-extmod-b-12fd33-openshift-default-846b7dbb9-pt6kj" Oct 30 12:42:23.799436 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:23.795249 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/396046e4-b596-4fed-943d-a5f42fea4f77-istio-envoy\") pod \"e2e-extmod-b-12fd33-openshift-default-846b7dbb9-pt6kj\" (UID: \"396046e4-b596-4fed-943d-a5f42fea4f77\") " pod="openshift-ingress/e2e-extmod-b-12fd33-openshift-default-846b7dbb9-pt6kj" Oct 30 12:42:23.804849 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:23.801463 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69z66\" (UniqueName: \"kubernetes.io/projected/396046e4-b596-4fed-943d-a5f42fea4f77-kube-api-access-69z66\") pod \"e2e-extmod-b-12fd33-openshift-default-846b7dbb9-pt6kj\" (UID: \"396046e4-b596-4fed-943d-a5f42fea4f77\") " pod="openshift-ingress/e2e-extmod-b-12fd33-openshift-default-846b7dbb9-pt6kj" Oct 30 12:42:23.810929 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:23.810486 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/396046e4-b596-4fed-943d-a5f42fea4f77-istio-token\") pod \"e2e-extmod-b-12fd33-openshift-default-846b7dbb9-pt6kj\" (UID: \"396046e4-b596-4fed-943d-a5f42fea4f77\") " pod="openshift-ingress/e2e-extmod-b-12fd33-openshift-default-846b7dbb9-pt6kj" Oct 30 12:42:23.849494 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:23.849448 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/396046e4-b596-4fed-943d-a5f42fea4f77-istio-podinfo\") pod \"e2e-extmod-b-12fd33-openshift-default-846b7dbb9-pt6kj\" (UID: \"396046e4-b596-4fed-943d-a5f42fea4f77\") " pod="openshift-ingress/e2e-extmod-b-12fd33-openshift-default-846b7dbb9-pt6kj" Oct 30 12:42:23.992953 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:23.992168 4908 patch_prober.go:29] interesting pod/e2e-extmod-a-a786af-openshift-default-f89f5ffc7-b62z2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" start-of-body= Oct 30 12:42:23.992953 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:23.992225 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a786af-openshift-default-f89f5ffc7-b62z2" podUID="20af951f-19a7-49e6-ab1d-3bdee930084d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" Oct 30 12:42:24.047734 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:24.047476 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-12fd33-openshift-default-846b7dbb9-pt6kj" Oct 30 12:42:24.354926 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:24.354313 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-0de4b9-openshift-default-5d57d77697-g7ct6"] Oct 30 12:42:24.384851 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:42:24.359055 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfd62d0e_d1b7_4695_8673_63c1d68f4f40.slice/crio-1280ef676eb0df69c00534c047b0fa44cc57b4b3e1d9247ff60476b090098f31 WatchSource:0}: Error finding container 1280ef676eb0df69c00534c047b0fa44cc57b4b3e1d9247ff60476b090098f31: Status 404 returned error can't find the container with id 1280ef676eb0df69c00534c047b0fa44cc57b4b3e1d9247ff60476b090098f31 Oct 30 12:42:24.384851 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:24.374186 4908 patch_prober.go:29] interesting pod/e2e-extmod-a-2545c7-openshift-default-69cf958465-vb4sg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" start-of-body= Oct 30 12:42:24.384851 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:24.374250 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-2545c7-openshift-default-69cf958465-vb4sg" podUID="136fdf42-e3f2-4e4a-83de-a2482e084e5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" Oct 30 12:42:24.405857 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:24.402307 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:42:24.405857 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:24.402398 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:42:24.405857 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:24.402444 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:42:24.630845 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:24.630162 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-12fd33-openshift-default-846b7dbb9-pt6kj"] Oct 30 12:42:24.641508 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:24.641466 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:42:24.641649 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:24.641559 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:42:24.641649 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:24.641601 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:42:25.186545 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:25.185282 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-a786af-openshift-default-f89f5ffc7-b62z2" Oct 30 12:42:25.186545 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:25.186485 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-a786af-openshift-default-f89f5ffc7-b62z2" Oct 30 12:42:25.189130 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:25.189095 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-12fd33-openshift-default-846b7dbb9-pt6kj" event={"ID":"396046e4-b596-4fed-943d-a5f42fea4f77","Type":"ContainerStarted","Data":"5c019e2dea5de51003c7ab7c190f0cc27089c921e8f8515c1f3935a5ef128412"} Oct 30 12:42:25.189315 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:25.189153 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-12fd33-openshift-default-846b7dbb9-pt6kj" event={"ID":"396046e4-b596-4fed-943d-a5f42fea4f77","Type":"ContainerStarted","Data":"f4d4c26cc4fd1588e9f9eae07216cf8e550ba28bd4ceeb3a9fa53177cb789ec6"} Oct 30 12:42:25.192696 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:25.192667 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-0de4b9-openshift-default-5d57d77697-g7ct6" event={"ID":"cfd62d0e-d1b7-4695-8673-63c1d68f4f40","Type":"ContainerStarted","Data":"1f42a26fc0287f52ffa90881714b4c18f92ff184a9d87eb82271f54bfc0d8b5d"} Oct 30 12:42:25.192805 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:25.192717 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-0de4b9-openshift-default-5d57d77697-g7ct6" event={"ID":"cfd62d0e-d1b7-4695-8673-63c1d68f4f40","Type":"ContainerStarted","Data":"1280ef676eb0df69c00534c047b0fa44cc57b4b3e1d9247ff60476b090098f31"} Oct 30 12:42:25.194485 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:25.194464 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-a786af-openshift-default-f89f5ffc7-b62z2" Oct 30 12:42:25.211598 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:25.211532 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-0de4b9-openshift-default-5d57d77697-g7ct6" Oct 30 12:42:25.213019 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:25.212992 4908 patch_prober.go:29] interesting pod/e2e-extmod-b-0de4b9-openshift-default-5d57d77697-g7ct6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" start-of-body= Oct 30 12:42:25.213156 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:25.213045 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0de4b9-openshift-default-5d57d77697-g7ct6" podUID="cfd62d0e-d1b7-4695-8673-63c1d68f4f40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" Oct 30 12:42:25.329008 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:25.328940 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-0de4b9-openshift-default-5d57d77697-g7ct6" podStartSLOduration=3.328921736 podStartE2EDuration="3.328921736s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:42:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:42:25.322539242 +0000 UTC m=+2901.836967987" watchObservedRunningTime="2026-10-30 12:42:25.328921736 +0000 UTC m=+2901.843350498" Oct 30 12:42:25.366348 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:25.366291 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-12fd33-openshift-default-846b7dbb9-pt6kj" podStartSLOduration=2.366271351 podStartE2EDuration="2.366271351s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:42:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:42:25.36565253 +0000 UTC m=+2901.880081316" watchObservedRunningTime="2026-10-30 12:42:25.366271351 +0000 UTC m=+2901.880700098" Oct 30 12:42:25.378909 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:25.378880 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-2545c7-openshift-default-69cf958465-vb4sg" Oct 30 12:42:25.381012 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:25.380990 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-2545c7-openshift-default-69cf958465-vb4sg" Oct 30 12:42:26.049224 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:26.049187 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-12fd33-openshift-default-846b7dbb9-pt6kj" Oct 30 12:42:26.056106 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:26.056080 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-12fd33-openshift-default-846b7dbb9-pt6kj" Oct 30 12:42:26.207848 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:26.207063 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-12fd33-openshift-default-846b7dbb9-pt6kj" Oct 30 12:42:26.208204 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:26.208176 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-12fd33-openshift-default-846b7dbb9-pt6kj" Oct 30 12:42:26.217135 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:26.217109 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-0de4b9-openshift-default-5d57d77697-g7ct6" Oct 30 12:42:27.214236 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:27.214197 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-0de4b9-openshift-default-5d57d77697-g7ct6" Oct 30 12:42:27.214881 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:27.214862 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-0de4b9-openshift-default-5d57d77697-g7ct6" Oct 30 12:42:49.684668 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:42:49.684582 4908 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vn6vv" containerName="extractor" Oct 30 12:42:51.102951 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:51.102917 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-ecb3aa-openshift-default-68d9fff7cc-xb5ct"] Oct 30 12:42:51.123716 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:51.123685 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-ecb3aa-openshift-default-68d9fff7cc-xb5ct"] Oct 30 12:42:51.124089 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:51.124014 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-ecb3aa-openshift-default-68d9fff7cc-xb5ct" Oct 30 12:42:51.177134 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:51.177103 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-ecb3aa-openshift-default-dockercfg-6vwpd\"" Oct 30 12:42:51.260847 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:51.204851 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e59c9971-42c6-4df7-94a0-fd6753d847ca-credential-socket\") pod \"e2e-extmod-b-ecb3aa-openshift-default-68d9fff7cc-xb5ct\" (UID: \"e59c9971-42c6-4df7-94a0-fd6753d847ca\") " pod="openshift-ingress/e2e-extmod-b-ecb3aa-openshift-default-68d9fff7cc-xb5ct" Oct 30 12:42:51.260847 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:51.204912 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e59c9971-42c6-4df7-94a0-fd6753d847ca-workload-socket\") pod \"e2e-extmod-b-ecb3aa-openshift-default-68d9fff7cc-xb5ct\" (UID: \"e59c9971-42c6-4df7-94a0-fd6753d847ca\") " pod="openshift-ingress/e2e-extmod-b-ecb3aa-openshift-default-68d9fff7cc-xb5ct" Oct 30 12:42:51.364765 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:51.364658 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e59c9971-42c6-4df7-94a0-fd6753d847ca-istio-podinfo\") pod \"e2e-extmod-b-ecb3aa-openshift-default-68d9fff7cc-xb5ct\" (UID: \"e59c9971-42c6-4df7-94a0-fd6753d847ca\") " pod="openshift-ingress/e2e-extmod-b-ecb3aa-openshift-default-68d9fff7cc-xb5ct" Oct 30 12:42:51.364765 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:51.364722 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e59c9971-42c6-4df7-94a0-fd6753d847ca-credential-socket\") pod \"e2e-extmod-b-ecb3aa-openshift-default-68d9fff7cc-xb5ct\" (UID: \"e59c9971-42c6-4df7-94a0-fd6753d847ca\") " pod="openshift-ingress/e2e-extmod-b-ecb3aa-openshift-default-68d9fff7cc-xb5ct" Oct 30 12:42:51.365040 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:51.364813 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e59c9971-42c6-4df7-94a0-fd6753d847ca-workload-socket\") pod \"e2e-extmod-b-ecb3aa-openshift-default-68d9fff7cc-xb5ct\" (UID: \"e59c9971-42c6-4df7-94a0-fd6753d847ca\") " pod="openshift-ingress/e2e-extmod-b-ecb3aa-openshift-default-68d9fff7cc-xb5ct" Oct 30 12:42:51.365040 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:51.364892 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e59c9971-42c6-4df7-94a0-fd6753d847ca-istiod-ca-cert\") pod \"e2e-extmod-b-ecb3aa-openshift-default-68d9fff7cc-xb5ct\" (UID: \"e59c9971-42c6-4df7-94a0-fd6753d847ca\") " pod="openshift-ingress/e2e-extmod-b-ecb3aa-openshift-default-68d9fff7cc-xb5ct" Oct 30 12:42:51.365040 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:51.364927 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sztmp\" (UniqueName: \"kubernetes.io/projected/e59c9971-42c6-4df7-94a0-fd6753d847ca-kube-api-access-sztmp\") pod \"e2e-extmod-b-ecb3aa-openshift-default-68d9fff7cc-xb5ct\" (UID: \"e59c9971-42c6-4df7-94a0-fd6753d847ca\") " pod="openshift-ingress/e2e-extmod-b-ecb3aa-openshift-default-68d9fff7cc-xb5ct" Oct 30 12:42:51.365040 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:51.365014 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e59c9971-42c6-4df7-94a0-fd6753d847ca-istio-envoy\") pod \"e2e-extmod-b-ecb3aa-openshift-default-68d9fff7cc-xb5ct\" (UID: \"e59c9971-42c6-4df7-94a0-fd6753d847ca\") " pod="openshift-ingress/e2e-extmod-b-ecb3aa-openshift-default-68d9fff7cc-xb5ct" Oct 30 12:42:51.365270 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:51.365074 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e59c9971-42c6-4df7-94a0-fd6753d847ca-istio-token\") pod \"e2e-extmod-b-ecb3aa-openshift-default-68d9fff7cc-xb5ct\" (UID: \"e59c9971-42c6-4df7-94a0-fd6753d847ca\") " pod="openshift-ingress/e2e-extmod-b-ecb3aa-openshift-default-68d9fff7cc-xb5ct" Oct 30 12:42:51.365270 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:51.365152 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e59c9971-42c6-4df7-94a0-fd6753d847ca-istio-data\") pod \"e2e-extmod-b-ecb3aa-openshift-default-68d9fff7cc-xb5ct\" (UID: \"e59c9971-42c6-4df7-94a0-fd6753d847ca\") " pod="openshift-ingress/e2e-extmod-b-ecb3aa-openshift-default-68d9fff7cc-xb5ct" Oct 30 12:42:51.365270 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:51.365254 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e59c9971-42c6-4df7-94a0-fd6753d847ca-workload-certs\") pod \"e2e-extmod-b-ecb3aa-openshift-default-68d9fff7cc-xb5ct\" (UID: \"e59c9971-42c6-4df7-94a0-fd6753d847ca\") " pod="openshift-ingress/e2e-extmod-b-ecb3aa-openshift-default-68d9fff7cc-xb5ct" Oct 30 12:42:51.365793 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:51.365767 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e59c9971-42c6-4df7-94a0-fd6753d847ca-credential-socket\") pod \"e2e-extmod-b-ecb3aa-openshift-default-68d9fff7cc-xb5ct\" (UID: \"e59c9971-42c6-4df7-94a0-fd6753d847ca\") " pod="openshift-ingress/e2e-extmod-b-ecb3aa-openshift-default-68d9fff7cc-xb5ct" Oct 30 12:42:51.365955 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:51.365808 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e59c9971-42c6-4df7-94a0-fd6753d847ca-workload-socket\") pod \"e2e-extmod-b-ecb3aa-openshift-default-68d9fff7cc-xb5ct\" (UID: \"e59c9971-42c6-4df7-94a0-fd6753d847ca\") " pod="openshift-ingress/e2e-extmod-b-ecb3aa-openshift-default-68d9fff7cc-xb5ct" Oct 30 12:42:51.466037 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:51.466008 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e59c9971-42c6-4df7-94a0-fd6753d847ca-istio-podinfo\") pod \"e2e-extmod-b-ecb3aa-openshift-default-68d9fff7cc-xb5ct\" (UID: \"e59c9971-42c6-4df7-94a0-fd6753d847ca\") " pod="openshift-ingress/e2e-extmod-b-ecb3aa-openshift-default-68d9fff7cc-xb5ct" Oct 30 12:42:51.466243 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:51.466055 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e59c9971-42c6-4df7-94a0-fd6753d847ca-istiod-ca-cert\") pod \"e2e-extmod-b-ecb3aa-openshift-default-68d9fff7cc-xb5ct\" (UID: \"e59c9971-42c6-4df7-94a0-fd6753d847ca\") " pod="openshift-ingress/e2e-extmod-b-ecb3aa-openshift-default-68d9fff7cc-xb5ct" Oct 30 12:42:51.466243 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:51.466075 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sztmp\" (UniqueName: \"kubernetes.io/projected/e59c9971-42c6-4df7-94a0-fd6753d847ca-kube-api-access-sztmp\") pod \"e2e-extmod-b-ecb3aa-openshift-default-68d9fff7cc-xb5ct\" (UID: \"e59c9971-42c6-4df7-94a0-fd6753d847ca\") " pod="openshift-ingress/e2e-extmod-b-ecb3aa-openshift-default-68d9fff7cc-xb5ct" Oct 30 12:42:51.466243 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:51.466140 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e59c9971-42c6-4df7-94a0-fd6753d847ca-istio-envoy\") pod \"e2e-extmod-b-ecb3aa-openshift-default-68d9fff7cc-xb5ct\" (UID: \"e59c9971-42c6-4df7-94a0-fd6753d847ca\") " pod="openshift-ingress/e2e-extmod-b-ecb3aa-openshift-default-68d9fff7cc-xb5ct" Oct 30 12:42:51.466243 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:51.466187 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e59c9971-42c6-4df7-94a0-fd6753d847ca-istio-token\") pod \"e2e-extmod-b-ecb3aa-openshift-default-68d9fff7cc-xb5ct\" (UID: \"e59c9971-42c6-4df7-94a0-fd6753d847ca\") " pod="openshift-ingress/e2e-extmod-b-ecb3aa-openshift-default-68d9fff7cc-xb5ct" Oct 30 12:42:51.466481 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:51.466252 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e59c9971-42c6-4df7-94a0-fd6753d847ca-istio-data\") pod \"e2e-extmod-b-ecb3aa-openshift-default-68d9fff7cc-xb5ct\" (UID: \"e59c9971-42c6-4df7-94a0-fd6753d847ca\") " pod="openshift-ingress/e2e-extmod-b-ecb3aa-openshift-default-68d9fff7cc-xb5ct" Oct 30 12:42:51.466481 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:51.466347 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e59c9971-42c6-4df7-94a0-fd6753d847ca-workload-certs\") pod \"e2e-extmod-b-ecb3aa-openshift-default-68d9fff7cc-xb5ct\" (UID: \"e59c9971-42c6-4df7-94a0-fd6753d847ca\") " pod="openshift-ingress/e2e-extmod-b-ecb3aa-openshift-default-68d9fff7cc-xb5ct" Oct 30 12:42:51.466736 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:51.466709 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e59c9971-42c6-4df7-94a0-fd6753d847ca-workload-certs\") pod \"e2e-extmod-b-ecb3aa-openshift-default-68d9fff7cc-xb5ct\" (UID: \"e59c9971-42c6-4df7-94a0-fd6753d847ca\") " pod="openshift-ingress/e2e-extmod-b-ecb3aa-openshift-default-68d9fff7cc-xb5ct" Oct 30 12:42:51.466854 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:51.466807 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e59c9971-42c6-4df7-94a0-fd6753d847ca-istiod-ca-cert\") pod \"e2e-extmod-b-ecb3aa-openshift-default-68d9fff7cc-xb5ct\" (UID: \"e59c9971-42c6-4df7-94a0-fd6753d847ca\") " pod="openshift-ingress/e2e-extmod-b-ecb3aa-openshift-default-68d9fff7cc-xb5ct" Oct 30 12:42:51.467045 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:51.467020 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e59c9971-42c6-4df7-94a0-fd6753d847ca-istio-data\") pod \"e2e-extmod-b-ecb3aa-openshift-default-68d9fff7cc-xb5ct\" (UID: \"e59c9971-42c6-4df7-94a0-fd6753d847ca\") " pod="openshift-ingress/e2e-extmod-b-ecb3aa-openshift-default-68d9fff7cc-xb5ct" Oct 30 12:42:51.468985 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:51.468955 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e59c9971-42c6-4df7-94a0-fd6753d847ca-istio-envoy\") pod \"e2e-extmod-b-ecb3aa-openshift-default-68d9fff7cc-xb5ct\" (UID: \"e59c9971-42c6-4df7-94a0-fd6753d847ca\") " pod="openshift-ingress/e2e-extmod-b-ecb3aa-openshift-default-68d9fff7cc-xb5ct" Oct 30 12:42:51.469406 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:51.469384 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e59c9971-42c6-4df7-94a0-fd6753d847ca-istio-podinfo\") pod \"e2e-extmod-b-ecb3aa-openshift-default-68d9fff7cc-xb5ct\" (UID: \"e59c9971-42c6-4df7-94a0-fd6753d847ca\") " pod="openshift-ingress/e2e-extmod-b-ecb3aa-openshift-default-68d9fff7cc-xb5ct" Oct 30 12:42:51.489118 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:51.489092 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e59c9971-42c6-4df7-94a0-fd6753d847ca-istio-token\") pod \"e2e-extmod-b-ecb3aa-openshift-default-68d9fff7cc-xb5ct\" (UID: \"e59c9971-42c6-4df7-94a0-fd6753d847ca\") " pod="openshift-ingress/e2e-extmod-b-ecb3aa-openshift-default-68d9fff7cc-xb5ct" Oct 30 12:42:51.494061 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:51.494039 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sztmp\" (UniqueName: \"kubernetes.io/projected/e59c9971-42c6-4df7-94a0-fd6753d847ca-kube-api-access-sztmp\") pod \"e2e-extmod-b-ecb3aa-openshift-default-68d9fff7cc-xb5ct\" (UID: \"e59c9971-42c6-4df7-94a0-fd6753d847ca\") " pod="openshift-ingress/e2e-extmod-b-ecb3aa-openshift-default-68d9fff7cc-xb5ct" Oct 30 12:42:51.497958 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:51.497939 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-ecb3aa-openshift-default-68d9fff7cc-xb5ct" Oct 30 12:42:51.891454 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:51.891367 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-ecb3aa-openshift-default-68d9fff7cc-xb5ct"] Oct 30 12:42:51.897320 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:42:51.897291 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode59c9971_42c6_4df7_94a0_fd6753d847ca.slice/crio-31c8b00537e28e5b9740ff12221a0dbfc2a3f94bbc7ad6717d2bec3c73513b68 WatchSource:0}: Error finding container 31c8b00537e28e5b9740ff12221a0dbfc2a3f94bbc7ad6717d2bec3c73513b68: Status 404 returned error can't find the container with id 31c8b00537e28e5b9740ff12221a0dbfc2a3f94bbc7ad6717d2bec3c73513b68 Oct 30 12:42:51.901427 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:51.901382 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:42:51.901540 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:51.901471 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:42:51.901540 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:51.901518 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:42:52.414539 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:52.414500 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-ecb3aa-openshift-default-68d9fff7cc-xb5ct" event={"ID":"e59c9971-42c6-4df7-94a0-fd6753d847ca","Type":"ContainerStarted","Data":"2964357075b0307575fbc5e3023d9e141572ec36c9be49b795754dcf26839c80"} Oct 30 12:42:52.414954 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:52.414586 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-ecb3aa-openshift-default-68d9fff7cc-xb5ct" event={"ID":"e59c9971-42c6-4df7-94a0-fd6753d847ca","Type":"ContainerStarted","Data":"31c8b00537e28e5b9740ff12221a0dbfc2a3f94bbc7ad6717d2bec3c73513b68"} Oct 30 12:42:52.473305 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:52.473249 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-ecb3aa-openshift-default-68d9fff7cc-xb5ct" podStartSLOduration=2.473230756 podStartE2EDuration="2.473230756s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:42:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:42:52.468572753 +0000 UTC m=+2928.983001511" watchObservedRunningTime="2026-10-30 12:42:52.473230756 +0000 UTC m=+2928.987659504" Oct 30 12:42:52.498554 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:52.498529 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-ecb3aa-openshift-default-68d9fff7cc-xb5ct" Oct 30 12:42:52.504027 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:52.504004 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-ecb3aa-openshift-default-68d9fff7cc-xb5ct" Oct 30 12:42:53.441263 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:53.441232 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-ecb3aa-openshift-default-68d9fff7cc-xb5ct" Oct 30 12:42:53.442479 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:53.442455 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-ecb3aa-openshift-default-68d9fff7cc-xb5ct" Oct 30 12:42:55.534910 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:42:55.533148 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9wwfc"] Oct 30 12:43:05.432078 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:05.432041 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-ac674b/autores-model-ac674b-kserve-554dc45f77-gzg4p"] Oct 30 12:43:05.437137 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:05.437112 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-ac674b/autores-model-ac674b-kserve-554dc45f77-gzg4p" Oct 30 12:43:05.458443 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:05.458410 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-ac674b/autores-model-ac674b-kserve-554dc45f77-gzg4p"] Oct 30 12:43:05.463170 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:05.463144 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-ac674b\"/\"default-dockercfg-kk66j\"" Oct 30 12:43:05.463965 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:05.463944 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-ac674b\"/\"autores-model-ac674b-kserve-self-signed-certs\"" Oct 30 12:43:05.464225 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:05.464191 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-ac674b\"/\"openshift-service-ca.crt\"" Oct 30 12:43:05.464313 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:05.464229 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-ac674b\"/\"kube-root-ca.crt\"" Oct 30 12:43:05.486419 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:05.486395 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af1fdb0d-774f-462f-ad3a-828685e5c4eb-tmp-dir\") pod \"autores-model-ac674b-kserve-554dc45f77-gzg4p\" (UID: \"af1fdb0d-774f-462f-ad3a-828685e5c4eb\") " pod="ai-tenant-e2e-autores-ac674b/autores-model-ac674b-kserve-554dc45f77-gzg4p" Oct 30 12:43:05.486612 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:05.486594 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tn7c\" (UniqueName: \"kubernetes.io/projected/af1fdb0d-774f-462f-ad3a-828685e5c4eb-kube-api-access-6tn7c\") pod \"autores-model-ac674b-kserve-554dc45f77-gzg4p\" (UID: \"af1fdb0d-774f-462f-ad3a-828685e5c4eb\") " pod="ai-tenant-e2e-autores-ac674b/autores-model-ac674b-kserve-554dc45f77-gzg4p" Oct 30 12:43:05.487123 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:05.487101 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af1fdb0d-774f-462f-ad3a-828685e5c4eb-dshm\") pod \"autores-model-ac674b-kserve-554dc45f77-gzg4p\" (UID: \"af1fdb0d-774f-462f-ad3a-828685e5c4eb\") " pod="ai-tenant-e2e-autores-ac674b/autores-model-ac674b-kserve-554dc45f77-gzg4p" Oct 30 12:43:05.487307 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:05.487291 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af1fdb0d-774f-462f-ad3a-828685e5c4eb-tls-certs\") pod \"autores-model-ac674b-kserve-554dc45f77-gzg4p\" (UID: \"af1fdb0d-774f-462f-ad3a-828685e5c4eb\") " pod="ai-tenant-e2e-autores-ac674b/autores-model-ac674b-kserve-554dc45f77-gzg4p" Oct 30 12:43:05.487490 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:05.487463 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af1fdb0d-774f-462f-ad3a-828685e5c4eb-home\") pod \"autores-model-ac674b-kserve-554dc45f77-gzg4p\" (UID: \"af1fdb0d-774f-462f-ad3a-828685e5c4eb\") " pod="ai-tenant-e2e-autores-ac674b/autores-model-ac674b-kserve-554dc45f77-gzg4p" Oct 30 12:43:05.487617 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:05.487604 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af1fdb0d-774f-462f-ad3a-828685e5c4eb-model-cache\") pod \"autores-model-ac674b-kserve-554dc45f77-gzg4p\" (UID: \"af1fdb0d-774f-462f-ad3a-828685e5c4eb\") " pod="ai-tenant-e2e-autores-ac674b/autores-model-ac674b-kserve-554dc45f77-gzg4p" Oct 30 12:43:05.589110 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:05.589075 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af1fdb0d-774f-462f-ad3a-828685e5c4eb-tmp-dir\") pod \"autores-model-ac674b-kserve-554dc45f77-gzg4p\" (UID: \"af1fdb0d-774f-462f-ad3a-828685e5c4eb\") " pod="ai-tenant-e2e-autores-ac674b/autores-model-ac674b-kserve-554dc45f77-gzg4p" Oct 30 12:43:05.589297 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:05.589264 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tn7c\" (UniqueName: \"kubernetes.io/projected/af1fdb0d-774f-462f-ad3a-828685e5c4eb-kube-api-access-6tn7c\") pod \"autores-model-ac674b-kserve-554dc45f77-gzg4p\" (UID: \"af1fdb0d-774f-462f-ad3a-828685e5c4eb\") " pod="ai-tenant-e2e-autores-ac674b/autores-model-ac674b-kserve-554dc45f77-gzg4p" Oct 30 12:43:05.589379 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:05.589345 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af1fdb0d-774f-462f-ad3a-828685e5c4eb-dshm\") pod \"autores-model-ac674b-kserve-554dc45f77-gzg4p\" (UID: \"af1fdb0d-774f-462f-ad3a-828685e5c4eb\") " pod="ai-tenant-e2e-autores-ac674b/autores-model-ac674b-kserve-554dc45f77-gzg4p" Oct 30 12:43:05.589440 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:05.589412 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af1fdb0d-774f-462f-ad3a-828685e5c4eb-tls-certs\") pod \"autores-model-ac674b-kserve-554dc45f77-gzg4p\" (UID: \"af1fdb0d-774f-462f-ad3a-828685e5c4eb\") " pod="ai-tenant-e2e-autores-ac674b/autores-model-ac674b-kserve-554dc45f77-gzg4p" Oct 30 12:43:05.589521 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:05.589501 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af1fdb0d-774f-462f-ad3a-828685e5c4eb-home\") pod \"autores-model-ac674b-kserve-554dc45f77-gzg4p\" (UID: \"af1fdb0d-774f-462f-ad3a-828685e5c4eb\") " pod="ai-tenant-e2e-autores-ac674b/autores-model-ac674b-kserve-554dc45f77-gzg4p" Oct 30 12:43:05.589576 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:05.589521 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af1fdb0d-774f-462f-ad3a-828685e5c4eb-tmp-dir\") pod \"autores-model-ac674b-kserve-554dc45f77-gzg4p\" (UID: \"af1fdb0d-774f-462f-ad3a-828685e5c4eb\") " pod="ai-tenant-e2e-autores-ac674b/autores-model-ac674b-kserve-554dc45f77-gzg4p" Oct 30 12:43:05.589576 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:05.589556 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af1fdb0d-774f-462f-ad3a-828685e5c4eb-model-cache\") pod \"autores-model-ac674b-kserve-554dc45f77-gzg4p\" (UID: \"af1fdb0d-774f-462f-ad3a-828685e5c4eb\") " pod="ai-tenant-e2e-autores-ac674b/autores-model-ac674b-kserve-554dc45f77-gzg4p" Oct 30 12:43:05.589971 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:05.589946 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af1fdb0d-774f-462f-ad3a-828685e5c4eb-model-cache\") pod \"autores-model-ac674b-kserve-554dc45f77-gzg4p\" (UID: \"af1fdb0d-774f-462f-ad3a-828685e5c4eb\") " pod="ai-tenant-e2e-autores-ac674b/autores-model-ac674b-kserve-554dc45f77-gzg4p" Oct 30 12:43:05.590901 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:05.590860 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af1fdb0d-774f-462f-ad3a-828685e5c4eb-home\") pod \"autores-model-ac674b-kserve-554dc45f77-gzg4p\" (UID: \"af1fdb0d-774f-462f-ad3a-828685e5c4eb\") " pod="ai-tenant-e2e-autores-ac674b/autores-model-ac674b-kserve-554dc45f77-gzg4p" Oct 30 12:43:05.596683 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:05.596386 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af1fdb0d-774f-462f-ad3a-828685e5c4eb-tls-certs\") pod \"autores-model-ac674b-kserve-554dc45f77-gzg4p\" (UID: \"af1fdb0d-774f-462f-ad3a-828685e5c4eb\") " pod="ai-tenant-e2e-autores-ac674b/autores-model-ac674b-kserve-554dc45f77-gzg4p" Oct 30 12:43:05.610079 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:05.610032 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af1fdb0d-774f-462f-ad3a-828685e5c4eb-dshm\") pod \"autores-model-ac674b-kserve-554dc45f77-gzg4p\" (UID: \"af1fdb0d-774f-462f-ad3a-828685e5c4eb\") " pod="ai-tenant-e2e-autores-ac674b/autores-model-ac674b-kserve-554dc45f77-gzg4p" Oct 30 12:43:05.614103 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:05.613999 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tn7c\" (UniqueName: \"kubernetes.io/projected/af1fdb0d-774f-462f-ad3a-828685e5c4eb-kube-api-access-6tn7c\") pod \"autores-model-ac674b-kserve-554dc45f77-gzg4p\" (UID: \"af1fdb0d-774f-462f-ad3a-828685e5c4eb\") " pod="ai-tenant-e2e-autores-ac674b/autores-model-ac674b-kserve-554dc45f77-gzg4p" Oct 30 12:43:05.752233 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:05.752128 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-ac674b/autores-model-ac674b-kserve-554dc45f77-gzg4p" Oct 30 12:43:06.183673 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:06.183640 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-ac674b/autores-model-ac674b-kserve-554dc45f77-gzg4p"] Oct 30 12:43:06.185313 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:43:06.185278 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaf1fdb0d_774f_462f_ad3a_828685e5c4eb.slice/crio-3cf58bce0567bff03cacce23c77334c14dde3973bd07f34aceed51285a70f72d WatchSource:0}: Error finding container 3cf58bce0567bff03cacce23c77334c14dde3973bd07f34aceed51285a70f72d: Status 404 returned error can't find the container with id 3cf58bce0567bff03cacce23c77334c14dde3973bd07f34aceed51285a70f72d Oct 30 12:43:06.635439 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:06.635402 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-ac674b/autores-model-ac674b-kserve-554dc45f77-gzg4p" event={"ID":"af1fdb0d-774f-462f-ad3a-828685e5c4eb","Type":"ContainerStarted","Data":"df0529212ff0f4f223bac4e4d4c719104b116a7858d6f57b508c7032e8d1e7ad"} Oct 30 12:43:06.636215 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:06.635452 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-ac674b/autores-model-ac674b-kserve-554dc45f77-gzg4p" event={"ID":"af1fdb0d-774f-462f-ad3a-828685e5c4eb","Type":"ContainerStarted","Data":"3cf58bce0567bff03cacce23c77334c14dde3973bd07f34aceed51285a70f72d"} Oct 30 12:43:06.637432 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:06.637377 4908 generic.go:356] "Generic (PLEG): container finished" podID="4a63f6b6-6cb7-4ed6-af39-c9fe090ebc87" containerID="9ce960f38f4c4100401ee91fbb26ddb7fea096e58551a875564449b6d515b345" exitCode=0 Oct 30 12:43:06.637432 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:06.637421 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-ac674b-openshift-default-798564ccc7-2nk9v" event={"ID":"4a63f6b6-6cb7-4ed6-af39-c9fe090ebc87","Type":"ContainerDied","Data":"9ce960f38f4c4100401ee91fbb26ddb7fea096e58551a875564449b6d515b345"} Oct 30 12:43:06.637807 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:06.637793 4908 scope.go:122] "RemoveContainer" containerID="9ce960f38f4c4100401ee91fbb26ddb7fea096e58551a875564449b6d515b345" Oct 30 12:43:06.639499 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:06.639462 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:43:06.639587 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:06.639540 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:43:06.639587 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:06.639571 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:43:06.690221 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:06.690168 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-autores-ac674b/autores-model-ac674b-kserve-554dc45f77-gzg4p" podStartSLOduration=1.690151248 podStartE2EDuration="1.690151248s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:43:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:43:06.687346771 +0000 UTC m=+2943.201775516" watchObservedRunningTime="2026-10-30 12:43:06.690151248 +0000 UTC m=+2943.204579989" Oct 30 12:43:06.753066 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:06.753008 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-ac674b/autores-model-ac674b-kserve-554dc45f77-gzg4p" Oct 30 12:43:07.646183 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:07.646144 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-ac674b-openshift-default-798564ccc7-2nk9v" event={"ID":"4a63f6b6-6cb7-4ed6-af39-c9fe090ebc87","Type":"ContainerStarted","Data":"3f4348813cb093d0971217b5f33073f4bde153ca653c9da4216a6c886c39ab38"} Oct 30 12:43:07.974908 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:07.973995 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-ac674b-openshift-default-798564ccc7-2nk9v" Oct 30 12:43:07.974908 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:07.974340 4908 patch_prober.go:29] interesting pod/e2e-autores-ac674b-openshift-default-798564ccc7-2nk9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" start-of-body= Oct 30 12:43:07.974908 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:07.974395 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-ac674b-openshift-default-798564ccc7-2nk9v" podUID="4a63f6b6-6cb7-4ed6-af39-c9fe090ebc87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" Oct 30 12:43:08.977185 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:08.977150 4908 patch_prober.go:29] interesting pod/e2e-autores-ac674b-openshift-default-798564ccc7-2nk9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" start-of-body= Oct 30 12:43:08.977764 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:08.977729 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-ac674b-openshift-default-798564ccc7-2nk9v" podUID="4a63f6b6-6cb7-4ed6-af39-c9fe090ebc87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" Oct 30 12:43:09.982360 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:09.982327 4908 patch_prober.go:29] interesting pod/e2e-autores-ac674b-openshift-default-798564ccc7-2nk9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" start-of-body= Oct 30 12:43:09.982972 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:09.982941 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-ac674b-openshift-default-798564ccc7-2nk9v" podUID="4a63f6b6-6cb7-4ed6-af39-c9fe090ebc87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" Oct 30 12:43:10.978853 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:10.978170 4908 patch_prober.go:29] interesting pod/e2e-autores-ac674b-openshift-default-798564ccc7-2nk9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" start-of-body= Oct 30 12:43:10.978853 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:10.978233 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-ac674b-openshift-default-798564ccc7-2nk9v" podUID="4a63f6b6-6cb7-4ed6-af39-c9fe090ebc87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" Oct 30 12:43:11.983851 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:11.975451 4908 patch_prober.go:29] interesting pod/e2e-autores-ac674b-openshift-default-798564ccc7-2nk9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" start-of-body= Oct 30 12:43:11.983851 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:11.975526 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-ac674b-openshift-default-798564ccc7-2nk9v" podUID="4a63f6b6-6cb7-4ed6-af39-c9fe090ebc87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" Oct 30 12:43:12.977853 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:12.975134 4908 patch_prober.go:29] interesting pod/e2e-autores-ac674b-openshift-default-798564ccc7-2nk9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" start-of-body= Oct 30 12:43:12.977853 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:12.975192 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-ac674b-openshift-default-798564ccc7-2nk9v" podUID="4a63f6b6-6cb7-4ed6-af39-c9fe090ebc87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" Oct 30 12:43:13.978949 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:13.978920 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-ac674b-openshift-default-798564ccc7-2nk9v" Oct 30 12:43:13.979408 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:13.979362 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-ac674b-openshift-default-798564ccc7-2nk9v" Oct 30 12:43:13.980144 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:13.980121 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-ac674b-openshift-default-798564ccc7-2nk9v" Oct 30 12:43:15.752622 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:15.752592 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-ac674b/autores-model-ac674b-kserve-554dc45f77-gzg4p" Oct 30 12:43:15.770625 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:15.770595 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-ac674b/autores-model-ac674b-kserve-554dc45f77-gzg4p" Oct 30 12:43:16.082878 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:16.082747 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-ac674b/autores-model-ac674b-kserve-554dc45f77-gzg4p" Oct 30 12:43:19.710165 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:43:19.710130 4908 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vn6vv" containerName="extractor" Oct 30 12:43:22.014914 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:22.014881 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-wq96b"] Oct 30 12:43:22.015416 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:22.015124 4908 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-wq96b" podUID="8aaac2b2-a4e5-4736-96ec-30f545797512" containerName="istio-proxy" containerID="cri-o://67f327e201fe008df2005b78d98003d09f2872c93d4faded29e77d32594d65e1" gracePeriod=30 Oct 30 12:43:24.536387 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:24.536346 4908 patch_prober.go:29] interesting pod/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-wq96b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 12:43:24.536757 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:24.536415 4908 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-wq96b" podUID="8aaac2b2-a4e5-4736-96ec-30f545797512" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 12:43:25.865983 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:25.865945 4908 generic.go:356] "Generic (PLEG): container finished" podID="3590e8ff-89a3-482b-b022-064bb84fe09c" containerID="e9795be9cfa47e1d277b621a66ccf1c4f0749ec3581d6f56ac16ca0e760718f7" exitCode=0 Oct 30 12:43:25.866539 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:25.866014 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-0f7d14-openshift-default-cf79d4dcc-mcrqq" event={"ID":"3590e8ff-89a3-482b-b022-064bb84fe09c","Type":"ContainerDied","Data":"e9795be9cfa47e1d277b621a66ccf1c4f0749ec3581d6f56ac16ca0e760718f7"} Oct 30 12:43:25.866539 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:25.866367 4908 scope.go:122] "RemoveContainer" containerID="e9795be9cfa47e1d277b621a66ccf1c4f0749ec3581d6f56ac16ca0e760718f7" Oct 30 12:43:25.868404 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:25.868375 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:43:25.868553 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:25.868445 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:43:25.868553 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:25.868486 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:43:26.891853 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:26.883388 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-0f7d14-openshift-default-cf79d4dcc-mcrqq" event={"ID":"3590e8ff-89a3-482b-b022-064bb84fe09c","Type":"ContainerStarted","Data":"7a32c67c0a333bd003cd5a52c0922d86d06dab5cab941208acab15e54b9e2919"} Oct 30 12:43:27.403496 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:27.403452 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-wq96b" Oct 30 12:43:27.504478 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:27.504389 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8aaac2b2-a4e5-4736-96ec-30f545797512-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8aaac2b2-a4e5-4736-96ec-30f545797512-workload-certs\") pod \"8aaac2b2-a4e5-4736-96ec-30f545797512\" (UID: \"8aaac2b2-a4e5-4736-96ec-30f545797512\") " Oct 30 12:43:27.504748 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:27.504730 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8aaac2b2-a4e5-4736-96ec-30f545797512-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8aaac2b2-a4e5-4736-96ec-30f545797512-istiod-ca-cert\") pod \"8aaac2b2-a4e5-4736-96ec-30f545797512\" (UID: \"8aaac2b2-a4e5-4736-96ec-30f545797512\") " Oct 30 12:43:27.505239 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:27.505220 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8aaac2b2-a4e5-4736-96ec-30f545797512-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8aaac2b2-a4e5-4736-96ec-30f545797512-istio-data\") pod \"8aaac2b2-a4e5-4736-96ec-30f545797512\" (UID: \"8aaac2b2-a4e5-4736-96ec-30f545797512\") " Oct 30 12:43:27.505400 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:27.505387 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8aaac2b2-a4e5-4736-96ec-30f545797512-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8aaac2b2-a4e5-4736-96ec-30f545797512-istio-envoy\") pod \"8aaac2b2-a4e5-4736-96ec-30f545797512\" (UID: \"8aaac2b2-a4e5-4736-96ec-30f545797512\") " Oct 30 12:43:27.505610 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:27.505593 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8aaac2b2-a4e5-4736-96ec-30f545797512-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8aaac2b2-a4e5-4736-96ec-30f545797512-credential-socket\") pod \"8aaac2b2-a4e5-4736-96ec-30f545797512\" (UID: \"8aaac2b2-a4e5-4736-96ec-30f545797512\") " Oct 30 12:43:27.505877 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:27.505237 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8aaac2b2-a4e5-4736-96ec-30f545797512-istiod-ca-cert" pod "8aaac2b2-a4e5-4736-96ec-30f545797512" (UID: "8aaac2b2-a4e5-4736-96ec-30f545797512"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:43:27.505877 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:27.505399 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8aaac2b2-a4e5-4736-96ec-30f545797512-istio-data" pod "8aaac2b2-a4e5-4736-96ec-30f545797512" (UID: "8aaac2b2-a4e5-4736-96ec-30f545797512"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:43:27.505877 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:27.505507 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8aaac2b2-a4e5-4736-96ec-30f545797512-workload-certs" pod "8aaac2b2-a4e5-4736-96ec-30f545797512" (UID: "8aaac2b2-a4e5-4736-96ec-30f545797512"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:43:27.506086 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:27.505943 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8aaac2b2-a4e5-4736-96ec-30f545797512-credential-socket" pod "8aaac2b2-a4e5-4736-96ec-30f545797512" (UID: "8aaac2b2-a4e5-4736-96ec-30f545797512"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:43:27.506086 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:27.506058 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8aaac2b2-a4e5-4736-96ec-30f545797512-workload-socket" pod "8aaac2b2-a4e5-4736-96ec-30f545797512" (UID: "8aaac2b2-a4e5-4736-96ec-30f545797512"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:43:27.506211 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:27.505744 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8aaac2b2-a4e5-4736-96ec-30f545797512-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8aaac2b2-a4e5-4736-96ec-30f545797512-workload-socket\") pod \"8aaac2b2-a4e5-4736-96ec-30f545797512\" (UID: \"8aaac2b2-a4e5-4736-96ec-30f545797512\") " Oct 30 12:43:27.506339 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:27.506326 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8aaac2b2-a4e5-4736-96ec-30f545797512-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8aaac2b2-a4e5-4736-96ec-30f545797512-istio-podinfo\") pod \"8aaac2b2-a4e5-4736-96ec-30f545797512\" (UID: \"8aaac2b2-a4e5-4736-96ec-30f545797512\") " Oct 30 12:43:27.506772 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:27.506754 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8aaac2b2-a4e5-4736-96ec-30f545797512-istio-token\" (UniqueName: \"kubernetes.io/projected/8aaac2b2-a4e5-4736-96ec-30f545797512-istio-token\") pod \"8aaac2b2-a4e5-4736-96ec-30f545797512\" (UID: \"8aaac2b2-a4e5-4736-96ec-30f545797512\") " Oct 30 12:43:27.507079 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:27.507062 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8aaac2b2-a4e5-4736-96ec-30f545797512-kube-api-access-l5mk6\" (UniqueName: \"kubernetes.io/projected/8aaac2b2-a4e5-4736-96ec-30f545797512-kube-api-access-l5mk6\") pod \"8aaac2b2-a4e5-4736-96ec-30f545797512\" (UID: \"8aaac2b2-a4e5-4736-96ec-30f545797512\") " Oct 30 12:43:27.511796 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:27.511092 4908 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8aaac2b2-a4e5-4736-96ec-30f545797512-workload-certs\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:43:27.511796 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:27.511131 4908 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8aaac2b2-a4e5-4736-96ec-30f545797512-istiod-ca-cert\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:43:27.511796 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:27.511145 4908 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8aaac2b2-a4e5-4736-96ec-30f545797512-istio-data\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:43:27.511796 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:27.511157 4908 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8aaac2b2-a4e5-4736-96ec-30f545797512-credential-socket\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:43:27.511796 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:27.511170 4908 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8aaac2b2-a4e5-4736-96ec-30f545797512-workload-socket\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:43:27.514865 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:27.512316 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8aaac2b2-a4e5-4736-96ec-30f545797512-istio-envoy" pod "8aaac2b2-a4e5-4736-96ec-30f545797512" (UID: "8aaac2b2-a4e5-4736-96ec-30f545797512"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:43:27.514865 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:27.512367 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8aaac2b2-a4e5-4736-96ec-30f545797512-istio-podinfo" pod "8aaac2b2-a4e5-4736-96ec-30f545797512" (UID: "8aaac2b2-a4e5-4736-96ec-30f545797512"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 12:43:27.514865 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:27.512450 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8aaac2b2-a4e5-4736-96ec-30f545797512-istio-token" pod "8aaac2b2-a4e5-4736-96ec-30f545797512" (UID: "8aaac2b2-a4e5-4736-96ec-30f545797512"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:43:27.529657 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:27.529591 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8aaac2b2-a4e5-4736-96ec-30f545797512-kube-api-access-l5mk6" pod "8aaac2b2-a4e5-4736-96ec-30f545797512" (UID: "8aaac2b2-a4e5-4736-96ec-30f545797512"). InnerVolumeSpecName "kube-api-access-l5mk6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:43:27.614865 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:27.613807 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5mk6\" (UniqueName: \"kubernetes.io/projected/8aaac2b2-a4e5-4736-96ec-30f545797512-kube-api-access-l5mk6\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:43:27.614865 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:27.613888 4908 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8aaac2b2-a4e5-4736-96ec-30f545797512-istio-envoy\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:43:27.614865 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:27.613904 4908 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8aaac2b2-a4e5-4736-96ec-30f545797512-istio-podinfo\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:43:27.614865 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:27.613916 4908 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8aaac2b2-a4e5-4736-96ec-30f545797512-istio-token\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:43:27.650855 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:27.650660 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-0f7d14-openshift-default-cf79d4dcc-mcrqq" Oct 30 12:43:27.651710 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:27.651687 4908 patch_prober.go:29] interesting pod/e2e-extmod-b-0f7d14-openshift-default-cf79d4dcc-mcrqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" start-of-body= Oct 30 12:43:27.651806 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:27.651731 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0f7d14-openshift-default-cf79d4dcc-mcrqq" podUID="3590e8ff-89a3-482b-b022-064bb84fe09c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" Oct 30 12:43:27.898968 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:27.898867 4908 generic.go:356] "Generic (PLEG): container finished" podID="8aaac2b2-a4e5-4736-96ec-30f545797512" containerID="67f327e201fe008df2005b78d98003d09f2872c93d4faded29e77d32594d65e1" exitCode=0 Oct 30 12:43:27.898968 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:27.898933 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-wq96b" event={"ID":"8aaac2b2-a4e5-4736-96ec-30f545797512","Type":"ContainerDied","Data":"67f327e201fe008df2005b78d98003d09f2872c93d4faded29e77d32594d65e1"} Oct 30 12:43:27.899495 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:27.899011 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-wq96b" event={"ID":"8aaac2b2-a4e5-4736-96ec-30f545797512","Type":"ContainerDied","Data":"38c07fda28d85a100a06f14d7d7295706bad8bfab68329943afe1b43df0930fb"} Oct 30 12:43:27.899495 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:27.899039 4908 scope.go:122] "RemoveContainer" containerID="67f327e201fe008df2005b78d98003d09f2872c93d4faded29e77d32594d65e1" Oct 30 12:43:27.899495 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:27.899477 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-wq96b" Oct 30 12:43:27.912560 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:27.912530 4908 scope.go:122] "RemoveContainer" containerID="67f327e201fe008df2005b78d98003d09f2872c93d4faded29e77d32594d65e1" Oct 30 12:43:27.912808 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:43:27.912789 4908 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67f327e201fe008df2005b78d98003d09f2872c93d4faded29e77d32594d65e1\": container with ID starting with 67f327e201fe008df2005b78d98003d09f2872c93d4faded29e77d32594d65e1 not found: ID does not exist" containerID="67f327e201fe008df2005b78d98003d09f2872c93d4faded29e77d32594d65e1" Oct 30 12:43:27.912912 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:27.912817 4908 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67f327e201fe008df2005b78d98003d09f2872c93d4faded29e77d32594d65e1"} err="failed to get container status \"67f327e201fe008df2005b78d98003d09f2872c93d4faded29e77d32594d65e1\": rpc error: code = NotFound desc = could not find container \"67f327e201fe008df2005b78d98003d09f2872c93d4faded29e77d32594d65e1\": container with ID starting with 67f327e201fe008df2005b78d98003d09f2872c93d4faded29e77d32594d65e1 not found: ID does not exist" Oct 30 12:43:27.954068 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:27.954043 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-wq96b"] Oct 30 12:43:27.984851 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:27.984153 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-wq96b"] Oct 30 12:43:28.651598 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:28.651566 4908 patch_prober.go:29] interesting pod/e2e-extmod-b-0f7d14-openshift-default-cf79d4dcc-mcrqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" start-of-body= Oct 30 12:43:28.651772 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:28.651614 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0f7d14-openshift-default-cf79d4dcc-mcrqq" podUID="3590e8ff-89a3-482b-b022-064bb84fe09c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" Oct 30 12:43:29.611377 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:29.611333 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8aaac2b2-a4e5-4736-96ec-30f545797512" path="/var/lib/kubelet/pods/8aaac2b2-a4e5-4736-96ec-30f545797512/volumes" Oct 30 12:43:29.650764 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:29.650733 4908 patch_prober.go:29] interesting pod/e2e-extmod-b-0f7d14-openshift-default-cf79d4dcc-mcrqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" start-of-body= Oct 30 12:43:29.650953 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:29.650788 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0f7d14-openshift-default-cf79d4dcc-mcrqq" podUID="3590e8ff-89a3-482b-b022-064bb84fe09c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" Oct 30 12:43:29.903185 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:29.903073 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-0f7d14-openshift-default-cf79d4dcc-mcrqq" Oct 30 12:43:30.651516 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:30.651481 4908 patch_prober.go:29] interesting pod/e2e-extmod-b-0f7d14-openshift-default-cf79d4dcc-mcrqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" start-of-body= Oct 30 12:43:30.651920 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:30.651529 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0f7d14-openshift-default-cf79d4dcc-mcrqq" podUID="3590e8ff-89a3-482b-b022-064bb84fe09c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" Oct 30 12:43:31.650918 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:31.650888 4908 patch_prober.go:29] interesting pod/e2e-extmod-b-0f7d14-openshift-default-cf79d4dcc-mcrqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" start-of-body= Oct 30 12:43:31.651087 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:31.650937 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0f7d14-openshift-default-cf79d4dcc-mcrqq" podUID="3590e8ff-89a3-482b-b022-064bb84fe09c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" Oct 30 12:43:32.656290 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:32.656255 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-0f7d14-openshift-default-cf79d4dcc-mcrqq" Oct 30 12:43:32.657231 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:32.657210 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-0f7d14-openshift-default-cf79d4dcc-mcrqq" Oct 30 12:43:34.990852 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:34.990797 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9wwfc"] Oct 30 12:43:35.590387 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:35.590355 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-ac674b/autores-model-ac674b-kserve-554dc45f77-gzg4p"] Oct 30 12:43:35.972975 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:35.972887 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-ac674b-ccrq5"] Oct 30 12:43:35.974084 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:35.974049 4908 cpu_manager.go:414] "containerMap: removing container" podUID="8aaac2b2-a4e5-4736-96ec-30f545797512" containerName="istio-proxy" Oct 30 12:43:35.974253 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:35.974236 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8aaac2b2-a4e5-4736-96ec-30f545797512" containerName="istio-proxy" Oct 30 12:43:35.974546 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:35.974530 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="8aaac2b2-a4e5-4736-96ec-30f545797512" containerName="istio-proxy" Oct 30 12:43:35.995985 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:35.990099 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-ac674b-ccrq5" Oct 30 12:43:36.006846 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:36.005236 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 12:43:36.006846 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:36.005351 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s75hq\"" Oct 30 12:43:36.076272 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:36.015923 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-ac674b-ccrq5"] Oct 30 12:43:36.244855 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:36.240179 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 12:43:36.244855 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:36.241745 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 12:43:36.252605 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:36.252466 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04afeca7-6667-4d76-aa58-33b9b37d9074-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-autores-ac674b-ccrq5\" (UID: \"04afeca7-6667-4d76-aa58-33b9b37d9074\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-ac674b-ccrq5" Oct 30 12:43:36.252879 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:36.252856 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04afeca7-6667-4d76-aa58-33b9b37d9074-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-autores-ac674b-ccrq5\" (UID: \"04afeca7-6667-4d76-aa58-33b9b37d9074\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-ac674b-ccrq5" Oct 30 12:43:36.253077 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:36.253061 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdbdh\" (UniqueName: \"kubernetes.io/projected/04afeca7-6667-4d76-aa58-33b9b37d9074-kube-api-access-sdbdh\") pod \"maas-api-revoke-keys-e2e-autores-ac674b-ccrq5\" (UID: \"04afeca7-6667-4d76-aa58-33b9b37d9074\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-ac674b-ccrq5" Oct 30 12:43:36.355087 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:36.355046 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04afeca7-6667-4d76-aa58-33b9b37d9074-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-autores-ac674b-ccrq5\" (UID: \"04afeca7-6667-4d76-aa58-33b9b37d9074\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-ac674b-ccrq5" Oct 30 12:43:36.355291 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:36.355102 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04afeca7-6667-4d76-aa58-33b9b37d9074-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-autores-ac674b-ccrq5\" (UID: \"04afeca7-6667-4d76-aa58-33b9b37d9074\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-ac674b-ccrq5" Oct 30 12:43:36.355291 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:36.355174 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdbdh\" (UniqueName: \"kubernetes.io/projected/04afeca7-6667-4d76-aa58-33b9b37d9074-kube-api-access-sdbdh\") pod \"maas-api-revoke-keys-e2e-autores-ac674b-ccrq5\" (UID: \"04afeca7-6667-4d76-aa58-33b9b37d9074\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-ac674b-ccrq5" Oct 30 12:43:36.355808 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:36.355777 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04afeca7-6667-4d76-aa58-33b9b37d9074-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-autores-ac674b-ccrq5\" (UID: \"04afeca7-6667-4d76-aa58-33b9b37d9074\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-ac674b-ccrq5" Oct 30 12:43:36.357792 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:36.357765 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04afeca7-6667-4d76-aa58-33b9b37d9074-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-autores-ac674b-ccrq5\" (UID: \"04afeca7-6667-4d76-aa58-33b9b37d9074\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-ac674b-ccrq5" Oct 30 12:43:36.379010 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:36.378984 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdbdh\" (UniqueName: \"kubernetes.io/projected/04afeca7-6667-4d76-aa58-33b9b37d9074-kube-api-access-sdbdh\") pod \"maas-api-revoke-keys-e2e-autores-ac674b-ccrq5\" (UID: \"04afeca7-6667-4d76-aa58-33b9b37d9074\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-ac674b-ccrq5" Oct 30 12:43:36.581367 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:36.581275 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-ac674b-ccrq5" Oct 30 12:43:36.930844 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:36.930808 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-ac674b-ccrq5"] Oct 30 12:43:36.931472 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:43:36.931434 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04afeca7_6667_4d76_aa58_33b9b37d9074.slice/crio-d6344562ce6d9754d343f4692ad78a2495fd1cb1d62e516080e25fbadd8bfbc8 WatchSource:0}: Error finding container d6344562ce6d9754d343f4692ad78a2495fd1cb1d62e516080e25fbadd8bfbc8: Status 404 returned error can't find the container with id d6344562ce6d9754d343f4692ad78a2495fd1cb1d62e516080e25fbadd8bfbc8 Oct 30 12:43:37.263690 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:37.263650 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-ac674b-ccrq5" event={"ID":"04afeca7-6667-4d76-aa58-33b9b37d9074","Type":"ContainerStarted","Data":"d6344562ce6d9754d343f4692ad78a2495fd1cb1d62e516080e25fbadd8bfbc8"} Oct 30 12:43:38.340387 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:38.340282 4908 generic.go:356] "Generic (PLEG): container finished" podID="04afeca7-6667-4d76-aa58-33b9b37d9074" containerID="2e4a993c889064abd572deb6699726c4c551100496810ade59ce8f88e2300604" exitCode=0 Oct 30 12:43:38.340906 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:38.340563 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-ac674b-ccrq5" event={"ID":"04afeca7-6667-4d76-aa58-33b9b37d9074","Type":"ContainerDied","Data":"2e4a993c889064abd572deb6699726c4c551100496810ade59ce8f88e2300604"} Oct 30 12:43:40.058064 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:40.058037 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-ac674b-ccrq5" Oct 30 12:43:40.227048 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:40.227013 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/04afeca7-6667-4d76-aa58-33b9b37d9074-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04afeca7-6667-4d76-aa58-33b9b37d9074-maas-api-xks-ca\") pod \"04afeca7-6667-4d76-aa58-33b9b37d9074\" (UID: \"04afeca7-6667-4d76-aa58-33b9b37d9074\") " Oct 30 12:43:40.227216 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:40.227085 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04afeca7-6667-4d76-aa58-33b9b37d9074-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04afeca7-6667-4d76-aa58-33b9b37d9074-maas-api-openshift-ca\") pod \"04afeca7-6667-4d76-aa58-33b9b37d9074\" (UID: \"04afeca7-6667-4d76-aa58-33b9b37d9074\") " Oct 30 12:43:40.227300 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:40.227244 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04afeca7-6667-4d76-aa58-33b9b37d9074-kube-api-access-sdbdh\" (UniqueName: \"kubernetes.io/projected/04afeca7-6667-4d76-aa58-33b9b37d9074-kube-api-access-sdbdh\") pod \"04afeca7-6667-4d76-aa58-33b9b37d9074\" (UID: \"04afeca7-6667-4d76-aa58-33b9b37d9074\") " Oct 30 12:43:40.227861 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:40.227806 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04afeca7-6667-4d76-aa58-33b9b37d9074-maas-api-openshift-ca" pod "04afeca7-6667-4d76-aa58-33b9b37d9074" (UID: "04afeca7-6667-4d76-aa58-33b9b37d9074"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:43:40.229225 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:40.229204 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04afeca7-6667-4d76-aa58-33b9b37d9074-kube-api-access-sdbdh" pod "04afeca7-6667-4d76-aa58-33b9b37d9074" (UID: "04afeca7-6667-4d76-aa58-33b9b37d9074"). InnerVolumeSpecName "kube-api-access-sdbdh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:43:40.229519 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:40.229497 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04afeca7-6667-4d76-aa58-33b9b37d9074-maas-api-xks-ca" pod "04afeca7-6667-4d76-aa58-33b9b37d9074" (UID: "04afeca7-6667-4d76-aa58-33b9b37d9074"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:43:40.329159 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:40.329063 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04afeca7-6667-4d76-aa58-33b9b37d9074-maas-api-openshift-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:43:40.329159 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:40.329101 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdbdh\" (UniqueName: \"kubernetes.io/projected/04afeca7-6667-4d76-aa58-33b9b37d9074-kube-api-access-sdbdh\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:43:40.329159 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:40.329119 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04afeca7-6667-4d76-aa58-33b9b37d9074-maas-api-xks-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:43:40.449981 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:40.449943 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-ac674b-ccrq5" event={"ID":"04afeca7-6667-4d76-aa58-33b9b37d9074","Type":"ContainerDied","Data":"d6344562ce6d9754d343f4692ad78a2495fd1cb1d62e516080e25fbadd8bfbc8"} Oct 30 12:43:40.450445 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:40.450423 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d6344562ce6d9754d343f4692ad78a2495fd1cb1d62e516080e25fbadd8bfbc8" Oct 30 12:43:40.450604 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:40.450372 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-ac674b-ccrq5" Oct 30 12:43:46.532216 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:46.532184 4908 generic.go:356] "Generic (PLEG): container finished" podID="cfd62d0e-d1b7-4695-8673-63c1d68f4f40" containerID="1f42a26fc0287f52ffa90881714b4c18f92ff184a9d87eb82271f54bfc0d8b5d" exitCode=0 Oct 30 12:43:46.532666 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:46.532326 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-0de4b9-openshift-default-5d57d77697-g7ct6" event={"ID":"cfd62d0e-d1b7-4695-8673-63c1d68f4f40","Type":"ContainerDied","Data":"1f42a26fc0287f52ffa90881714b4c18f92ff184a9d87eb82271f54bfc0d8b5d"} Oct 30 12:43:46.532748 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:46.532732 4908 scope.go:122] "RemoveContainer" containerID="1f42a26fc0287f52ffa90881714b4c18f92ff184a9d87eb82271f54bfc0d8b5d" Oct 30 12:43:46.534894 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:46.534862 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:43:46.535030 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:46.534951 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:43:46.535030 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:46.535000 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:43:47.541326 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:47.541292 4908 generic.go:356] "Generic (PLEG): container finished" podID="396046e4-b596-4fed-943d-a5f42fea4f77" containerID="5c019e2dea5de51003c7ab7c190f0cc27089c921e8f8515c1f3935a5ef128412" exitCode=0 Oct 30 12:43:47.541774 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:47.541366 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-12fd33-openshift-default-846b7dbb9-pt6kj" event={"ID":"396046e4-b596-4fed-943d-a5f42fea4f77","Type":"ContainerDied","Data":"5c019e2dea5de51003c7ab7c190f0cc27089c921e8f8515c1f3935a5ef128412"} Oct 30 12:43:47.541877 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:47.541792 4908 scope.go:122] "RemoveContainer" containerID="5c019e2dea5de51003c7ab7c190f0cc27089c921e8f8515c1f3935a5ef128412" Oct 30 12:43:47.543691 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:47.543664 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-0de4b9-openshift-default-5d57d77697-g7ct6" event={"ID":"cfd62d0e-d1b7-4695-8673-63c1d68f4f40","Type":"ContainerStarted","Data":"43ef3fa0e82eba6bb1b8aef922b2c49ee38df108ebba18778528befd76f665f5"} Oct 30 12:43:47.543792 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:47.543751 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:43:47.543861 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:47.543842 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:43:47.543914 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:47.543896 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:43:48.212109 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:48.212070 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-0de4b9-openshift-default-5d57d77697-g7ct6" Oct 30 12:43:48.212276 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:48.212202 4908 patch_prober.go:29] interesting pod/e2e-extmod-b-0de4b9-openshift-default-5d57d77697-g7ct6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" start-of-body= Oct 30 12:43:48.212276 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:48.212246 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0de4b9-openshift-default-5d57d77697-g7ct6" podUID="cfd62d0e-d1b7-4695-8673-63c1d68f4f40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" Oct 30 12:43:48.558230 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:48.558188 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-12fd33-openshift-default-846b7dbb9-pt6kj" event={"ID":"396046e4-b596-4fed-943d-a5f42fea4f77","Type":"ContainerStarted","Data":"f3fe57816ba5c65b5b600a2b3b6c2845d5ecb71b822de4f93e925e2573fc40f6"} Oct 30 12:43:49.049010 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:49.048975 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-12fd33-openshift-default-846b7dbb9-pt6kj" Oct 30 12:43:49.049183 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:49.049156 4908 patch_prober.go:29] interesting pod/e2e-extmod-b-12fd33-openshift-default-846b7dbb9-pt6kj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.235:15021/healthz/ready\": dial tcp 10.132.0.235:15021: connect: connection refused" start-of-body= Oct 30 12:43:49.049228 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:49.049201 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-12fd33-openshift-default-846b7dbb9-pt6kj" podUID="396046e4-b596-4fed-943d-a5f42fea4f77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.235:15021/healthz/ready\": dial tcp 10.132.0.235:15021: connect: connection refused" Oct 30 12:43:49.217850 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:49.213336 4908 patch_prober.go:29] interesting pod/e2e-extmod-b-0de4b9-openshift-default-5d57d77697-g7ct6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" start-of-body= Oct 30 12:43:49.217850 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:49.213426 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0de4b9-openshift-default-5d57d77697-g7ct6" podUID="cfd62d0e-d1b7-4695-8673-63c1d68f4f40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" Oct 30 12:43:49.747727 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:43:49.747685 4908 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vn6vv" containerName="extractor" Oct 30 12:43:50.048651 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:50.048553 4908 patch_prober.go:29] interesting pod/e2e-extmod-b-12fd33-openshift-default-846b7dbb9-pt6kj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.235:15021/healthz/ready\": dial tcp 10.132.0.235:15021: connect: connection refused" start-of-body= Oct 30 12:43:50.048651 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:50.048613 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-12fd33-openshift-default-846b7dbb9-pt6kj" podUID="396046e4-b596-4fed-943d-a5f42fea4f77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.235:15021/healthz/ready\": dial tcp 10.132.0.235:15021: connect: connection refused" Oct 30 12:43:50.214185 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:50.214150 4908 patch_prober.go:29] interesting pod/e2e-extmod-b-0de4b9-openshift-default-5d57d77697-g7ct6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" start-of-body= Oct 30 12:43:50.214460 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:50.214432 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0de4b9-openshift-default-5d57d77697-g7ct6" podUID="cfd62d0e-d1b7-4695-8673-63c1d68f4f40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" Oct 30 12:43:50.631234 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:50.630989 4908 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-autores-ac674b/autores-model-ac674b-kserve-554dc45f77-gzg4p" podUID="af1fdb0d-774f-462f-ad3a-828685e5c4eb" containerName="main" containerID="cri-o://df0529212ff0f4f223bac4e4d4c719104b116a7858d6f57b508c7032e8d1e7ad" gracePeriod=45 Oct 30 12:43:50.782862 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:50.780092 4908 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-autores-ac674b/autores-model-ac674b-kserve-554dc45f77-gzg4p" podUID="af1fdb0d-774f-462f-ad3a-828685e5c4eb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.237:8000/ready\": dial tcp 10.132.0.237:8000: connect: connection refused" Oct 30 12:43:51.048731 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:51.048699 4908 patch_prober.go:29] interesting pod/e2e-extmod-b-12fd33-openshift-default-846b7dbb9-pt6kj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.235:15021/healthz/ready\": dial tcp 10.132.0.235:15021: connect: connection refused" start-of-body= Oct 30 12:43:51.048930 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:51.048762 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-12fd33-openshift-default-846b7dbb9-pt6kj" podUID="396046e4-b596-4fed-943d-a5f42fea4f77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.235:15021/healthz/ready\": dial tcp 10.132.0.235:15021: connect: connection refused" Oct 30 12:43:51.132560 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:51.132243 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-ac674b/autores-model-ac674b-kserve-554dc45f77-gzg4p" Oct 30 12:43:51.271815 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:51.259704 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af1fdb0d-774f-462f-ad3a-828685e5c4eb-kube-api-access-6tn7c\" (UniqueName: \"kubernetes.io/projected/af1fdb0d-774f-462f-ad3a-828685e5c4eb-kube-api-access-6tn7c\") pod \"af1fdb0d-774f-462f-ad3a-828685e5c4eb\" (UID: \"af1fdb0d-774f-462f-ad3a-828685e5c4eb\") " Oct 30 12:43:51.271815 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:51.259801 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af1fdb0d-774f-462f-ad3a-828685e5c4eb-tls-certs\" (UniqueName: \"kubernetes.io/secret/af1fdb0d-774f-462f-ad3a-828685e5c4eb-tls-certs\") pod \"af1fdb0d-774f-462f-ad3a-828685e5c4eb\" (UID: \"af1fdb0d-774f-462f-ad3a-828685e5c4eb\") " Oct 30 12:43:51.271815 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:51.259970 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af1fdb0d-774f-462f-ad3a-828685e5c4eb-home\" (UniqueName: \"kubernetes.io/empty-dir/af1fdb0d-774f-462f-ad3a-828685e5c4eb-home\") pod \"af1fdb0d-774f-462f-ad3a-828685e5c4eb\" (UID: \"af1fdb0d-774f-462f-ad3a-828685e5c4eb\") " Oct 30 12:43:51.271815 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:51.260090 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af1fdb0d-774f-462f-ad3a-828685e5c4eb-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af1fdb0d-774f-462f-ad3a-828685e5c4eb-tmp-dir\") pod \"af1fdb0d-774f-462f-ad3a-828685e5c4eb\" (UID: \"af1fdb0d-774f-462f-ad3a-828685e5c4eb\") " Oct 30 12:43:51.271815 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:51.260333 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af1fdb0d-774f-462f-ad3a-828685e5c4eb-dshm\" (UniqueName: \"kubernetes.io/empty-dir/af1fdb0d-774f-462f-ad3a-828685e5c4eb-dshm\") pod \"af1fdb0d-774f-462f-ad3a-828685e5c4eb\" (UID: \"af1fdb0d-774f-462f-ad3a-828685e5c4eb\") " Oct 30 12:43:51.271815 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:51.260418 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af1fdb0d-774f-462f-ad3a-828685e5c4eb-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af1fdb0d-774f-462f-ad3a-828685e5c4eb-model-cache\") pod \"af1fdb0d-774f-462f-ad3a-828685e5c4eb\" (UID: \"af1fdb0d-774f-462f-ad3a-828685e5c4eb\") " Oct 30 12:43:51.308902 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:51.283534 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af1fdb0d-774f-462f-ad3a-828685e5c4eb-model-cache" pod "af1fdb0d-774f-462f-ad3a-828685e5c4eb" (UID: "af1fdb0d-774f-462f-ad3a-828685e5c4eb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:43:51.308902 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:51.292447 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af1fdb0d-774f-462f-ad3a-828685e5c4eb-tmp-dir" pod "af1fdb0d-774f-462f-ad3a-828685e5c4eb" (UID: "af1fdb0d-774f-462f-ad3a-828685e5c4eb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:43:51.309280 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:51.309224 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af1fdb0d-774f-462f-ad3a-828685e5c4eb-home" pod "af1fdb0d-774f-462f-ad3a-828685e5c4eb" (UID: "af1fdb0d-774f-462f-ad3a-828685e5c4eb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:43:51.309521 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:51.309501 4908 patch_prober.go:29] interesting pod/e2e-extmod-b-0de4b9-openshift-default-5d57d77697-g7ct6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" start-of-body= Oct 30 12:43:51.309665 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:51.309643 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0de4b9-openshift-default-5d57d77697-g7ct6" podUID="cfd62d0e-d1b7-4695-8673-63c1d68f4f40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" Oct 30 12:43:51.354777 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:51.354194 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af1fdb0d-774f-462f-ad3a-828685e5c4eb-kube-api-access-6tn7c" pod "af1fdb0d-774f-462f-ad3a-828685e5c4eb" (UID: "af1fdb0d-774f-462f-ad3a-828685e5c4eb"). InnerVolumeSpecName "kube-api-access-6tn7c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:43:51.354777 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:51.354284 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af1fdb0d-774f-462f-ad3a-828685e5c4eb-dshm" pod "af1fdb0d-774f-462f-ad3a-828685e5c4eb" (UID: "af1fdb0d-774f-462f-ad3a-828685e5c4eb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:43:51.358184 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:51.358160 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af1fdb0d-774f-462f-ad3a-828685e5c4eb-tls-certs" pod "af1fdb0d-774f-462f-ad3a-828685e5c4eb" (UID: "af1fdb0d-774f-462f-ad3a-828685e5c4eb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:43:51.409974 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:51.367881 4908 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af1fdb0d-774f-462f-ad3a-828685e5c4eb-model-cache\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:43:51.409974 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:51.367914 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6tn7c\" (UniqueName: \"kubernetes.io/projected/af1fdb0d-774f-462f-ad3a-828685e5c4eb-kube-api-access-6tn7c\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:43:51.409974 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:51.367931 4908 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af1fdb0d-774f-462f-ad3a-828685e5c4eb-tls-certs\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:43:51.409974 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:51.367944 4908 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af1fdb0d-774f-462f-ad3a-828685e5c4eb-home\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:43:51.409974 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:51.367956 4908 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af1fdb0d-774f-462f-ad3a-828685e5c4eb-tmp-dir\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:43:51.409974 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:51.367968 4908 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af1fdb0d-774f-462f-ad3a-828685e5c4eb-dshm\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:43:51.647105 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:51.647031 4908 generic.go:356] "Generic (PLEG): container finished" podID="af1fdb0d-774f-462f-ad3a-828685e5c4eb" containerID="df0529212ff0f4f223bac4e4d4c719104b116a7858d6f57b508c7032e8d1e7ad" exitCode=0 Oct 30 12:43:51.647275 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:51.647200 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-ac674b/autores-model-ac674b-kserve-554dc45f77-gzg4p" Oct 30 12:43:51.647848 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:51.647802 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-ac674b/autores-model-ac674b-kserve-554dc45f77-gzg4p" event={"ID":"af1fdb0d-774f-462f-ad3a-828685e5c4eb","Type":"ContainerDied","Data":"df0529212ff0f4f223bac4e4d4c719104b116a7858d6f57b508c7032e8d1e7ad"} Oct 30 12:43:51.647933 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:51.647868 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-ac674b/autores-model-ac674b-kserve-554dc45f77-gzg4p" event={"ID":"af1fdb0d-774f-462f-ad3a-828685e5c4eb","Type":"ContainerDied","Data":"3cf58bce0567bff03cacce23c77334c14dde3973bd07f34aceed51285a70f72d"} Oct 30 12:43:51.647933 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:51.647894 4908 scope.go:122] "RemoveContainer" containerID="df0529212ff0f4f223bac4e4d4c719104b116a7858d6f57b508c7032e8d1e7ad" Oct 30 12:43:51.693357 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:51.685989 4908 scope.go:122] "RemoveContainer" containerID="df0529212ff0f4f223bac4e4d4c719104b116a7858d6f57b508c7032e8d1e7ad" Oct 30 12:43:51.693357 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:43:51.693221 4908 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df0529212ff0f4f223bac4e4d4c719104b116a7858d6f57b508c7032e8d1e7ad\": container with ID starting with df0529212ff0f4f223bac4e4d4c719104b116a7858d6f57b508c7032e8d1e7ad not found: ID does not exist" containerID="df0529212ff0f4f223bac4e4d4c719104b116a7858d6f57b508c7032e8d1e7ad" Oct 30 12:43:51.693357 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:51.693264 4908 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df0529212ff0f4f223bac4e4d4c719104b116a7858d6f57b508c7032e8d1e7ad"} err="failed to get container status \"df0529212ff0f4f223bac4e4d4c719104b116a7858d6f57b508c7032e8d1e7ad\": rpc error: code = NotFound desc = could not find container \"df0529212ff0f4f223bac4e4d4c719104b116a7858d6f57b508c7032e8d1e7ad\": container with ID starting with df0529212ff0f4f223bac4e4d4c719104b116a7858d6f57b508c7032e8d1e7ad not found: ID does not exist" Oct 30 12:43:51.754158 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:51.754103 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-ac674b/autores-model-ac674b-kserve-554dc45f77-gzg4p"] Oct 30 12:43:51.835005 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:51.834952 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-ac674b/autores-model-ac674b-kserve-554dc45f77-gzg4p"] Oct 30 12:43:52.049170 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:52.049136 4908 patch_prober.go:29] interesting pod/e2e-extmod-b-12fd33-openshift-default-846b7dbb9-pt6kj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.235:15021/healthz/ready\": dial tcp 10.132.0.235:15021: connect: connection refused" start-of-body= Oct 30 12:43:52.049389 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:52.049195 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-12fd33-openshift-default-846b7dbb9-pt6kj" podUID="396046e4-b596-4fed-943d-a5f42fea4f77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.235:15021/healthz/ready\": dial tcp 10.132.0.235:15021: connect: connection refused" Oct 30 12:43:52.212851 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:52.212194 4908 patch_prober.go:29] interesting pod/e2e-extmod-b-0de4b9-openshift-default-5d57d77697-g7ct6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" start-of-body= Oct 30 12:43:52.212851 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:52.212276 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0de4b9-openshift-default-5d57d77697-g7ct6" podUID="cfd62d0e-d1b7-4695-8673-63c1d68f4f40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" Oct 30 12:43:53.050854 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:53.048717 4908 patch_prober.go:29] interesting pod/e2e-extmod-b-12fd33-openshift-default-846b7dbb9-pt6kj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.235:15021/healthz/ready\": dial tcp 10.132.0.235:15021: connect: connection refused" start-of-body= Oct 30 12:43:53.050854 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:53.048776 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-12fd33-openshift-default-846b7dbb9-pt6kj" podUID="396046e4-b596-4fed-943d-a5f42fea4f77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.235:15021/healthz/ready\": dial tcp 10.132.0.235:15021: connect: connection refused" Oct 30 12:43:53.224871 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:53.217178 4908 patch_prober.go:29] interesting pod/e2e-extmod-b-0de4b9-openshift-default-5d57d77697-g7ct6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" start-of-body= Oct 30 12:43:53.224871 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:53.217236 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0de4b9-openshift-default-5d57d77697-g7ct6" podUID="cfd62d0e-d1b7-4695-8673-63c1d68f4f40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" Oct 30 12:43:53.620220 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:53.620182 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af1fdb0d-774f-462f-ad3a-828685e5c4eb" path="/var/lib/kubelet/pods/af1fdb0d-774f-462f-ad3a-828685e5c4eb/volumes" Oct 30 12:43:54.049166 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:54.049132 4908 patch_prober.go:29] interesting pod/e2e-extmod-b-12fd33-openshift-default-846b7dbb9-pt6kj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.235:15021/healthz/ready\": dial tcp 10.132.0.235:15021: connect: connection refused" start-of-body= Oct 30 12:43:54.049373 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:54.049185 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-12fd33-openshift-default-846b7dbb9-pt6kj" podUID="396046e4-b596-4fed-943d-a5f42fea4f77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.235:15021/healthz/ready\": dial tcp 10.132.0.235:15021: connect: connection refused" Oct 30 12:43:54.217077 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:54.217041 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-0de4b9-openshift-default-5d57d77697-g7ct6" Oct 30 12:43:54.217558 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:54.217534 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-0de4b9-openshift-default-5d57d77697-g7ct6" Oct 30 12:43:54.218385 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:54.218364 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-0de4b9-openshift-default-5d57d77697-g7ct6" Oct 30 12:43:55.053370 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:55.053342 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-12fd33-openshift-default-846b7dbb9-pt6kj" Oct 30 12:43:55.053718 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:55.053693 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-12fd33-openshift-default-846b7dbb9-pt6kj" Oct 30 12:43:55.054365 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:43:55.054346 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-12fd33-openshift-default-846b7dbb9-pt6kj" Oct 30 12:44:06.172595 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:44:06.172557 4908 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-plczt_a70ee11c-c20e-4a4f-8476-ac5eb02816d0/console-operator/3.log" Oct 30 12:44:06.181657 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:44:06.179253 4908 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b8bxz_644e19b3-10ad-4145-9ed3-595d19ed9b07/ovn-acl-logging/0.log" Oct 30 12:44:06.867514 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:44:06.867487 4908 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-plczt_a70ee11c-c20e-4a4f-8476-ac5eb02816d0/console-operator/3.log" Oct 30 12:44:06.876052 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:44:06.876021 4908 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b8bxz_644e19b3-10ad-4145-9ed3-595d19ed9b07/ovn-acl-logging/0.log" Oct 30 12:44:07.640892 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:44:07.640684 4908 scope.go:122] "RemoveContainer" containerID="9da6929b078acf015a15b8646c7a7e0b4e32efaf60e554d7e3f278e8d1cf6518" Oct 30 12:44:07.660361 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:44:07.660325 4908 scope.go:122] "RemoveContainer" containerID="afeae25d85adb6805166772afe122092971f5d6d811cb082ee3b7dd28f40508f" Oct 30 12:44:08.460861 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:44:08.448043 4908 patch_prober.go:29] interesting pod/e2e-extmod-b-ecb3aa-openshift-default-68d9fff7cc-xb5ct container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused" start-of-body= Oct 30 12:44:08.460861 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:44:08.448125 4908 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-ecb3aa-openshift-default-68d9fff7cc-xb5ct" podUID="e59c9971-42c6-4df7-94a0-fd6753d847ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused" Oct 30 12:44:08.884528 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:44:08.884493 4908 generic.go:356] "Generic (PLEG): container finished" podID="e59c9971-42c6-4df7-94a0-fd6753d847ca" containerID="2964357075b0307575fbc5e3023d9e141572ec36c9be49b795754dcf26839c80" exitCode=0 Oct 30 12:44:08.884984 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:44:08.884567 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-ecb3aa-openshift-default-68d9fff7cc-xb5ct" event={"ID":"e59c9971-42c6-4df7-94a0-fd6753d847ca","Type":"ContainerDied","Data":"2964357075b0307575fbc5e3023d9e141572ec36c9be49b795754dcf26839c80"} Oct 30 12:44:08.885166 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:44:08.885139 4908 scope.go:122] "RemoveContainer" containerID="2964357075b0307575fbc5e3023d9e141572ec36c9be49b795754dcf26839c80" Oct 30 12:44:08.887465 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:44:08.887416 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:44:08.887567 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:44:08.887522 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:44:08.887619 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:44:08.887568 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:44:09.898775 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:44:09.898715 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-ecb3aa-openshift-default-68d9fff7cc-xb5ct" event={"ID":"e59c9971-42c6-4df7-94a0-fd6753d847ca","Type":"ContainerStarted","Data":"8ce08c9d4597660bc5284fdf10c7b97e8d93f9c4e7db07260f38a845bb89a568"} Oct 30 12:44:10.508372 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:44:10.502187 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-ecb3aa-openshift-default-68d9fff7cc-xb5ct" Oct 30 12:44:10.508372 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:44:10.502318 4908 patch_prober.go:29] interesting pod/e2e-extmod-b-ecb3aa-openshift-default-68d9fff7cc-xb5ct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused" start-of-body= Oct 30 12:44:10.508372 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:44:10.502362 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-ecb3aa-openshift-default-68d9fff7cc-xb5ct" podUID="e59c9971-42c6-4df7-94a0-fd6753d847ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused" Oct 30 12:44:11.499089 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:44:11.499049 4908 patch_prober.go:29] interesting pod/e2e-extmod-b-ecb3aa-openshift-default-68d9fff7cc-xb5ct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused" start-of-body= Oct 30 12:44:11.499507 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:44:11.499127 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-ecb3aa-openshift-default-68d9fff7cc-xb5ct" podUID="e59c9971-42c6-4df7-94a0-fd6753d847ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused" Oct 30 12:44:12.498360 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:44:12.498327 4908 patch_prober.go:29] interesting pod/e2e-extmod-b-ecb3aa-openshift-default-68d9fff7cc-xb5ct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused" start-of-body= Oct 30 12:44:12.498617 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:44:12.498585 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-ecb3aa-openshift-default-68d9fff7cc-xb5ct" podUID="e59c9971-42c6-4df7-94a0-fd6753d847ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused" Oct 30 12:44:13.498979 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:44:13.498937 4908 patch_prober.go:29] interesting pod/e2e-extmod-b-ecb3aa-openshift-default-68d9fff7cc-xb5ct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused" start-of-body= Oct 30 12:44:13.499377 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:44:13.498994 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-ecb3aa-openshift-default-68d9fff7cc-xb5ct" podUID="e59c9971-42c6-4df7-94a0-fd6753d847ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused" Oct 30 12:44:14.499122 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:44:14.499086 4908 patch_prober.go:29] interesting pod/e2e-extmod-b-ecb3aa-openshift-default-68d9fff7cc-xb5ct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused" start-of-body= Oct 30 12:44:14.499517 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:44:14.499137 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-ecb3aa-openshift-default-68d9fff7cc-xb5ct" podUID="e59c9971-42c6-4df7-94a0-fd6753d847ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused" Oct 30 12:44:15.503642 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:44:15.503612 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-ecb3aa-openshift-default-68d9fff7cc-xb5ct" Oct 30 12:44:15.504085 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:44:15.504060 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-ecb3aa-openshift-default-68d9fff7cc-xb5ct" Oct 30 12:44:15.504756 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:44:15.504734 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-ecb3aa-openshift-default-68d9fff7cc-xb5ct" Oct 30 12:44:19.684674 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:44:19.684637 4908 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vn6vv" containerName="extractor" Oct 30 12:44:29.924148 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:44:29.924108 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9wwfc"] Oct 30 12:44:32.887884 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:44:32.887808 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-ac674b-ccrq5"] Oct 30 12:44:32.891752 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:44:32.891726 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-ac674b-ccrq5"] Oct 30 12:44:33.610004 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:44:33.609966 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04afeca7-6667-4d76-aa58-33b9b37d9074" path="/var/lib/kubelet/pods/04afeca7-6667-4d76-aa58-33b9b37d9074/volumes" Oct 30 12:44:41.020038 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:44:41.020000 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-ac674b-openshift-default-798564ccc7-2nk9v"] Oct 30 12:44:41.020558 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:44:41.020192 4908 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-ac674b-openshift-default-798564ccc7-2nk9v" podUID="4a63f6b6-6cb7-4ed6-af39-c9fe090ebc87" containerName="istio-proxy" containerID="cri-o://3f4348813cb093d0971217b5f33073f4bde153ca653c9da4216a6c886c39ab38" gracePeriod=30 Oct 30 12:44:43.375227 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:44:43.375186 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-10f364-openshift-default-965d4bfd5-4b57q"] Oct 30 12:44:43.375917 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:44:43.375897 4908 cpu_manager.go:414] "containerMap: removing container" podUID="af1fdb0d-774f-462f-ad3a-828685e5c4eb" containerName="main" Oct 30 12:44:43.375917 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:44:43.375917 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af1fdb0d-774f-462f-ad3a-828685e5c4eb" containerName="main" Oct 30 12:44:43.376100 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:44:43.375964 4908 cpu_manager.go:414] "containerMap: removing container" podUID="04afeca7-6667-4d76-aa58-33b9b37d9074" containerName="revoke-keys" Oct 30 12:44:43.376100 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:44:43.375972 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04afeca7-6667-4d76-aa58-33b9b37d9074" containerName="revoke-keys" Oct 30 12:44:43.376224 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:44:43.376101 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="af1fdb0d-774f-462f-ad3a-828685e5c4eb" containerName="main" Oct 30 12:44:43.376224 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:44:43.376118 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="04afeca7-6667-4d76-aa58-33b9b37d9074" containerName="revoke-keys" Oct 30 12:44:43.379885 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:44:43.379865 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-10f364-openshift-default-965d4bfd5-4b57q" Oct 30 12:44:43.394231 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:44:43.394200 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-10f364-openshift-default-dockercfg-x4h7d\"" Oct 30 12:44:43.407033 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:44:43.406998 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa19be03-336a-4af4-aba5-e089b4933bf9-istio-token\") pod \"e2e-extmod-a-10f364-openshift-default-965d4bfd5-4b57q\" (UID: \"aa19be03-336a-4af4-aba5-e089b4933bf9\") " pod="openshift-ingress/e2e-extmod-a-10f364-openshift-default-965d4bfd5-4b57q" Oct 30 12:44:43.407033 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:44:43.407031 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa19be03-336a-4af4-aba5-e089b4933bf9-workload-certs\") pod \"e2e-extmod-a-10f364-openshift-default-965d4bfd5-4b57q\" (UID: \"aa19be03-336a-4af4-aba5-e089b4933bf9\") " pod="openshift-ingress/e2e-extmod-a-10f364-openshift-default-965d4bfd5-4b57q" Oct 30 12:44:43.407235 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:44:43.407061 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa19be03-336a-4af4-aba5-e089b4933bf9-credential-socket\") pod \"e2e-extmod-a-10f364-openshift-default-965d4bfd5-4b57q\" (UID: \"aa19be03-336a-4af4-aba5-e089b4933bf9\") " pod="openshift-ingress/e2e-extmod-a-10f364-openshift-default-965d4bfd5-4b57q" Oct 30 12:44:43.407235 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:44:43.407137 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa19be03-336a-4af4-aba5-e089b4933bf9-istio-envoy\") pod \"e2e-extmod-a-10f364-openshift-default-965d4bfd5-4b57q\" (UID: \"aa19be03-336a-4af4-aba5-e089b4933bf9\") " pod="openshift-ingress/e2e-extmod-a-10f364-openshift-default-965d4bfd5-4b57q" Oct 30 12:44:43.407358 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:44:43.407262 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa19be03-336a-4af4-aba5-e089b4933bf9-istio-data\") pod \"e2e-extmod-a-10f364-openshift-default-965d4bfd5-4b57q\" (UID: \"aa19be03-336a-4af4-aba5-e089b4933bf9\") " pod="openshift-ingress/e2e-extmod-a-10f364-openshift-default-965d4bfd5-4b57q" Oct 30 12:44:43.407417 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:44:43.407355 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa19be03-336a-4af4-aba5-e089b4933bf9-workload-socket\") pod \"e2e-extmod-a-10f364-openshift-default-965d4bfd5-4b57q\" (UID: \"aa19be03-336a-4af4-aba5-e089b4933bf9\") " pod="openshift-ingress/e2e-extmod-a-10f364-openshift-default-965d4bfd5-4b57q" Oct 30 12:44:43.407488 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:44:43.407444 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k587c\" (UniqueName: \"kubernetes.io/projected/aa19be03-336a-4af4-aba5-e089b4933bf9-kube-api-access-k587c\") pod \"e2e-extmod-a-10f364-openshift-default-965d4bfd5-4b57q\" (UID: \"aa19be03-336a-4af4-aba5-e089b4933bf9\") " pod="openshift-ingress/e2e-extmod-a-10f364-openshift-default-965d4bfd5-4b57q" Oct 30 12:44:43.407547 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:44:43.407486 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa19be03-336a-4af4-aba5-e089b4933bf9-istio-podinfo\") pod \"e2e-extmod-a-10f364-openshift-default-965d4bfd5-4b57q\" (UID: \"aa19be03-336a-4af4-aba5-e089b4933bf9\") " pod="openshift-ingress/e2e-extmod-a-10f364-openshift-default-965d4bfd5-4b57q" Oct 30 12:44:43.407669 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:44:43.407650 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa19be03-336a-4af4-aba5-e089b4933bf9-istiod-ca-cert\") pod \"e2e-extmod-a-10f364-openshift-default-965d4bfd5-4b57q\" (UID: \"aa19be03-336a-4af4-aba5-e089b4933bf9\") " pod="openshift-ingress/e2e-extmod-a-10f364-openshift-default-965d4bfd5-4b57q" Oct 30 12:44:43.422886 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:44:43.422858 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-10f364-openshift-default-965d4bfd5-4b57q"] Oct 30 12:44:43.515040 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:44:43.514987 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa19be03-336a-4af4-aba5-e089b4933bf9-istiod-ca-cert\") pod \"e2e-extmod-a-10f364-openshift-default-965d4bfd5-4b57q\" (UID: \"aa19be03-336a-4af4-aba5-e089b4933bf9\") " pod="openshift-ingress/e2e-extmod-a-10f364-openshift-default-965d4bfd5-4b57q" Oct 30 12:44:43.537792 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:44:43.537755 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa19be03-336a-4af4-aba5-e089b4933bf9-istio-token\") pod \"e2e-extmod-a-10f364-openshift-default-965d4bfd5-4b57q\" (UID: \"aa19be03-336a-4af4-aba5-e089b4933bf9\") " pod="openshift-ingress/e2e-extmod-a-10f364-openshift-default-965d4bfd5-4b57q" Oct 30 12:44:43.541953 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:44:43.541904 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa19be03-336a-4af4-aba5-e089b4933bf9-workload-certs\") pod \"e2e-extmod-a-10f364-openshift-default-965d4bfd5-4b57q\" (UID: \"aa19be03-336a-4af4-aba5-e089b4933bf9\") " pod="openshift-ingress/e2e-extmod-a-10f364-openshift-default-965d4bfd5-4b57q" Oct 30 12:44:43.546106 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:44:43.546061 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa19be03-336a-4af4-aba5-e089b4933bf9-credential-socket\") pod \"e2e-extmod-a-10f364-openshift-default-965d4bfd5-4b57q\" (UID: \"aa19be03-336a-4af4-aba5-e089b4933bf9\") " pod="openshift-ingress/e2e-extmod-a-10f364-openshift-default-965d4bfd5-4b57q" Oct 30 12:44:43.579737 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:44:43.554906 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa19be03-336a-4af4-aba5-e089b4933bf9-istio-envoy\") pod \"e2e-extmod-a-10f364-openshift-default-965d4bfd5-4b57q\" (UID: \"aa19be03-336a-4af4-aba5-e089b4933bf9\") " pod="openshift-ingress/e2e-extmod-a-10f364-openshift-default-965d4bfd5-4b57q" Oct 30 12:44:43.580075 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:44:43.580050 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa19be03-336a-4af4-aba5-e089b4933bf9-istio-data\") pod \"e2e-extmod-a-10f364-openshift-default-965d4bfd5-4b57q\" (UID: \"aa19be03-336a-4af4-aba5-e089b4933bf9\") " pod="openshift-ingress/e2e-extmod-a-10f364-openshift-default-965d4bfd5-4b57q" Oct 30 12:44:43.580316 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:44:43.580298 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa19be03-336a-4af4-aba5-e089b4933bf9-workload-socket\") pod \"e2e-extmod-a-10f364-openshift-default-965d4bfd5-4b57q\" (UID: \"aa19be03-336a-4af4-aba5-e089b4933bf9\") " pod="openshift-ingress/e2e-extmod-a-10f364-openshift-default-965d4bfd5-4b57q" Oct 30 12:44:43.580523 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:44:43.580508 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k587c\" (UniqueName: \"kubernetes.io/projected/aa19be03-336a-4af4-aba5-e089b4933bf9-kube-api-access-k587c\") pod \"e2e-extmod-a-10f364-openshift-default-965d4bfd5-4b57q\" (UID: \"aa19be03-336a-4af4-aba5-e089b4933bf9\") " pod="openshift-ingress/e2e-extmod-a-10f364-openshift-default-965d4bfd5-4b57q" Oct 30 12:44:43.580671 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:44:43.580654 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa19be03-336a-4af4-aba5-e089b4933bf9-istio-podinfo\") pod \"e2e-extmod-a-10f364-openshift-default-965d4bfd5-4b57q\" (UID: \"aa19be03-336a-4af4-aba5-e089b4933bf9\") " pod="openshift-ingress/e2e-extmod-a-10f364-openshift-default-965d4bfd5-4b57q" Oct 30 12:44:43.582103 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:44:43.537591 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa19be03-336a-4af4-aba5-e089b4933bf9-istiod-ca-cert\") pod \"e2e-extmod-a-10f364-openshift-default-965d4bfd5-4b57q\" (UID: \"aa19be03-336a-4af4-aba5-e089b4933bf9\") " pod="openshift-ingress/e2e-extmod-a-10f364-openshift-default-965d4bfd5-4b57q" Oct 30 12:44:43.582103 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:44:43.555366 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa19be03-336a-4af4-aba5-e089b4933bf9-credential-socket\") pod \"e2e-extmod-a-10f364-openshift-default-965d4bfd5-4b57q\" (UID: \"aa19be03-336a-4af4-aba5-e089b4933bf9\") " pod="openshift-ingress/e2e-extmod-a-10f364-openshift-default-965d4bfd5-4b57q" Oct 30 12:44:43.582103 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:44:43.581508 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa19be03-336a-4af4-aba5-e089b4933bf9-istio-data\") pod \"e2e-extmod-a-10f364-openshift-default-965d4bfd5-4b57q\" (UID: \"aa19be03-336a-4af4-aba5-e089b4933bf9\") " pod="openshift-ingress/e2e-extmod-a-10f364-openshift-default-965d4bfd5-4b57q" Oct 30 12:44:43.582103 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:44:43.556196 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa19be03-336a-4af4-aba5-e089b4933bf9-workload-certs\") pod \"e2e-extmod-a-10f364-openshift-default-965d4bfd5-4b57q\" (UID: \"aa19be03-336a-4af4-aba5-e089b4933bf9\") " pod="openshift-ingress/e2e-extmod-a-10f364-openshift-default-965d4bfd5-4b57q" Oct 30 12:44:43.582103 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:44:43.581777 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa19be03-336a-4af4-aba5-e089b4933bf9-workload-socket\") pod \"e2e-extmod-a-10f364-openshift-default-965d4bfd5-4b57q\" (UID: \"aa19be03-336a-4af4-aba5-e089b4933bf9\") " pod="openshift-ingress/e2e-extmod-a-10f364-openshift-default-965d4bfd5-4b57q" Oct 30 12:44:43.582770 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:44:43.582715 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa19be03-336a-4af4-aba5-e089b4933bf9-istio-envoy\") pod \"e2e-extmod-a-10f364-openshift-default-965d4bfd5-4b57q\" (UID: \"aa19be03-336a-4af4-aba5-e089b4933bf9\") " pod="openshift-ingress/e2e-extmod-a-10f364-openshift-default-965d4bfd5-4b57q" Oct 30 12:44:43.653021 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:44:43.652152 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa19be03-336a-4af4-aba5-e089b4933bf9-istio-token\") pod \"e2e-extmod-a-10f364-openshift-default-965d4bfd5-4b57q\" (UID: \"aa19be03-336a-4af4-aba5-e089b4933bf9\") " pod="openshift-ingress/e2e-extmod-a-10f364-openshift-default-965d4bfd5-4b57q" Oct 30 12:44:43.667857 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:44:43.667815 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa19be03-336a-4af4-aba5-e089b4933bf9-istio-podinfo\") pod \"e2e-extmod-a-10f364-openshift-default-965d4bfd5-4b57q\" (UID: \"aa19be03-336a-4af4-aba5-e089b4933bf9\") " pod="openshift-ingress/e2e-extmod-a-10f364-openshift-default-965d4bfd5-4b57q" Oct 30 12:44:43.735610 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:44:43.735518 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k587c\" (UniqueName: \"kubernetes.io/projected/aa19be03-336a-4af4-aba5-e089b4933bf9-kube-api-access-k587c\") pod \"e2e-extmod-a-10f364-openshift-default-965d4bfd5-4b57q\" (UID: \"aa19be03-336a-4af4-aba5-e089b4933bf9\") " pod="openshift-ingress/e2e-extmod-a-10f364-openshift-default-965d4bfd5-4b57q" Oct 30 12:44:43.982584 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:44:43.982537 4908 patch_prober.go:29] interesting pod/e2e-autores-ac674b-openshift-default-798564ccc7-2nk9v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 12:44:43.982791 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:44:43.982614 4908 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-autores-ac674b-openshift-default-798564ccc7-2nk9v" podUID="4a63f6b6-6cb7-4ed6-af39-c9fe090ebc87" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 12:44:43.995315 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:44:43.995284 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-10f364-openshift-default-965d4bfd5-4b57q" Oct 30 12:44:44.381973 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:44:44.381895 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-10f364-openshift-default-965d4bfd5-4b57q"] Oct 30 12:44:44.386677 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:44:44.386637 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa19be03_336a_4af4_aba5_e089b4933bf9.slice/crio-247a7c751781c9cfb4fff10470dcca3e23dd7d58d500dd5348bd0f9d7e9998a1 WatchSource:0}: Error finding container 247a7c751781c9cfb4fff10470dcca3e23dd7d58d500dd5348bd0f9d7e9998a1: Status 404 returned error can't find the container with id 247a7c751781c9cfb4fff10470dcca3e23dd7d58d500dd5348bd0f9d7e9998a1 Oct 30 12:44:44.389886 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:44:44.389823 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:44:44.390020 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:44:44.389920 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:44:44.390020 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:44:44.389952 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:44:45.268488 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:44:45.268447 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-10f364-openshift-default-965d4bfd5-4b57q" event={"ID":"aa19be03-336a-4af4-aba5-e089b4933bf9","Type":"ContainerStarted","Data":"240af9c35b77a33c0f8b0ff549be18a2e92490e84c801042f95eed7a94cdc084"} Oct 30 12:44:45.268672 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:44:45.268506 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-10f364-openshift-default-965d4bfd5-4b57q" event={"ID":"aa19be03-336a-4af4-aba5-e089b4933bf9","Type":"ContainerStarted","Data":"247a7c751781c9cfb4fff10470dcca3e23dd7d58d500dd5348bd0f9d7e9998a1"} Oct 30 12:44:45.316983 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:44:45.316915 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-10f364-openshift-default-965d4bfd5-4b57q" podStartSLOduration=2.3168976470000002 podStartE2EDuration="2.316897647s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:44:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:44:45.312644048 +0000 UTC m=+3041.827072794" watchObservedRunningTime="2026-10-30 12:44:45.316897647 +0000 UTC m=+3041.831326393" Oct 30 12:44:45.996071 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:44:45.996035 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-10f364-openshift-default-965d4bfd5-4b57q" Oct 30 12:44:46.003254 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:44:46.003229 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-10f364-openshift-default-965d4bfd5-4b57q" Oct 30 12:44:46.278416 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:44:46.278326 4908 generic.go:356] "Generic (PLEG): container finished" podID="4a63f6b6-6cb7-4ed6-af39-c9fe090ebc87" containerID="3f4348813cb093d0971217b5f33073f4bde153ca653c9da4216a6c886c39ab38" exitCode=0 Oct 30 12:44:46.278533 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:44:46.278444 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-ac674b-openshift-default-798564ccc7-2nk9v" event={"ID":"4a63f6b6-6cb7-4ed6-af39-c9fe090ebc87","Type":"ContainerDied","Data":"3f4348813cb093d0971217b5f33073f4bde153ca653c9da4216a6c886c39ab38"} Oct 30 12:44:46.278533 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:44:46.278501 4908 scope.go:122] "RemoveContainer" containerID="9ce960f38f4c4100401ee91fbb26ddb7fea096e58551a875564449b6d515b345" Oct 30 12:44:46.278882 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:44:46.278859 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-10f364-openshift-default-965d4bfd5-4b57q" Oct 30 12:44:46.285375 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:44:46.285354 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-10f364-openshift-default-965d4bfd5-4b57q" Oct 30 12:44:46.306767 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:44:46.306739 4908 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ce960f38f4c4100401ee91fbb26ddb7fea096e58551a875564449b6d515b345\": container with ID starting with 9ce960f38f4c4100401ee91fbb26ddb7fea096e58551a875564449b6d515b345 not found: ID does not exist" containerID="9ce960f38f4c4100401ee91fbb26ddb7fea096e58551a875564449b6d515b345" Oct 30 12:44:46.317655 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:44:46.310489 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-ac674b-openshift-default-798564ccc7-2nk9v" Oct 30 12:44:46.449767 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:44:46.449723 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a63f6b6-6cb7-4ed6-af39-c9fe090ebc87-kube-api-access-64gvm\" (UniqueName: \"kubernetes.io/projected/4a63f6b6-6cb7-4ed6-af39-c9fe090ebc87-kube-api-access-64gvm\") pod \"4a63f6b6-6cb7-4ed6-af39-c9fe090ebc87\" (UID: \"4a63f6b6-6cb7-4ed6-af39-c9fe090ebc87\") " Oct 30 12:44:46.450002 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:44:46.449800 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a63f6b6-6cb7-4ed6-af39-c9fe090ebc87-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a63f6b6-6cb7-4ed6-af39-c9fe090ebc87-credential-socket\") pod \"4a63f6b6-6cb7-4ed6-af39-c9fe090ebc87\" (UID: \"4a63f6b6-6cb7-4ed6-af39-c9fe090ebc87\") " Oct 30 12:44:46.450002 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:44:46.449876 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a63f6b6-6cb7-4ed6-af39-c9fe090ebc87-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a63f6b6-6cb7-4ed6-af39-c9fe090ebc87-istiod-ca-cert\") pod \"4a63f6b6-6cb7-4ed6-af39-c9fe090ebc87\" (UID: \"4a63f6b6-6cb7-4ed6-af39-c9fe090ebc87\") " Oct 30 12:44:46.450136 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:44:46.449992 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4a63f6b6-6cb7-4ed6-af39-c9fe090ebc87-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a63f6b6-6cb7-4ed6-af39-c9fe090ebc87-istio-podinfo\") pod \"4a63f6b6-6cb7-4ed6-af39-c9fe090ebc87\" (UID: \"4a63f6b6-6cb7-4ed6-af39-c9fe090ebc87\") " Oct 30 12:44:46.450136 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:44:46.450045 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a63f6b6-6cb7-4ed6-af39-c9fe090ebc87-istio-token\" (UniqueName: \"kubernetes.io/projected/4a63f6b6-6cb7-4ed6-af39-c9fe090ebc87-istio-token\") pod \"4a63f6b6-6cb7-4ed6-af39-c9fe090ebc87\" (UID: \"4a63f6b6-6cb7-4ed6-af39-c9fe090ebc87\") " Oct 30 12:44:46.450136 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:44:46.450090 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a63f6b6-6cb7-4ed6-af39-c9fe090ebc87-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a63f6b6-6cb7-4ed6-af39-c9fe090ebc87-workload-socket\") pod \"4a63f6b6-6cb7-4ed6-af39-c9fe090ebc87\" (UID: \"4a63f6b6-6cb7-4ed6-af39-c9fe090ebc87\") " Oct 30 12:44:46.450356 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:44:46.450154 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a63f6b6-6cb7-4ed6-af39-c9fe090ebc87-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a63f6b6-6cb7-4ed6-af39-c9fe090ebc87-istio-envoy\") pod \"4a63f6b6-6cb7-4ed6-af39-c9fe090ebc87\" (UID: \"4a63f6b6-6cb7-4ed6-af39-c9fe090ebc87\") " Oct 30 12:44:46.450356 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:44:46.450163 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a63f6b6-6cb7-4ed6-af39-c9fe090ebc87-credential-socket" pod "4a63f6b6-6cb7-4ed6-af39-c9fe090ebc87" (UID: "4a63f6b6-6cb7-4ed6-af39-c9fe090ebc87"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:44:46.450356 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:44:46.450241 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a63f6b6-6cb7-4ed6-af39-c9fe090ebc87-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a63f6b6-6cb7-4ed6-af39-c9fe090ebc87-workload-certs\") pod \"4a63f6b6-6cb7-4ed6-af39-c9fe090ebc87\" (UID: \"4a63f6b6-6cb7-4ed6-af39-c9fe090ebc87\") " Oct 30 12:44:46.450356 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:44:46.450245 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a63f6b6-6cb7-4ed6-af39-c9fe090ebc87-istiod-ca-cert" pod "4a63f6b6-6cb7-4ed6-af39-c9fe090ebc87" (UID: "4a63f6b6-6cb7-4ed6-af39-c9fe090ebc87"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:44:46.450356 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:44:46.450297 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a63f6b6-6cb7-4ed6-af39-c9fe090ebc87-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a63f6b6-6cb7-4ed6-af39-c9fe090ebc87-istio-data\") pod \"4a63f6b6-6cb7-4ed6-af39-c9fe090ebc87\" (UID: \"4a63f6b6-6cb7-4ed6-af39-c9fe090ebc87\") " Oct 30 12:44:46.450981 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:44:46.450838 4908 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a63f6b6-6cb7-4ed6-af39-c9fe090ebc87-credential-socket\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:44:46.450981 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:44:46.450866 4908 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a63f6b6-6cb7-4ed6-af39-c9fe090ebc87-istiod-ca-cert\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:44:46.450981 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:44:46.450941 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a63f6b6-6cb7-4ed6-af39-c9fe090ebc87-workload-socket" pod "4a63f6b6-6cb7-4ed6-af39-c9fe090ebc87" (UID: "4a63f6b6-6cb7-4ed6-af39-c9fe090ebc87"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:44:46.451245 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:44:46.451021 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a63f6b6-6cb7-4ed6-af39-c9fe090ebc87-workload-certs" pod "4a63f6b6-6cb7-4ed6-af39-c9fe090ebc87" (UID: "4a63f6b6-6cb7-4ed6-af39-c9fe090ebc87"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:44:46.451245 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:44:46.451074 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a63f6b6-6cb7-4ed6-af39-c9fe090ebc87-istio-data" pod "4a63f6b6-6cb7-4ed6-af39-c9fe090ebc87" (UID: "4a63f6b6-6cb7-4ed6-af39-c9fe090ebc87"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:44:46.457874 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:44:46.452882 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a63f6b6-6cb7-4ed6-af39-c9fe090ebc87-istio-token" pod "4a63f6b6-6cb7-4ed6-af39-c9fe090ebc87" (UID: "4a63f6b6-6cb7-4ed6-af39-c9fe090ebc87"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:44:46.457874 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:44:46.453050 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4a63f6b6-6cb7-4ed6-af39-c9fe090ebc87-istio-podinfo" pod "4a63f6b6-6cb7-4ed6-af39-c9fe090ebc87" (UID: "4a63f6b6-6cb7-4ed6-af39-c9fe090ebc87"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 12:44:46.457874 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:44:46.453276 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a63f6b6-6cb7-4ed6-af39-c9fe090ebc87-istio-envoy" pod "4a63f6b6-6cb7-4ed6-af39-c9fe090ebc87" (UID: "4a63f6b6-6cb7-4ed6-af39-c9fe090ebc87"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:44:46.458653 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:44:46.458624 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a63f6b6-6cb7-4ed6-af39-c9fe090ebc87-kube-api-access-64gvm" pod "4a63f6b6-6cb7-4ed6-af39-c9fe090ebc87" (UID: "4a63f6b6-6cb7-4ed6-af39-c9fe090ebc87"). InnerVolumeSpecName "kube-api-access-64gvm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:44:46.552506 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:44:46.552418 4908 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a63f6b6-6cb7-4ed6-af39-c9fe090ebc87-istio-token\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:44:46.552506 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:44:46.552447 4908 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a63f6b6-6cb7-4ed6-af39-c9fe090ebc87-workload-socket\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:44:46.552506 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:44:46.552459 4908 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a63f6b6-6cb7-4ed6-af39-c9fe090ebc87-istio-envoy\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:44:46.552506 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:44:46.552468 4908 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a63f6b6-6cb7-4ed6-af39-c9fe090ebc87-workload-certs\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:44:46.552506 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:44:46.552476 4908 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a63f6b6-6cb7-4ed6-af39-c9fe090ebc87-istio-data\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:44:46.552506 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:44:46.552487 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64gvm\" (UniqueName: \"kubernetes.io/projected/4a63f6b6-6cb7-4ed6-af39-c9fe090ebc87-kube-api-access-64gvm\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:44:46.552506 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:44:46.552496 4908 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a63f6b6-6cb7-4ed6-af39-c9fe090ebc87-istio-podinfo\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:44:47.289445 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:44:47.289409 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-ac674b-openshift-default-798564ccc7-2nk9v" Oct 30 12:44:47.289968 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:44:47.289414 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-ac674b-openshift-default-798564ccc7-2nk9v" event={"ID":"4a63f6b6-6cb7-4ed6-af39-c9fe090ebc87","Type":"ContainerDied","Data":"5b58a286df337aabc133243b1fd729691ea5849889a530aa4340e8be9a8a2c6c"} Oct 30 12:44:47.289968 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:44:47.289572 4908 scope.go:122] "RemoveContainer" containerID="3f4348813cb093d0971217b5f33073f4bde153ca653c9da4216a6c886c39ab38" Oct 30 12:44:47.356605 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:44:47.356574 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-ac674b-openshift-default-798564ccc7-2nk9v"] Oct 30 12:44:47.367999 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:44:47.367971 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-ac674b-openshift-default-798564ccc7-2nk9v"] Oct 30 12:44:47.606611 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:44:47.606530 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a63f6b6-6cb7-4ed6-af39-c9fe090ebc87" path="/var/lib/kubelet/pods/4a63f6b6-6cb7-4ed6-af39-c9fe090ebc87/volumes" Oct 30 12:44:49.687987 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:44:49.687943 4908 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vn6vv" containerName="extractor" Oct 30 12:45:00.171311 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:00.171273 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6ad1da-29889405-bbbrz"] Oct 30 12:45:00.172167 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:00.172141 4908 cpu_manager.go:414] "containerMap: removing container" podUID="4a63f6b6-6cb7-4ed6-af39-c9fe090ebc87" containerName="istio-proxy" Oct 30 12:45:00.172276 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:00.172168 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a63f6b6-6cb7-4ed6-af39-c9fe090ebc87" containerName="istio-proxy" Oct 30 12:45:00.172276 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:00.172249 4908 cpu_manager.go:414] "containerMap: removing container" podUID="4a63f6b6-6cb7-4ed6-af39-c9fe090ebc87" containerName="istio-proxy" Oct 30 12:45:00.172276 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:00.172259 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a63f6b6-6cb7-4ed6-af39-c9fe090ebc87" containerName="istio-proxy" Oct 30 12:45:00.172491 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:00.172423 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a63f6b6-6cb7-4ed6-af39-c9fe090ebc87" containerName="istio-proxy" Oct 30 12:45:00.177979 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:00.177949 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2545c7-29889405-x55rn"] Oct 30 12:45:00.178114 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:00.178078 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6ad1da-29889405-bbbrz" Oct 30 12:45:00.179132 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:00.179086 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a63f6b6-6cb7-4ed6-af39-c9fe090ebc87" containerName="istio-proxy" Oct 30 12:45:00.183445 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:00.183423 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0de4b9-29889405-4pk7b"] Oct 30 12:45:00.183882 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:00.183680 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2545c7-29889405-x55rn" Oct 30 12:45:00.188189 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:00.188170 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0f7d14-29889405-c7tnx"] Oct 30 12:45:00.188627 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:00.188396 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0de4b9-29889405-4pk7b" Oct 30 12:45:00.193794 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:00.193776 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ecb3aa-29889405-srcpd"] Oct 30 12:45:00.193907 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:00.193892 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0f7d14-29889405-c7tnx" Oct 30 12:45:00.197536 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:00.197518 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ecb3aa-29889405-srcpd" Oct 30 12:45:00.228378 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:00.228356 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 12:45:00.228500 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:00.228481 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 12:45:00.229134 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:00.229116 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s75hq\"" Oct 30 12:45:00.239980 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:00.239866 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6ad1da-29889405-bbbrz"] Oct 30 12:45:00.251365 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:00.251336 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66mc5\" (UniqueName: \"kubernetes.io/projected/12c31795-7957-4dff-83c3-d4d2c1d31aac-kube-api-access-66mc5\") pod \"maas-api-key-cleanup-e2e-extmod-b-0de4b9-29889405-4pk7b\" (UID: \"12c31795-7957-4dff-83c3-d4d2c1d31aac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0de4b9-29889405-4pk7b" Oct 30 12:45:00.251551 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:00.251532 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2jrz\" (UniqueName: \"kubernetes.io/projected/0b9ecbb7-1e58-4ac9-b913-ff46c0a02210-kube-api-access-w2jrz\") pod \"maas-api-key-cleanup-e2e-extmod-a-2545c7-29889405-x55rn\" (UID: \"0b9ecbb7-1e58-4ac9-b913-ff46c0a02210\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2545c7-29889405-x55rn" Oct 30 12:45:00.251635 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:00.251577 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3eef63f0-3b6c-4e62-bdc9-594ee9d4bd17-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-6ad1da-29889405-bbbrz\" (UID: \"3eef63f0-3b6c-4e62-bdc9-594ee9d4bd17\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6ad1da-29889405-bbbrz" Oct 30 12:45:00.251698 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:00.251646 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5l7d\" (UniqueName: \"kubernetes.io/projected/3eef63f0-3b6c-4e62-bdc9-594ee9d4bd17-kube-api-access-h5l7d\") pod \"maas-api-key-cleanup-e2e-extmod-a-6ad1da-29889405-bbbrz\" (UID: \"3eef63f0-3b6c-4e62-bdc9-594ee9d4bd17\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6ad1da-29889405-bbbrz" Oct 30 12:45:00.251759 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:00.251708 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/12c31795-7957-4dff-83c3-d4d2c1d31aac-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-0de4b9-29889405-4pk7b\" (UID: \"12c31795-7957-4dff-83c3-d4d2c1d31aac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0de4b9-29889405-4pk7b" Oct 30 12:45:00.251819 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:00.251772 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b9ecbb7-1e58-4ac9-b913-ff46c0a02210-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-2545c7-29889405-x55rn\" (UID: \"0b9ecbb7-1e58-4ac9-b913-ff46c0a02210\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2545c7-29889405-x55rn" Oct 30 12:45:00.262257 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:00.262224 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0de4b9-29889405-4pk7b"] Oct 30 12:45:00.264435 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:00.264405 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0f7d14-29889405-c7tnx"] Oct 30 12:45:00.267922 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:00.267893 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ecb3aa-29889405-srcpd"] Oct 30 12:45:00.271141 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:00.271117 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-9d45fb-29889405-2gz7s"] Oct 30 12:45:00.276309 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:00.276288 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f7464f-29889405-zzjr7"] Oct 30 12:45:00.276436 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:00.276416 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-9d45fb-29889405-2gz7s" Oct 30 12:45:00.316079 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:00.316043 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-73bafc-29889405-l27f6"] Oct 30 12:45:00.318745 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:00.318714 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f7464f-29889405-zzjr7" Oct 30 12:45:00.320863 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:00.320840 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-dd0628-29889405-cj8wq"] Oct 30 12:45:00.322126 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:00.322106 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-73bafc-29889405-l27f6" Oct 30 12:45:00.331018 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:00.327930 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-812990-29889405-stgg2"] Oct 30 12:45:00.331018 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:00.328018 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-dd0628-29889405-cj8wq" Oct 30 12:45:00.334224 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:00.334202 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f7464f-29889405-zzjr7"] Oct 30 12:45:00.334315 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:00.334245 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2545c7-29889405-x55rn"] Oct 30 12:45:00.334315 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:00.334271 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-9d45fb-29889405-2gz7s"] Oct 30 12:45:00.334315 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:00.334295 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-dd0628-29889405-cj8wq"] Oct 30 12:45:00.334476 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:00.334323 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-812990-29889405-stgg2" Oct 30 12:45:00.334850 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:00.334807 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-73bafc-29889405-l27f6"] Oct 30 12:45:00.340546 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:00.340526 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-812990-29889405-stgg2"] Oct 30 12:45:00.349398 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:00.349372 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-8d2170-29889405-82rzm"] Oct 30 12:45:00.352784 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:00.352746 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2jrz\" (UniqueName: \"kubernetes.io/projected/0b9ecbb7-1e58-4ac9-b913-ff46c0a02210-kube-api-access-w2jrz\") pod \"maas-api-key-cleanup-e2e-extmod-a-2545c7-29889405-x55rn\" (UID: \"0b9ecbb7-1e58-4ac9-b913-ff46c0a02210\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2545c7-29889405-x55rn" Oct 30 12:45:00.353100 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:00.353080 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3eef63f0-3b6c-4e62-bdc9-594ee9d4bd17-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-6ad1da-29889405-bbbrz\" (UID: \"3eef63f0-3b6c-4e62-bdc9-594ee9d4bd17\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6ad1da-29889405-bbbrz" Oct 30 12:45:00.353291 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:00.353274 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fcc90cb3-e12e-4b12-a50d-de740f552b40-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-9d45fb-29889405-2gz7s\" (UID: \"fcc90cb3-e12e-4b12-a50d-de740f552b40\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-9d45fb-29889405-2gz7s" Oct 30 12:45:00.353403 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:00.353392 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tllf9\" (UniqueName: \"kubernetes.io/projected/4d181a49-62cd-454a-9e03-380198da3a08-kube-api-access-tllf9\") pod \"maas-api-key-cleanup-e2e-extmod-b-0f7d14-29889405-c7tnx\" (UID: \"4d181a49-62cd-454a-9e03-380198da3a08\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0f7d14-29889405-c7tnx" Oct 30 12:45:00.353599 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:00.353542 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5l7d\" (UniqueName: \"kubernetes.io/projected/3eef63f0-3b6c-4e62-bdc9-594ee9d4bd17-kube-api-access-h5l7d\") pod \"maas-api-key-cleanup-e2e-extmod-a-6ad1da-29889405-bbbrz\" (UID: \"3eef63f0-3b6c-4e62-bdc9-594ee9d4bd17\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6ad1da-29889405-bbbrz" Oct 30 12:45:00.353691 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:00.353679 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3a015cd-f16d-42b4-abae-9e2dd114093e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-ecb3aa-29889405-srcpd\" (UID: \"a3a015cd-f16d-42b4-abae-9e2dd114093e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ecb3aa-29889405-srcpd" Oct 30 12:45:00.353780 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:00.353760 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3eef63f0-3b6c-4e62-bdc9-594ee9d4bd17-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-6ad1da-29889405-bbbrz\" (UID: \"3eef63f0-3b6c-4e62-bdc9-594ee9d4bd17\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6ad1da-29889405-bbbrz" Oct 30 12:45:00.353898 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:00.353772 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/12c31795-7957-4dff-83c3-d4d2c1d31aac-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-0de4b9-29889405-4pk7b\" (UID: \"12c31795-7957-4dff-83c3-d4d2c1d31aac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0de4b9-29889405-4pk7b" Oct 30 12:45:00.353975 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:00.353914 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55mhq\" (UniqueName: \"kubernetes.io/projected/a3a015cd-f16d-42b4-abae-9e2dd114093e-kube-api-access-55mhq\") pod \"maas-api-key-cleanup-e2e-extmod-b-ecb3aa-29889405-srcpd\" (UID: \"a3a015cd-f16d-42b4-abae-9e2dd114093e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ecb3aa-29889405-srcpd" Oct 30 12:45:00.353975 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:00.353960 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b9ecbb7-1e58-4ac9-b913-ff46c0a02210-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-2545c7-29889405-x55rn\" (UID: \"0b9ecbb7-1e58-4ac9-b913-ff46c0a02210\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2545c7-29889405-x55rn" Oct 30 12:45:00.354101 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:00.354041 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d181a49-62cd-454a-9e03-380198da3a08-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-0f7d14-29889405-c7tnx\" (UID: \"4d181a49-62cd-454a-9e03-380198da3a08\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0f7d14-29889405-c7tnx" Oct 30 12:45:00.354101 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:00.354061 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66mc5\" (UniqueName: \"kubernetes.io/projected/12c31795-7957-4dff-83c3-d4d2c1d31aac-kube-api-access-66mc5\") pod \"maas-api-key-cleanup-e2e-extmod-b-0de4b9-29889405-4pk7b\" (UID: \"12c31795-7957-4dff-83c3-d4d2c1d31aac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0de4b9-29889405-4pk7b" Oct 30 12:45:00.354101 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:00.354085 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfrb2\" (UniqueName: \"kubernetes.io/projected/fcc90cb3-e12e-4b12-a50d-de740f552b40-kube-api-access-dfrb2\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-9d45fb-29889405-2gz7s\" (UID: \"fcc90cb3-e12e-4b12-a50d-de740f552b40\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-9d45fb-29889405-2gz7s" Oct 30 12:45:00.354279 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:00.354113 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-8d2170-29889405-82rzm" Oct 30 12:45:00.354279 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:00.354235 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-327e31-29889405-lr598"] Oct 30 12:45:00.355117 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:00.355096 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/12c31795-7957-4dff-83c3-d4d2c1d31aac-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-0de4b9-29889405-4pk7b\" (UID: \"12c31795-7957-4dff-83c3-d4d2c1d31aac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0de4b9-29889405-4pk7b" Oct 30 12:45:00.357494 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:00.357311 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b9ecbb7-1e58-4ac9-b913-ff46c0a02210-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-2545c7-29889405-x55rn\" (UID: \"0b9ecbb7-1e58-4ac9-b913-ff46c0a02210\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2545c7-29889405-x55rn" Oct 30 12:45:00.358854 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:00.358818 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-327e31-29889405-lr598" Oct 30 12:45:00.359236 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:00.359215 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2675c0-29889405-n4dv7"] Oct 30 12:45:00.382022 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:00.381986 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-327e31-29889405-lr598"] Oct 30 12:45:00.382261 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:00.382248 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0bda67-29889405-q74jd"] Oct 30 12:45:00.382436 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:00.382101 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2675c0-29889405-n4dv7" Oct 30 12:45:00.389619 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:00.389600 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f920f7-29889405-fp46s"] Oct 30 12:45:00.389878 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:00.389704 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0bda67-29889405-q74jd" Oct 30 12:45:00.396760 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:00.396700 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5l7d\" (UniqueName: \"kubernetes.io/projected/3eef63f0-3b6c-4e62-bdc9-594ee9d4bd17-kube-api-access-h5l7d\") pod \"maas-api-key-cleanup-e2e-extmod-a-6ad1da-29889405-bbbrz\" (UID: \"3eef63f0-3b6c-4e62-bdc9-594ee9d4bd17\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6ad1da-29889405-bbbrz" Oct 30 12:45:00.398258 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:00.398235 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f920f7-29889405-fp46s" Oct 30 12:45:00.398944 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:00.398922 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66mc5\" (UniqueName: \"kubernetes.io/projected/12c31795-7957-4dff-83c3-d4d2c1d31aac-kube-api-access-66mc5\") pod \"maas-api-key-cleanup-e2e-extmod-b-0de4b9-29889405-4pk7b\" (UID: \"12c31795-7957-4dff-83c3-d4d2c1d31aac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0de4b9-29889405-4pk7b" Oct 30 12:45:00.403597 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:00.403541 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2675c0-29889405-n4dv7"] Oct 30 12:45:00.410878 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:00.410860 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-8d2170-29889405-82rzm"] Oct 30 12:45:00.425575 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:00.425512 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2jrz\" (UniqueName: \"kubernetes.io/projected/0b9ecbb7-1e58-4ac9-b913-ff46c0a02210-kube-api-access-w2jrz\") pod \"maas-api-key-cleanup-e2e-extmod-a-2545c7-29889405-x55rn\" (UID: \"0b9ecbb7-1e58-4ac9-b913-ff46c0a02210\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2545c7-29889405-x55rn" Oct 30 12:45:00.462503 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:00.462476 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0bda67-29889405-q74jd"] Oct 30 12:45:00.463903 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:00.463871 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/27d77dc7-aee3-45a4-8ad9-a48560889798-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-327e31-29889405-lr598\" (UID: \"27d77dc7-aee3-45a4-8ad9-a48560889798\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-327e31-29889405-lr598" Oct 30 12:45:00.464158 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:00.464139 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d181a49-62cd-454a-9e03-380198da3a08-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-0f7d14-29889405-c7tnx\" (UID: \"4d181a49-62cd-454a-9e03-380198da3a08\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0f7d14-29889405-c7tnx" Oct 30 12:45:00.464283 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:00.464267 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brfbn\" (UniqueName: \"kubernetes.io/projected/ccd6f000-a55d-4214-a830-faf2d200b688-kube-api-access-brfbn\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-2675c0-29889405-n4dv7\" (UID: \"ccd6f000-a55d-4214-a830-faf2d200b688\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2675c0-29889405-n4dv7" Oct 30 12:45:00.464393 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:00.464380 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfrb2\" (UniqueName: \"kubernetes.io/projected/fcc90cb3-e12e-4b12-a50d-de740f552b40-kube-api-access-dfrb2\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-9d45fb-29889405-2gz7s\" (UID: \"fcc90cb3-e12e-4b12-a50d-de740f552b40\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-9d45fb-29889405-2gz7s" Oct 30 12:45:00.464497 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:00.464484 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab18f563-fbfd-4dbb-8b81-277c223a30f4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-f7464f-29889405-zzjr7\" (UID: \"ab18f563-fbfd-4dbb-8b81-277c223a30f4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f7464f-29889405-zzjr7" Oct 30 12:45:00.464605 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:00.464591 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6j5w\" (UniqueName: \"kubernetes.io/projected/240c2309-8812-4591-9871-b16dcb9b6ff3-kube-api-access-z6j5w\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-dd0628-29889405-cj8wq\" (UID: \"240c2309-8812-4591-9871-b16dcb9b6ff3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-dd0628-29889405-cj8wq" Oct 30 12:45:00.464728 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:00.464716 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/240c2309-8812-4591-9871-b16dcb9b6ff3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-dd0628-29889405-cj8wq\" (UID: \"240c2309-8812-4591-9871-b16dcb9b6ff3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-dd0628-29889405-cj8wq" Oct 30 12:45:00.464848 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:00.464817 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mg5n2\" (UniqueName: \"kubernetes.io/projected/958d7012-017f-4cd4-a637-a44ec84babc6-kube-api-access-mg5n2\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-0bda67-29889405-q74jd\" (UID: \"958d7012-017f-4cd4-a637-a44ec84babc6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0bda67-29889405-q74jd" Oct 30 12:45:00.464970 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:00.464954 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlbzs\" (UniqueName: \"kubernetes.io/projected/1aa1b39f-f88b-4ddf-8ee1-cf567d9d480a-kube-api-access-mlbzs\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-73bafc-29889405-l27f6\" (UID: \"1aa1b39f-f88b-4ddf-8ee1-cf567d9d480a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-73bafc-29889405-l27f6" Oct 30 12:45:00.465084 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:00.465069 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1aa1b39f-f88b-4ddf-8ee1-cf567d9d480a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-73bafc-29889405-l27f6\" (UID: \"1aa1b39f-f88b-4ddf-8ee1-cf567d9d480a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-73bafc-29889405-l27f6" Oct 30 12:45:00.465197 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:00.465182 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbdcr\" (UniqueName: \"kubernetes.io/projected/ab18f563-fbfd-4dbb-8b81-277c223a30f4-kube-api-access-nbdcr\") pod \"maas-api-key-cleanup-e2e-extmod-a-f7464f-29889405-zzjr7\" (UID: \"ab18f563-fbfd-4dbb-8b81-277c223a30f4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f7464f-29889405-zzjr7" Oct 30 12:45:00.465296 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:00.465283 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lr9w\" (UniqueName: \"kubernetes.io/projected/e354caaa-fac9-485a-b960-7488fea0df0d-kube-api-access-4lr9w\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-812990-29889405-stgg2\" (UID: \"e354caaa-fac9-485a-b960-7488fea0df0d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-812990-29889405-stgg2" Oct 30 12:45:00.465444 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:00.465431 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2z45r\" (UniqueName: \"kubernetes.io/projected/ea9095cb-47a4-4ea4-b143-069f736c29b6-kube-api-access-2z45r\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-8d2170-29889405-82rzm\" (UID: \"ea9095cb-47a4-4ea4-b143-069f736c29b6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-8d2170-29889405-82rzm" Oct 30 12:45:00.465569 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:00.465557 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fcc90cb3-e12e-4b12-a50d-de740f552b40-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-9d45fb-29889405-2gz7s\" (UID: \"fcc90cb3-e12e-4b12-a50d-de740f552b40\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-9d45fb-29889405-2gz7s" Oct 30 12:45:00.465670 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:00.465658 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tllf9\" (UniqueName: \"kubernetes.io/projected/4d181a49-62cd-454a-9e03-380198da3a08-kube-api-access-tllf9\") pod \"maas-api-key-cleanup-e2e-extmod-b-0f7d14-29889405-c7tnx\" (UID: \"4d181a49-62cd-454a-9e03-380198da3a08\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0f7d14-29889405-c7tnx" Oct 30 12:45:00.465802 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:00.465790 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjbp8\" (UniqueName: \"kubernetes.io/projected/27d77dc7-aee3-45a4-8ad9-a48560889798-kube-api-access-sjbp8\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-327e31-29889405-lr598\" (UID: \"27d77dc7-aee3-45a4-8ad9-a48560889798\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-327e31-29889405-lr598" Oct 30 12:45:00.465926 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:00.465911 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ccd6f000-a55d-4214-a830-faf2d200b688-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-2675c0-29889405-n4dv7\" (UID: \"ccd6f000-a55d-4214-a830-faf2d200b688\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2675c0-29889405-n4dv7" Oct 30 12:45:00.466033 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:00.466020 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3a015cd-f16d-42b4-abae-9e2dd114093e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-ecb3aa-29889405-srcpd\" (UID: \"a3a015cd-f16d-42b4-abae-9e2dd114093e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ecb3aa-29889405-srcpd" Oct 30 12:45:00.466132 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:00.466119 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e354caaa-fac9-485a-b960-7488fea0df0d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-812990-29889405-stgg2\" (UID: \"e354caaa-fac9-485a-b960-7488fea0df0d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-812990-29889405-stgg2" Oct 30 12:45:00.466269 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:00.466257 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55mhq\" (UniqueName: \"kubernetes.io/projected/a3a015cd-f16d-42b4-abae-9e2dd114093e-kube-api-access-55mhq\") pod \"maas-api-key-cleanup-e2e-extmod-b-ecb3aa-29889405-srcpd\" (UID: \"a3a015cd-f16d-42b4-abae-9e2dd114093e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ecb3aa-29889405-srcpd" Oct 30 12:45:00.466409 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:00.466395 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea9095cb-47a4-4ea4-b143-069f736c29b6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-8d2170-29889405-82rzm\" (UID: \"ea9095cb-47a4-4ea4-b143-069f736c29b6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-8d2170-29889405-82rzm" Oct 30 12:45:00.466522 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:00.466510 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/958d7012-017f-4cd4-a637-a44ec84babc6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-0bda67-29889405-q74jd\" (UID: \"958d7012-017f-4cd4-a637-a44ec84babc6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0bda67-29889405-q74jd" Oct 30 12:45:00.467403 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:00.467378 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d181a49-62cd-454a-9e03-380198da3a08-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-0f7d14-29889405-c7tnx\" (UID: \"4d181a49-62cd-454a-9e03-380198da3a08\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0f7d14-29889405-c7tnx" Oct 30 12:45:00.469646 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:00.469546 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3a015cd-f16d-42b4-abae-9e2dd114093e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-ecb3aa-29889405-srcpd\" (UID: \"a3a015cd-f16d-42b4-abae-9e2dd114093e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ecb3aa-29889405-srcpd" Oct 30 12:45:00.470168 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:00.470142 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fcc90cb3-e12e-4b12-a50d-de740f552b40-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-9d45fb-29889405-2gz7s\" (UID: \"fcc90cb3-e12e-4b12-a50d-de740f552b40\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-9d45fb-29889405-2gz7s" Oct 30 12:45:00.471981 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:00.471963 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f920f7-29889405-fp46s"] Oct 30 12:45:00.492354 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:00.492325 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c1b1e8-29889405-5fkdv"] Oct 30 12:45:00.497209 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:00.497176 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c1b1e8-29889405-5fkdv" Oct 30 12:45:00.499529 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:00.499509 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6ad1da-29889405-bbbrz" Oct 30 12:45:00.510251 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:00.510101 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2545c7-29889405-x55rn" Oct 30 12:45:00.521168 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:00.520867 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfrb2\" (UniqueName: \"kubernetes.io/projected/fcc90cb3-e12e-4b12-a50d-de740f552b40-kube-api-access-dfrb2\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-9d45fb-29889405-2gz7s\" (UID: \"fcc90cb3-e12e-4b12-a50d-de740f552b40\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-9d45fb-29889405-2gz7s" Oct 30 12:45:00.521168 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:00.521020 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0de4b9-29889405-4pk7b" Oct 30 12:45:00.521412 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:00.521356 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55mhq\" (UniqueName: \"kubernetes.io/projected/a3a015cd-f16d-42b4-abae-9e2dd114093e-kube-api-access-55mhq\") pod \"maas-api-key-cleanup-e2e-extmod-b-ecb3aa-29889405-srcpd\" (UID: \"a3a015cd-f16d-42b4-abae-9e2dd114093e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ecb3aa-29889405-srcpd" Oct 30 12:45:00.522423 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:00.522402 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tllf9\" (UniqueName: \"kubernetes.io/projected/4d181a49-62cd-454a-9e03-380198da3a08-kube-api-access-tllf9\") pod \"maas-api-key-cleanup-e2e-extmod-b-0f7d14-29889405-c7tnx\" (UID: \"4d181a49-62cd-454a-9e03-380198da3a08\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0f7d14-29889405-c7tnx" Oct 30 12:45:00.543418 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:00.543379 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ecb3aa-29889405-srcpd" Oct 30 12:45:00.567601 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:00.567565 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/27d77dc7-aee3-45a4-8ad9-a48560889798-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-327e31-29889405-lr598\" (UID: \"27d77dc7-aee3-45a4-8ad9-a48560889798\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-327e31-29889405-lr598" Oct 30 12:45:00.568819 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:00.568452 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/27d77dc7-aee3-45a4-8ad9-a48560889798-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-327e31-29889405-lr598\" (UID: \"27d77dc7-aee3-45a4-8ad9-a48560889798\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-327e31-29889405-lr598" Oct 30 12:45:00.569034 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:00.569004 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fae2258-d203-4719-a547-868d7b04070d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-f920f7-29889405-fp46s\" (UID: \"7fae2258-d203-4719-a547-868d7b04070d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f920f7-29889405-fp46s" Oct 30 12:45:00.569210 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:00.569189 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brfbn\" (UniqueName: \"kubernetes.io/projected/ccd6f000-a55d-4214-a830-faf2d200b688-kube-api-access-brfbn\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-2675c0-29889405-n4dv7\" (UID: \"ccd6f000-a55d-4214-a830-faf2d200b688\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2675c0-29889405-n4dv7" Oct 30 12:45:00.569589 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:00.569573 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab18f563-fbfd-4dbb-8b81-277c223a30f4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-f7464f-29889405-zzjr7\" (UID: \"ab18f563-fbfd-4dbb-8b81-277c223a30f4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f7464f-29889405-zzjr7" Oct 30 12:45:00.570302 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:00.570286 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6j5w\" (UniqueName: \"kubernetes.io/projected/240c2309-8812-4591-9871-b16dcb9b6ff3-kube-api-access-z6j5w\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-dd0628-29889405-cj8wq\" (UID: \"240c2309-8812-4591-9871-b16dcb9b6ff3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-dd0628-29889405-cj8wq" Oct 30 12:45:00.571245 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:00.570666 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/240c2309-8812-4591-9871-b16dcb9b6ff3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-dd0628-29889405-cj8wq\" (UID: \"240c2309-8812-4591-9871-b16dcb9b6ff3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-dd0628-29889405-cj8wq" Oct 30 12:45:00.571420 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:00.571405 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mg5n2\" (UniqueName: \"kubernetes.io/projected/958d7012-017f-4cd4-a637-a44ec84babc6-kube-api-access-mg5n2\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-0bda67-29889405-q74jd\" (UID: \"958d7012-017f-4cd4-a637-a44ec84babc6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0bda67-29889405-q74jd" Oct 30 12:45:00.571902 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:00.571885 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlbzs\" (UniqueName: \"kubernetes.io/projected/1aa1b39f-f88b-4ddf-8ee1-cf567d9d480a-kube-api-access-mlbzs\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-73bafc-29889405-l27f6\" (UID: \"1aa1b39f-f88b-4ddf-8ee1-cf567d9d480a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-73bafc-29889405-l27f6" Oct 30 12:45:00.572274 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:00.572247 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1aa1b39f-f88b-4ddf-8ee1-cf567d9d480a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-73bafc-29889405-l27f6\" (UID: \"1aa1b39f-f88b-4ddf-8ee1-cf567d9d480a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-73bafc-29889405-l27f6" Oct 30 12:45:00.573095 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:00.573079 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbdcr\" (UniqueName: \"kubernetes.io/projected/ab18f563-fbfd-4dbb-8b81-277c223a30f4-kube-api-access-nbdcr\") pod \"maas-api-key-cleanup-e2e-extmod-a-f7464f-29889405-zzjr7\" (UID: \"ab18f563-fbfd-4dbb-8b81-277c223a30f4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f7464f-29889405-zzjr7" Oct 30 12:45:00.573647 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:00.573441 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lr9w\" (UniqueName: \"kubernetes.io/projected/e354caaa-fac9-485a-b960-7488fea0df0d-kube-api-access-4lr9w\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-812990-29889405-stgg2\" (UID: \"e354caaa-fac9-485a-b960-7488fea0df0d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-812990-29889405-stgg2" Oct 30 12:45:00.573873 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:00.573854 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2z45r\" (UniqueName: \"kubernetes.io/projected/ea9095cb-47a4-4ea4-b143-069f736c29b6-kube-api-access-2z45r\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-8d2170-29889405-82rzm\" (UID: \"ea9095cb-47a4-4ea4-b143-069f736c29b6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-8d2170-29889405-82rzm" Oct 30 12:45:00.574376 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:00.574355 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjbp8\" (UniqueName: \"kubernetes.io/projected/27d77dc7-aee3-45a4-8ad9-a48560889798-kube-api-access-sjbp8\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-327e31-29889405-lr598\" (UID: \"27d77dc7-aee3-45a4-8ad9-a48560889798\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-327e31-29889405-lr598" Oct 30 12:45:00.575642 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:00.574794 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ccd6f000-a55d-4214-a830-faf2d200b688-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-2675c0-29889405-n4dv7\" (UID: \"ccd6f000-a55d-4214-a830-faf2d200b688\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2675c0-29889405-n4dv7" Oct 30 12:45:00.576425 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:00.576404 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e354caaa-fac9-485a-b960-7488fea0df0d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-812990-29889405-stgg2\" (UID: \"e354caaa-fac9-485a-b960-7488fea0df0d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-812990-29889405-stgg2" Oct 30 12:45:00.624573 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:00.615167 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4g9l\" (UniqueName: \"kubernetes.io/projected/7fae2258-d203-4719-a547-868d7b04070d-kube-api-access-n4g9l\") pod \"maas-api-key-cleanup-e2e-worker-w0-f920f7-29889405-fp46s\" (UID: \"7fae2258-d203-4719-a547-868d7b04070d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f920f7-29889405-fp46s" Oct 30 12:45:00.624573 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:00.615257 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea9095cb-47a4-4ea4-b143-069f736c29b6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-8d2170-29889405-82rzm\" (UID: \"ea9095cb-47a4-4ea4-b143-069f736c29b6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-8d2170-29889405-82rzm" Oct 30 12:45:00.624573 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:00.615301 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/958d7012-017f-4cd4-a637-a44ec84babc6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-0bda67-29889405-q74jd\" (UID: \"958d7012-017f-4cd4-a637-a44ec84babc6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0bda67-29889405-q74jd" Oct 30 12:45:00.624573 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:00.573037 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1aa1b39f-f88b-4ddf-8ee1-cf567d9d480a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-73bafc-29889405-l27f6\" (UID: \"1aa1b39f-f88b-4ddf-8ee1-cf567d9d480a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-73bafc-29889405-l27f6" Oct 30 12:45:00.624573 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:00.606009 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-9d45fb-29889405-2gz7s" Oct 30 12:45:00.624573 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:00.616169 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e354caaa-fac9-485a-b960-7488fea0df0d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-812990-29889405-stgg2\" (UID: \"e354caaa-fac9-485a-b960-7488fea0df0d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-812990-29889405-stgg2" Oct 30 12:45:00.624573 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:00.571196 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/240c2309-8812-4591-9871-b16dcb9b6ff3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-dd0628-29889405-cj8wq\" (UID: \"240c2309-8812-4591-9871-b16dcb9b6ff3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-dd0628-29889405-cj8wq" Oct 30 12:45:00.624573 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:00.570222 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab18f563-fbfd-4dbb-8b81-277c223a30f4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-f7464f-29889405-zzjr7\" (UID: \"ab18f563-fbfd-4dbb-8b81-277c223a30f4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f7464f-29889405-zzjr7" Oct 30 12:45:00.624573 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:00.616873 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea9095cb-47a4-4ea4-b143-069f736c29b6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-8d2170-29889405-82rzm\" (UID: \"ea9095cb-47a4-4ea4-b143-069f736c29b6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-8d2170-29889405-82rzm" Oct 30 12:45:00.624573 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:00.575495 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ccd6f000-a55d-4214-a830-faf2d200b688-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-2675c0-29889405-n4dv7\" (UID: \"ccd6f000-a55d-4214-a830-faf2d200b688\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2675c0-29889405-n4dv7" Oct 30 12:45:00.624573 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:00.623068 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/958d7012-017f-4cd4-a637-a44ec84babc6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-0bda67-29889405-q74jd\" (UID: \"958d7012-017f-4cd4-a637-a44ec84babc6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0bda67-29889405-q74jd" Oct 30 12:45:00.631014 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:00.630499 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjbp8\" (UniqueName: \"kubernetes.io/projected/27d77dc7-aee3-45a4-8ad9-a48560889798-kube-api-access-sjbp8\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-327e31-29889405-lr598\" (UID: \"27d77dc7-aee3-45a4-8ad9-a48560889798\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-327e31-29889405-lr598" Oct 30 12:45:00.631014 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:00.630540 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c1b1e8-29889405-5fkdv"] Oct 30 12:45:00.631627 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:00.631593 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brfbn\" (UniqueName: \"kubernetes.io/projected/ccd6f000-a55d-4214-a830-faf2d200b688-kube-api-access-brfbn\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-2675c0-29889405-n4dv7\" (UID: \"ccd6f000-a55d-4214-a830-faf2d200b688\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2675c0-29889405-n4dv7" Oct 30 12:45:00.632849 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:00.632771 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2z45r\" (UniqueName: \"kubernetes.io/projected/ea9095cb-47a4-4ea4-b143-069f736c29b6-kube-api-access-2z45r\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-8d2170-29889405-82rzm\" (UID: \"ea9095cb-47a4-4ea4-b143-069f736c29b6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-8d2170-29889405-82rzm" Oct 30 12:45:00.638311 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:00.638259 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlbzs\" (UniqueName: \"kubernetes.io/projected/1aa1b39f-f88b-4ddf-8ee1-cf567d9d480a-kube-api-access-mlbzs\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-73bafc-29889405-l27f6\" (UID: \"1aa1b39f-f88b-4ddf-8ee1-cf567d9d480a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-73bafc-29889405-l27f6" Oct 30 12:45:00.641897 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:00.641796 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbdcr\" (UniqueName: \"kubernetes.io/projected/ab18f563-fbfd-4dbb-8b81-277c223a30f4-kube-api-access-nbdcr\") pod \"maas-api-key-cleanup-e2e-extmod-a-f7464f-29889405-zzjr7\" (UID: \"ab18f563-fbfd-4dbb-8b81-277c223a30f4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f7464f-29889405-zzjr7" Oct 30 12:45:00.642859 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:00.642711 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mg5n2\" (UniqueName: \"kubernetes.io/projected/958d7012-017f-4cd4-a637-a44ec84babc6-kube-api-access-mg5n2\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-0bda67-29889405-q74jd\" (UID: \"958d7012-017f-4cd4-a637-a44ec84babc6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0bda67-29889405-q74jd" Oct 30 12:45:00.643227 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:00.643203 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lr9w\" (UniqueName: \"kubernetes.io/projected/e354caaa-fac9-485a-b960-7488fea0df0d-kube-api-access-4lr9w\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-812990-29889405-stgg2\" (UID: \"e354caaa-fac9-485a-b960-7488fea0df0d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-812990-29889405-stgg2" Oct 30 12:45:00.645122 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:00.645079 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6j5w\" (UniqueName: \"kubernetes.io/projected/240c2309-8812-4591-9871-b16dcb9b6ff3-kube-api-access-z6j5w\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-dd0628-29889405-cj8wq\" (UID: \"240c2309-8812-4591-9871-b16dcb9b6ff3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-dd0628-29889405-cj8wq" Oct 30 12:45:00.645963 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:00.645711 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f7464f-29889405-zzjr7" Oct 30 12:45:00.657632 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:00.657432 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-73bafc-29889405-l27f6" Oct 30 12:45:00.672256 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:00.672070 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-dd0628-29889405-cj8wq" Oct 30 12:45:00.711130 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:00.709163 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-812990-29889405-stgg2" Oct 30 12:45:00.716033 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:00.715524 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-8d2170-29889405-82rzm" Oct 30 12:45:00.716895 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:00.716608 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4g9l\" (UniqueName: \"kubernetes.io/projected/7fae2258-d203-4719-a547-868d7b04070d-kube-api-access-n4g9l\") pod \"maas-api-key-cleanup-e2e-worker-w0-f920f7-29889405-fp46s\" (UID: \"7fae2258-d203-4719-a547-868d7b04070d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f920f7-29889405-fp46s" Oct 30 12:45:00.716895 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:00.716746 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fae2258-d203-4719-a547-868d7b04070d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-f920f7-29889405-fp46s\" (UID: \"7fae2258-d203-4719-a547-868d7b04070d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f920f7-29889405-fp46s" Oct 30 12:45:00.716895 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:00.716789 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x44n2\" (UniqueName: \"kubernetes.io/projected/81dec304-04f6-4c40-8bf3-7fa3c4e0994d-kube-api-access-x44n2\") pod \"maas-api-key-cleanup-e2e-worker-w1-c1b1e8-29889405-5fkdv\" (UID: \"81dec304-04f6-4c40-8bf3-7fa3c4e0994d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c1b1e8-29889405-5fkdv" Oct 30 12:45:00.716895 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:00.716842 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81dec304-04f6-4c40-8bf3-7fa3c4e0994d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-c1b1e8-29889405-5fkdv\" (UID: \"81dec304-04f6-4c40-8bf3-7fa3c4e0994d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c1b1e8-29889405-5fkdv" Oct 30 12:45:00.718442 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:00.718309 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fae2258-d203-4719-a547-868d7b04070d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-f920f7-29889405-fp46s\" (UID: \"7fae2258-d203-4719-a547-868d7b04070d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f920f7-29889405-fp46s" Oct 30 12:45:00.745718 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:00.744126 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4g9l\" (UniqueName: \"kubernetes.io/projected/7fae2258-d203-4719-a547-868d7b04070d-kube-api-access-n4g9l\") pod \"maas-api-key-cleanup-e2e-worker-w0-f920f7-29889405-fp46s\" (UID: \"7fae2258-d203-4719-a547-868d7b04070d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f920f7-29889405-fp46s" Oct 30 12:45:00.767010 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:00.762661 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-327e31-29889405-lr598" Oct 30 12:45:00.786877 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:00.783654 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2675c0-29889405-n4dv7" Oct 30 12:45:00.797812 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:00.797501 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0bda67-29889405-q74jd" Oct 30 12:45:00.818413 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:00.818355 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f920f7-29889405-fp46s" Oct 30 12:45:00.826028 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:00.820483 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x44n2\" (UniqueName: \"kubernetes.io/projected/81dec304-04f6-4c40-8bf3-7fa3c4e0994d-kube-api-access-x44n2\") pod \"maas-api-key-cleanup-e2e-worker-w1-c1b1e8-29889405-5fkdv\" (UID: \"81dec304-04f6-4c40-8bf3-7fa3c4e0994d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c1b1e8-29889405-5fkdv" Oct 30 12:45:00.826028 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:00.820540 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81dec304-04f6-4c40-8bf3-7fa3c4e0994d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-c1b1e8-29889405-5fkdv\" (UID: \"81dec304-04f6-4c40-8bf3-7fa3c4e0994d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c1b1e8-29889405-5fkdv" Oct 30 12:45:00.826028 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:00.821428 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81dec304-04f6-4c40-8bf3-7fa3c4e0994d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-c1b1e8-29889405-5fkdv\" (UID: \"81dec304-04f6-4c40-8bf3-7fa3c4e0994d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c1b1e8-29889405-5fkdv" Oct 30 12:45:00.833741 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:00.833720 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0f7d14-29889405-c7tnx" Oct 30 12:45:00.886585 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:00.886523 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x44n2\" (UniqueName: \"kubernetes.io/projected/81dec304-04f6-4c40-8bf3-7fa3c4e0994d-kube-api-access-x44n2\") pod \"maas-api-key-cleanup-e2e-worker-w1-c1b1e8-29889405-5fkdv\" (UID: \"81dec304-04f6-4c40-8bf3-7fa3c4e0994d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c1b1e8-29889405-5fkdv" Oct 30 12:45:00.899587 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:00.899479 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6ad1da-29889405-bbbrz"] Oct 30 12:45:00.918198 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:00.918114 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2545c7-29889405-x55rn"] Oct 30 12:45:01.166493 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:01.166460 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c1b1e8-29889405-5fkdv" Oct 30 12:45:01.183195 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:01.183160 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ecb3aa-29889405-srcpd"] Oct 30 12:45:01.194469 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:01.194437 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0de4b9-29889405-4pk7b"] Oct 30 12:45:01.315060 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:45:01.310215 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcc90cb3_e12e_4b12_a50d_de740f552b40.slice/crio-c717b1d24445cc627f9f30f19a007e4f704be4508c07f3d141e1e84c09cf7bcb WatchSource:0}: Error finding container c717b1d24445cc627f9f30f19a007e4f704be4508c07f3d141e1e84c09cf7bcb: Status 404 returned error can't find the container with id c717b1d24445cc627f9f30f19a007e4f704be4508c07f3d141e1e84c09cf7bcb Oct 30 12:45:01.315060 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:01.313983 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-327e31-29889405-lr598"] Oct 30 12:45:01.321519 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:01.319955 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-812990-29889405-stgg2"] Oct 30 12:45:01.341653 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:01.336337 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-9d45fb-29889405-2gz7s"] Oct 30 12:45:01.341653 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:45:01.339467 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27d77dc7_aee3_45a4_8ad9_a48560889798.slice/crio-29c950ef5f0ad024e67aeb49d0de82f1bc142eeb4120cd65e01acf19657b9d63 WatchSource:0}: Error finding container 29c950ef5f0ad024e67aeb49d0de82f1bc142eeb4120cd65e01acf19657b9d63: Status 404 returned error can't find the container with id 29c950ef5f0ad024e67aeb49d0de82f1bc142eeb4120cd65e01acf19657b9d63 Oct 30 12:45:01.386994 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:01.386961 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0f7d14-29889405-c7tnx"] Oct 30 12:45:01.392513 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:01.392426 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-8d2170-29889405-82rzm"] Oct 30 12:45:01.399330 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:01.399308 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f920f7-29889405-fp46s"] Oct 30 12:45:01.403506 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:01.403472 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f7464f-29889405-zzjr7"] Oct 30 12:45:01.408785 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:01.408714 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-73bafc-29889405-l27f6"] Oct 30 12:45:01.412442 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:01.412419 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0bda67-29889405-q74jd"] Oct 30 12:45:01.426410 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:01.426385 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-9d45fb-29889405-2gz7s" event={"ID":"fcc90cb3-e12e-4b12-a50d-de740f552b40","Type":"ContainerStarted","Data":"c717b1d24445cc627f9f30f19a007e4f704be4508c07f3d141e1e84c09cf7bcb"} Oct 30 12:45:01.427746 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:01.427720 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2545c7-29889405-x55rn" event={"ID":"0b9ecbb7-1e58-4ac9-b913-ff46c0a02210","Type":"ContainerStarted","Data":"462412b4821f160bbcc534f284980c28b544f41249feed7768d6a0dd92b5f62a"} Oct 30 12:45:01.428992 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:01.428967 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0de4b9-29889405-4pk7b" event={"ID":"12c31795-7957-4dff-83c3-d4d2c1d31aac","Type":"ContainerStarted","Data":"6597ee431c0dca5339e454ce4c5a183e927240fc03846aaba3a3713bd658c5d9"} Oct 30 12:45:01.430172 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:01.430149 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-327e31-29889405-lr598" event={"ID":"27d77dc7-aee3-45a4-8ad9-a48560889798","Type":"ContainerStarted","Data":"29c950ef5f0ad024e67aeb49d0de82f1bc142eeb4120cd65e01acf19657b9d63"} Oct 30 12:45:01.431345 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:01.431328 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ecb3aa-29889405-srcpd" event={"ID":"a3a015cd-f16d-42b4-abae-9e2dd114093e","Type":"ContainerStarted","Data":"2d40f1812c1461813fc4609be9d3050bbe8d9ec6851bf6040eea2874428c703d"} Oct 30 12:45:01.432475 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:01.432456 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-812990-29889405-stgg2" event={"ID":"e354caaa-fac9-485a-b960-7488fea0df0d","Type":"ContainerStarted","Data":"509040ff6853c09788717a7bcace7c47a6c8298b5f82c443a8b874b9f7976039"} Oct 30 12:45:01.433593 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:01.433576 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6ad1da-29889405-bbbrz" event={"ID":"3eef63f0-3b6c-4e62-bdc9-594ee9d4bd17","Type":"ContainerStarted","Data":"763cefcc3b66f8c23e9a6446b658e6eba031064ee534423c1b8818b639cf8222"} Oct 30 12:45:01.447981 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:01.447962 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-dd0628-29889405-cj8wq"] Oct 30 12:45:01.466242 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:45:01.465182 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d181a49_62cd_454a_9e03_380198da3a08.slice/crio-bd4ce623329eb5574d52d7bc9e3ad8d1e824a4685cc1631986aa3bd1b893b1d4 WatchSource:0}: Error finding container bd4ce623329eb5574d52d7bc9e3ad8d1e824a4685cc1631986aa3bd1b893b1d4: Status 404 returned error can't find the container with id bd4ce623329eb5574d52d7bc9e3ad8d1e824a4685cc1631986aa3bd1b893b1d4 Oct 30 12:45:01.466242 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:45:01.466020 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea9095cb_47a4_4ea4_b143_069f736c29b6.slice/crio-10ae2400903d389be55371923b49d31602eb80961c30fccaf97d2bcd69106c46 WatchSource:0}: Error finding container 10ae2400903d389be55371923b49d31602eb80961c30fccaf97d2bcd69106c46: Status 404 returned error can't find the container with id 10ae2400903d389be55371923b49d31602eb80961c30fccaf97d2bcd69106c46 Oct 30 12:45:01.467776 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:45:01.467694 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fae2258_d203_4719_a547_868d7b04070d.slice/crio-8bdfc71c07e1df036e8e134113b55bb7baf1f84fd40b05ba0a25326b74f36eb9 WatchSource:0}: Error finding container 8bdfc71c07e1df036e8e134113b55bb7baf1f84fd40b05ba0a25326b74f36eb9: Status 404 returned error can't find the container with id 8bdfc71c07e1df036e8e134113b55bb7baf1f84fd40b05ba0a25326b74f36eb9 Oct 30 12:45:01.469416 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:45:01.469360 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab18f563_fbfd_4dbb_8b81_277c223a30f4.slice/crio-935b800fa225e1b686f14ca8806c51a6893667808fc93285ff1e713b93360df8 WatchSource:0}: Error finding container 935b800fa225e1b686f14ca8806c51a6893667808fc93285ff1e713b93360df8: Status 404 returned error can't find the container with id 935b800fa225e1b686f14ca8806c51a6893667808fc93285ff1e713b93360df8 Oct 30 12:45:01.473110 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:45:01.473085 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1aa1b39f_f88b_4ddf_8ee1_cf567d9d480a.slice/crio-2fb626744864ae24c5462ec7c1e1eafa50c43793af2281b294aa092c8d360d6d WatchSource:0}: Error finding container 2fb626744864ae24c5462ec7c1e1eafa50c43793af2281b294aa092c8d360d6d: Status 404 returned error can't find the container with id 2fb626744864ae24c5462ec7c1e1eafa50c43793af2281b294aa092c8d360d6d Oct 30 12:45:01.475371 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:45:01.475346 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod958d7012_017f_4cd4_a637_a44ec84babc6.slice/crio-db31531d80b98920ca82ab5894044210627b48e06a7442ee4cfb8a5b2aed0000 WatchSource:0}: Error finding container db31531d80b98920ca82ab5894044210627b48e06a7442ee4cfb8a5b2aed0000: Status 404 returned error can't find the container with id db31531d80b98920ca82ab5894044210627b48e06a7442ee4cfb8a5b2aed0000 Oct 30 12:45:01.539858 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:01.539812 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2675c0-29889405-n4dv7"] Oct 30 12:45:01.555086 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:45:01.555048 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccd6f000_a55d_4214_a830_faf2d200b688.slice/crio-a6f05fa1bc7093a9c685192649e3969c40987fd095b36d7cbe3345fba01cfdeb WatchSource:0}: Error finding container a6f05fa1bc7093a9c685192649e3969c40987fd095b36d7cbe3345fba01cfdeb: Status 404 returned error can't find the container with id a6f05fa1bc7093a9c685192649e3969c40987fd095b36d7cbe3345fba01cfdeb Oct 30 12:45:01.618626 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:01.618593 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c1b1e8-29889405-5fkdv"] Oct 30 12:45:02.451877 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:02.451808 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0bda67-29889405-q74jd" event={"ID":"958d7012-017f-4cd4-a637-a44ec84babc6","Type":"ContainerStarted","Data":"ee351ebdd0e398981adfce246cda40da62f2369c29bf167560d216084c38ff44"} Oct 30 12:45:02.452407 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:02.451942 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0bda67-29889405-q74jd" event={"ID":"958d7012-017f-4cd4-a637-a44ec84babc6","Type":"ContainerStarted","Data":"db31531d80b98920ca82ab5894044210627b48e06a7442ee4cfb8a5b2aed0000"} Oct 30 12:45:02.453814 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:02.453777 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-73bafc-29889405-l27f6" event={"ID":"1aa1b39f-f88b-4ddf-8ee1-cf567d9d480a","Type":"ContainerStarted","Data":"164ea7d97d9a02b357bcae630e979f5f48771699019b30de5c563b5de11a646d"} Oct 30 12:45:02.453993 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:02.453878 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-73bafc-29889405-l27f6" event={"ID":"1aa1b39f-f88b-4ddf-8ee1-cf567d9d480a","Type":"ContainerStarted","Data":"2fb626744864ae24c5462ec7c1e1eafa50c43793af2281b294aa092c8d360d6d"} Oct 30 12:45:02.455523 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:02.455498 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ecb3aa-29889405-srcpd" event={"ID":"a3a015cd-f16d-42b4-abae-9e2dd114093e","Type":"ContainerStarted","Data":"1869c3db5e4ba9f3b5c63a47ac21ddac18b6e93f18ed4c40e798573a2b000573"} Oct 30 12:45:02.457282 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:02.457263 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2675c0-29889405-n4dv7" event={"ID":"ccd6f000-a55d-4214-a830-faf2d200b688","Type":"ContainerStarted","Data":"decd72c23aa5f4b6f95688e5a092a99c348125afe45caaadc04f44aa636d4ecc"} Oct 30 12:45:02.457338 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:02.457299 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2675c0-29889405-n4dv7" event={"ID":"ccd6f000-a55d-4214-a830-faf2d200b688","Type":"ContainerStarted","Data":"a6f05fa1bc7093a9c685192649e3969c40987fd095b36d7cbe3345fba01cfdeb"} Oct 30 12:45:02.463230 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:02.462602 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f920f7-29889405-fp46s" event={"ID":"7fae2258-d203-4719-a547-868d7b04070d","Type":"ContainerStarted","Data":"b979df0dad047b0a4595d0cf7c51e2f478d26d0ce44d4a2ef681253dcb9f8b7d"} Oct 30 12:45:02.463230 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:02.462645 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f920f7-29889405-fp46s" event={"ID":"7fae2258-d203-4719-a547-868d7b04070d","Type":"ContainerStarted","Data":"8bdfc71c07e1df036e8e134113b55bb7baf1f84fd40b05ba0a25326b74f36eb9"} Oct 30 12:45:02.466521 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:02.466111 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-812990-29889405-stgg2" event={"ID":"e354caaa-fac9-485a-b960-7488fea0df0d","Type":"ContainerStarted","Data":"1d8956b8eb4b9c6037bbb0dd342c70f4341b49883fc8c24464d9b3a669c1a857"} Oct 30 12:45:02.470370 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:02.469818 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-dd0628-29889405-cj8wq" event={"ID":"240c2309-8812-4591-9871-b16dcb9b6ff3","Type":"ContainerStarted","Data":"79904a7e29e33a4c04415b3116e53b390ff2fc38bc2b3ff2fc6ec608f13d332d"} Oct 30 12:45:02.470370 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:02.469892 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-dd0628-29889405-cj8wq" event={"ID":"240c2309-8812-4591-9871-b16dcb9b6ff3","Type":"ContainerStarted","Data":"59ab3c41971091f710d7032586bed62e7f33798adafea82937ab31e7e44de78a"} Oct 30 12:45:02.471964 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:02.471807 4908 generic.go:356] "Generic (PLEG): container finished" podID="3eef63f0-3b6c-4e62-bdc9-594ee9d4bd17" containerID="9e84965f30b15989bc596893f03aaff2a44a097ab15403c3bfdb7d74c2a4847c" exitCode=60 Oct 30 12:45:02.472066 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:02.472033 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6ad1da-29889405-bbbrz" event={"ID":"3eef63f0-3b6c-4e62-bdc9-594ee9d4bd17","Type":"ContainerDied","Data":"9e84965f30b15989bc596893f03aaff2a44a097ab15403c3bfdb7d74c2a4847c"} Oct 30 12:45:02.472923 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:02.472797 4908 scope.go:122] "RemoveContainer" containerID="9e84965f30b15989bc596893f03aaff2a44a097ab15403c3bfdb7d74c2a4847c" Oct 30 12:45:02.474673 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:02.474059 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c1b1e8-29889405-5fkdv" event={"ID":"81dec304-04f6-4c40-8bf3-7fa3c4e0994d","Type":"ContainerStarted","Data":"88d7753f48f3e94eb67177f17f6c2dd76d9ef0ed508452d5da87a577ac16672b"} Oct 30 12:45:02.474673 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:02.474118 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c1b1e8-29889405-5fkdv" event={"ID":"81dec304-04f6-4c40-8bf3-7fa3c4e0994d","Type":"ContainerStarted","Data":"09a83966c96f5083ef306912f7e20c1358c3d10b12847efdd8585d1e137f2993"} Oct 30 12:45:02.477570 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:02.475767 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-9d45fb-29889405-2gz7s" event={"ID":"fcc90cb3-e12e-4b12-a50d-de740f552b40","Type":"ContainerStarted","Data":"c5e46a758af64e78f65e3485def44369aed65ef5ee6b55c4c03d0814599d7980"} Oct 30 12:45:02.478472 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:02.478448 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0f7d14-29889405-c7tnx" event={"ID":"4d181a49-62cd-454a-9e03-380198da3a08","Type":"ContainerStarted","Data":"a4473b6680c4f2a1f709ccf446cca4710973253807c932f2be3b2bf5f21f3e73"} Oct 30 12:45:02.478572 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:02.478491 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0f7d14-29889405-c7tnx" event={"ID":"4d181a49-62cd-454a-9e03-380198da3a08","Type":"ContainerStarted","Data":"bd4ce623329eb5574d52d7bc9e3ad8d1e824a4685cc1631986aa3bd1b893b1d4"} Oct 30 12:45:02.480223 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:02.480116 4908 generic.go:356] "Generic (PLEG): container finished" podID="0b9ecbb7-1e58-4ac9-b913-ff46c0a02210" containerID="1076689cee52b02e299046cf6f1248a2a74375ae54e30f293991124e07042a19" exitCode=60 Oct 30 12:45:02.480223 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:02.480190 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2545c7-29889405-x55rn" event={"ID":"0b9ecbb7-1e58-4ac9-b913-ff46c0a02210","Type":"ContainerDied","Data":"1076689cee52b02e299046cf6f1248a2a74375ae54e30f293991124e07042a19"} Oct 30 12:45:02.480355 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:02.480341 4908 scope.go:122] "RemoveContainer" containerID="1076689cee52b02e299046cf6f1248a2a74375ae54e30f293991124e07042a19" Oct 30 12:45:02.482464 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:02.482248 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-8d2170-29889405-82rzm" event={"ID":"ea9095cb-47a4-4ea4-b143-069f736c29b6","Type":"ContainerStarted","Data":"fc1ff9baa10a8f937d3a69b86eaa9cea4c31c29f6b6d50a20db9ef10331297fe"} Oct 30 12:45:02.482464 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:02.482289 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-8d2170-29889405-82rzm" event={"ID":"ea9095cb-47a4-4ea4-b143-069f736c29b6","Type":"ContainerStarted","Data":"10ae2400903d389be55371923b49d31602eb80961c30fccaf97d2bcd69106c46"} Oct 30 12:45:02.484091 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:02.483986 4908 generic.go:356] "Generic (PLEG): container finished" podID="12c31795-7957-4dff-83c3-d4d2c1d31aac" containerID="6b816b264d7b0a7fb786e3ebfe7d03124dc10df019b4cfeb3cec260013bfb8c8" exitCode=60 Oct 30 12:45:02.484091 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:02.484051 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0de4b9-29889405-4pk7b" event={"ID":"12c31795-7957-4dff-83c3-d4d2c1d31aac","Type":"ContainerDied","Data":"6b816b264d7b0a7fb786e3ebfe7d03124dc10df019b4cfeb3cec260013bfb8c8"} Oct 30 12:45:02.484384 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:02.484188 4908 scope.go:122] "RemoveContainer" containerID="6b816b264d7b0a7fb786e3ebfe7d03124dc10df019b4cfeb3cec260013bfb8c8" Oct 30 12:45:02.485822 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:02.485791 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f7464f-29889405-zzjr7" event={"ID":"ab18f563-fbfd-4dbb-8b81-277c223a30f4","Type":"ContainerStarted","Data":"4e6410f56cc6daf833cad17ee73fe201859a48ea70858e7f27134d84e54fefda"} Oct 30 12:45:02.485960 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:02.485884 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f7464f-29889405-zzjr7" event={"ID":"ab18f563-fbfd-4dbb-8b81-277c223a30f4","Type":"ContainerStarted","Data":"935b800fa225e1b686f14ca8806c51a6893667808fc93285ff1e713b93360df8"} Oct 30 12:45:02.487714 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:02.487691 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-327e31-29889405-lr598" event={"ID":"27d77dc7-aee3-45a4-8ad9-a48560889798","Type":"ContainerStarted","Data":"70ad774f590b7af18dd6fc14ded449920a6a91d42b06a979e3b19094610b9b5d"} Oct 30 12:45:02.523564 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:02.523521 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0bda67-29889405-q74jd" podStartSLOduration=2.52351087 podStartE2EDuration="2.52351087s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:45:02.520260489 +0000 UTC m=+3059.034689254" watchObservedRunningTime="2026-10-30 12:45:02.52351087 +0000 UTC m=+3059.037939615" Oct 30 12:45:02.598908 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:02.598798 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-8d2170-29889405-82rzm" podStartSLOduration=2.59878055 podStartE2EDuration="2.59878055s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:45:02.593849147 +0000 UTC m=+3059.108277891" watchObservedRunningTime="2026-10-30 12:45:02.59878055 +0000 UTC m=+3059.113209298" Oct 30 12:45:02.639231 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:02.639184 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c1b1e8-29889405-5fkdv" podStartSLOduration=2.639170526 podStartE2EDuration="2.639170526s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:45:02.638285292 +0000 UTC m=+3059.152714037" watchObservedRunningTime="2026-10-30 12:45:02.639170526 +0000 UTC m=+3059.153599272" Oct 30 12:45:02.680348 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:02.680293 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-812990-29889405-stgg2" podStartSLOduration=2.680275374 podStartE2EDuration="2.680275374s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:45:02.679361172 +0000 UTC m=+3059.193789940" watchObservedRunningTime="2026-10-30 12:45:02.680275374 +0000 UTC m=+3059.194704121" Oct 30 12:45:02.733111 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:02.733058 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f920f7-29889405-fp46s" podStartSLOduration=2.733043827 podStartE2EDuration="2.733043827s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:45:02.729178525 +0000 UTC m=+3059.243607274" watchObservedRunningTime="2026-10-30 12:45:02.733043827 +0000 UTC m=+3059.247472573" Oct 30 12:45:02.790814 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:02.790762 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-9d45fb-29889405-2gz7s" podStartSLOduration=2.790748196 podStartE2EDuration="2.790748196s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:45:02.785860343 +0000 UTC m=+3059.300289102" watchObservedRunningTime="2026-10-30 12:45:02.790748196 +0000 UTC m=+3059.305176942" Oct 30 12:45:02.860519 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:02.859330 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0f7d14-29889405-c7tnx" podStartSLOduration=2.859311865 podStartE2EDuration="2.859311865s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:45:02.857237564 +0000 UTC m=+3059.371666310" watchObservedRunningTime="2026-10-30 12:45:02.859311865 +0000 UTC m=+3059.373740612" Oct 30 12:45:03.023094 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:03.023044 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-73bafc-29889405-l27f6" podStartSLOduration=3.023028945 podStartE2EDuration="3.023028945s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:45:03.019488779 +0000 UTC m=+3059.533917524" watchObservedRunningTime="2026-10-30 12:45:03.023028945 +0000 UTC m=+3059.537457685" Oct 30 12:45:03.111560 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:03.111505 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f7464f-29889405-zzjr7" podStartSLOduration=3.111491458 podStartE2EDuration="3.111491458s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:45:03.107534151 +0000 UTC m=+3059.621962899" watchObservedRunningTime="2026-10-30 12:45:03.111491458 +0000 UTC m=+3059.625920219" Oct 30 12:45:03.112989 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:03.112958 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ecb3aa-29889405-srcpd" podStartSLOduration=3.112945975 podStartE2EDuration="3.112945975s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:45:03.062724584 +0000 UTC m=+3059.577153330" watchObservedRunningTime="2026-10-30 12:45:03.112945975 +0000 UTC m=+3059.627374721" Oct 30 12:45:03.196249 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:03.196198 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-327e31-29889405-lr598" podStartSLOduration=3.19618522 podStartE2EDuration="3.19618522s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:45:03.190933122 +0000 UTC m=+3059.705361868" watchObservedRunningTime="2026-10-30 12:45:03.19618522 +0000 UTC m=+3059.710613967" Oct 30 12:45:03.229217 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:03.229169 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2675c0-29889405-n4dv7" podStartSLOduration=3.229155437 podStartE2EDuration="3.229155437s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:45:03.227887918 +0000 UTC m=+3059.742316666" watchObservedRunningTime="2026-10-30 12:45:03.229155437 +0000 UTC m=+3059.743584183" Oct 30 12:45:03.269499 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:03.269445 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-dd0628-29889405-cj8wq" podStartSLOduration=3.269430608 podStartE2EDuration="3.269430608s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:45:03.265432666 +0000 UTC m=+3059.779861414" watchObservedRunningTime="2026-10-30 12:45:03.269430608 +0000 UTC m=+3059.783859390" Oct 30 12:45:03.500123 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:03.500086 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0de4b9-29889405-4pk7b" event={"ID":"12c31795-7957-4dff-83c3-d4d2c1d31aac","Type":"ContainerStarted","Data":"cb5f9257cacca16f6e1ed8a7bd8255de113bc93d89da2b25b6e3c466c9531bcb"} Oct 30 12:45:03.501969 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:03.501944 4908 generic.go:356] "Generic (PLEG): container finished" podID="ab18f563-fbfd-4dbb-8b81-277c223a30f4" containerID="4e6410f56cc6daf833cad17ee73fe201859a48ea70858e7f27134d84e54fefda" exitCode=60 Oct 30 12:45:03.502087 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:03.502021 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f7464f-29889405-zzjr7" event={"ID":"ab18f563-fbfd-4dbb-8b81-277c223a30f4","Type":"ContainerDied","Data":"4e6410f56cc6daf833cad17ee73fe201859a48ea70858e7f27134d84e54fefda"} Oct 30 12:45:03.502287 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:03.502272 4908 scope.go:122] "RemoveContainer" containerID="4e6410f56cc6daf833cad17ee73fe201859a48ea70858e7f27134d84e54fefda" Oct 30 12:45:03.504008 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:03.503984 4908 generic.go:356] "Generic (PLEG): container finished" podID="27d77dc7-aee3-45a4-8ad9-a48560889798" containerID="70ad774f590b7af18dd6fc14ded449920a6a91d42b06a979e3b19094610b9b5d" exitCode=60 Oct 30 12:45:03.504120 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:03.504010 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-327e31-29889405-lr598" event={"ID":"27d77dc7-aee3-45a4-8ad9-a48560889798","Type":"ContainerDied","Data":"70ad774f590b7af18dd6fc14ded449920a6a91d42b06a979e3b19094610b9b5d"} Oct 30 12:45:03.504338 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:03.504308 4908 scope.go:122] "RemoveContainer" containerID="70ad774f590b7af18dd6fc14ded449920a6a91d42b06a979e3b19094610b9b5d" Oct 30 12:45:03.506376 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:03.506354 4908 generic.go:356] "Generic (PLEG): container finished" podID="958d7012-017f-4cd4-a637-a44ec84babc6" containerID="ee351ebdd0e398981adfce246cda40da62f2369c29bf167560d216084c38ff44" exitCode=60 Oct 30 12:45:03.506546 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:03.506466 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0bda67-29889405-q74jd" event={"ID":"958d7012-017f-4cd4-a637-a44ec84babc6","Type":"ContainerDied","Data":"ee351ebdd0e398981adfce246cda40da62f2369c29bf167560d216084c38ff44"} Oct 30 12:45:03.506696 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:03.506675 4908 scope.go:122] "RemoveContainer" containerID="ee351ebdd0e398981adfce246cda40da62f2369c29bf167560d216084c38ff44" Oct 30 12:45:03.508441 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:03.508418 4908 generic.go:356] "Generic (PLEG): container finished" podID="1aa1b39f-f88b-4ddf-8ee1-cf567d9d480a" containerID="164ea7d97d9a02b357bcae630e979f5f48771699019b30de5c563b5de11a646d" exitCode=60 Oct 30 12:45:03.508536 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:03.508498 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-73bafc-29889405-l27f6" event={"ID":"1aa1b39f-f88b-4ddf-8ee1-cf567d9d480a","Type":"ContainerDied","Data":"164ea7d97d9a02b357bcae630e979f5f48771699019b30de5c563b5de11a646d"} Oct 30 12:45:03.508716 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:03.508702 4908 scope.go:122] "RemoveContainer" containerID="164ea7d97d9a02b357bcae630e979f5f48771699019b30de5c563b5de11a646d" Oct 30 12:45:03.510312 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:03.510292 4908 generic.go:356] "Generic (PLEG): container finished" podID="a3a015cd-f16d-42b4-abae-9e2dd114093e" containerID="1869c3db5e4ba9f3b5c63a47ac21ddac18b6e93f18ed4c40e798573a2b000573" exitCode=60 Oct 30 12:45:03.510414 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:03.510337 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ecb3aa-29889405-srcpd" event={"ID":"a3a015cd-f16d-42b4-abae-9e2dd114093e","Type":"ContainerDied","Data":"1869c3db5e4ba9f3b5c63a47ac21ddac18b6e93f18ed4c40e798573a2b000573"} Oct 30 12:45:03.510601 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:03.510567 4908 scope.go:122] "RemoveContainer" containerID="1869c3db5e4ba9f3b5c63a47ac21ddac18b6e93f18ed4c40e798573a2b000573" Oct 30 12:45:03.511944 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:03.511923 4908 generic.go:356] "Generic (PLEG): container finished" podID="ccd6f000-a55d-4214-a830-faf2d200b688" containerID="decd72c23aa5f4b6f95688e5a092a99c348125afe45caaadc04f44aa636d4ecc" exitCode=60 Oct 30 12:45:03.512029 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:03.511998 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2675c0-29889405-n4dv7" event={"ID":"ccd6f000-a55d-4214-a830-faf2d200b688","Type":"ContainerDied","Data":"decd72c23aa5f4b6f95688e5a092a99c348125afe45caaadc04f44aa636d4ecc"} Oct 30 12:45:03.512188 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:03.512172 4908 scope.go:122] "RemoveContainer" containerID="decd72c23aa5f4b6f95688e5a092a99c348125afe45caaadc04f44aa636d4ecc" Oct 30 12:45:03.513556 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:03.513537 4908 generic.go:356] "Generic (PLEG): container finished" podID="7fae2258-d203-4719-a547-868d7b04070d" containerID="b979df0dad047b0a4595d0cf7c51e2f478d26d0ce44d4a2ef681253dcb9f8b7d" exitCode=60 Oct 30 12:45:03.513666 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:03.513611 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f920f7-29889405-fp46s" event={"ID":"7fae2258-d203-4719-a547-868d7b04070d","Type":"ContainerDied","Data":"b979df0dad047b0a4595d0cf7c51e2f478d26d0ce44d4a2ef681253dcb9f8b7d"} Oct 30 12:45:03.513967 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:03.513947 4908 scope.go:122] "RemoveContainer" containerID="b979df0dad047b0a4595d0cf7c51e2f478d26d0ce44d4a2ef681253dcb9f8b7d" Oct 30 12:45:03.515868 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:03.515844 4908 generic.go:356] "Generic (PLEG): container finished" podID="e354caaa-fac9-485a-b960-7488fea0df0d" containerID="1d8956b8eb4b9c6037bbb0dd342c70f4341b49883fc8c24464d9b3a669c1a857" exitCode=60 Oct 30 12:45:03.515943 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:03.515916 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-812990-29889405-stgg2" event={"ID":"e354caaa-fac9-485a-b960-7488fea0df0d","Type":"ContainerDied","Data":"1d8956b8eb4b9c6037bbb0dd342c70f4341b49883fc8c24464d9b3a669c1a857"} Oct 30 12:45:03.516076 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:03.516061 4908 scope.go:122] "RemoveContainer" containerID="1d8956b8eb4b9c6037bbb0dd342c70f4341b49883fc8c24464d9b3a669c1a857" Oct 30 12:45:03.517599 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:03.517579 4908 generic.go:356] "Generic (PLEG): container finished" podID="240c2309-8812-4591-9871-b16dcb9b6ff3" containerID="79904a7e29e33a4c04415b3116e53b390ff2fc38bc2b3ff2fc6ec608f13d332d" exitCode=60 Oct 30 12:45:03.517705 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:03.517667 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-dd0628-29889405-cj8wq" event={"ID":"240c2309-8812-4591-9871-b16dcb9b6ff3","Type":"ContainerDied","Data":"79904a7e29e33a4c04415b3116e53b390ff2fc38bc2b3ff2fc6ec608f13d332d"} Oct 30 12:45:03.517868 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:03.517849 4908 scope.go:122] "RemoveContainer" containerID="79904a7e29e33a4c04415b3116e53b390ff2fc38bc2b3ff2fc6ec608f13d332d" Oct 30 12:45:03.519692 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:03.519672 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6ad1da-29889405-bbbrz" event={"ID":"3eef63f0-3b6c-4e62-bdc9-594ee9d4bd17","Type":"ContainerStarted","Data":"86e7231e8ea8407171fbf6e4bcc6b0d2c0a4943b1f1e7e7a4fe32e6ee6214b25"} Oct 30 12:45:03.521472 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:03.521452 4908 generic.go:356] "Generic (PLEG): container finished" podID="81dec304-04f6-4c40-8bf3-7fa3c4e0994d" containerID="88d7753f48f3e94eb67177f17f6c2dd76d9ef0ed508452d5da87a577ac16672b" exitCode=60 Oct 30 12:45:03.521571 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:03.521529 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c1b1e8-29889405-5fkdv" event={"ID":"81dec304-04f6-4c40-8bf3-7fa3c4e0994d","Type":"ContainerDied","Data":"88d7753f48f3e94eb67177f17f6c2dd76d9ef0ed508452d5da87a577ac16672b"} Oct 30 12:45:03.521695 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:03.521681 4908 scope.go:122] "RemoveContainer" containerID="88d7753f48f3e94eb67177f17f6c2dd76d9ef0ed508452d5da87a577ac16672b" Oct 30 12:45:03.525685 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:03.525664 4908 generic.go:356] "Generic (PLEG): container finished" podID="fcc90cb3-e12e-4b12-a50d-de740f552b40" containerID="c5e46a758af64e78f65e3485def44369aed65ef5ee6b55c4c03d0814599d7980" exitCode=60 Oct 30 12:45:03.525805 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:03.525733 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-9d45fb-29889405-2gz7s" event={"ID":"fcc90cb3-e12e-4b12-a50d-de740f552b40","Type":"ContainerDied","Data":"c5e46a758af64e78f65e3485def44369aed65ef5ee6b55c4c03d0814599d7980"} Oct 30 12:45:03.526152 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:03.526042 4908 scope.go:122] "RemoveContainer" containerID="c5e46a758af64e78f65e3485def44369aed65ef5ee6b55c4c03d0814599d7980" Oct 30 12:45:03.542717 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:03.542664 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0de4b9-29889405-4pk7b" podStartSLOduration=3.542648161 podStartE2EDuration="3.542648161s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:45:03.540789824 +0000 UTC m=+3060.055218572" watchObservedRunningTime="2026-10-30 12:45:03.542648161 +0000 UTC m=+3060.057076907" Oct 30 12:45:03.552800 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:03.552765 4908 generic.go:356] "Generic (PLEG): container finished" podID="4d181a49-62cd-454a-9e03-380198da3a08" containerID="a4473b6680c4f2a1f709ccf446cca4710973253807c932f2be3b2bf5f21f3e73" exitCode=60 Oct 30 12:45:03.554113 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:03.554089 4908 scope.go:122] "RemoveContainer" containerID="a4473b6680c4f2a1f709ccf446cca4710973253807c932f2be3b2bf5f21f3e73" Oct 30 12:45:03.554275 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:03.554141 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0f7d14-29889405-c7tnx" event={"ID":"4d181a49-62cd-454a-9e03-380198da3a08","Type":"ContainerDied","Data":"a4473b6680c4f2a1f709ccf446cca4710973253807c932f2be3b2bf5f21f3e73"} Oct 30 12:45:03.568870 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:03.568802 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2545c7-29889405-x55rn" event={"ID":"0b9ecbb7-1e58-4ac9-b913-ff46c0a02210","Type":"ContainerStarted","Data":"b57e0271d18afb9707f6556c49708c97241eedf42eed2fa54b1916215d67ed7f"} Oct 30 12:45:03.573209 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:03.573173 4908 generic.go:356] "Generic (PLEG): container finished" podID="ea9095cb-47a4-4ea4-b143-069f736c29b6" containerID="fc1ff9baa10a8f937d3a69b86eaa9cea4c31c29f6b6d50a20db9ef10331297fe" exitCode=60 Oct 30 12:45:03.573379 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:03.573256 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-8d2170-29889405-82rzm" event={"ID":"ea9095cb-47a4-4ea4-b143-069f736c29b6","Type":"ContainerDied","Data":"fc1ff9baa10a8f937d3a69b86eaa9cea4c31c29f6b6d50a20db9ef10331297fe"} Oct 30 12:45:03.573543 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:03.573519 4908 scope.go:122] "RemoveContainer" containerID="fc1ff9baa10a8f937d3a69b86eaa9cea4c31c29f6b6d50a20db9ef10331297fe" Oct 30 12:45:03.632723 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:03.632373 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6ad1da-29889405-bbbrz" podStartSLOduration=3.632355001 podStartE2EDuration="3.632355001s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:45:03.630317545 +0000 UTC m=+3060.144746291" watchObservedRunningTime="2026-10-30 12:45:03.632355001 +0000 UTC m=+3060.146783748" Oct 30 12:45:04.246397 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:04.246340 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2545c7-29889405-x55rn" podStartSLOduration=4.246326702 podStartE2EDuration="4.246326702s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:45:04.245453378 +0000 UTC m=+3060.759882124" watchObservedRunningTime="2026-10-30 12:45:04.246326702 +0000 UTC m=+3060.760755448" Oct 30 12:45:04.585823 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:04.585785 4908 generic.go:356] "Generic (PLEG): container finished" podID="0b9ecbb7-1e58-4ac9-b913-ff46c0a02210" containerID="b57e0271d18afb9707f6556c49708c97241eedf42eed2fa54b1916215d67ed7f" exitCode=60 Oct 30 12:45:04.586364 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:04.585903 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2545c7-29889405-x55rn" event={"ID":"0b9ecbb7-1e58-4ac9-b913-ff46c0a02210","Type":"ContainerDied","Data":"b57e0271d18afb9707f6556c49708c97241eedf42eed2fa54b1916215d67ed7f"} Oct 30 12:45:04.586364 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:04.585983 4908 scope.go:122] "RemoveContainer" containerID="1076689cee52b02e299046cf6f1248a2a74375ae54e30f293991124e07042a19" Oct 30 12:45:04.586364 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:04.586328 4908 scope.go:122] "RemoveContainer" containerID="b57e0271d18afb9707f6556c49708c97241eedf42eed2fa54b1916215d67ed7f" Oct 30 12:45:04.586698 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:45:04.586678 4908 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-2545c7-29889405-x55rn_odh-ai-gateway-infra(0b9ecbb7-1e58-4ac9-b913-ff46c0a02210)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2545c7-29889405-x55rn" podUID="0b9ecbb7-1e58-4ac9-b913-ff46c0a02210" Oct 30 12:45:04.589169 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:04.589142 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-8d2170-29889405-82rzm" event={"ID":"ea9095cb-47a4-4ea4-b143-069f736c29b6","Type":"ContainerStarted","Data":"87f808e36def3853eb9b8e107bc2b422a33cf0406724742f3cf405244ef8f07a"} Oct 30 12:45:04.592175 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:04.592141 4908 generic.go:356] "Generic (PLEG): container finished" podID="12c31795-7957-4dff-83c3-d4d2c1d31aac" containerID="cb5f9257cacca16f6e1ed8a7bd8255de113bc93d89da2b25b6e3c466c9531bcb" exitCode=60 Oct 30 12:45:04.592300 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:04.592225 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0de4b9-29889405-4pk7b" event={"ID":"12c31795-7957-4dff-83c3-d4d2c1d31aac","Type":"ContainerDied","Data":"cb5f9257cacca16f6e1ed8a7bd8255de113bc93d89da2b25b6e3c466c9531bcb"} Oct 30 12:45:04.592464 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:04.592446 4908 scope.go:122] "RemoveContainer" containerID="cb5f9257cacca16f6e1ed8a7bd8255de113bc93d89da2b25b6e3c466c9531bcb" Oct 30 12:45:04.592752 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:45:04.592730 4908 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-0de4b9-29889405-4pk7b_odh-ai-gateway-infra(12c31795-7957-4dff-83c3-d4d2c1d31aac)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0de4b9-29889405-4pk7b" podUID="12c31795-7957-4dff-83c3-d4d2c1d31aac" Oct 30 12:45:04.595023 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:04.594992 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f7464f-29889405-zzjr7" event={"ID":"ab18f563-fbfd-4dbb-8b81-277c223a30f4","Type":"ContainerStarted","Data":"b60b948e805f3a9a626013f5f4e0d50b3a64f57dd24888a1b6fc8d447b9c3ee6"} Oct 30 12:45:04.597442 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:04.597407 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-327e31-29889405-lr598" event={"ID":"27d77dc7-aee3-45a4-8ad9-a48560889798","Type":"ContainerStarted","Data":"97b9267ff8f1831eb7fe9024663eb52367837e40ede0c7fac187246461049edd"} Oct 30 12:45:04.599619 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:04.599596 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0bda67-29889405-q74jd" event={"ID":"958d7012-017f-4cd4-a637-a44ec84babc6","Type":"ContainerStarted","Data":"3cd62c631dd03cd2b4a63e38adba127303c8eb05ddfc0a92a00b2069f8df1022"} Oct 30 12:45:04.601961 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:04.601936 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-73bafc-29889405-l27f6" event={"ID":"1aa1b39f-f88b-4ddf-8ee1-cf567d9d480a","Type":"ContainerStarted","Data":"d4ce5f0ceb4f692801b346b4615ab3e5da5b06c74931968f35feb2892fef54c5"} Oct 30 12:45:04.604282 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:04.604257 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ecb3aa-29889405-srcpd" event={"ID":"a3a015cd-f16d-42b4-abae-9e2dd114093e","Type":"ContainerStarted","Data":"b1d7afd7341510e4a5f062c50b50b2c3998054904a942041810b9ea14645ffb5"} Oct 30 12:45:04.604994 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:04.604957 4908 scope.go:122] "RemoveContainer" containerID="6b816b264d7b0a7fb786e3ebfe7d03124dc10df019b4cfeb3cec260013bfb8c8" Oct 30 12:45:04.607328 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:04.607307 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2675c0-29889405-n4dv7" event={"ID":"ccd6f000-a55d-4214-a830-faf2d200b688","Type":"ContainerStarted","Data":"d12a6301d143fec5dc8f58abcb4ddc4fd2c435b86527bb63f1337233d17f245e"} Oct 30 12:45:04.610467 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:04.610443 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f920f7-29889405-fp46s" event={"ID":"7fae2258-d203-4719-a547-868d7b04070d","Type":"ContainerStarted","Data":"c268c9a5d2c266fd8722ca28f0ece47033655dc2089510a949c285d1a063cb91"} Oct 30 12:45:04.614462 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:04.614433 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-812990-29889405-stgg2" event={"ID":"e354caaa-fac9-485a-b960-7488fea0df0d","Type":"ContainerStarted","Data":"11a5433ba0f063525be4e327392af63b16b82e15a007833dd007a491f7bb307f"} Oct 30 12:45:04.617552 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:04.617520 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-dd0628-29889405-cj8wq" event={"ID":"240c2309-8812-4591-9871-b16dcb9b6ff3","Type":"ContainerStarted","Data":"a38d646ff0f49f138829d6b4d6e9a32dc95422d184c175f842e78d1bedbe529f"} Oct 30 12:45:04.619469 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:04.619437 4908 generic.go:356] "Generic (PLEG): container finished" podID="3eef63f0-3b6c-4e62-bdc9-594ee9d4bd17" containerID="86e7231e8ea8407171fbf6e4bcc6b0d2c0a4943b1f1e7e7a4fe32e6ee6214b25" exitCode=60 Oct 30 12:45:04.619579 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:04.619503 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6ad1da-29889405-bbbrz" event={"ID":"3eef63f0-3b6c-4e62-bdc9-594ee9d4bd17","Type":"ContainerDied","Data":"86e7231e8ea8407171fbf6e4bcc6b0d2c0a4943b1f1e7e7a4fe32e6ee6214b25"} Oct 30 12:45:04.619779 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:04.619760 4908 scope.go:122] "RemoveContainer" containerID="86e7231e8ea8407171fbf6e4bcc6b0d2c0a4943b1f1e7e7a4fe32e6ee6214b25" Oct 30 12:45:04.620097 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:45:04.620074 4908 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-6ad1da-29889405-bbbrz_odh-ai-gateway-infra(3eef63f0-3b6c-4e62-bdc9-594ee9d4bd17)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6ad1da-29889405-bbbrz" podUID="3eef63f0-3b6c-4e62-bdc9-594ee9d4bd17" Oct 30 12:45:04.621470 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:04.621451 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c1b1e8-29889405-5fkdv" event={"ID":"81dec304-04f6-4c40-8bf3-7fa3c4e0994d","Type":"ContainerStarted","Data":"28cfc981683a877d68c5d5c87b357dc4ee99e46a5677a9a40e3caf7a18f071ea"} Oct 30 12:45:04.623187 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:04.623167 4908 scope.go:122] "RemoveContainer" containerID="9e84965f30b15989bc596893f03aaff2a44a097ab15403c3bfdb7d74c2a4847c" Oct 30 12:45:04.623392 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:04.623372 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-9d45fb-29889405-2gz7s" event={"ID":"fcc90cb3-e12e-4b12-a50d-de740f552b40","Type":"ContainerStarted","Data":"c5975e955216f73550a5fe8f9a1a4026b404886285cd3f4af674b4c65c60b820"} Oct 30 12:45:04.625375 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:04.625352 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0f7d14-29889405-c7tnx" event={"ID":"4d181a49-62cd-454a-9e03-380198da3a08","Type":"ContainerStarted","Data":"1ecc2f2193c9794de7252a33ef2b637a0fb771ad739ee9dba3f7c6ac1bfcef69"} Oct 30 12:45:05.635859 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:05.635761 4908 generic.go:356] "Generic (PLEG): container finished" podID="27d77dc7-aee3-45a4-8ad9-a48560889798" containerID="97b9267ff8f1831eb7fe9024663eb52367837e40ede0c7fac187246461049edd" exitCode=60 Oct 30 12:45:05.636289 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:05.635860 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-327e31-29889405-lr598" event={"ID":"27d77dc7-aee3-45a4-8ad9-a48560889798","Type":"ContainerDied","Data":"97b9267ff8f1831eb7fe9024663eb52367837e40ede0c7fac187246461049edd"} Oct 30 12:45:05.636289 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:05.635935 4908 scope.go:122] "RemoveContainer" containerID="70ad774f590b7af18dd6fc14ded449920a6a91d42b06a979e3b19094610b9b5d" Oct 30 12:45:05.636289 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:05.636273 4908 scope.go:122] "RemoveContainer" containerID="97b9267ff8f1831eb7fe9024663eb52367837e40ede0c7fac187246461049edd" Oct 30 12:45:05.636661 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:45:05.636637 4908 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-327e31-29889405-lr598_odh-ai-gateway-infra(27d77dc7-aee3-45a4-8ad9-a48560889798)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-327e31-29889405-lr598" podUID="27d77dc7-aee3-45a4-8ad9-a48560889798" Oct 30 12:45:05.637888 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:05.637866 4908 generic.go:356] "Generic (PLEG): container finished" podID="958d7012-017f-4cd4-a637-a44ec84babc6" containerID="3cd62c631dd03cd2b4a63e38adba127303c8eb05ddfc0a92a00b2069f8df1022" exitCode=60 Oct 30 12:45:05.637972 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:05.637895 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0bda67-29889405-q74jd" event={"ID":"958d7012-017f-4cd4-a637-a44ec84babc6","Type":"ContainerDied","Data":"3cd62c631dd03cd2b4a63e38adba127303c8eb05ddfc0a92a00b2069f8df1022"} Oct 30 12:45:05.638228 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:05.638210 4908 scope.go:122] "RemoveContainer" containerID="3cd62c631dd03cd2b4a63e38adba127303c8eb05ddfc0a92a00b2069f8df1022" Oct 30 12:45:05.638596 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:45:05.638575 4908 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-0bda67-29889405-q74jd_odh-ai-gateway-infra(958d7012-017f-4cd4-a637-a44ec84babc6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0bda67-29889405-q74jd" podUID="958d7012-017f-4cd4-a637-a44ec84babc6" Oct 30 12:45:05.639866 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:05.639844 4908 generic.go:356] "Generic (PLEG): container finished" podID="1aa1b39f-f88b-4ddf-8ee1-cf567d9d480a" containerID="d4ce5f0ceb4f692801b346b4615ab3e5da5b06c74931968f35feb2892fef54c5" exitCode=60 Oct 30 12:45:05.639960 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:05.639916 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-73bafc-29889405-l27f6" event={"ID":"1aa1b39f-f88b-4ddf-8ee1-cf567d9d480a","Type":"ContainerDied","Data":"d4ce5f0ceb4f692801b346b4615ab3e5da5b06c74931968f35feb2892fef54c5"} Oct 30 12:45:05.640149 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:05.640132 4908 scope.go:122] "RemoveContainer" containerID="d4ce5f0ceb4f692801b346b4615ab3e5da5b06c74931968f35feb2892fef54c5" Oct 30 12:45:05.640472 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:45:05.640444 4908 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-73bafc-29889405-l27f6_odh-ai-gateway-infra(1aa1b39f-f88b-4ddf-8ee1-cf567d9d480a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-73bafc-29889405-l27f6" podUID="1aa1b39f-f88b-4ddf-8ee1-cf567d9d480a" Oct 30 12:45:05.642004 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:05.641986 4908 generic.go:356] "Generic (PLEG): container finished" podID="a3a015cd-f16d-42b4-abae-9e2dd114093e" containerID="b1d7afd7341510e4a5f062c50b50b2c3998054904a942041810b9ea14645ffb5" exitCode=60 Oct 30 12:45:05.642084 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:05.642063 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ecb3aa-29889405-srcpd" event={"ID":"a3a015cd-f16d-42b4-abae-9e2dd114093e","Type":"ContainerDied","Data":"b1d7afd7341510e4a5f062c50b50b2c3998054904a942041810b9ea14645ffb5"} Oct 30 12:45:05.642345 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:05.642323 4908 scope.go:122] "RemoveContainer" containerID="b1d7afd7341510e4a5f062c50b50b2c3998054904a942041810b9ea14645ffb5" Oct 30 12:45:05.642680 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:45:05.642655 4908 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-ecb3aa-29889405-srcpd_odh-ai-gateway-infra(a3a015cd-f16d-42b4-abae-9e2dd114093e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ecb3aa-29889405-srcpd" podUID="a3a015cd-f16d-42b4-abae-9e2dd114093e" Oct 30 12:45:05.643803 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:05.643784 4908 generic.go:356] "Generic (PLEG): container finished" podID="ccd6f000-a55d-4214-a830-faf2d200b688" containerID="d12a6301d143fec5dc8f58abcb4ddc4fd2c435b86527bb63f1337233d17f245e" exitCode=60 Oct 30 12:45:05.643919 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:05.643812 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2675c0-29889405-n4dv7" event={"ID":"ccd6f000-a55d-4214-a830-faf2d200b688","Type":"ContainerDied","Data":"d12a6301d143fec5dc8f58abcb4ddc4fd2c435b86527bb63f1337233d17f245e"} Oct 30 12:45:05.644094 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:05.644076 4908 scope.go:122] "RemoveContainer" containerID="d12a6301d143fec5dc8f58abcb4ddc4fd2c435b86527bb63f1337233d17f245e" Oct 30 12:45:05.644371 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:45:05.644350 4908 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-2675c0-29889405-n4dv7_odh-ai-gateway-infra(ccd6f000-a55d-4214-a830-faf2d200b688)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2675c0-29889405-n4dv7" podUID="ccd6f000-a55d-4214-a830-faf2d200b688" Oct 30 12:45:05.645761 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:05.645738 4908 generic.go:356] "Generic (PLEG): container finished" podID="7fae2258-d203-4719-a547-868d7b04070d" containerID="c268c9a5d2c266fd8722ca28f0ece47033655dc2089510a949c285d1a063cb91" exitCode=60 Oct 30 12:45:05.645856 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:05.645812 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f920f7-29889405-fp46s" event={"ID":"7fae2258-d203-4719-a547-868d7b04070d","Type":"ContainerDied","Data":"c268c9a5d2c266fd8722ca28f0ece47033655dc2089510a949c285d1a063cb91"} Oct 30 12:45:05.646012 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:05.645996 4908 scope.go:122] "RemoveContainer" containerID="c268c9a5d2c266fd8722ca28f0ece47033655dc2089510a949c285d1a063cb91" Oct 30 12:45:05.646233 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:45:05.646216 4908 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-f920f7-29889405-fp46s_odh-ai-gateway-infra(7fae2258-d203-4719-a547-868d7b04070d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f920f7-29889405-fp46s" podUID="7fae2258-d203-4719-a547-868d7b04070d" Oct 30 12:45:05.647678 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:05.647659 4908 generic.go:356] "Generic (PLEG): container finished" podID="e354caaa-fac9-485a-b960-7488fea0df0d" containerID="11a5433ba0f063525be4e327392af63b16b82e15a007833dd007a491f7bb307f" exitCode=60 Oct 30 12:45:05.647753 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:05.647722 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-812990-29889405-stgg2" event={"ID":"e354caaa-fac9-485a-b960-7488fea0df0d","Type":"ContainerDied","Data":"11a5433ba0f063525be4e327392af63b16b82e15a007833dd007a491f7bb307f"} Oct 30 12:45:05.648016 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:05.647998 4908 scope.go:122] "RemoveContainer" containerID="11a5433ba0f063525be4e327392af63b16b82e15a007833dd007a491f7bb307f" Oct 30 12:45:05.648258 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:45:05.648234 4908 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-812990-29889405-stgg2_odh-ai-gateway-infra(e354caaa-fac9-485a-b960-7488fea0df0d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-812990-29889405-stgg2" podUID="e354caaa-fac9-485a-b960-7488fea0df0d" Oct 30 12:45:05.649871 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:05.649804 4908 generic.go:356] "Generic (PLEG): container finished" podID="240c2309-8812-4591-9871-b16dcb9b6ff3" containerID="a38d646ff0f49f138829d6b4d6e9a32dc95422d184c175f842e78d1bedbe529f" exitCode=60 Oct 30 12:45:05.649871 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:05.649852 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-dd0628-29889405-cj8wq" event={"ID":"240c2309-8812-4591-9871-b16dcb9b6ff3","Type":"ContainerDied","Data":"a38d646ff0f49f138829d6b4d6e9a32dc95422d184c175f842e78d1bedbe529f"} Oct 30 12:45:05.650129 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:05.650111 4908 scope.go:122] "RemoveContainer" containerID="a38d646ff0f49f138829d6b4d6e9a32dc95422d184c175f842e78d1bedbe529f" Oct 30 12:45:05.650341 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:45:05.650321 4908 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-dd0628-29889405-cj8wq_odh-ai-gateway-infra(240c2309-8812-4591-9871-b16dcb9b6ff3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-dd0628-29889405-cj8wq" podUID="240c2309-8812-4591-9871-b16dcb9b6ff3" Oct 30 12:45:05.650810 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:05.650791 4908 scope.go:122] "RemoveContainer" containerID="ee351ebdd0e398981adfce246cda40da62f2369c29bf167560d216084c38ff44" Oct 30 12:45:05.651774 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:05.651759 4908 scope.go:122] "RemoveContainer" containerID="86e7231e8ea8407171fbf6e4bcc6b0d2c0a4943b1f1e7e7a4fe32e6ee6214b25" Oct 30 12:45:05.652044 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:45:05.652021 4908 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-6ad1da-29889405-bbbrz_odh-ai-gateway-infra(3eef63f0-3b6c-4e62-bdc9-594ee9d4bd17)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6ad1da-29889405-bbbrz" podUID="3eef63f0-3b6c-4e62-bdc9-594ee9d4bd17" Oct 30 12:45:05.653338 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:05.653321 4908 generic.go:356] "Generic (PLEG): container finished" podID="81dec304-04f6-4c40-8bf3-7fa3c4e0994d" containerID="28cfc981683a877d68c5d5c87b357dc4ee99e46a5677a9a40e3caf7a18f071ea" exitCode=60 Oct 30 12:45:05.653544 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:05.653343 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c1b1e8-29889405-5fkdv" event={"ID":"81dec304-04f6-4c40-8bf3-7fa3c4e0994d","Type":"ContainerDied","Data":"28cfc981683a877d68c5d5c87b357dc4ee99e46a5677a9a40e3caf7a18f071ea"} Oct 30 12:45:05.653604 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:05.653592 4908 scope.go:122] "RemoveContainer" containerID="28cfc981683a877d68c5d5c87b357dc4ee99e46a5677a9a40e3caf7a18f071ea" Oct 30 12:45:05.653786 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:45:05.653769 4908 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-c1b1e8-29889405-5fkdv_odh-ai-gateway-infra(81dec304-04f6-4c40-8bf3-7fa3c4e0994d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c1b1e8-29889405-5fkdv" podUID="81dec304-04f6-4c40-8bf3-7fa3c4e0994d" Oct 30 12:45:05.655182 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:05.655160 4908 generic.go:356] "Generic (PLEG): container finished" podID="fcc90cb3-e12e-4b12-a50d-de740f552b40" containerID="c5975e955216f73550a5fe8f9a1a4026b404886285cd3f4af674b4c65c60b820" exitCode=60 Oct 30 12:45:05.655256 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:05.655227 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-9d45fb-29889405-2gz7s" event={"ID":"fcc90cb3-e12e-4b12-a50d-de740f552b40","Type":"ContainerDied","Data":"c5975e955216f73550a5fe8f9a1a4026b404886285cd3f4af674b4c65c60b820"} Oct 30 12:45:05.655428 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:05.655398 4908 scope.go:122] "RemoveContainer" containerID="c5975e955216f73550a5fe8f9a1a4026b404886285cd3f4af674b4c65c60b820" Oct 30 12:45:05.655659 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:45:05.655640 4908 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-9d45fb-29889405-2gz7s_odh-ai-gateway-infra(fcc90cb3-e12e-4b12-a50d-de740f552b40)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-9d45fb-29889405-2gz7s" podUID="fcc90cb3-e12e-4b12-a50d-de740f552b40" Oct 30 12:45:05.657146 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:05.657123 4908 generic.go:356] "Generic (PLEG): container finished" podID="4d181a49-62cd-454a-9e03-380198da3a08" containerID="1ecc2f2193c9794de7252a33ef2b637a0fb771ad739ee9dba3f7c6ac1bfcef69" exitCode=60 Oct 30 12:45:05.657236 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:05.657192 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0f7d14-29889405-c7tnx" event={"ID":"4d181a49-62cd-454a-9e03-380198da3a08","Type":"ContainerDied","Data":"1ecc2f2193c9794de7252a33ef2b637a0fb771ad739ee9dba3f7c6ac1bfcef69"} Oct 30 12:45:05.657403 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:05.657387 4908 scope.go:122] "RemoveContainer" containerID="1ecc2f2193c9794de7252a33ef2b637a0fb771ad739ee9dba3f7c6ac1bfcef69" Oct 30 12:45:05.657619 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:45:05.657590 4908 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-0f7d14-29889405-c7tnx_odh-ai-gateway-infra(4d181a49-62cd-454a-9e03-380198da3a08)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0f7d14-29889405-c7tnx" podUID="4d181a49-62cd-454a-9e03-380198da3a08" Oct 30 12:45:05.659033 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:05.659015 4908 scope.go:122] "RemoveContainer" containerID="b57e0271d18afb9707f6556c49708c97241eedf42eed2fa54b1916215d67ed7f" Oct 30 12:45:05.659251 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:45:05.659231 4908 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-2545c7-29889405-x55rn_odh-ai-gateway-infra(0b9ecbb7-1e58-4ac9-b913-ff46c0a02210)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2545c7-29889405-x55rn" podUID="0b9ecbb7-1e58-4ac9-b913-ff46c0a02210" Oct 30 12:45:05.661118 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:05.661095 4908 generic.go:356] "Generic (PLEG): container finished" podID="ea9095cb-47a4-4ea4-b143-069f736c29b6" containerID="87f808e36def3853eb9b8e107bc2b422a33cf0406724742f3cf405244ef8f07a" exitCode=60 Oct 30 12:45:05.661230 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:05.661180 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-8d2170-29889405-82rzm" event={"ID":"ea9095cb-47a4-4ea4-b143-069f736c29b6","Type":"ContainerDied","Data":"87f808e36def3853eb9b8e107bc2b422a33cf0406724742f3cf405244ef8f07a"} Oct 30 12:45:05.661382 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:05.661365 4908 scope.go:122] "RemoveContainer" containerID="87f808e36def3853eb9b8e107bc2b422a33cf0406724742f3cf405244ef8f07a" Oct 30 12:45:05.661610 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:45:05.661589 4908 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-8d2170-29889405-82rzm_odh-ai-gateway-infra(ea9095cb-47a4-4ea4-b143-069f736c29b6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-8d2170-29889405-82rzm" podUID="ea9095cb-47a4-4ea4-b143-069f736c29b6" Oct 30 12:45:05.663220 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:05.663200 4908 scope.go:122] "RemoveContainer" containerID="cb5f9257cacca16f6e1ed8a7bd8255de113bc93d89da2b25b6e3c466c9531bcb" Oct 30 12:45:05.663558 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:45:05.663532 4908 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-0de4b9-29889405-4pk7b_odh-ai-gateway-infra(12c31795-7957-4dff-83c3-d4d2c1d31aac)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0de4b9-29889405-4pk7b" podUID="12c31795-7957-4dff-83c3-d4d2c1d31aac" Oct 30 12:45:05.665244 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:05.665221 4908 generic.go:356] "Generic (PLEG): container finished" podID="ab18f563-fbfd-4dbb-8b81-277c223a30f4" containerID="b60b948e805f3a9a626013f5f4e0d50b3a64f57dd24888a1b6fc8d447b9c3ee6" exitCode=60 Oct 30 12:45:05.665346 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:05.665299 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f7464f-29889405-zzjr7" event={"ID":"ab18f563-fbfd-4dbb-8b81-277c223a30f4","Type":"ContainerDied","Data":"b60b948e805f3a9a626013f5f4e0d50b3a64f57dd24888a1b6fc8d447b9c3ee6"} Oct 30 12:45:05.665462 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:05.665448 4908 scope.go:122] "RemoveContainer" containerID="b60b948e805f3a9a626013f5f4e0d50b3a64f57dd24888a1b6fc8d447b9c3ee6" Oct 30 12:45:05.665642 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:45:05.665625 4908 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-f7464f-29889405-zzjr7_odh-ai-gateway-infra(ab18f563-fbfd-4dbb-8b81-277c223a30f4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f7464f-29889405-zzjr7" podUID="ab18f563-fbfd-4dbb-8b81-277c223a30f4" Oct 30 12:45:05.666362 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:05.666342 4908 scope.go:122] "RemoveContainer" containerID="164ea7d97d9a02b357bcae630e979f5f48771699019b30de5c563b5de11a646d" Oct 30 12:45:05.682692 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:05.682672 4908 scope.go:122] "RemoveContainer" containerID="1869c3db5e4ba9f3b5c63a47ac21ddac18b6e93f18ed4c40e798573a2b000573" Oct 30 12:45:05.698301 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:05.698278 4908 scope.go:122] "RemoveContainer" containerID="decd72c23aa5f4b6f95688e5a092a99c348125afe45caaadc04f44aa636d4ecc" Oct 30 12:45:05.713420 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:05.713399 4908 scope.go:122] "RemoveContainer" containerID="b979df0dad047b0a4595d0cf7c51e2f478d26d0ce44d4a2ef681253dcb9f8b7d" Oct 30 12:45:05.727535 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:05.727514 4908 scope.go:122] "RemoveContainer" containerID="1d8956b8eb4b9c6037bbb0dd342c70f4341b49883fc8c24464d9b3a669c1a857" Oct 30 12:45:05.744429 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:05.743537 4908 scope.go:122] "RemoveContainer" containerID="79904a7e29e33a4c04415b3116e53b390ff2fc38bc2b3ff2fc6ec608f13d332d" Oct 30 12:45:05.758962 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:05.758938 4908 scope.go:122] "RemoveContainer" containerID="88d7753f48f3e94eb67177f17f6c2dd76d9ef0ed508452d5da87a577ac16672b" Oct 30 12:45:05.774721 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:05.774693 4908 scope.go:122] "RemoveContainer" containerID="c5e46a758af64e78f65e3485def44369aed65ef5ee6b55c4c03d0814599d7980" Oct 30 12:45:05.792042 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:05.792014 4908 scope.go:122] "RemoveContainer" containerID="a4473b6680c4f2a1f709ccf446cca4710973253807c932f2be3b2bf5f21f3e73" Oct 30 12:45:05.806225 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:05.806201 4908 scope.go:122] "RemoveContainer" containerID="fc1ff9baa10a8f937d3a69b86eaa9cea4c31c29f6b6d50a20db9ef10331297fe" Oct 30 12:45:05.822010 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:05.821963 4908 scope.go:122] "RemoveContainer" containerID="4e6410f56cc6daf833cad17ee73fe201859a48ea70858e7f27134d84e54fefda" Oct 30 12:45:06.646615 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:06.646574 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9wwfc"] Oct 30 12:45:06.676279 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:06.676242 4908 scope.go:122] "RemoveContainer" containerID="d4ce5f0ceb4f692801b346b4615ab3e5da5b06c74931968f35feb2892fef54c5" Oct 30 12:45:06.676636 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:45:06.676610 4908 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-73bafc-29889405-l27f6_odh-ai-gateway-infra(1aa1b39f-f88b-4ddf-8ee1-cf567d9d480a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-73bafc-29889405-l27f6" podUID="1aa1b39f-f88b-4ddf-8ee1-cf567d9d480a" Oct 30 12:45:06.678188 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:06.678169 4908 scope.go:122] "RemoveContainer" containerID="b1d7afd7341510e4a5f062c50b50b2c3998054904a942041810b9ea14645ffb5" Oct 30 12:45:06.678625 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:45:06.678603 4908 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-ecb3aa-29889405-srcpd_odh-ai-gateway-infra(a3a015cd-f16d-42b4-abae-9e2dd114093e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ecb3aa-29889405-srcpd" podUID="a3a015cd-f16d-42b4-abae-9e2dd114093e" Oct 30 12:45:06.680194 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:06.680172 4908 scope.go:122] "RemoveContainer" containerID="d12a6301d143fec5dc8f58abcb4ddc4fd2c435b86527bb63f1337233d17f245e" Oct 30 12:45:06.680485 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:45:06.680456 4908 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-2675c0-29889405-n4dv7_odh-ai-gateway-infra(ccd6f000-a55d-4214-a830-faf2d200b688)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2675c0-29889405-n4dv7" podUID="ccd6f000-a55d-4214-a830-faf2d200b688" Oct 30 12:45:06.682108 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:06.682090 4908 scope.go:122] "RemoveContainer" containerID="c268c9a5d2c266fd8722ca28f0ece47033655dc2089510a949c285d1a063cb91" Oct 30 12:45:06.682342 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:45:06.682320 4908 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-f920f7-29889405-fp46s_odh-ai-gateway-infra(7fae2258-d203-4719-a547-868d7b04070d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f920f7-29889405-fp46s" podUID="7fae2258-d203-4719-a547-868d7b04070d" Oct 30 12:45:06.683815 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:06.683797 4908 scope.go:122] "RemoveContainer" containerID="11a5433ba0f063525be4e327392af63b16b82e15a007833dd007a491f7bb307f" Oct 30 12:45:06.684036 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:45:06.684012 4908 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-812990-29889405-stgg2_odh-ai-gateway-infra(e354caaa-fac9-485a-b960-7488fea0df0d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-812990-29889405-stgg2" podUID="e354caaa-fac9-485a-b960-7488fea0df0d" Oct 30 12:45:06.685466 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:06.685449 4908 scope.go:122] "RemoveContainer" containerID="a38d646ff0f49f138829d6b4d6e9a32dc95422d184c175f842e78d1bedbe529f" Oct 30 12:45:06.685694 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:45:06.685665 4908 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-dd0628-29889405-cj8wq_odh-ai-gateway-infra(240c2309-8812-4591-9871-b16dcb9b6ff3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-dd0628-29889405-cj8wq" podUID="240c2309-8812-4591-9871-b16dcb9b6ff3" Oct 30 12:45:06.687240 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:06.687218 4908 scope.go:122] "RemoveContainer" containerID="28cfc981683a877d68c5d5c87b357dc4ee99e46a5677a9a40e3caf7a18f071ea" Oct 30 12:45:06.687521 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:45:06.687497 4908 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-c1b1e8-29889405-5fkdv_odh-ai-gateway-infra(81dec304-04f6-4c40-8bf3-7fa3c4e0994d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c1b1e8-29889405-5fkdv" podUID="81dec304-04f6-4c40-8bf3-7fa3c4e0994d" Oct 30 12:45:06.688926 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:06.688906 4908 scope.go:122] "RemoveContainer" containerID="c5975e955216f73550a5fe8f9a1a4026b404886285cd3f4af674b4c65c60b820" Oct 30 12:45:06.689118 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:45:06.689097 4908 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-9d45fb-29889405-2gz7s_odh-ai-gateway-infra(fcc90cb3-e12e-4b12-a50d-de740f552b40)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-9d45fb-29889405-2gz7s" podUID="fcc90cb3-e12e-4b12-a50d-de740f552b40" Oct 30 12:45:06.690633 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:06.690608 4908 scope.go:122] "RemoveContainer" containerID="1ecc2f2193c9794de7252a33ef2b637a0fb771ad739ee9dba3f7c6ac1bfcef69" Oct 30 12:45:06.690822 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:45:06.690797 4908 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-0f7d14-29889405-c7tnx_odh-ai-gateway-infra(4d181a49-62cd-454a-9e03-380198da3a08)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0f7d14-29889405-c7tnx" podUID="4d181a49-62cd-454a-9e03-380198da3a08" Oct 30 12:45:06.692711 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:06.692689 4908 scope.go:122] "RemoveContainer" containerID="87f808e36def3853eb9b8e107bc2b422a33cf0406724742f3cf405244ef8f07a" Oct 30 12:45:06.692940 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:45:06.692917 4908 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-8d2170-29889405-82rzm_odh-ai-gateway-infra(ea9095cb-47a4-4ea4-b143-069f736c29b6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-8d2170-29889405-82rzm" podUID="ea9095cb-47a4-4ea4-b143-069f736c29b6" Oct 30 12:45:06.694482 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:06.694459 4908 scope.go:122] "RemoveContainer" containerID="b60b948e805f3a9a626013f5f4e0d50b3a64f57dd24888a1b6fc8d447b9c3ee6" Oct 30 12:45:06.694739 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:45:06.694711 4908 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-f7464f-29889405-zzjr7_odh-ai-gateway-infra(ab18f563-fbfd-4dbb-8b81-277c223a30f4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f7464f-29889405-zzjr7" podUID="ab18f563-fbfd-4dbb-8b81-277c223a30f4" Oct 30 12:45:06.696179 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:06.696161 4908 scope.go:122] "RemoveContainer" containerID="97b9267ff8f1831eb7fe9024663eb52367837e40ede0c7fac187246461049edd" Oct 30 12:45:06.696372 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:45:06.696352 4908 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-327e31-29889405-lr598_odh-ai-gateway-infra(27d77dc7-aee3-45a4-8ad9-a48560889798)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-327e31-29889405-lr598" podUID="27d77dc7-aee3-45a4-8ad9-a48560889798" Oct 30 12:45:06.697991 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:06.697973 4908 scope.go:122] "RemoveContainer" containerID="3cd62c631dd03cd2b4a63e38adba127303c8eb05ddfc0a92a00b2069f8df1022" Oct 30 12:45:06.698192 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:45:06.698172 4908 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-0bda67-29889405-q74jd_odh-ai-gateway-infra(958d7012-017f-4cd4-a637-a44ec84babc6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0bda67-29889405-q74jd" podUID="958d7012-017f-4cd4-a637-a44ec84babc6" Oct 30 12:45:15.606941 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:15.606909 4908 scope.go:122] "RemoveContainer" containerID="c268c9a5d2c266fd8722ca28f0ece47033655dc2089510a949c285d1a063cb91" Oct 30 12:45:15.607523 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:15.607003 4908 scope.go:122] "RemoveContainer" containerID="c5975e955216f73550a5fe8f9a1a4026b404886285cd3f4af674b4c65c60b820" Oct 30 12:45:15.607523 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:15.607137 4908 scope.go:122] "RemoveContainer" containerID="86e7231e8ea8407171fbf6e4bcc6b0d2c0a4943b1f1e7e7a4fe32e6ee6214b25" Oct 30 12:45:16.598133 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:16.598106 4908 scope.go:122] "RemoveContainer" containerID="d4ce5f0ceb4f692801b346b4615ab3e5da5b06c74931968f35feb2892fef54c5" Oct 30 12:45:16.599025 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:16.598535 4908 scope.go:122] "RemoveContainer" containerID="b1d7afd7341510e4a5f062c50b50b2c3998054904a942041810b9ea14645ffb5" Oct 30 12:45:16.599025 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:16.598774 4908 scope.go:122] "RemoveContainer" containerID="87f808e36def3853eb9b8e107bc2b422a33cf0406724742f3cf405244ef8f07a" Oct 30 12:45:16.599025 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:16.598877 4908 scope.go:122] "RemoveContainer" containerID="b60b948e805f3a9a626013f5f4e0d50b3a64f57dd24888a1b6fc8d447b9c3ee6" Oct 30 12:45:16.599385 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:16.599206 4908 scope.go:122] "RemoveContainer" containerID="11a5433ba0f063525be4e327392af63b16b82e15a007833dd007a491f7bb307f" Oct 30 12:45:16.784896 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:16.784860 4908 generic.go:356] "Generic (PLEG): container finished" podID="7fae2258-d203-4719-a547-868d7b04070d" containerID="a48055d08ff0ef02e29b1f320ff1f0fb327b5cf26b890c8a5930704cd83abbc9" exitCode=60 Oct 30 12:45:16.785300 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:16.784898 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f920f7-29889405-fp46s" event={"ID":"7fae2258-d203-4719-a547-868d7b04070d","Type":"ContainerDied","Data":"a48055d08ff0ef02e29b1f320ff1f0fb327b5cf26b890c8a5930704cd83abbc9"} Oct 30 12:45:16.785300 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:16.784959 4908 scope.go:122] "RemoveContainer" containerID="c268c9a5d2c266fd8722ca28f0ece47033655dc2089510a949c285d1a063cb91" Oct 30 12:45:16.785435 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:16.785380 4908 scope.go:122] "RemoveContainer" containerID="a48055d08ff0ef02e29b1f320ff1f0fb327b5cf26b890c8a5930704cd83abbc9" Oct 30 12:45:16.785721 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:45:16.785693 4908 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-f920f7-29889405-fp46s_odh-ai-gateway-infra(7fae2258-d203-4719-a547-868d7b04070d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f920f7-29889405-fp46s" podUID="7fae2258-d203-4719-a547-868d7b04070d" Oct 30 12:45:16.787343 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:16.787318 4908 generic.go:356] "Generic (PLEG): container finished" podID="3eef63f0-3b6c-4e62-bdc9-594ee9d4bd17" containerID="75acb3ade9024b8c465db2b72202513859ea6a62c3ae64de32a3b275b1119874" exitCode=60 Oct 30 12:45:16.787451 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:16.787387 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6ad1da-29889405-bbbrz" event={"ID":"3eef63f0-3b6c-4e62-bdc9-594ee9d4bd17","Type":"ContainerDied","Data":"75acb3ade9024b8c465db2b72202513859ea6a62c3ae64de32a3b275b1119874"} Oct 30 12:45:16.787778 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:16.787757 4908 scope.go:122] "RemoveContainer" containerID="75acb3ade9024b8c465db2b72202513859ea6a62c3ae64de32a3b275b1119874" Oct 30 12:45:16.788043 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:45:16.788024 4908 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-6ad1da-29889405-bbbrz_odh-ai-gateway-infra(3eef63f0-3b6c-4e62-bdc9-594ee9d4bd17)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6ad1da-29889405-bbbrz" podUID="3eef63f0-3b6c-4e62-bdc9-594ee9d4bd17" Oct 30 12:45:16.789413 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:16.789390 4908 generic.go:356] "Generic (PLEG): container finished" podID="fcc90cb3-e12e-4b12-a50d-de740f552b40" containerID="f47cc524c2a11573f89b6496491b5fca8cd98651f07fa7e24a0a516c96b8a6da" exitCode=60 Oct 30 12:45:16.789507 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:16.789450 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-9d45fb-29889405-2gz7s" event={"ID":"fcc90cb3-e12e-4b12-a50d-de740f552b40","Type":"ContainerDied","Data":"f47cc524c2a11573f89b6496491b5fca8cd98651f07fa7e24a0a516c96b8a6da"} Oct 30 12:45:16.789869 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:16.789820 4908 scope.go:122] "RemoveContainer" containerID="f47cc524c2a11573f89b6496491b5fca8cd98651f07fa7e24a0a516c96b8a6da" Oct 30 12:45:16.790141 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:45:16.790117 4908 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-9d45fb-29889405-2gz7s_odh-ai-gateway-infra(fcc90cb3-e12e-4b12-a50d-de740f552b40)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-9d45fb-29889405-2gz7s" podUID="fcc90cb3-e12e-4b12-a50d-de740f552b40" Oct 30 12:45:16.803093 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:16.803074 4908 scope.go:122] "RemoveContainer" containerID="86e7231e8ea8407171fbf6e4bcc6b0d2c0a4943b1f1e7e7a4fe32e6ee6214b25" Oct 30 12:45:16.817550 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:16.817540 4908 scope.go:122] "RemoveContainer" containerID="c5975e955216f73550a5fe8f9a1a4026b404886285cd3f4af674b4c65c60b820" Oct 30 12:45:17.611872 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:17.611844 4908 scope.go:122] "RemoveContainer" containerID="28cfc981683a877d68c5d5c87b357dc4ee99e46a5677a9a40e3caf7a18f071ea" Oct 30 12:45:17.612053 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:17.611922 4908 scope.go:122] "RemoveContainer" containerID="d12a6301d143fec5dc8f58abcb4ddc4fd2c435b86527bb63f1337233d17f245e" Oct 30 12:45:17.612104 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:17.612053 4908 scope.go:122] "RemoveContainer" containerID="3cd62c631dd03cd2b4a63e38adba127303c8eb05ddfc0a92a00b2069f8df1022" Oct 30 12:45:17.612289 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:17.612267 4908 scope.go:122] "RemoveContainer" containerID="b57e0271d18afb9707f6556c49708c97241eedf42eed2fa54b1916215d67ed7f" Oct 30 12:45:17.612608 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:17.612549 4908 scope.go:122] "RemoveContainer" containerID="cb5f9257cacca16f6e1ed8a7bd8255de113bc93d89da2b25b6e3c466c9531bcb" Oct 30 12:45:17.800581 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:17.800552 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-8d2170-29889405-82rzm" event={"ID":"ea9095cb-47a4-4ea4-b143-069f736c29b6","Type":"ContainerStarted","Data":"686e7872d70672f7517cc90c9bdca39c8d842129a2fc80dced64c60d1255f160"} Oct 30 12:45:17.802973 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:17.802947 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f7464f-29889405-zzjr7" event={"ID":"ab18f563-fbfd-4dbb-8b81-277c223a30f4","Type":"ContainerStarted","Data":"f927a23d07841776e2b829e96d7eb0e864a79bd0c7aef3e5736fca413595587d"} Oct 30 12:45:17.805225 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:17.805200 4908 generic.go:356] "Generic (PLEG): container finished" podID="1aa1b39f-f88b-4ddf-8ee1-cf567d9d480a" containerID="caeef42f67821699eb55280dec7dfeb4ffbd3714cf3995dfbb422a4d79bfc3cb" exitCode=60 Oct 30 12:45:17.805320 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:17.805263 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-73bafc-29889405-l27f6" event={"ID":"1aa1b39f-f88b-4ddf-8ee1-cf567d9d480a","Type":"ContainerDied","Data":"caeef42f67821699eb55280dec7dfeb4ffbd3714cf3995dfbb422a4d79bfc3cb"} Oct 30 12:45:17.805320 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:17.805311 4908 scope.go:122] "RemoveContainer" containerID="d4ce5f0ceb4f692801b346b4615ab3e5da5b06c74931968f35feb2892fef54c5" Oct 30 12:45:17.805623 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:17.805607 4908 scope.go:122] "RemoveContainer" containerID="caeef42f67821699eb55280dec7dfeb4ffbd3714cf3995dfbb422a4d79bfc3cb" Oct 30 12:45:17.806030 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:45:17.805927 4908 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-73bafc-29889405-l27f6_odh-ai-gateway-infra(1aa1b39f-f88b-4ddf-8ee1-cf567d9d480a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-73bafc-29889405-l27f6" podUID="1aa1b39f-f88b-4ddf-8ee1-cf567d9d480a" Oct 30 12:45:17.808078 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:17.808050 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ecb3aa-29889405-srcpd" event={"ID":"a3a015cd-f16d-42b4-abae-9e2dd114093e","Type":"ContainerStarted","Data":"fd593bdf2d871963b56351aaac5a7059482f1aa1df835dc0c1b426ef8bed9523"} Oct 30 12:45:17.811691 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:17.811669 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-812990-29889405-stgg2" event={"ID":"e354caaa-fac9-485a-b960-7488fea0df0d","Type":"ContainerStarted","Data":"a69cd4f91636e0b0ca9c46bd61743170f0ab51a258797840d778a6fa487c108b"} Oct 30 12:45:17.842680 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:17.842660 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f920f7-29889405-fp46s"] Oct 30 12:45:17.889052 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:17.889019 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6ad1da-29889405-bbbrz"] Oct 30 12:45:17.937041 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:17.937009 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-9d45fb-29889405-2gz7s"] Oct 30 12:45:18.081062 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:18.078763 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53111ea1-6dc7q"] Oct 30 12:45:18.087452 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:18.087414 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53111ea1-6dc7q"] Oct 30 12:45:18.193260 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:18.193220 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f920f7-29889405-fp46s" Oct 30 12:45:18.274409 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:18.274387 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6ad1da-29889405-bbbrz" Oct 30 12:45:18.280382 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:18.280358 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-9d45fb-29889405-2gz7s" Oct 30 12:45:18.313355 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:18.313330 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3eef63f0-3b6c-4e62-bdc9-594ee9d4bd17-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3eef63f0-3b6c-4e62-bdc9-594ee9d4bd17-maas-api-service-ca\") pod \"3eef63f0-3b6c-4e62-bdc9-594ee9d4bd17\" (UID: \"3eef63f0-3b6c-4e62-bdc9-594ee9d4bd17\") " Oct 30 12:45:18.313484 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:18.313374 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fae2258-d203-4719-a547-868d7b04070d-kube-api-access-n4g9l\" (UniqueName: \"kubernetes.io/projected/7fae2258-d203-4719-a547-868d7b04070d-kube-api-access-n4g9l\") pod \"7fae2258-d203-4719-a547-868d7b04070d\" (UID: \"7fae2258-d203-4719-a547-868d7b04070d\") " Oct 30 12:45:18.313484 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:18.313444 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fcc90cb3-e12e-4b12-a50d-de740f552b40-kube-api-access-dfrb2\" (UniqueName: \"kubernetes.io/projected/fcc90cb3-e12e-4b12-a50d-de740f552b40-kube-api-access-dfrb2\") pod \"fcc90cb3-e12e-4b12-a50d-de740f552b40\" (UID: \"fcc90cb3-e12e-4b12-a50d-de740f552b40\") " Oct 30 12:45:18.313604 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:18.313549 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7fae2258-d203-4719-a547-868d7b04070d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fae2258-d203-4719-a547-868d7b04070d-maas-api-service-ca\") pod \"7fae2258-d203-4719-a547-868d7b04070d\" (UID: \"7fae2258-d203-4719-a547-868d7b04070d\") " Oct 30 12:45:18.313604 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:18.313590 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fcc90cb3-e12e-4b12-a50d-de740f552b40-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fcc90cb3-e12e-4b12-a50d-de740f552b40-maas-api-service-ca\") pod \"fcc90cb3-e12e-4b12-a50d-de740f552b40\" (UID: \"fcc90cb3-e12e-4b12-a50d-de740f552b40\") " Oct 30 12:45:18.313708 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:18.313640 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3eef63f0-3b6c-4e62-bdc9-594ee9d4bd17-kube-api-access-h5l7d\" (UniqueName: \"kubernetes.io/projected/3eef63f0-3b6c-4e62-bdc9-594ee9d4bd17-kube-api-access-h5l7d\") pod \"3eef63f0-3b6c-4e62-bdc9-594ee9d4bd17\" (UID: \"3eef63f0-3b6c-4e62-bdc9-594ee9d4bd17\") " Oct 30 12:45:18.314110 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:18.314080 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fae2258-d203-4719-a547-868d7b04070d-maas-api-service-ca" pod "7fae2258-d203-4719-a547-868d7b04070d" (UID: "7fae2258-d203-4719-a547-868d7b04070d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:45:18.314263 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:18.314230 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcc90cb3-e12e-4b12-a50d-de740f552b40-maas-api-service-ca" pod "fcc90cb3-e12e-4b12-a50d-de740f552b40" (UID: "fcc90cb3-e12e-4b12-a50d-de740f552b40"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:45:18.314415 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:18.314395 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fae2258-d203-4719-a547-868d7b04070d-maas-api-service-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:45:18.314605 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:18.314417 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fcc90cb3-e12e-4b12-a50d-de740f552b40-maas-api-service-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:45:18.315081 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:18.315048 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3eef63f0-3b6c-4e62-bdc9-594ee9d4bd17-maas-api-service-ca" pod "3eef63f0-3b6c-4e62-bdc9-594ee9d4bd17" (UID: "3eef63f0-3b6c-4e62-bdc9-594ee9d4bd17"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:45:18.316738 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:18.316713 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fae2258-d203-4719-a547-868d7b04070d-kube-api-access-n4g9l" pod "7fae2258-d203-4719-a547-868d7b04070d" (UID: "7fae2258-d203-4719-a547-868d7b04070d"). InnerVolumeSpecName "kube-api-access-n4g9l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:45:18.316976 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:18.316869 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcc90cb3-e12e-4b12-a50d-de740f552b40-kube-api-access-dfrb2" pod "fcc90cb3-e12e-4b12-a50d-de740f552b40" (UID: "fcc90cb3-e12e-4b12-a50d-de740f552b40"). InnerVolumeSpecName "kube-api-access-dfrb2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:45:18.317385 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:18.317365 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3eef63f0-3b6c-4e62-bdc9-594ee9d4bd17-kube-api-access-h5l7d" pod "3eef63f0-3b6c-4e62-bdc9-594ee9d4bd17" (UID: "3eef63f0-3b6c-4e62-bdc9-594ee9d4bd17"). InnerVolumeSpecName "kube-api-access-h5l7d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:45:18.416618 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:18.416544 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5l7d\" (UniqueName: \"kubernetes.io/projected/3eef63f0-3b6c-4e62-bdc9-594ee9d4bd17-kube-api-access-h5l7d\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:45:18.416618 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:18.416570 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3eef63f0-3b6c-4e62-bdc9-594ee9d4bd17-maas-api-service-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:45:18.416618 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:18.416590 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4g9l\" (UniqueName: \"kubernetes.io/projected/7fae2258-d203-4719-a547-868d7b04070d-kube-api-access-n4g9l\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:45:18.416618 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:18.416599 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dfrb2\" (UniqueName: \"kubernetes.io/projected/fcc90cb3-e12e-4b12-a50d-de740f552b40-kube-api-access-dfrb2\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:45:18.603517 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:18.603473 4908 scope.go:122] "RemoveContainer" containerID="a38d646ff0f49f138829d6b4d6e9a32dc95422d184c175f842e78d1bedbe529f" Oct 30 12:45:18.604364 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:18.604339 4908 scope.go:122] "RemoveContainer" containerID="1ecc2f2193c9794de7252a33ef2b637a0fb771ad739ee9dba3f7c6ac1bfcef69" Oct 30 12:45:18.605004 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:18.604982 4908 scope.go:122] "RemoveContainer" containerID="97b9267ff8f1831eb7fe9024663eb52367837e40ede0c7fac187246461049edd" Oct 30 12:45:18.826035 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:18.826011 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6ad1da-29889405-bbbrz" Oct 30 12:45:18.826035 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:18.826016 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6ad1da-29889405-bbbrz" event={"ID":"3eef63f0-3b6c-4e62-bdc9-594ee9d4bd17","Type":"ContainerDied","Data":"763cefcc3b66f8c23e9a6446b658e6eba031064ee534423c1b8818b639cf8222"} Oct 30 12:45:18.826675 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:18.826077 4908 scope.go:122] "RemoveContainer" containerID="75acb3ade9024b8c465db2b72202513859ea6a62c3ae64de32a3b275b1119874" Oct 30 12:45:18.829213 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:18.828654 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c1b1e8-29889405-5fkdv" event={"ID":"81dec304-04f6-4c40-8bf3-7fa3c4e0994d","Type":"ContainerStarted","Data":"8ea21b30baa0fcf3080805b940e57eb6e3c776c5050d4f257a4f75ca9aa84976"} Oct 30 12:45:18.830191 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:18.830174 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-9d45fb-29889405-2gz7s" Oct 30 12:45:18.830296 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:18.830221 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-9d45fb-29889405-2gz7s" event={"ID":"fcc90cb3-e12e-4b12-a50d-de740f552b40","Type":"ContainerDied","Data":"c717b1d24445cc627f9f30f19a007e4f704be4508c07f3d141e1e84c09cf7bcb"} Oct 30 12:45:18.832428 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:18.832406 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2545c7-29889405-x55rn" event={"ID":"0b9ecbb7-1e58-4ac9-b913-ff46c0a02210","Type":"ContainerStarted","Data":"a563f3897d86ffd0f5f7da590f69431333ff2cc537696b8d43ccd40698809c16"} Oct 30 12:45:18.834712 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:18.834687 4908 generic.go:356] "Generic (PLEG): container finished" podID="ea9095cb-47a4-4ea4-b143-069f736c29b6" containerID="686e7872d70672f7517cc90c9bdca39c8d842129a2fc80dced64c60d1255f160" exitCode=60 Oct 30 12:45:18.834811 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:18.834708 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-8d2170-29889405-82rzm" event={"ID":"ea9095cb-47a4-4ea4-b143-069f736c29b6","Type":"ContainerDied","Data":"686e7872d70672f7517cc90c9bdca39c8d842129a2fc80dced64c60d1255f160"} Oct 30 12:45:18.835053 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:18.835033 4908 scope.go:122] "RemoveContainer" containerID="686e7872d70672f7517cc90c9bdca39c8d842129a2fc80dced64c60d1255f160" Oct 30 12:45:18.835264 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:45:18.835246 4908 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-8d2170-29889405-82rzm_odh-ai-gateway-infra(ea9095cb-47a4-4ea4-b143-069f736c29b6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-8d2170-29889405-82rzm" podUID="ea9095cb-47a4-4ea4-b143-069f736c29b6" Oct 30 12:45:18.837761 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:18.837739 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0de4b9-29889405-4pk7b" event={"ID":"12c31795-7957-4dff-83c3-d4d2c1d31aac","Type":"ContainerStarted","Data":"ba59444e58c6b2529c1b830a77fb361c0d5716a949ea3be9772154b17bca0552"} Oct 30 12:45:18.839207 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:18.839190 4908 scope.go:122] "RemoveContainer" containerID="f47cc524c2a11573f89b6496491b5fca8cd98651f07fa7e24a0a516c96b8a6da" Oct 30 12:45:18.840221 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:18.840182 4908 generic.go:356] "Generic (PLEG): container finished" podID="ab18f563-fbfd-4dbb-8b81-277c223a30f4" containerID="f927a23d07841776e2b829e96d7eb0e864a79bd0c7aef3e5736fca413595587d" exitCode=60 Oct 30 12:45:18.840335 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:18.840308 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f7464f-29889405-zzjr7" event={"ID":"ab18f563-fbfd-4dbb-8b81-277c223a30f4","Type":"ContainerDied","Data":"f927a23d07841776e2b829e96d7eb0e864a79bd0c7aef3e5736fca413595587d"} Oct 30 12:45:18.840946 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:18.840928 4908 scope.go:122] "RemoveContainer" containerID="f927a23d07841776e2b829e96d7eb0e864a79bd0c7aef3e5736fca413595587d" Oct 30 12:45:18.841283 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:45:18.841256 4908 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-f7464f-29889405-zzjr7_odh-ai-gateway-infra(ab18f563-fbfd-4dbb-8b81-277c223a30f4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f7464f-29889405-zzjr7" podUID="ab18f563-fbfd-4dbb-8b81-277c223a30f4" Oct 30 12:45:18.842611 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:18.842587 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0bda67-29889405-q74jd" event={"ID":"958d7012-017f-4cd4-a637-a44ec84babc6","Type":"ContainerStarted","Data":"ce06be9927a31d27c7fb0c45e90592c4c2b0985479cce1b21f4d15cd061c8bb3"} Oct 30 12:45:18.847538 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:18.847519 4908 generic.go:356] "Generic (PLEG): container finished" podID="a3a015cd-f16d-42b4-abae-9e2dd114093e" containerID="fd593bdf2d871963b56351aaac5a7059482f1aa1df835dc0c1b426ef8bed9523" exitCode=60 Oct 30 12:45:18.847622 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:18.847593 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ecb3aa-29889405-srcpd" event={"ID":"a3a015cd-f16d-42b4-abae-9e2dd114093e","Type":"ContainerDied","Data":"fd593bdf2d871963b56351aaac5a7059482f1aa1df835dc0c1b426ef8bed9523"} Oct 30 12:45:18.848001 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:18.847978 4908 scope.go:122] "RemoveContainer" containerID="fd593bdf2d871963b56351aaac5a7059482f1aa1df835dc0c1b426ef8bed9523" Oct 30 12:45:18.848256 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:45:18.848230 4908 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-ecb3aa-29889405-srcpd_odh-ai-gateway-infra(a3a015cd-f16d-42b4-abae-9e2dd114093e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ecb3aa-29889405-srcpd" podUID="a3a015cd-f16d-42b4-abae-9e2dd114093e" Oct 30 12:45:18.849475 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:18.849452 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2675c0-29889405-n4dv7" event={"ID":"ccd6f000-a55d-4214-a830-faf2d200b688","Type":"ContainerStarted","Data":"1c6c4d604e413c8e41440f7a726fa9cbd054de04098fa9f8be8513163f17a9f8"} Oct 30 12:45:18.850890 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:18.850866 4908 scope.go:122] "RemoveContainer" containerID="87f808e36def3853eb9b8e107bc2b422a33cf0406724742f3cf405244ef8f07a" Oct 30 12:45:18.851152 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:18.850927 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f920f7-29889405-fp46s" Oct 30 12:45:18.851152 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:18.850937 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f920f7-29889405-fp46s" event={"ID":"7fae2258-d203-4719-a547-868d7b04070d","Type":"ContainerDied","Data":"8bdfc71c07e1df036e8e134113b55bb7baf1f84fd40b05ba0a25326b74f36eb9"} Oct 30 12:45:18.874289 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:18.874240 4908 generic.go:356] "Generic (PLEG): container finished" podID="e354caaa-fac9-485a-b960-7488fea0df0d" containerID="a69cd4f91636e0b0ca9c46bd61743170f0ab51a258797840d778a6fa487c108b" exitCode=60 Oct 30 12:45:18.874417 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:18.874341 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-812990-29889405-stgg2" event={"ID":"e354caaa-fac9-485a-b960-7488fea0df0d","Type":"ContainerDied","Data":"a69cd4f91636e0b0ca9c46bd61743170f0ab51a258797840d778a6fa487c108b"} Oct 30 12:45:18.875010 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:18.874987 4908 scope.go:122] "RemoveContainer" containerID="a69cd4f91636e0b0ca9c46bd61743170f0ab51a258797840d778a6fa487c108b" Oct 30 12:45:18.875744 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:45:18.875674 4908 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-812990-29889405-stgg2_odh-ai-gateway-infra(e354caaa-fac9-485a-b960-7488fea0df0d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-812990-29889405-stgg2" podUID="e354caaa-fac9-485a-b960-7488fea0df0d" Oct 30 12:45:18.888936 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:18.888910 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-8d2170-29889405-82rzm"] Oct 30 12:45:18.936189 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:18.936139 4908 scope.go:122] "RemoveContainer" containerID="b60b948e805f3a9a626013f5f4e0d50b3a64f57dd24888a1b6fc8d447b9c3ee6" Oct 30 12:45:18.938818 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:18.938797 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-812990-29889405-stgg2"] Oct 30 12:45:18.950882 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:18.950822 4908 scope.go:122] "RemoveContainer" containerID="b1d7afd7341510e4a5f062c50b50b2c3998054904a942041810b9ea14645ffb5" Oct 30 12:45:18.969893 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:18.969755 4908 scope.go:122] "RemoveContainer" containerID="a48055d08ff0ef02e29b1f320ff1f0fb327b5cf26b890c8a5930704cd83abbc9" Oct 30 12:45:19.032233 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:19.032203 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6ad1da-29889405-bbbrz"] Oct 30 12:45:19.040606 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:19.040577 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6ad1da-29889405-bbbrz"] Oct 30 12:45:19.071031 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:19.071005 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-73bafc-29889405-l27f6"] Oct 30 12:45:19.078652 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:19.078625 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-288d314b-qtpns"] Oct 30 12:45:19.085703 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:19.085676 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-288d314b-qtpns"] Oct 30 12:45:19.117233 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:19.113884 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ecb3aa-29889405-srcpd"] Oct 30 12:45:19.251166 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:19.251129 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f7464f-29889405-zzjr7"] Oct 30 12:45:19.294353 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:19.294324 4908 scope.go:122] "RemoveContainer" containerID="11a5433ba0f063525be4e327392af63b16b82e15a007833dd007a491f7bb307f" Oct 30 12:45:19.332125 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:19.332099 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-73bafc-29889405-l27f6" Oct 30 12:45:19.354173 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:19.354148 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1aa1b39f-f88b-4ddf-8ee1-cf567d9d480a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1aa1b39f-f88b-4ddf-8ee1-cf567d9d480a-maas-api-service-ca\") pod \"1aa1b39f-f88b-4ddf-8ee1-cf567d9d480a\" (UID: \"1aa1b39f-f88b-4ddf-8ee1-cf567d9d480a\") " Oct 30 12:45:19.354302 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:19.354214 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1aa1b39f-f88b-4ddf-8ee1-cf567d9d480a-kube-api-access-mlbzs\" (UniqueName: \"kubernetes.io/projected/1aa1b39f-f88b-4ddf-8ee1-cf567d9d480a-kube-api-access-mlbzs\") pod \"1aa1b39f-f88b-4ddf-8ee1-cf567d9d480a\" (UID: \"1aa1b39f-f88b-4ddf-8ee1-cf567d9d480a\") " Oct 30 12:45:19.354522 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:19.354499 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1aa1b39f-f88b-4ddf-8ee1-cf567d9d480a-maas-api-service-ca" pod "1aa1b39f-f88b-4ddf-8ee1-cf567d9d480a" (UID: "1aa1b39f-f88b-4ddf-8ee1-cf567d9d480a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:45:19.354807 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:19.354787 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1aa1b39f-f88b-4ddf-8ee1-cf567d9d480a-maas-api-service-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:45:19.356438 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:19.356407 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1aa1b39f-f88b-4ddf-8ee1-cf567d9d480a-kube-api-access-mlbzs" pod "1aa1b39f-f88b-4ddf-8ee1-cf567d9d480a" (UID: "1aa1b39f-f88b-4ddf-8ee1-cf567d9d480a"). InnerVolumeSpecName "kube-api-access-mlbzs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:45:19.373943 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:19.373911 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-9d45fb-29889405-2gz7s"] Oct 30 12:45:19.379450 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:19.379428 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-9d45fb-29889405-2gz7s"] Oct 30 12:45:19.455943 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:19.455850 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mlbzs\" (UniqueName: \"kubernetes.io/projected/1aa1b39f-f88b-4ddf-8ee1-cf567d9d480a-kube-api-access-mlbzs\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:45:19.573354 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:19.573324 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f920f7-29889405-fp46s"] Oct 30 12:45:19.580386 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:19.580360 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f920f7-29889405-fp46s"] Oct 30 12:45:19.608065 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:19.608027 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="359a81f9-7309-414e-924c-f82fbfa4037d" path="/var/lib/kubelet/pods/359a81f9-7309-414e-924c-f82fbfa4037d/volumes" Oct 30 12:45:19.608653 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:19.608620 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3eef63f0-3b6c-4e62-bdc9-594ee9d4bd17" path="/var/lib/kubelet/pods/3eef63f0-3b6c-4e62-bdc9-594ee9d4bd17/volumes" Oct 30 12:45:19.609202 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:19.609183 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7fae2258-d203-4719-a547-868d7b04070d" path="/var/lib/kubelet/pods/7fae2258-d203-4719-a547-868d7b04070d/volumes" Oct 30 12:45:19.610008 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:19.609991 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aaccb644-e26d-4562-8cf1-bc344cd801e8" path="/var/lib/kubelet/pods/aaccb644-e26d-4562-8cf1-bc344cd801e8/volumes" Oct 30 12:45:19.610406 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:19.610386 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fcc90cb3-e12e-4b12-a50d-de740f552b40" path="/var/lib/kubelet/pods/fcc90cb3-e12e-4b12-a50d-de740f552b40/volumes" Oct 30 12:45:19.702059 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:45:19.702024 4908 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vn6vv" containerName="extractor" Oct 30 12:45:19.885268 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:19.885234 4908 generic.go:356] "Generic (PLEG): container finished" podID="958d7012-017f-4cd4-a637-a44ec84babc6" containerID="ce06be9927a31d27c7fb0c45e90592c4c2b0985479cce1b21f4d15cd061c8bb3" exitCode=60 Oct 30 12:45:19.885772 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:19.885311 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0bda67-29889405-q74jd" event={"ID":"958d7012-017f-4cd4-a637-a44ec84babc6","Type":"ContainerDied","Data":"ce06be9927a31d27c7fb0c45e90592c4c2b0985479cce1b21f4d15cd061c8bb3"} Oct 30 12:45:19.885772 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:19.885389 4908 scope.go:122] "RemoveContainer" containerID="3cd62c631dd03cd2b4a63e38adba127303c8eb05ddfc0a92a00b2069f8df1022" Oct 30 12:45:19.885941 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:19.885865 4908 scope.go:122] "RemoveContainer" containerID="ce06be9927a31d27c7fb0c45e90592c4c2b0985479cce1b21f4d15cd061c8bb3" Oct 30 12:45:19.886245 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:45:19.886218 4908 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-0bda67-29889405-q74jd_odh-ai-gateway-infra(958d7012-017f-4cd4-a637-a44ec84babc6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0bda67-29889405-q74jd" podUID="958d7012-017f-4cd4-a637-a44ec84babc6" Oct 30 12:45:19.887410 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:19.887384 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-73bafc-29889405-l27f6" Oct 30 12:45:19.887806 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:19.887414 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-73bafc-29889405-l27f6" event={"ID":"1aa1b39f-f88b-4ddf-8ee1-cf567d9d480a","Type":"ContainerDied","Data":"2fb626744864ae24c5462ec7c1e1eafa50c43793af2281b294aa092c8d360d6d"} Oct 30 12:45:19.893403 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:19.893378 4908 generic.go:356] "Generic (PLEG): container finished" podID="ccd6f000-a55d-4214-a830-faf2d200b688" containerID="1c6c4d604e413c8e41440f7a726fa9cbd054de04098fa9f8be8513163f17a9f8" exitCode=60 Oct 30 12:45:19.893506 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:19.893461 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2675c0-29889405-n4dv7" event={"ID":"ccd6f000-a55d-4214-a830-faf2d200b688","Type":"ContainerDied","Data":"1c6c4d604e413c8e41440f7a726fa9cbd054de04098fa9f8be8513163f17a9f8"} Oct 30 12:45:19.893965 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:19.893858 4908 scope.go:122] "RemoveContainer" containerID="1c6c4d604e413c8e41440f7a726fa9cbd054de04098fa9f8be8513163f17a9f8" Oct 30 12:45:19.894154 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:45:19.894130 4908 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-2675c0-29889405-n4dv7_odh-ai-gateway-infra(ccd6f000-a55d-4214-a830-faf2d200b688)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2675c0-29889405-n4dv7" podUID="ccd6f000-a55d-4214-a830-faf2d200b688" Oct 30 12:45:19.899819 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:19.899792 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-dd0628-29889405-cj8wq" event={"ID":"240c2309-8812-4591-9871-b16dcb9b6ff3","Type":"ContainerStarted","Data":"2993e9e9844db094f64fc24c584739e51be507cc6e33f86be0f49f1f4ca8937d"} Oct 30 12:45:19.903342 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:19.903311 4908 generic.go:356] "Generic (PLEG): container finished" podID="81dec304-04f6-4c40-8bf3-7fa3c4e0994d" containerID="8ea21b30baa0fcf3080805b940e57eb6e3c776c5050d4f257a4f75ca9aa84976" exitCode=60 Oct 30 12:45:19.903453 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:19.903379 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c1b1e8-29889405-5fkdv" event={"ID":"81dec304-04f6-4c40-8bf3-7fa3c4e0994d","Type":"ContainerDied","Data":"8ea21b30baa0fcf3080805b940e57eb6e3c776c5050d4f257a4f75ca9aa84976"} Oct 30 12:45:19.903739 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:19.903720 4908 scope.go:122] "RemoveContainer" containerID="8ea21b30baa0fcf3080805b940e57eb6e3c776c5050d4f257a4f75ca9aa84976" Oct 30 12:45:19.904086 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:45:19.904053 4908 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-c1b1e8-29889405-5fkdv_odh-ai-gateway-infra(81dec304-04f6-4c40-8bf3-7fa3c4e0994d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c1b1e8-29889405-5fkdv" podUID="81dec304-04f6-4c40-8bf3-7fa3c4e0994d" Oct 30 12:45:19.904173 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:19.904156 4908 scope.go:122] "RemoveContainer" containerID="caeef42f67821699eb55280dec7dfeb4ffbd3714cf3995dfbb422a4d79bfc3cb" Oct 30 12:45:19.909781 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:19.909757 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0f7d14-29889405-c7tnx" event={"ID":"4d181a49-62cd-454a-9e03-380198da3a08","Type":"ContainerDied","Data":"06beadb6ba5df908ed5c15f4ea75f97ef4160fe779a4e8bdc31857a1b84ebb3a"} Oct 30 12:45:19.909928 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:19.909901 4908 generic.go:356] "Generic (PLEG): container finished" podID="4d181a49-62cd-454a-9e03-380198da3a08" containerID="06beadb6ba5df908ed5c15f4ea75f97ef4160fe779a4e8bdc31857a1b84ebb3a" exitCode=60 Oct 30 12:45:19.910244 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:19.910222 4908 scope.go:122] "RemoveContainer" containerID="06beadb6ba5df908ed5c15f4ea75f97ef4160fe779a4e8bdc31857a1b84ebb3a" Oct 30 12:45:19.910614 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:45:19.910526 4908 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-0f7d14-29889405-c7tnx_odh-ai-gateway-infra(4d181a49-62cd-454a-9e03-380198da3a08)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0f7d14-29889405-c7tnx" podUID="4d181a49-62cd-454a-9e03-380198da3a08" Oct 30 12:45:19.912303 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:19.912283 4908 generic.go:356] "Generic (PLEG): container finished" podID="0b9ecbb7-1e58-4ac9-b913-ff46c0a02210" containerID="a563f3897d86ffd0f5f7da590f69431333ff2cc537696b8d43ccd40698809c16" exitCode=60 Oct 30 12:45:19.912405 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:19.912361 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2545c7-29889405-x55rn" event={"ID":"0b9ecbb7-1e58-4ac9-b913-ff46c0a02210","Type":"ContainerDied","Data":"a563f3897d86ffd0f5f7da590f69431333ff2cc537696b8d43ccd40698809c16"} Oct 30 12:45:19.912757 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:19.912738 4908 scope.go:122] "RemoveContainer" containerID="a563f3897d86ffd0f5f7da590f69431333ff2cc537696b8d43ccd40698809c16" Oct 30 12:45:19.913130 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:45:19.913063 4908 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-2545c7-29889405-x55rn_odh-ai-gateway-infra(0b9ecbb7-1e58-4ac9-b913-ff46c0a02210)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2545c7-29889405-x55rn" podUID="0b9ecbb7-1e58-4ac9-b913-ff46c0a02210" Oct 30 12:45:19.919672 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:19.919653 4908 scope.go:122] "RemoveContainer" containerID="d12a6301d143fec5dc8f58abcb4ddc4fd2c435b86527bb63f1337233d17f245e" Oct 30 12:45:19.922617 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:19.922596 4908 generic.go:356] "Generic (PLEG): container finished" podID="12c31795-7957-4dff-83c3-d4d2c1d31aac" containerID="ba59444e58c6b2529c1b830a77fb361c0d5716a949ea3be9772154b17bca0552" exitCode=60 Oct 30 12:45:19.922744 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:19.922669 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0de4b9-29889405-4pk7b" event={"ID":"12c31795-7957-4dff-83c3-d4d2c1d31aac","Type":"ContainerDied","Data":"ba59444e58c6b2529c1b830a77fb361c0d5716a949ea3be9772154b17bca0552"} Oct 30 12:45:19.923071 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:19.923052 4908 scope.go:122] "RemoveContainer" containerID="ba59444e58c6b2529c1b830a77fb361c0d5716a949ea3be9772154b17bca0552" Oct 30 12:45:19.923370 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:45:19.923349 4908 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-0de4b9-29889405-4pk7b_odh-ai-gateway-infra(12c31795-7957-4dff-83c3-d4d2c1d31aac)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0de4b9-29889405-4pk7b" podUID="12c31795-7957-4dff-83c3-d4d2c1d31aac" Oct 30 12:45:19.928776 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:19.928744 4908 generic.go:356] "Generic (PLEG): container finished" podID="27d77dc7-aee3-45a4-8ad9-a48560889798" containerID="4cb2a69c42c2726ea6a3c7a36a90923f43bf154857e7fe630094cbfe2acf65ff" exitCode=60 Oct 30 12:45:19.928897 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:19.928804 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-327e31-29889405-lr598" event={"ID":"27d77dc7-aee3-45a4-8ad9-a48560889798","Type":"ContainerDied","Data":"4cb2a69c42c2726ea6a3c7a36a90923f43bf154857e7fe630094cbfe2acf65ff"} Oct 30 12:45:19.929280 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:19.929256 4908 scope.go:122] "RemoveContainer" containerID="4cb2a69c42c2726ea6a3c7a36a90923f43bf154857e7fe630094cbfe2acf65ff" Oct 30 12:45:19.929640 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:45:19.929612 4908 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-327e31-29889405-lr598_odh-ai-gateway-infra(27d77dc7-aee3-45a4-8ad9-a48560889798)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-327e31-29889405-lr598" podUID="27d77dc7-aee3-45a4-8ad9-a48560889798" Oct 30 12:45:19.957856 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:19.954902 4908 scope.go:122] "RemoveContainer" containerID="28cfc981683a877d68c5d5c87b357dc4ee99e46a5677a9a40e3caf7a18f071ea" Oct 30 12:45:20.033748 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:20.033711 4908 scope.go:122] "RemoveContainer" containerID="1ecc2f2193c9794de7252a33ef2b637a0fb771ad739ee9dba3f7c6ac1bfcef69" Oct 30 12:45:20.064914 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:20.064884 4908 scope.go:122] "RemoveContainer" containerID="b57e0271d18afb9707f6556c49708c97241eedf42eed2fa54b1916215d67ed7f" Oct 30 12:45:20.090896 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:20.087977 4908 scope.go:122] "RemoveContainer" containerID="cb5f9257cacca16f6e1ed8a7bd8255de113bc93d89da2b25b6e3c466c9531bcb" Oct 30 12:45:20.114233 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:20.114067 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2675c0-29889405-n4dv7"] Oct 30 12:45:20.122014 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:20.120968 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-73bafc-29889405-l27f6"] Oct 30 12:45:20.145174 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:20.145096 4908 scope.go:122] "RemoveContainer" containerID="97b9267ff8f1831eb7fe9024663eb52367837e40ede0c7fac187246461049edd" Oct 30 12:45:20.153286 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:20.151848 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ecb3aa-29889405-srcpd" Oct 30 12:45:20.158923 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:20.156411 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-73bafc-29889405-l27f6"] Oct 30 12:45:20.178169 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:20.178131 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0bda67-29889405-q74jd"] Oct 30 12:45:20.191892 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:20.191799 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3a015cd-f16d-42b4-abae-9e2dd114093e-kube-api-access-55mhq\" (UniqueName: \"kubernetes.io/projected/a3a015cd-f16d-42b4-abae-9e2dd114093e-kube-api-access-55mhq\") pod \"a3a015cd-f16d-42b4-abae-9e2dd114093e\" (UID: \"a3a015cd-f16d-42b4-abae-9e2dd114093e\") " Oct 30 12:45:20.192605 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:20.192581 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3a015cd-f16d-42b4-abae-9e2dd114093e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3a015cd-f16d-42b4-abae-9e2dd114093e-maas-api-service-ca\") pod \"a3a015cd-f16d-42b4-abae-9e2dd114093e\" (UID: \"a3a015cd-f16d-42b4-abae-9e2dd114093e\") " Oct 30 12:45:20.192985 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:20.192955 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3a015cd-f16d-42b4-abae-9e2dd114093e-maas-api-service-ca" pod "a3a015cd-f16d-42b4-abae-9e2dd114093e" (UID: "a3a015cd-f16d-42b4-abae-9e2dd114093e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:45:20.193739 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:20.193710 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3a015cd-f16d-42b4-abae-9e2dd114093e-maas-api-service-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:45:20.196009 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:20.195981 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3a015cd-f16d-42b4-abae-9e2dd114093e-kube-api-access-55mhq" pod "a3a015cd-f16d-42b4-abae-9e2dd114093e" (UID: "a3a015cd-f16d-42b4-abae-9e2dd114093e"). InnerVolumeSpecName "kube-api-access-55mhq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:45:20.222213 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:20.222146 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-812990-29889405-stgg2" Oct 30 12:45:20.254544 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:20.254518 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-8d2170-29889405-82rzm" Oct 30 12:45:20.257796 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:20.257774 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f7464f-29889405-zzjr7" Oct 30 12:45:20.295228 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:20.295149 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e354caaa-fac9-485a-b960-7488fea0df0d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e354caaa-fac9-485a-b960-7488fea0df0d-maas-api-service-ca\") pod \"e354caaa-fac9-485a-b960-7488fea0df0d\" (UID: \"e354caaa-fac9-485a-b960-7488fea0df0d\") " Oct 30 12:45:20.295396 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:20.295251 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e354caaa-fac9-485a-b960-7488fea0df0d-kube-api-access-4lr9w\" (UniqueName: \"kubernetes.io/projected/e354caaa-fac9-485a-b960-7488fea0df0d-kube-api-access-4lr9w\") pod \"e354caaa-fac9-485a-b960-7488fea0df0d\" (UID: \"e354caaa-fac9-485a-b960-7488fea0df0d\") " Oct 30 12:45:20.295396 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:20.295290 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea9095cb-47a4-4ea4-b143-069f736c29b6-kube-api-access-2z45r\" (UniqueName: \"kubernetes.io/projected/ea9095cb-47a4-4ea4-b143-069f736c29b6-kube-api-access-2z45r\") pod \"ea9095cb-47a4-4ea4-b143-069f736c29b6\" (UID: \"ea9095cb-47a4-4ea4-b143-069f736c29b6\") " Oct 30 12:45:20.295396 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:20.295341 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab18f563-fbfd-4dbb-8b81-277c223a30f4-kube-api-access-nbdcr\" (UniqueName: \"kubernetes.io/projected/ab18f563-fbfd-4dbb-8b81-277c223a30f4-kube-api-access-nbdcr\") pod \"ab18f563-fbfd-4dbb-8b81-277c223a30f4\" (UID: \"ab18f563-fbfd-4dbb-8b81-277c223a30f4\") " Oct 30 12:45:20.295396 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:20.295380 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab18f563-fbfd-4dbb-8b81-277c223a30f4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab18f563-fbfd-4dbb-8b81-277c223a30f4-maas-api-service-ca\") pod \"ab18f563-fbfd-4dbb-8b81-277c223a30f4\" (UID: \"ab18f563-fbfd-4dbb-8b81-277c223a30f4\") " Oct 30 12:45:20.295625 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:20.295533 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea9095cb-47a4-4ea4-b143-069f736c29b6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea9095cb-47a4-4ea4-b143-069f736c29b6-maas-api-service-ca\") pod \"ea9095cb-47a4-4ea4-b143-069f736c29b6\" (UID: \"ea9095cb-47a4-4ea4-b143-069f736c29b6\") " Oct 30 12:45:20.295625 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:20.295588 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e354caaa-fac9-485a-b960-7488fea0df0d-maas-api-service-ca" pod "e354caaa-fac9-485a-b960-7488fea0df0d" (UID: "e354caaa-fac9-485a-b960-7488fea0df0d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:45:20.295998 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:20.295969 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea9095cb-47a4-4ea4-b143-069f736c29b6-maas-api-service-ca" pod "ea9095cb-47a4-4ea4-b143-069f736c29b6" (UID: "ea9095cb-47a4-4ea4-b143-069f736c29b6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:45:20.296090 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:20.296073 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea9095cb-47a4-4ea4-b143-069f736c29b6-maas-api-service-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:45:20.296153 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:20.296096 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e354caaa-fac9-485a-b960-7488fea0df0d-maas-api-service-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:45:20.296153 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:20.296110 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55mhq\" (UniqueName: \"kubernetes.io/projected/a3a015cd-f16d-42b4-abae-9e2dd114093e-kube-api-access-55mhq\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:45:20.296153 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:20.296138 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab18f563-fbfd-4dbb-8b81-277c223a30f4-maas-api-service-ca" pod "ab18f563-fbfd-4dbb-8b81-277c223a30f4" (UID: "ab18f563-fbfd-4dbb-8b81-277c223a30f4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:45:20.297791 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:20.297765 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e354caaa-fac9-485a-b960-7488fea0df0d-kube-api-access-4lr9w" pod "e354caaa-fac9-485a-b960-7488fea0df0d" (UID: "e354caaa-fac9-485a-b960-7488fea0df0d"). InnerVolumeSpecName "kube-api-access-4lr9w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:45:20.297928 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:20.297894 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea9095cb-47a4-4ea4-b143-069f736c29b6-kube-api-access-2z45r" pod "ea9095cb-47a4-4ea4-b143-069f736c29b6" (UID: "ea9095cb-47a4-4ea4-b143-069f736c29b6"). InnerVolumeSpecName "kube-api-access-2z45r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:45:20.298012 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:20.297995 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab18f563-fbfd-4dbb-8b81-277c223a30f4-kube-api-access-nbdcr" pod "ab18f563-fbfd-4dbb-8b81-277c223a30f4" (UID: "ab18f563-fbfd-4dbb-8b81-277c223a30f4"). InnerVolumeSpecName "kube-api-access-nbdcr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:45:20.338066 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:20.338031 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c1b1e8-29889405-5fkdv"] Oct 30 12:45:20.397624 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:20.397590 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4lr9w\" (UniqueName: \"kubernetes.io/projected/e354caaa-fac9-485a-b960-7488fea0df0d-kube-api-access-4lr9w\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:45:20.397624 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:20.397620 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2z45r\" (UniqueName: \"kubernetes.io/projected/ea9095cb-47a4-4ea4-b143-069f736c29b6-kube-api-access-2z45r\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:45:20.397624 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:20.397630 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nbdcr\" (UniqueName: \"kubernetes.io/projected/ab18f563-fbfd-4dbb-8b81-277c223a30f4-kube-api-access-nbdcr\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:45:20.397876 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:20.397643 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab18f563-fbfd-4dbb-8b81-277c223a30f4-maas-api-service-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:45:20.448449 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:20.448419 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2545c7-29889405-x55rn"] Oct 30 12:45:20.529976 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:20.529942 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0de4b9-29889405-4pk7b"] Oct 30 12:45:20.937841 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:20.937742 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-812990-29889405-stgg2" event={"ID":"e354caaa-fac9-485a-b960-7488fea0df0d","Type":"ContainerDied","Data":"509040ff6853c09788717a7bcace7c47a6c8298b5f82c443a8b874b9f7976039"} Oct 30 12:45:20.937841 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:20.937760 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-812990-29889405-stgg2" Oct 30 12:45:20.938281 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:20.937872 4908 scope.go:122] "RemoveContainer" containerID="a69cd4f91636e0b0ca9c46bd61743170f0ab51a258797840d778a6fa487c108b" Oct 30 12:45:20.939977 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:20.939939 4908 generic.go:356] "Generic (PLEG): container finished" podID="240c2309-8812-4591-9871-b16dcb9b6ff3" containerID="2993e9e9844db094f64fc24c584739e51be507cc6e33f86be0f49f1f4ca8937d" exitCode=60 Oct 30 12:45:20.940103 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:20.939980 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-dd0628-29889405-cj8wq" event={"ID":"240c2309-8812-4591-9871-b16dcb9b6ff3","Type":"ContainerDied","Data":"2993e9e9844db094f64fc24c584739e51be507cc6e33f86be0f49f1f4ca8937d"} Oct 30 12:45:20.940490 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:20.940455 4908 scope.go:122] "RemoveContainer" containerID="2993e9e9844db094f64fc24c584739e51be507cc6e33f86be0f49f1f4ca8937d" Oct 30 12:45:20.940810 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:45:20.940778 4908 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-dd0628-29889405-cj8wq_odh-ai-gateway-infra(240c2309-8812-4591-9871-b16dcb9b6ff3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-dd0628-29889405-cj8wq" podUID="240c2309-8812-4591-9871-b16dcb9b6ff3" Oct 30 12:45:20.951511 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:20.951484 4908 scope.go:122] "RemoveContainer" containerID="a38d646ff0f49f138829d6b4d6e9a32dc95422d184c175f842e78d1bedbe529f" Oct 30 12:45:20.952105 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:20.952080 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-8d2170-29889405-82rzm" event={"ID":"ea9095cb-47a4-4ea4-b143-069f736c29b6","Type":"ContainerDied","Data":"10ae2400903d389be55371923b49d31602eb80961c30fccaf97d2bcd69106c46"} Oct 30 12:45:20.952228 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:20.952112 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-8d2170-29889405-82rzm" Oct 30 12:45:20.956856 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:20.956181 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f7464f-29889405-zzjr7" Oct 30 12:45:20.956856 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:20.956739 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f7464f-29889405-zzjr7" event={"ID":"ab18f563-fbfd-4dbb-8b81-277c223a30f4","Type":"ContainerDied","Data":"935b800fa225e1b686f14ca8806c51a6893667808fc93285ff1e713b93360df8"} Oct 30 12:45:20.964439 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:20.964413 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ecb3aa-29889405-srcpd" event={"ID":"a3a015cd-f16d-42b4-abae-9e2dd114093e","Type":"ContainerDied","Data":"2d40f1812c1461813fc4609be9d3050bbe8d9ec6851bf6040eea2874428c703d"} Oct 30 12:45:20.964559 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:20.964524 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ecb3aa-29889405-srcpd" Oct 30 12:45:21.012618 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:21.006960 4908 scope.go:122] "RemoveContainer" containerID="686e7872d70672f7517cc90c9bdca39c8d842129a2fc80dced64c60d1255f160" Oct 30 12:45:21.029251 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:21.029108 4908 scope.go:122] "RemoveContainer" containerID="f927a23d07841776e2b829e96d7eb0e864a79bd0c7aef3e5736fca413595587d" Oct 30 12:45:21.049148 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:21.047953 4908 scope.go:122] "RemoveContainer" containerID="fd593bdf2d871963b56351aaac5a7059482f1aa1df835dc0c1b426ef8bed9523" Oct 30 12:45:21.125343 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:21.125313 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-812990-29889405-stgg2"] Oct 30 12:45:21.140245 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:21.140208 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-812990-29889405-stgg2"] Oct 30 12:45:21.225017 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:21.224977 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c1b1e8-29889405-5fkdv" Oct 30 12:45:21.303654 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:21.303610 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ecb3aa-29889405-srcpd"] Oct 30 12:45:21.351684 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:21.339689 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0de4b9-29889405-4pk7b" Oct 30 12:45:21.369363 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:21.369288 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ecb3aa-29889405-srcpd"] Oct 30 12:45:21.377038 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:21.376741 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-327e31-29889405-lr598"] Oct 30 12:45:21.418631 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:21.418604 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f7464f-29889405-zzjr7"] Oct 30 12:45:21.430224 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:21.430197 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f7464f-29889405-zzjr7"] Oct 30 12:45:21.433318 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:21.433285 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81dec304-04f6-4c40-8bf3-7fa3c4e0994d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81dec304-04f6-4c40-8bf3-7fa3c4e0994d-maas-api-service-ca\") pod \"81dec304-04f6-4c40-8bf3-7fa3c4e0994d\" (UID: \"81dec304-04f6-4c40-8bf3-7fa3c4e0994d\") " Oct 30 12:45:21.434188 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:21.433578 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81dec304-04f6-4c40-8bf3-7fa3c4e0994d-kube-api-access-x44n2\" (UniqueName: \"kubernetes.io/projected/81dec304-04f6-4c40-8bf3-7fa3c4e0994d-kube-api-access-x44n2\") pod \"81dec304-04f6-4c40-8bf3-7fa3c4e0994d\" (UID: \"81dec304-04f6-4c40-8bf3-7fa3c4e0994d\") " Oct 30 12:45:21.434188 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:21.433873 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81dec304-04f6-4c40-8bf3-7fa3c4e0994d-maas-api-service-ca" pod "81dec304-04f6-4c40-8bf3-7fa3c4e0994d" (UID: "81dec304-04f6-4c40-8bf3-7fa3c4e0994d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:45:21.434188 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:21.434128 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81dec304-04f6-4c40-8bf3-7fa3c4e0994d-maas-api-service-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:45:21.434524 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:21.434502 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2545c7-29889405-x55rn" Oct 30 12:45:21.436798 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:21.436769 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81dec304-04f6-4c40-8bf3-7fa3c4e0994d-kube-api-access-x44n2" pod "81dec304-04f6-4c40-8bf3-7fa3c4e0994d" (UID: "81dec304-04f6-4c40-8bf3-7fa3c4e0994d"). InnerVolumeSpecName "kube-api-access-x44n2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:45:21.443565 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:21.441861 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-dd0628-29889405-cj8wq"] Oct 30 12:45:21.446115 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:21.446093 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0bda67-29889405-q74jd" Oct 30 12:45:21.448189 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:21.448171 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2675c0-29889405-n4dv7" Oct 30 12:45:21.535360 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:21.535325 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b9ecbb7-1e58-4ac9-b913-ff46c0a02210-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b9ecbb7-1e58-4ac9-b913-ff46c0a02210-maas-api-service-ca\") pod \"0b9ecbb7-1e58-4ac9-b913-ff46c0a02210\" (UID: \"0b9ecbb7-1e58-4ac9-b913-ff46c0a02210\") " Oct 30 12:45:21.535527 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:21.535386 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b9ecbb7-1e58-4ac9-b913-ff46c0a02210-kube-api-access-w2jrz\" (UniqueName: \"kubernetes.io/projected/0b9ecbb7-1e58-4ac9-b913-ff46c0a02210-kube-api-access-w2jrz\") pod \"0b9ecbb7-1e58-4ac9-b913-ff46c0a02210\" (UID: \"0b9ecbb7-1e58-4ac9-b913-ff46c0a02210\") " Oct 30 12:45:21.535527 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:21.535455 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12c31795-7957-4dff-83c3-d4d2c1d31aac-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/12c31795-7957-4dff-83c3-d4d2c1d31aac-maas-api-service-ca\") pod \"12c31795-7957-4dff-83c3-d4d2c1d31aac\" (UID: \"12c31795-7957-4dff-83c3-d4d2c1d31aac\") " Oct 30 12:45:21.535691 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:21.535569 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12c31795-7957-4dff-83c3-d4d2c1d31aac-kube-api-access-66mc5\" (UniqueName: \"kubernetes.io/projected/12c31795-7957-4dff-83c3-d4d2c1d31aac-kube-api-access-66mc5\") pod \"12c31795-7957-4dff-83c3-d4d2c1d31aac\" (UID: \"12c31795-7957-4dff-83c3-d4d2c1d31aac\") " Oct 30 12:45:21.535844 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:21.535797 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b9ecbb7-1e58-4ac9-b913-ff46c0a02210-maas-api-service-ca" pod "0b9ecbb7-1e58-4ac9-b913-ff46c0a02210" (UID: "0b9ecbb7-1e58-4ac9-b913-ff46c0a02210"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:45:21.535955 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:21.535895 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12c31795-7957-4dff-83c3-d4d2c1d31aac-maas-api-service-ca" pod "12c31795-7957-4dff-83c3-d4d2c1d31aac" (UID: "12c31795-7957-4dff-83c3-d4d2c1d31aac"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:45:21.536380 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:21.536359 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b9ecbb7-1e58-4ac9-b913-ff46c0a02210-maas-api-service-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:45:21.536380 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:21.536379 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/12c31795-7957-4dff-83c3-d4d2c1d31aac-maas-api-service-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:45:21.536556 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:21.536388 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x44n2\" (UniqueName: \"kubernetes.io/projected/81dec304-04f6-4c40-8bf3-7fa3c4e0994d-kube-api-access-x44n2\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:45:21.537555 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:21.537530 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b9ecbb7-1e58-4ac9-b913-ff46c0a02210-kube-api-access-w2jrz" pod "0b9ecbb7-1e58-4ac9-b913-ff46c0a02210" (UID: "0b9ecbb7-1e58-4ac9-b913-ff46c0a02210"). InnerVolumeSpecName "kube-api-access-w2jrz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:45:21.537628 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:21.537604 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12c31795-7957-4dff-83c3-d4d2c1d31aac-kube-api-access-66mc5" pod "12c31795-7957-4dff-83c3-d4d2c1d31aac" (UID: "12c31795-7957-4dff-83c3-d4d2c1d31aac"). InnerVolumeSpecName "kube-api-access-66mc5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:45:21.553600 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:21.553576 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-327e31-29889405-lr598" Oct 30 12:45:21.569420 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:21.569397 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-8d2170-29889405-82rzm"] Oct 30 12:45:21.588313 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:21.588288 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-8d2170-29889405-82rzm"] Oct 30 12:45:21.610447 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:21.610423 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1aa1b39f-f88b-4ddf-8ee1-cf567d9d480a" path="/var/lib/kubelet/pods/1aa1b39f-f88b-4ddf-8ee1-cf567d9d480a/volumes" Oct 30 12:45:21.610915 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:21.610901 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3a015cd-f16d-42b4-abae-9e2dd114093e" path="/var/lib/kubelet/pods/a3a015cd-f16d-42b4-abae-9e2dd114093e/volumes" Oct 30 12:45:21.611297 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:21.611285 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab18f563-fbfd-4dbb-8b81-277c223a30f4" path="/var/lib/kubelet/pods/ab18f563-fbfd-4dbb-8b81-277c223a30f4/volumes" Oct 30 12:45:21.612001 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:21.611986 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e354caaa-fac9-485a-b960-7488fea0df0d" path="/var/lib/kubelet/pods/e354caaa-fac9-485a-b960-7488fea0df0d/volumes" Oct 30 12:45:21.612472 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:21.612454 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea9095cb-47a4-4ea4-b143-069f736c29b6" path="/var/lib/kubelet/pods/ea9095cb-47a4-4ea4-b143-069f736c29b6/volumes" Oct 30 12:45:21.636944 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:21.636920 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ccd6f000-a55d-4214-a830-faf2d200b688-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ccd6f000-a55d-4214-a830-faf2d200b688-maas-api-service-ca\") pod \"ccd6f000-a55d-4214-a830-faf2d200b688\" (UID: \"ccd6f000-a55d-4214-a830-faf2d200b688\") " Oct 30 12:45:21.637072 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:21.637057 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/958d7012-017f-4cd4-a637-a44ec84babc6-kube-api-access-mg5n2\" (UniqueName: \"kubernetes.io/projected/958d7012-017f-4cd4-a637-a44ec84babc6-kube-api-access-mg5n2\") pod \"958d7012-017f-4cd4-a637-a44ec84babc6\" (UID: \"958d7012-017f-4cd4-a637-a44ec84babc6\") " Oct 30 12:45:21.637224 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:21.637195 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/958d7012-017f-4cd4-a637-a44ec84babc6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/958d7012-017f-4cd4-a637-a44ec84babc6-maas-api-service-ca\") pod \"958d7012-017f-4cd4-a637-a44ec84babc6\" (UID: \"958d7012-017f-4cd4-a637-a44ec84babc6\") " Oct 30 12:45:21.637366 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:21.637251 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccd6f000-a55d-4214-a830-faf2d200b688-kube-api-access-brfbn\" (UniqueName: \"kubernetes.io/projected/ccd6f000-a55d-4214-a830-faf2d200b688-kube-api-access-brfbn\") pod \"ccd6f000-a55d-4214-a830-faf2d200b688\" (UID: \"ccd6f000-a55d-4214-a830-faf2d200b688\") " Oct 30 12:45:21.637366 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:21.637266 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccd6f000-a55d-4214-a830-faf2d200b688-maas-api-service-ca" pod "ccd6f000-a55d-4214-a830-faf2d200b688" (UID: "ccd6f000-a55d-4214-a830-faf2d200b688"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:45:21.637550 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:21.637525 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/958d7012-017f-4cd4-a637-a44ec84babc6-maas-api-service-ca" pod "958d7012-017f-4cd4-a637-a44ec84babc6" (UID: "958d7012-017f-4cd4-a637-a44ec84babc6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:45:21.637722 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:21.637703 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66mc5\" (UniqueName: \"kubernetes.io/projected/12c31795-7957-4dff-83c3-d4d2c1d31aac-kube-api-access-66mc5\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:45:21.637787 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:21.637726 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ccd6f000-a55d-4214-a830-faf2d200b688-maas-api-service-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:45:21.637787 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:21.637740 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2jrz\" (UniqueName: \"kubernetes.io/projected/0b9ecbb7-1e58-4ac9-b913-ff46c0a02210-kube-api-access-w2jrz\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:45:21.637787 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:21.637753 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/958d7012-017f-4cd4-a637-a44ec84babc6-maas-api-service-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:45:21.639644 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:21.639608 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccd6f000-a55d-4214-a830-faf2d200b688-kube-api-access-brfbn" pod "ccd6f000-a55d-4214-a830-faf2d200b688" (UID: "ccd6f000-a55d-4214-a830-faf2d200b688"). InnerVolumeSpecName "kube-api-access-brfbn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:45:21.639752 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:21.639726 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/958d7012-017f-4cd4-a637-a44ec84babc6-kube-api-access-mg5n2" pod "958d7012-017f-4cd4-a637-a44ec84babc6" (UID: "958d7012-017f-4cd4-a637-a44ec84babc6"). InnerVolumeSpecName "kube-api-access-mg5n2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:45:21.690518 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:21.690479 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0f7d14-29889405-c7tnx"] Oct 30 12:45:21.739231 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:21.739188 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27d77dc7-aee3-45a4-8ad9-a48560889798-kube-api-access-sjbp8\" (UniqueName: \"kubernetes.io/projected/27d77dc7-aee3-45a4-8ad9-a48560889798-kube-api-access-sjbp8\") pod \"27d77dc7-aee3-45a4-8ad9-a48560889798\" (UID: \"27d77dc7-aee3-45a4-8ad9-a48560889798\") " Oct 30 12:45:21.739501 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:21.739479 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27d77dc7-aee3-45a4-8ad9-a48560889798-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/27d77dc7-aee3-45a4-8ad9-a48560889798-maas-api-service-ca\") pod \"27d77dc7-aee3-45a4-8ad9-a48560889798\" (UID: \"27d77dc7-aee3-45a4-8ad9-a48560889798\") " Oct 30 12:45:21.740130 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:21.740107 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brfbn\" (UniqueName: \"kubernetes.io/projected/ccd6f000-a55d-4214-a830-faf2d200b688-kube-api-access-brfbn\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:45:21.740228 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:21.740135 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mg5n2\" (UniqueName: \"kubernetes.io/projected/958d7012-017f-4cd4-a637-a44ec84babc6-kube-api-access-mg5n2\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:45:21.741306 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:21.741280 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27d77dc7-aee3-45a4-8ad9-a48560889798-maas-api-service-ca" pod "27d77dc7-aee3-45a4-8ad9-a48560889798" (UID: "27d77dc7-aee3-45a4-8ad9-a48560889798"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:45:21.745304 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:21.745285 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27d77dc7-aee3-45a4-8ad9-a48560889798-kube-api-access-sjbp8" pod "27d77dc7-aee3-45a4-8ad9-a48560889798" (UID: "27d77dc7-aee3-45a4-8ad9-a48560889798"). InnerVolumeSpecName "kube-api-access-sjbp8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:45:21.841846 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:21.841800 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/27d77dc7-aee3-45a4-8ad9-a48560889798-maas-api-service-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:45:21.841846 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:21.841849 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjbp8\" (UniqueName: \"kubernetes.io/projected/27d77dc7-aee3-45a4-8ad9-a48560889798-kube-api-access-sjbp8\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:45:21.921278 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:21.921249 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0f7d14-29889405-c7tnx" Oct 30 12:45:21.943079 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:21.943042 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d181a49-62cd-454a-9e03-380198da3a08-kube-api-access-tllf9\" (UniqueName: \"kubernetes.io/projected/4d181a49-62cd-454a-9e03-380198da3a08-kube-api-access-tllf9\") pod \"4d181a49-62cd-454a-9e03-380198da3a08\" (UID: \"4d181a49-62cd-454a-9e03-380198da3a08\") " Oct 30 12:45:21.945981 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:21.945941 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d181a49-62cd-454a-9e03-380198da3a08-kube-api-access-tllf9" pod "4d181a49-62cd-454a-9e03-380198da3a08" (UID: "4d181a49-62cd-454a-9e03-380198da3a08"). InnerVolumeSpecName "kube-api-access-tllf9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:45:21.979912 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:21.979868 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0bda67-29889405-q74jd" event={"ID":"958d7012-017f-4cd4-a637-a44ec84babc6","Type":"ContainerDied","Data":"db31531d80b98920ca82ab5894044210627b48e06a7442ee4cfb8a5b2aed0000"} Oct 30 12:45:21.980087 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:21.979980 4908 scope.go:122] "RemoveContainer" containerID="ce06be9927a31d27c7fb0c45e90592c4c2b0985479cce1b21f4d15cd061c8bb3" Oct 30 12:45:21.980148 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:21.980128 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0bda67-29889405-q74jd" Oct 30 12:45:21.985993 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:21.985959 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2675c0-29889405-n4dv7" event={"ID":"ccd6f000-a55d-4214-a830-faf2d200b688","Type":"ContainerDied","Data":"a6f05fa1bc7093a9c685192649e3969c40987fd095b36d7cbe3345fba01cfdeb"} Oct 30 12:45:21.986286 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:21.986266 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2675c0-29889405-n4dv7" Oct 30 12:45:22.008753 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:22.008723 4908 scope.go:122] "RemoveContainer" containerID="1c6c4d604e413c8e41440f7a726fa9cbd054de04098fa9f8be8513163f17a9f8" Oct 30 12:45:22.010216 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:22.010182 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c1b1e8-29889405-5fkdv" event={"ID":"81dec304-04f6-4c40-8bf3-7fa3c4e0994d","Type":"ContainerDied","Data":"09a83966c96f5083ef306912f7e20c1358c3d10b12847efdd8585d1e137f2993"} Oct 30 12:45:22.010327 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:22.010224 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c1b1e8-29889405-5fkdv" Oct 30 12:45:22.015740 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:22.015333 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0f7d14-29889405-c7tnx" Oct 30 12:45:22.015931 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:22.015743 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0f7d14-29889405-c7tnx" event={"ID":"4d181a49-62cd-454a-9e03-380198da3a08","Type":"ContainerDied","Data":"bd4ce623329eb5574d52d7bc9e3ad8d1e824a4685cc1631986aa3bd1b893b1d4"} Oct 30 12:45:22.017987 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:22.017635 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2545c7-29889405-x55rn" event={"ID":"0b9ecbb7-1e58-4ac9-b913-ff46c0a02210","Type":"ContainerDied","Data":"462412b4821f160bbcc534f284980c28b544f41249feed7768d6a0dd92b5f62a"} Oct 30 12:45:22.017987 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:22.017643 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2545c7-29889405-x55rn" Oct 30 12:45:22.023663 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:22.022875 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0de4b9-29889405-4pk7b" event={"ID":"12c31795-7957-4dff-83c3-d4d2c1d31aac","Type":"ContainerDied","Data":"6597ee431c0dca5339e454ce4c5a183e927240fc03846aaba3a3713bd658c5d9"} Oct 30 12:45:22.023663 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:22.023017 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0de4b9-29889405-4pk7b" Oct 30 12:45:22.027188 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:22.027136 4908 scope.go:122] "RemoveContainer" containerID="8ea21b30baa0fcf3080805b940e57eb6e3c776c5050d4f257a4f75ca9aa84976" Oct 30 12:45:22.027516 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:22.027487 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-327e31-29889405-lr598" event={"ID":"27d77dc7-aee3-45a4-8ad9-a48560889798","Type":"ContainerDied","Data":"29c950ef5f0ad024e67aeb49d0de82f1bc142eeb4120cd65e01acf19657b9d63"} Oct 30 12:45:22.027740 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:22.027722 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-327e31-29889405-lr598" Oct 30 12:45:22.044575 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:22.044540 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d181a49-62cd-454a-9e03-380198da3a08-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d181a49-62cd-454a-9e03-380198da3a08-maas-api-service-ca\") pod \"4d181a49-62cd-454a-9e03-380198da3a08\" (UID: \"4d181a49-62cd-454a-9e03-380198da3a08\") " Oct 30 12:45:22.045151 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:22.045128 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tllf9\" (UniqueName: \"kubernetes.io/projected/4d181a49-62cd-454a-9e03-380198da3a08-kube-api-access-tllf9\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:45:22.045506 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:22.045482 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d181a49-62cd-454a-9e03-380198da3a08-maas-api-service-ca" pod "4d181a49-62cd-454a-9e03-380198da3a08" (UID: "4d181a49-62cd-454a-9e03-380198da3a08"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:45:22.047468 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:22.047450 4908 scope.go:122] "RemoveContainer" containerID="06beadb6ba5df908ed5c15f4ea75f97ef4160fe779a4e8bdc31857a1b84ebb3a" Oct 30 12:45:22.058464 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:22.058443 4908 scope.go:122] "RemoveContainer" containerID="a563f3897d86ffd0f5f7da590f69431333ff2cc537696b8d43ccd40698809c16" Oct 30 12:45:22.075526 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:22.075494 4908 scope.go:122] "RemoveContainer" containerID="ba59444e58c6b2529c1b830a77fb361c0d5716a949ea3be9772154b17bca0552" Oct 30 12:45:22.091628 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:22.091606 4908 scope.go:122] "RemoveContainer" containerID="4cb2a69c42c2726ea6a3c7a36a90923f43bf154857e7fe630094cbfe2acf65ff" Oct 30 12:45:22.111924 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:22.111893 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0bda67-29889405-q74jd"] Oct 30 12:45:22.114509 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:22.114483 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0bda67-29889405-q74jd"] Oct 30 12:45:22.146368 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:22.146330 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d181a49-62cd-454a-9e03-380198da3a08-maas-api-service-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:45:22.154124 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:22.154092 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-327e31-29889405-lr598"] Oct 30 12:45:22.159454 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:22.159429 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-327e31-29889405-lr598"] Oct 30 12:45:22.184264 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:22.184237 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-dd0628-29889405-cj8wq" Oct 30 12:45:22.198613 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:22.198586 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2675c0-29889405-n4dv7"] Oct 30 12:45:22.207597 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:22.207570 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2675c0-29889405-n4dv7"] Oct 30 12:45:22.247778 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:22.247679 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/240c2309-8812-4591-9871-b16dcb9b6ff3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/240c2309-8812-4591-9871-b16dcb9b6ff3-maas-api-service-ca\") pod \"240c2309-8812-4591-9871-b16dcb9b6ff3\" (UID: \"240c2309-8812-4591-9871-b16dcb9b6ff3\") " Oct 30 12:45:22.247998 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:22.247789 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/240c2309-8812-4591-9871-b16dcb9b6ff3-kube-api-access-z6j5w\" (UniqueName: \"kubernetes.io/projected/240c2309-8812-4591-9871-b16dcb9b6ff3-kube-api-access-z6j5w\") pod \"240c2309-8812-4591-9871-b16dcb9b6ff3\" (UID: \"240c2309-8812-4591-9871-b16dcb9b6ff3\") " Oct 30 12:45:22.248256 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:22.248227 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/240c2309-8812-4591-9871-b16dcb9b6ff3-maas-api-service-ca" pod "240c2309-8812-4591-9871-b16dcb9b6ff3" (UID: "240c2309-8812-4591-9871-b16dcb9b6ff3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:45:22.248398 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:22.248348 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/240c2309-8812-4591-9871-b16dcb9b6ff3-maas-api-service-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:45:22.250077 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:22.250055 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/240c2309-8812-4591-9871-b16dcb9b6ff3-kube-api-access-z6j5w" pod "240c2309-8812-4591-9871-b16dcb9b6ff3" (UID: "240c2309-8812-4591-9871-b16dcb9b6ff3"). InnerVolumeSpecName "kube-api-access-z6j5w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:45:22.286796 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:22.286760 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c1b1e8-29889405-5fkdv"] Oct 30 12:45:22.298909 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:22.298875 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c1b1e8-29889405-5fkdv"] Oct 30 12:45:22.349458 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:22.349427 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6j5w\" (UniqueName: \"kubernetes.io/projected/240c2309-8812-4591-9871-b16dcb9b6ff3-kube-api-access-z6j5w\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:45:22.353054 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:22.353025 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2545c7-29889405-x55rn"] Oct 30 12:45:22.370122 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:22.370097 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2545c7-29889405-x55rn"] Oct 30 12:45:22.414483 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:22.414457 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0de4b9-29889405-4pk7b"] Oct 30 12:45:22.422247 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:22.422225 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0de4b9-29889405-4pk7b"] Oct 30 12:45:22.509564 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:22.509477 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0f7d14-29889405-c7tnx"] Oct 30 12:45:22.515947 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:22.515915 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0f7d14-29889405-c7tnx"] Oct 30 12:45:23.037646 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:23.037618 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-dd0628-29889405-cj8wq" Oct 30 12:45:23.037646 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:23.037633 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-dd0628-29889405-cj8wq" event={"ID":"240c2309-8812-4591-9871-b16dcb9b6ff3","Type":"ContainerDied","Data":"59ab3c41971091f710d7032586bed62e7f33798adafea82937ab31e7e44de78a"} Oct 30 12:45:23.038186 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:23.037698 4908 scope.go:122] "RemoveContainer" containerID="2993e9e9844db094f64fc24c584739e51be507cc6e33f86be0f49f1f4ca8937d" Oct 30 12:45:23.100448 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:23.094859 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-dd0628-29889405-cj8wq"] Oct 30 12:45:23.108241 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:23.108210 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-dd0628-29889405-cj8wq"] Oct 30 12:45:23.609270 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:23.609231 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b9ecbb7-1e58-4ac9-b913-ff46c0a02210" path="/var/lib/kubelet/pods/0b9ecbb7-1e58-4ac9-b913-ff46c0a02210/volumes" Oct 30 12:45:23.609691 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:23.609676 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12c31795-7957-4dff-83c3-d4d2c1d31aac" path="/var/lib/kubelet/pods/12c31795-7957-4dff-83c3-d4d2c1d31aac/volumes" Oct 30 12:45:23.610124 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:23.610108 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="240c2309-8812-4591-9871-b16dcb9b6ff3" path="/var/lib/kubelet/pods/240c2309-8812-4591-9871-b16dcb9b6ff3/volumes" Oct 30 12:45:23.610790 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:23.610773 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27d77dc7-aee3-45a4-8ad9-a48560889798" path="/var/lib/kubelet/pods/27d77dc7-aee3-45a4-8ad9-a48560889798/volumes" Oct 30 12:45:23.611204 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:23.611190 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d181a49-62cd-454a-9e03-380198da3a08" path="/var/lib/kubelet/pods/4d181a49-62cd-454a-9e03-380198da3a08/volumes" Oct 30 12:45:23.611687 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:23.611670 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81dec304-04f6-4c40-8bf3-7fa3c4e0994d" path="/var/lib/kubelet/pods/81dec304-04f6-4c40-8bf3-7fa3c4e0994d/volumes" Oct 30 12:45:23.612265 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:23.612248 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="958d7012-017f-4cd4-a637-a44ec84babc6" path="/var/lib/kubelet/pods/958d7012-017f-4cd4-a637-a44ec84babc6/volumes" Oct 30 12:45:23.612635 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:23.612622 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ccd6f000-a55d-4214-a830-faf2d200b688" path="/var/lib/kubelet/pods/ccd6f000-a55d-4214-a830-faf2d200b688/volumes" Oct 30 12:45:28.040684 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:28.040651 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42836380-lr6fl"] Oct 30 12:45:28.051448 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:28.051411 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42836380-lr6fl"] Oct 30 12:45:28.730346 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:28.730308 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44f58a36-r5vts"] Oct 30 12:45:28.731114 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:28.731092 4908 cpu_manager.go:414] "containerMap: removing container" podUID="81dec304-04f6-4c40-8bf3-7fa3c4e0994d" containerName="cleanup" Oct 30 12:45:28.731114 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:28.731114 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81dec304-04f6-4c40-8bf3-7fa3c4e0994d" containerName="cleanup" Oct 30 12:45:28.731305 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:28.731129 4908 cpu_manager.go:414] "containerMap: removing container" podUID="958d7012-017f-4cd4-a637-a44ec84babc6" containerName="cleanup" Oct 30 12:45:28.731305 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:28.731159 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="958d7012-017f-4cd4-a637-a44ec84babc6" containerName="cleanup" Oct 30 12:45:28.731305 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:28.731168 4908 cpu_manager.go:414] "containerMap: removing container" podUID="fcc90cb3-e12e-4b12-a50d-de740f552b40" containerName="cleanup" Oct 30 12:45:28.731305 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:28.731174 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fcc90cb3-e12e-4b12-a50d-de740f552b40" containerName="cleanup" Oct 30 12:45:28.731305 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:28.731190 4908 cpu_manager.go:414] "containerMap: removing container" podUID="240c2309-8812-4591-9871-b16dcb9b6ff3" containerName="cleanup" Oct 30 12:45:28.731305 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:28.731195 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="240c2309-8812-4591-9871-b16dcb9b6ff3" containerName="cleanup" Oct 30 12:45:28.731305 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:28.731227 4908 cpu_manager.go:414] "containerMap: removing container" podUID="e354caaa-fac9-485a-b960-7488fea0df0d" containerName="cleanup" Oct 30 12:45:28.731305 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:28.731232 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e354caaa-fac9-485a-b960-7488fea0df0d" containerName="cleanup" Oct 30 12:45:28.731305 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:28.731240 4908 cpu_manager.go:414] "containerMap: removing container" podUID="a3a015cd-f16d-42b4-abae-9e2dd114093e" containerName="cleanup" Oct 30 12:45:28.731305 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:28.731245 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3a015cd-f16d-42b4-abae-9e2dd114093e" containerName="cleanup" Oct 30 12:45:28.731305 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:28.731257 4908 cpu_manager.go:414] "containerMap: removing container" podUID="3eef63f0-3b6c-4e62-bdc9-594ee9d4bd17" containerName="cleanup" Oct 30 12:45:28.731305 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:28.731266 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3eef63f0-3b6c-4e62-bdc9-594ee9d4bd17" containerName="cleanup" Oct 30 12:45:28.731305 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:28.731277 4908 cpu_manager.go:414] "containerMap: removing container" podUID="1aa1b39f-f88b-4ddf-8ee1-cf567d9d480a" containerName="cleanup" Oct 30 12:45:28.731305 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:28.731283 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1aa1b39f-f88b-4ddf-8ee1-cf567d9d480a" containerName="cleanup" Oct 30 12:45:28.731305 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:28.731290 4908 cpu_manager.go:414] "containerMap: removing container" podUID="1aa1b39f-f88b-4ddf-8ee1-cf567d9d480a" containerName="cleanup" Oct 30 12:45:28.731305 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:28.731297 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1aa1b39f-f88b-4ddf-8ee1-cf567d9d480a" containerName="cleanup" Oct 30 12:45:28.731305 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:28.731306 4908 cpu_manager.go:414] "containerMap: removing container" podUID="12c31795-7957-4dff-83c3-d4d2c1d31aac" containerName="cleanup" Oct 30 12:45:28.731305 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:28.731313 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12c31795-7957-4dff-83c3-d4d2c1d31aac" containerName="cleanup" Oct 30 12:45:28.732004 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:28.731319 4908 cpu_manager.go:414] "containerMap: removing container" podUID="ab18f563-fbfd-4dbb-8b81-277c223a30f4" containerName="cleanup" Oct 30 12:45:28.732004 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:28.731323 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab18f563-fbfd-4dbb-8b81-277c223a30f4" containerName="cleanup" Oct 30 12:45:28.732004 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:28.731329 4908 cpu_manager.go:414] "containerMap: removing container" podUID="a3a015cd-f16d-42b4-abae-9e2dd114093e" containerName="cleanup" Oct 30 12:45:28.732004 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:28.731336 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3a015cd-f16d-42b4-abae-9e2dd114093e" containerName="cleanup" Oct 30 12:45:28.732004 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:28.731348 4908 cpu_manager.go:414] "containerMap: removing container" podUID="27d77dc7-aee3-45a4-8ad9-a48560889798" containerName="cleanup" Oct 30 12:45:28.732004 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:28.731355 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27d77dc7-aee3-45a4-8ad9-a48560889798" containerName="cleanup" Oct 30 12:45:28.732004 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:28.731363 4908 cpu_manager.go:414] "containerMap: removing container" podUID="ea9095cb-47a4-4ea4-b143-069f736c29b6" containerName="cleanup" Oct 30 12:45:28.732004 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:28.731368 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea9095cb-47a4-4ea4-b143-069f736c29b6" containerName="cleanup" Oct 30 12:45:28.732004 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:28.731383 4908 cpu_manager.go:414] "containerMap: removing container" podUID="ccd6f000-a55d-4214-a830-faf2d200b688" containerName="cleanup" Oct 30 12:45:28.732004 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:28.731387 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccd6f000-a55d-4214-a830-faf2d200b688" containerName="cleanup" Oct 30 12:45:28.732004 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:28.731397 4908 cpu_manager.go:414] "containerMap: removing container" podUID="958d7012-017f-4cd4-a637-a44ec84babc6" containerName="cleanup" Oct 30 12:45:28.732004 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:28.731404 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="958d7012-017f-4cd4-a637-a44ec84babc6" containerName="cleanup" Oct 30 12:45:28.732004 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:28.731413 4908 cpu_manager.go:414] "containerMap: removing container" podUID="27d77dc7-aee3-45a4-8ad9-a48560889798" containerName="cleanup" Oct 30 12:45:28.732004 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:28.731418 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27d77dc7-aee3-45a4-8ad9-a48560889798" containerName="cleanup" Oct 30 12:45:28.732004 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:28.731432 4908 cpu_manager.go:414] "containerMap: removing container" podUID="e354caaa-fac9-485a-b960-7488fea0df0d" containerName="cleanup" Oct 30 12:45:28.732004 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:28.731440 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e354caaa-fac9-485a-b960-7488fea0df0d" containerName="cleanup" Oct 30 12:45:28.732004 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:28.731450 4908 cpu_manager.go:414] "containerMap: removing container" podUID="7fae2258-d203-4719-a547-868d7b04070d" containerName="cleanup" Oct 30 12:45:28.732004 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:28.731454 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7fae2258-d203-4719-a547-868d7b04070d" containerName="cleanup" Oct 30 12:45:28.732004 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:28.731462 4908 cpu_manager.go:414] "containerMap: removing container" podUID="e354caaa-fac9-485a-b960-7488fea0df0d" containerName="cleanup" Oct 30 12:45:28.732004 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:28.731466 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e354caaa-fac9-485a-b960-7488fea0df0d" containerName="cleanup" Oct 30 12:45:28.732004 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:28.731472 4908 cpu_manager.go:414] "containerMap: removing container" podUID="4d181a49-62cd-454a-9e03-380198da3a08" containerName="cleanup" Oct 30 12:45:28.732004 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:28.731476 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d181a49-62cd-454a-9e03-380198da3a08" containerName="cleanup" Oct 30 12:45:28.732004 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:28.731483 4908 cpu_manager.go:414] "containerMap: removing container" podUID="0b9ecbb7-1e58-4ac9-b913-ff46c0a02210" containerName="cleanup" Oct 30 12:45:28.732004 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:28.731487 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b9ecbb7-1e58-4ac9-b913-ff46c0a02210" containerName="cleanup" Oct 30 12:45:28.732004 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:28.731497 4908 cpu_manager.go:414] "containerMap: removing container" podUID="ea9095cb-47a4-4ea4-b143-069f736c29b6" containerName="cleanup" Oct 30 12:45:28.732004 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:28.731511 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea9095cb-47a4-4ea4-b143-069f736c29b6" containerName="cleanup" Oct 30 12:45:28.732004 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:28.731523 4908 cpu_manager.go:414] "containerMap: removing container" podUID="ccd6f000-a55d-4214-a830-faf2d200b688" containerName="cleanup" Oct 30 12:45:28.732004 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:28.731531 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccd6f000-a55d-4214-a830-faf2d200b688" containerName="cleanup" Oct 30 12:45:28.732004 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:28.731542 4908 cpu_manager.go:414] "containerMap: removing container" podUID="a3a015cd-f16d-42b4-abae-9e2dd114093e" containerName="cleanup" Oct 30 12:45:28.732004 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:28.731549 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3a015cd-f16d-42b4-abae-9e2dd114093e" containerName="cleanup" Oct 30 12:45:28.732004 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:28.731559 4908 cpu_manager.go:414] "containerMap: removing container" podUID="ea9095cb-47a4-4ea4-b143-069f736c29b6" containerName="cleanup" Oct 30 12:45:28.732004 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:28.731566 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea9095cb-47a4-4ea4-b143-069f736c29b6" containerName="cleanup" Oct 30 12:45:28.732004 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:28.731575 4908 cpu_manager.go:414] "containerMap: removing container" podUID="3eef63f0-3b6c-4e62-bdc9-594ee9d4bd17" containerName="cleanup" Oct 30 12:45:28.732004 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:28.731580 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3eef63f0-3b6c-4e62-bdc9-594ee9d4bd17" containerName="cleanup" Oct 30 12:45:28.732004 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:28.731585 4908 cpu_manager.go:414] "containerMap: removing container" podUID="ab18f563-fbfd-4dbb-8b81-277c223a30f4" containerName="cleanup" Oct 30 12:45:28.732004 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:28.731589 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab18f563-fbfd-4dbb-8b81-277c223a30f4" containerName="cleanup" Oct 30 12:45:28.732004 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:28.731595 4908 cpu_manager.go:414] "containerMap: removing container" podUID="7fae2258-d203-4719-a547-868d7b04070d" containerName="cleanup" Oct 30 12:45:28.732004 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:28.731601 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7fae2258-d203-4719-a547-868d7b04070d" containerName="cleanup" Oct 30 12:45:28.732004 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:28.731612 4908 cpu_manager.go:414] "containerMap: removing container" podUID="4d181a49-62cd-454a-9e03-380198da3a08" containerName="cleanup" Oct 30 12:45:28.732004 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:28.731617 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d181a49-62cd-454a-9e03-380198da3a08" containerName="cleanup" Oct 30 12:45:28.732004 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:28.731631 4908 cpu_manager.go:414] "containerMap: removing container" podUID="7fae2258-d203-4719-a547-868d7b04070d" containerName="cleanup" Oct 30 12:45:28.732004 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:28.731637 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7fae2258-d203-4719-a547-868d7b04070d" containerName="cleanup" Oct 30 12:45:28.732004 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:28.731644 4908 cpu_manager.go:414] "containerMap: removing container" podUID="0b9ecbb7-1e58-4ac9-b913-ff46c0a02210" containerName="cleanup" Oct 30 12:45:28.732004 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:28.731648 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b9ecbb7-1e58-4ac9-b913-ff46c0a02210" containerName="cleanup" Oct 30 12:45:28.732004 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:28.731658 4908 cpu_manager.go:414] "containerMap: removing container" podUID="12c31795-7957-4dff-83c3-d4d2c1d31aac" containerName="cleanup" Oct 30 12:45:28.732004 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:28.731663 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12c31795-7957-4dff-83c3-d4d2c1d31aac" containerName="cleanup" Oct 30 12:45:28.732004 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:28.731669 4908 cpu_manager.go:414] "containerMap: removing container" podUID="240c2309-8812-4591-9871-b16dcb9b6ff3" containerName="cleanup" Oct 30 12:45:28.732004 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:28.731676 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="240c2309-8812-4591-9871-b16dcb9b6ff3" containerName="cleanup" Oct 30 12:45:28.732004 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:28.731684 4908 cpu_manager.go:414] "containerMap: removing container" podUID="ab18f563-fbfd-4dbb-8b81-277c223a30f4" containerName="cleanup" Oct 30 12:45:28.732004 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:28.731691 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab18f563-fbfd-4dbb-8b81-277c223a30f4" containerName="cleanup" Oct 30 12:45:28.732004 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:28.731702 4908 cpu_manager.go:414] "containerMap: removing container" podUID="ccd6f000-a55d-4214-a830-faf2d200b688" containerName="cleanup" Oct 30 12:45:28.732004 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:28.731709 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccd6f000-a55d-4214-a830-faf2d200b688" containerName="cleanup" Oct 30 12:45:28.732004 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:28.731719 4908 cpu_manager.go:414] "containerMap: removing container" podUID="fcc90cb3-e12e-4b12-a50d-de740f552b40" containerName="cleanup" Oct 30 12:45:28.732004 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:28.731727 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fcc90cb3-e12e-4b12-a50d-de740f552b40" containerName="cleanup" Oct 30 12:45:28.732004 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:28.731737 4908 cpu_manager.go:414] "containerMap: removing container" podUID="81dec304-04f6-4c40-8bf3-7fa3c4e0994d" containerName="cleanup" Oct 30 12:45:28.732004 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:28.731742 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81dec304-04f6-4c40-8bf3-7fa3c4e0994d" containerName="cleanup" Oct 30 12:45:28.732004 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:28.731942 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="e354caaa-fac9-485a-b960-7488fea0df0d" containerName="cleanup" Oct 30 12:45:28.732004 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:28.731956 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3a015cd-f16d-42b4-abae-9e2dd114093e" containerName="cleanup" Oct 30 12:45:28.732004 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:28.731968 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="3eef63f0-3b6c-4e62-bdc9-594ee9d4bd17" containerName="cleanup" Oct 30 12:45:28.732004 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:28.731978 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="1aa1b39f-f88b-4ddf-8ee1-cf567d9d480a" containerName="cleanup" Oct 30 12:45:28.732004 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:28.731985 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="1aa1b39f-f88b-4ddf-8ee1-cf567d9d480a" containerName="cleanup" Oct 30 12:45:28.732004 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:28.731990 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="12c31795-7957-4dff-83c3-d4d2c1d31aac" containerName="cleanup" Oct 30 12:45:28.732004 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:28.731997 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab18f563-fbfd-4dbb-8b81-277c223a30f4" containerName="cleanup" Oct 30 12:45:28.732004 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:28.732002 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3a015cd-f16d-42b4-abae-9e2dd114093e" containerName="cleanup" Oct 30 12:45:28.732004 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:28.732007 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="27d77dc7-aee3-45a4-8ad9-a48560889798" containerName="cleanup" Oct 30 12:45:28.732004 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:28.732013 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea9095cb-47a4-4ea4-b143-069f736c29b6" containerName="cleanup" Oct 30 12:45:28.732004 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:28.732020 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="ccd6f000-a55d-4214-a830-faf2d200b688" containerName="cleanup" Oct 30 12:45:28.732004 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:28.732027 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="958d7012-017f-4cd4-a637-a44ec84babc6" containerName="cleanup" Oct 30 12:45:28.732004 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:28.732033 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="fcc90cb3-e12e-4b12-a50d-de740f552b40" containerName="cleanup" Oct 30 12:45:28.732004 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:28.732038 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="27d77dc7-aee3-45a4-8ad9-a48560889798" containerName="cleanup" Oct 30 12:45:28.732004 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:28.732043 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="240c2309-8812-4591-9871-b16dcb9b6ff3" containerName="cleanup" Oct 30 12:45:28.732004 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:28.732048 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="27d77dc7-aee3-45a4-8ad9-a48560889798" containerName="cleanup" Oct 30 12:45:28.732004 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:28.732054 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="3eef63f0-3b6c-4e62-bdc9-594ee9d4bd17" containerName="cleanup" Oct 30 12:45:28.734565 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:28.732061 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="1aa1b39f-f88b-4ddf-8ee1-cf567d9d480a" containerName="cleanup" Oct 30 12:45:28.734565 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:28.732067 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="e354caaa-fac9-485a-b960-7488fea0df0d" containerName="cleanup" Oct 30 12:45:28.734565 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:28.732081 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="12c31795-7957-4dff-83c3-d4d2c1d31aac" containerName="cleanup" Oct 30 12:45:28.734565 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:28.732091 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="7fae2258-d203-4719-a547-868d7b04070d" containerName="cleanup" Oct 30 12:45:28.734565 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:28.732100 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="e354caaa-fac9-485a-b960-7488fea0df0d" containerName="cleanup" Oct 30 12:45:28.734565 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:28.732107 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="81dec304-04f6-4c40-8bf3-7fa3c4e0994d" containerName="cleanup" Oct 30 12:45:28.734565 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:28.732112 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d181a49-62cd-454a-9e03-380198da3a08" containerName="cleanup" Oct 30 12:45:28.734565 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:28.732119 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b9ecbb7-1e58-4ac9-b913-ff46c0a02210" containerName="cleanup" Oct 30 12:45:28.734565 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:28.732126 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea9095cb-47a4-4ea4-b143-069f736c29b6" containerName="cleanup" Oct 30 12:45:28.734565 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:28.732132 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="ccd6f000-a55d-4214-a830-faf2d200b688" containerName="cleanup" Oct 30 12:45:28.734565 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:28.732137 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3a015cd-f16d-42b4-abae-9e2dd114093e" containerName="cleanup" Oct 30 12:45:28.734565 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:28.732143 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea9095cb-47a4-4ea4-b143-069f736c29b6" containerName="cleanup" Oct 30 12:45:28.734565 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:28.732148 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab18f563-fbfd-4dbb-8b81-277c223a30f4" containerName="cleanup" Oct 30 12:45:28.734565 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:28.732153 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="7fae2258-d203-4719-a547-868d7b04070d" containerName="cleanup" Oct 30 12:45:28.734565 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:28.732159 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d181a49-62cd-454a-9e03-380198da3a08" containerName="cleanup" Oct 30 12:45:28.734565 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:28.732165 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="7fae2258-d203-4719-a547-868d7b04070d" containerName="cleanup" Oct 30 12:45:28.734565 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:28.732169 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b9ecbb7-1e58-4ac9-b913-ff46c0a02210" containerName="cleanup" Oct 30 12:45:28.734565 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:28.732176 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d181a49-62cd-454a-9e03-380198da3a08" containerName="cleanup" Oct 30 12:45:28.734565 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:28.732183 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="240c2309-8812-4591-9871-b16dcb9b6ff3" containerName="cleanup" Oct 30 12:45:28.734565 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:28.732188 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="ccd6f000-a55d-4214-a830-faf2d200b688" containerName="cleanup" Oct 30 12:45:28.734565 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:28.732192 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="958d7012-017f-4cd4-a637-a44ec84babc6" containerName="cleanup" Oct 30 12:45:28.734565 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:28.732198 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="81dec304-04f6-4c40-8bf3-7fa3c4e0994d" containerName="cleanup" Oct 30 12:45:28.734565 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:28.732203 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b9ecbb7-1e58-4ac9-b913-ff46c0a02210" containerName="cleanup" Oct 30 12:45:28.734565 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:28.732208 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="81dec304-04f6-4c40-8bf3-7fa3c4e0994d" containerName="cleanup" Oct 30 12:45:28.734565 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:28.732214 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="958d7012-017f-4cd4-a637-a44ec84babc6" containerName="cleanup" Oct 30 12:45:28.734565 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:28.732220 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="fcc90cb3-e12e-4b12-a50d-de740f552b40" containerName="cleanup" Oct 30 12:45:28.734565 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:28.732227 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="240c2309-8812-4591-9871-b16dcb9b6ff3" containerName="cleanup" Oct 30 12:45:28.738099 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:28.738065 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44f58a36-r5vts" Oct 30 12:45:28.758499 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:28.758466 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44f58a36-r5vts"] Oct 30 12:45:28.767878 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:28.764227 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 12:45:28.767878 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:28.764246 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s75hq\"" Oct 30 12:45:28.767878 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:28.764278 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 12:45:28.767878 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:28.764381 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 12:45:28.914898 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:28.914812 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea780d26-11ed-426e-aef1-42f1779d5697-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-44f58a36-r5vts\" (UID: \"ea780d26-11ed-426e-aef1-42f1779d5697\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44f58a36-r5vts" Oct 30 12:45:28.915110 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:28.915093 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea780d26-11ed-426e-aef1-42f1779d5697-maas-api-xks-ca\") pod \"maas-api-revoke-sub-44f58a36-r5vts\" (UID: \"ea780d26-11ed-426e-aef1-42f1779d5697\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44f58a36-r5vts" Oct 30 12:45:28.915156 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:28.915126 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54pjs\" (UniqueName: \"kubernetes.io/projected/ea780d26-11ed-426e-aef1-42f1779d5697-kube-api-access-54pjs\") pod \"maas-api-revoke-sub-44f58a36-r5vts\" (UID: \"ea780d26-11ed-426e-aef1-42f1779d5697\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44f58a36-r5vts" Oct 30 12:45:29.016793 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:29.016684 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea780d26-11ed-426e-aef1-42f1779d5697-maas-api-xks-ca\") pod \"maas-api-revoke-sub-44f58a36-r5vts\" (UID: \"ea780d26-11ed-426e-aef1-42f1779d5697\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44f58a36-r5vts" Oct 30 12:45:29.016793 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:29.016727 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54pjs\" (UniqueName: \"kubernetes.io/projected/ea780d26-11ed-426e-aef1-42f1779d5697-kube-api-access-54pjs\") pod \"maas-api-revoke-sub-44f58a36-r5vts\" (UID: \"ea780d26-11ed-426e-aef1-42f1779d5697\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44f58a36-r5vts" Oct 30 12:45:29.017064 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:29.016863 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea780d26-11ed-426e-aef1-42f1779d5697-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-44f58a36-r5vts\" (UID: \"ea780d26-11ed-426e-aef1-42f1779d5697\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44f58a36-r5vts" Oct 30 12:45:29.017667 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:29.017645 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea780d26-11ed-426e-aef1-42f1779d5697-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-44f58a36-r5vts\" (UID: \"ea780d26-11ed-426e-aef1-42f1779d5697\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44f58a36-r5vts" Oct 30 12:45:29.019314 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:29.019275 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea780d26-11ed-426e-aef1-42f1779d5697-maas-api-xks-ca\") pod \"maas-api-revoke-sub-44f58a36-r5vts\" (UID: \"ea780d26-11ed-426e-aef1-42f1779d5697\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44f58a36-r5vts" Oct 30 12:45:29.081853 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:29.073523 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-54pjs\" (UniqueName: \"kubernetes.io/projected/ea780d26-11ed-426e-aef1-42f1779d5697-kube-api-access-54pjs\") pod \"maas-api-revoke-sub-44f58a36-r5vts\" (UID: \"ea780d26-11ed-426e-aef1-42f1779d5697\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44f58a36-r5vts" Oct 30 12:45:29.171879 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:29.171269 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-94b0bf54-jxsz2"] Oct 30 12:45:29.176556 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:29.176518 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-94b0bf54-jxsz2"] Oct 30 12:45:29.360510 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:29.360343 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44f58a36-r5vts" Oct 30 12:45:29.595856 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:29.595373 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f7df0891-6ppbs"] Oct 30 12:45:29.620102 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:29.596260 4908 cpu_manager.go:414] "containerMap: removing container" podUID="4d181a49-62cd-454a-9e03-380198da3a08" containerName="cleanup" Oct 30 12:45:29.620102 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:29.596283 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d181a49-62cd-454a-9e03-380198da3a08" containerName="cleanup" Oct 30 12:45:29.620102 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:29.596308 4908 cpu_manager.go:414] "containerMap: removing container" podUID="0b9ecbb7-1e58-4ac9-b913-ff46c0a02210" containerName="cleanup" Oct 30 12:45:29.620102 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:29.596316 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b9ecbb7-1e58-4ac9-b913-ff46c0a02210" containerName="cleanup" Oct 30 12:45:29.620102 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:29.597094 4908 cpu_manager.go:414] "containerMap: removing container" podUID="12c31795-7957-4dff-83c3-d4d2c1d31aac" containerName="cleanup" Oct 30 12:45:29.620102 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:29.597111 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12c31795-7957-4dff-83c3-d4d2c1d31aac" containerName="cleanup" Oct 30 12:45:29.620102 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:29.597126 4908 cpu_manager.go:414] "containerMap: removing container" podUID="240c2309-8812-4591-9871-b16dcb9b6ff3" containerName="cleanup" Oct 30 12:45:29.620102 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:29.597133 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="240c2309-8812-4591-9871-b16dcb9b6ff3" containerName="cleanup" Oct 30 12:45:29.620102 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:29.597157 4908 cpu_manager.go:414] "containerMap: removing container" podUID="81dec304-04f6-4c40-8bf3-7fa3c4e0994d" containerName="cleanup" Oct 30 12:45:29.620102 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:29.597165 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81dec304-04f6-4c40-8bf3-7fa3c4e0994d" containerName="cleanup" Oct 30 12:45:29.620102 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:29.597179 4908 cpu_manager.go:414] "containerMap: removing container" podUID="fcc90cb3-e12e-4b12-a50d-de740f552b40" containerName="cleanup" Oct 30 12:45:29.620102 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:29.597185 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fcc90cb3-e12e-4b12-a50d-de740f552b40" containerName="cleanup" Oct 30 12:45:29.620102 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:29.597216 4908 cpu_manager.go:414] "containerMap: removing container" podUID="3eef63f0-3b6c-4e62-bdc9-594ee9d4bd17" containerName="cleanup" Oct 30 12:45:29.620102 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:29.597224 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3eef63f0-3b6c-4e62-bdc9-594ee9d4bd17" containerName="cleanup" Oct 30 12:45:29.620102 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:29.597244 4908 cpu_manager.go:414] "containerMap: removing container" podUID="1aa1b39f-f88b-4ddf-8ee1-cf567d9d480a" containerName="cleanup" Oct 30 12:45:29.620102 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:29.597251 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1aa1b39f-f88b-4ddf-8ee1-cf567d9d480a" containerName="cleanup" Oct 30 12:45:29.620102 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:29.597274 4908 cpu_manager.go:414] "containerMap: removing container" podUID="958d7012-017f-4cd4-a637-a44ec84babc6" containerName="cleanup" Oct 30 12:45:29.620102 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:29.597283 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="958d7012-017f-4cd4-a637-a44ec84babc6" containerName="cleanup" Oct 30 12:45:29.620102 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:29.597295 4908 cpu_manager.go:414] "containerMap: removing container" podUID="27d77dc7-aee3-45a4-8ad9-a48560889798" containerName="cleanup" Oct 30 12:45:29.620102 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:29.597303 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27d77dc7-aee3-45a4-8ad9-a48560889798" containerName="cleanup" Oct 30 12:45:29.620102 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:29.597475 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="12c31795-7957-4dff-83c3-d4d2c1d31aac" containerName="cleanup" Oct 30 12:45:29.620102 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:29.597490 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab18f563-fbfd-4dbb-8b81-277c223a30f4" containerName="cleanup" Oct 30 12:45:29.620102 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:29.597517 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="fcc90cb3-e12e-4b12-a50d-de740f552b40" containerName="cleanup" Oct 30 12:45:29.620102 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:29.597527 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="3eef63f0-3b6c-4e62-bdc9-594ee9d4bd17" containerName="cleanup" Oct 30 12:45:29.734969 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:29.715782 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f7df0891-6ppbs"] Oct 30 12:45:29.734969 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:29.732047 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7df0891-6ppbs" Oct 30 12:45:29.770456 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:29.769301 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="985ef4ea-15fc-4ae1-94fd-c71c7dae96ca" path="/var/lib/kubelet/pods/985ef4ea-15fc-4ae1-94fd-c71c7dae96ca/volumes" Oct 30 12:45:29.770456 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:29.769923 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec4c8aae-7586-4fdf-bddd-9ba01d54b071" path="/var/lib/kubelet/pods/ec4c8aae-7586-4fdf-bddd-9ba01d54b071/volumes" Oct 30 12:45:29.866788 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:29.866747 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ba9de25-ec88-4cf8-b374-dc779b854834-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f7df0891-6ppbs\" (UID: \"0ba9de25-ec88-4cf8-b374-dc779b854834\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7df0891-6ppbs" Oct 30 12:45:29.867048 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:29.867014 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7qss\" (UniqueName: \"kubernetes.io/projected/0ba9de25-ec88-4cf8-b374-dc779b854834-kube-api-access-p7qss\") pod \"maas-api-revoke-sub-f7df0891-6ppbs\" (UID: \"0ba9de25-ec88-4cf8-b374-dc779b854834\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7df0891-6ppbs" Oct 30 12:45:29.867097 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:29.867083 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ba9de25-ec88-4cf8-b374-dc779b854834-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f7df0891-6ppbs\" (UID: \"0ba9de25-ec88-4cf8-b374-dc779b854834\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7df0891-6ppbs" Oct 30 12:45:29.884609 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:29.884534 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44f58a36-r5vts"] Oct 30 12:45:29.888262 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:45:29.888216 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea780d26_11ed_426e_aef1_42f1779d5697.slice/crio-eeff831722d9be12720c1eb570af9791205f60da2d90ba1ddcc166fe22c347ec WatchSource:0}: Error finding container eeff831722d9be12720c1eb570af9791205f60da2d90ba1ddcc166fe22c347ec: Status 404 returned error can't find the container with id eeff831722d9be12720c1eb570af9791205f60da2d90ba1ddcc166fe22c347ec Oct 30 12:45:29.968197 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:29.968164 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7qss\" (UniqueName: \"kubernetes.io/projected/0ba9de25-ec88-4cf8-b374-dc779b854834-kube-api-access-p7qss\") pod \"maas-api-revoke-sub-f7df0891-6ppbs\" (UID: \"0ba9de25-ec88-4cf8-b374-dc779b854834\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7df0891-6ppbs" Oct 30 12:45:29.968348 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:29.968210 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ba9de25-ec88-4cf8-b374-dc779b854834-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f7df0891-6ppbs\" (UID: \"0ba9de25-ec88-4cf8-b374-dc779b854834\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7df0891-6ppbs" Oct 30 12:45:29.968437 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:29.968406 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ba9de25-ec88-4cf8-b374-dc779b854834-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f7df0891-6ppbs\" (UID: \"0ba9de25-ec88-4cf8-b374-dc779b854834\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7df0891-6ppbs" Oct 30 12:45:29.969084 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:29.969059 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ba9de25-ec88-4cf8-b374-dc779b854834-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f7df0891-6ppbs\" (UID: \"0ba9de25-ec88-4cf8-b374-dc779b854834\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7df0891-6ppbs" Oct 30 12:45:29.970761 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:29.970743 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ba9de25-ec88-4cf8-b374-dc779b854834-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f7df0891-6ppbs\" (UID: \"0ba9de25-ec88-4cf8-b374-dc779b854834\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7df0891-6ppbs" Oct 30 12:45:29.991068 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:29.991039 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7qss\" (UniqueName: \"kubernetes.io/projected/0ba9de25-ec88-4cf8-b374-dc779b854834-kube-api-access-p7qss\") pod \"maas-api-revoke-sub-f7df0891-6ppbs\" (UID: \"0ba9de25-ec88-4cf8-b374-dc779b854834\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7df0891-6ppbs" Oct 30 12:45:30.059428 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:30.059386 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7df0891-6ppbs" Oct 30 12:45:30.181115 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:30.181074 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44f58a36-r5vts" event={"ID":"ea780d26-11ed-426e-aef1-42f1779d5697","Type":"ContainerStarted","Data":"eeff831722d9be12720c1eb570af9791205f60da2d90ba1ddcc166fe22c347ec"} Oct 30 12:45:30.367922 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:30.367621 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f7df0891-6ppbs"] Oct 30 12:45:30.370603 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:45:30.370569 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ba9de25_ec88_4cf8_b374_dc779b854834.slice/crio-40b4667dc92e19169fdf736f1ffbed16aac09365898ffff895666100a8c3741c WatchSource:0}: Error finding container 40b4667dc92e19169fdf736f1ffbed16aac09365898ffff895666100a8c3741c: Status 404 returned error can't find the container with id 40b4667dc92e19169fdf736f1ffbed16aac09365898ffff895666100a8c3741c Oct 30 12:45:31.193115 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:31.193025 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7df0891-6ppbs" event={"ID":"0ba9de25-ec88-4cf8-b374-dc779b854834","Type":"ContainerStarted","Data":"a939d98d58a7c943a93eeaa4f8f63483f5fbd67d33cabcbaa9eceacad46e323a"} Oct 30 12:45:31.193564 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:31.193114 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7df0891-6ppbs" event={"ID":"0ba9de25-ec88-4cf8-b374-dc779b854834","Type":"ContainerStarted","Data":"40b4667dc92e19169fdf736f1ffbed16aac09365898ffff895666100a8c3741c"} Oct 30 12:45:31.220471 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:31.219541 4908 generic.go:356] "Generic (PLEG): container finished" podID="ea780d26-11ed-426e-aef1-42f1779d5697" containerID="6bb731e41d13376b40da40e3940bbfe21c236a896d7adc0260fdc71e13262b2e" exitCode=0 Oct 30 12:45:31.220471 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:31.219646 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44f58a36-r5vts" event={"ID":"ea780d26-11ed-426e-aef1-42f1779d5697","Type":"ContainerDied","Data":"6bb731e41d13376b40da40e3940bbfe21c236a896d7adc0260fdc71e13262b2e"} Oct 30 12:45:31.257791 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:31.257719 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7df0891-6ppbs" podStartSLOduration=2.257699662 podStartE2EDuration="2.257699662s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:45:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:45:31.252858836 +0000 UTC m=+3087.767287582" watchObservedRunningTime="2026-10-30 12:45:31.257699662 +0000 UTC m=+3087.772128409" Oct 30 12:45:31.742103 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:31.742068 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-6ad1da/payload-processing-external-model-e2e-extmod-a-6ad1da-59c7f4475"] Oct 30 12:45:31.746334 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:31.746310 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6ad1da/payload-processing-external-model-e2e-extmod-a-6ad1da-59c7f4475" Oct 30 12:45:31.765994 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:31.765959 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-6ad1da\"/\"routing-overlay\"" Oct 30 12:45:31.766880 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:31.766489 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-6ad1da\"/\"openshift-service-ca.crt\"" Oct 30 12:45:31.766880 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:31.766486 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-6ad1da\"/\"kube-root-ca.crt\"" Oct 30 12:45:31.766880 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:31.766553 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-6ad1da\"/\"payload-processing-external-model-e2e--da1b9d00-dockercfg-88b4c\"" Oct 30 12:45:31.766880 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:31.766579 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-6ad1da\"/\"agc-cred-62a730bed018\"" Oct 30 12:45:31.766880 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:31.766840 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-6ad1da\"/\"payload-processing-external-model-plugins-e2e-extmod-a-6ad1da\"" Oct 30 12:45:31.769763 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:31.769736 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-6ad1da/payload-processing-external-model-e2e-extmod-a-6ad1da-59c7f4475"] Oct 30 12:45:31.838852 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:31.831761 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85b89442-972b-4fef-83cf-23875def30d1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6ad1da-59c7f4475\" (UID: \"85b89442-972b-4fef-83cf-23875def30d1\") " pod="ai-tenant-e2e-extmod-a-6ad1da/payload-processing-external-model-e2e-extmod-a-6ad1da-59c7f4475" Oct 30 12:45:31.838852 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:31.831889 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/85b89442-972b-4fef-83cf-23875def30d1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-6ad1da-59c7f4475\" (UID: \"85b89442-972b-4fef-83cf-23875def30d1\") " pod="ai-tenant-e2e-extmod-a-6ad1da/payload-processing-external-model-e2e-extmod-a-6ad1da-59c7f4475" Oct 30 12:45:31.838852 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:31.832009 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/85b89442-972b-4fef-83cf-23875def30d1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6ad1da-59c7f4475\" (UID: \"85b89442-972b-4fef-83cf-23875def30d1\") " pod="ai-tenant-e2e-extmod-a-6ad1da/payload-processing-external-model-e2e-extmod-a-6ad1da-59c7f4475" Oct 30 12:45:31.838852 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:31.832068 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5lt9\" (UniqueName: \"kubernetes.io/projected/85b89442-972b-4fef-83cf-23875def30d1-kube-api-access-x5lt9\") pod \"payload-processing-external-model-e2e-extmod-a-6ad1da-59c7f4475\" (UID: \"85b89442-972b-4fef-83cf-23875def30d1\") " pod="ai-tenant-e2e-extmod-a-6ad1da/payload-processing-external-model-e2e-extmod-a-6ad1da-59c7f4475" Oct 30 12:45:31.950852 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:31.936685 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85b89442-972b-4fef-83cf-23875def30d1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6ad1da-59c7f4475\" (UID: \"85b89442-972b-4fef-83cf-23875def30d1\") " pod="ai-tenant-e2e-extmod-a-6ad1da/payload-processing-external-model-e2e-extmod-a-6ad1da-59c7f4475" Oct 30 12:45:31.950852 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:31.936791 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/85b89442-972b-4fef-83cf-23875def30d1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-6ad1da-59c7f4475\" (UID: \"85b89442-972b-4fef-83cf-23875def30d1\") " pod="ai-tenant-e2e-extmod-a-6ad1da/payload-processing-external-model-e2e-extmod-a-6ad1da-59c7f4475" Oct 30 12:45:31.950852 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:31.936929 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/85b89442-972b-4fef-83cf-23875def30d1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6ad1da-59c7f4475\" (UID: \"85b89442-972b-4fef-83cf-23875def30d1\") " pod="ai-tenant-e2e-extmod-a-6ad1da/payload-processing-external-model-e2e-extmod-a-6ad1da-59c7f4475" Oct 30 12:45:31.950852 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:31.936991 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5lt9\" (UniqueName: \"kubernetes.io/projected/85b89442-972b-4fef-83cf-23875def30d1-kube-api-access-x5lt9\") pod \"payload-processing-external-model-e2e-extmod-a-6ad1da-59c7f4475\" (UID: \"85b89442-972b-4fef-83cf-23875def30d1\") " pod="ai-tenant-e2e-extmod-a-6ad1da/payload-processing-external-model-e2e-extmod-a-6ad1da-59c7f4475" Oct 30 12:45:31.950852 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:31.938354 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85b89442-972b-4fef-83cf-23875def30d1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6ad1da-59c7f4475\" (UID: \"85b89442-972b-4fef-83cf-23875def30d1\") " pod="ai-tenant-e2e-extmod-a-6ad1da/payload-processing-external-model-e2e-extmod-a-6ad1da-59c7f4475" Oct 30 12:45:31.950852 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:31.939227 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/85b89442-972b-4fef-83cf-23875def30d1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6ad1da-59c7f4475\" (UID: \"85b89442-972b-4fef-83cf-23875def30d1\") " pod="ai-tenant-e2e-extmod-a-6ad1da/payload-processing-external-model-e2e-extmod-a-6ad1da-59c7f4475" Oct 30 12:45:31.989170 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:31.989123 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5lt9\" (UniqueName: \"kubernetes.io/projected/85b89442-972b-4fef-83cf-23875def30d1-kube-api-access-x5lt9\") pod \"payload-processing-external-model-e2e-extmod-a-6ad1da-59c7f4475\" (UID: \"85b89442-972b-4fef-83cf-23875def30d1\") " pod="ai-tenant-e2e-extmod-a-6ad1da/payload-processing-external-model-e2e-extmod-a-6ad1da-59c7f4475" Oct 30 12:45:31.998423 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:31.998346 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/85b89442-972b-4fef-83cf-23875def30d1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-6ad1da-59c7f4475\" (UID: \"85b89442-972b-4fef-83cf-23875def30d1\") " pod="ai-tenant-e2e-extmod-a-6ad1da/payload-processing-external-model-e2e-extmod-a-6ad1da-59c7f4475" Oct 30 12:45:32.060918 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:32.060883 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6ad1da/payload-processing-external-model-e2e-extmod-a-6ad1da-59c7f4475" Oct 30 12:45:32.229179 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:32.229146 4908 generic.go:356] "Generic (PLEG): container finished" podID="0ba9de25-ec88-4cf8-b374-dc779b854834" containerID="a939d98d58a7c943a93eeaa4f8f63483f5fbd67d33cabcbaa9eceacad46e323a" exitCode=0 Oct 30 12:45:32.229560 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:32.229223 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7df0891-6ppbs" event={"ID":"0ba9de25-ec88-4cf8-b374-dc779b854834","Type":"ContainerDied","Data":"a939d98d58a7c943a93eeaa4f8f63483f5fbd67d33cabcbaa9eceacad46e323a"} Oct 30 12:45:32.373680 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:32.373635 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-6ad1da/payload-processing-external-model-e2e-extmod-a-6ad1da-59c7f4475"] Oct 30 12:45:32.380393 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:32.380364 4908 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 12:45:32.410597 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:32.410567 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44f58a36-r5vts" Oct 30 12:45:32.545633 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:32.545555 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea780d26-11ed-426e-aef1-42f1779d5697-kube-api-access-54pjs\" (UniqueName: \"kubernetes.io/projected/ea780d26-11ed-426e-aef1-42f1779d5697-kube-api-access-54pjs\") pod \"ea780d26-11ed-426e-aef1-42f1779d5697\" (UID: \"ea780d26-11ed-426e-aef1-42f1779d5697\") " Oct 30 12:45:32.545782 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:32.545689 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea780d26-11ed-426e-aef1-42f1779d5697-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea780d26-11ed-426e-aef1-42f1779d5697-maas-api-openshift-ca\") pod \"ea780d26-11ed-426e-aef1-42f1779d5697\" (UID: \"ea780d26-11ed-426e-aef1-42f1779d5697\") " Oct 30 12:45:32.545782 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:32.545769 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ea780d26-11ed-426e-aef1-42f1779d5697-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea780d26-11ed-426e-aef1-42f1779d5697-maas-api-xks-ca\") pod \"ea780d26-11ed-426e-aef1-42f1779d5697\" (UID: \"ea780d26-11ed-426e-aef1-42f1779d5697\") " Oct 30 12:45:32.546066 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:32.546040 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea780d26-11ed-426e-aef1-42f1779d5697-maas-api-openshift-ca" pod "ea780d26-11ed-426e-aef1-42f1779d5697" (UID: "ea780d26-11ed-426e-aef1-42f1779d5697"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:45:32.546249 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:32.546222 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea780d26-11ed-426e-aef1-42f1779d5697-maas-api-openshift-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:45:32.547816 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:32.547791 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea780d26-11ed-426e-aef1-42f1779d5697-kube-api-access-54pjs" pod "ea780d26-11ed-426e-aef1-42f1779d5697" (UID: "ea780d26-11ed-426e-aef1-42f1779d5697"). InnerVolumeSpecName "kube-api-access-54pjs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:45:32.547975 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:32.547949 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea780d26-11ed-426e-aef1-42f1779d5697-maas-api-xks-ca" pod "ea780d26-11ed-426e-aef1-42f1779d5697" (UID: "ea780d26-11ed-426e-aef1-42f1779d5697"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:45:32.647767 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:32.647722 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54pjs\" (UniqueName: \"kubernetes.io/projected/ea780d26-11ed-426e-aef1-42f1779d5697-kube-api-access-54pjs\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:45:32.647767 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:32.647758 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea780d26-11ed-426e-aef1-42f1779d5697-maas-api-xks-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:45:33.242565 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:33.242523 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-6ad1da/payload-processing-external-model-e2e-extmod-a-6ad1da-59c7f4475" event={"ID":"85b89442-972b-4fef-83cf-23875def30d1","Type":"ContainerStarted","Data":"d83eb64387d4e050d33eb099f38fe83373b517bebca583792b676e3b55be050f"} Oct 30 12:45:33.244571 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:33.244544 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44f58a36-r5vts" Oct 30 12:45:33.244719 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:33.244563 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44f58a36-r5vts" event={"ID":"ea780d26-11ed-426e-aef1-42f1779d5697","Type":"ContainerDied","Data":"eeff831722d9be12720c1eb570af9791205f60da2d90ba1ddcc166fe22c347ec"} Oct 30 12:45:33.244719 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:33.244612 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eeff831722d9be12720c1eb570af9791205f60da2d90ba1ddcc166fe22c347ec" Oct 30 12:45:33.442225 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:33.441944 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7df0891-6ppbs" Oct 30 12:45:33.560610 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:33.560531 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ba9de25-ec88-4cf8-b374-dc779b854834-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ba9de25-ec88-4cf8-b374-dc779b854834-maas-api-openshift-ca\") pod \"0ba9de25-ec88-4cf8-b374-dc779b854834\" (UID: \"0ba9de25-ec88-4cf8-b374-dc779b854834\") " Oct 30 12:45:33.560610 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:33.560590 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0ba9de25-ec88-4cf8-b374-dc779b854834-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ba9de25-ec88-4cf8-b374-dc779b854834-maas-api-xks-ca\") pod \"0ba9de25-ec88-4cf8-b374-dc779b854834\" (UID: \"0ba9de25-ec88-4cf8-b374-dc779b854834\") " Oct 30 12:45:33.560864 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:33.560704 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ba9de25-ec88-4cf8-b374-dc779b854834-kube-api-access-p7qss\" (UniqueName: \"kubernetes.io/projected/0ba9de25-ec88-4cf8-b374-dc779b854834-kube-api-access-p7qss\") pod \"0ba9de25-ec88-4cf8-b374-dc779b854834\" (UID: \"0ba9de25-ec88-4cf8-b374-dc779b854834\") " Oct 30 12:45:33.560980 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:33.560954 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ba9de25-ec88-4cf8-b374-dc779b854834-maas-api-openshift-ca" pod "0ba9de25-ec88-4cf8-b374-dc779b854834" (UID: "0ba9de25-ec88-4cf8-b374-dc779b854834"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:45:33.561329 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:33.561304 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ba9de25-ec88-4cf8-b374-dc779b854834-maas-api-openshift-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:45:33.563488 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:33.563456 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ba9de25-ec88-4cf8-b374-dc779b854834-kube-api-access-p7qss" pod "0ba9de25-ec88-4cf8-b374-dc779b854834" (UID: "0ba9de25-ec88-4cf8-b374-dc779b854834"). InnerVolumeSpecName "kube-api-access-p7qss". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:45:33.566781 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:33.566754 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ba9de25-ec88-4cf8-b374-dc779b854834-maas-api-xks-ca" pod "0ba9de25-ec88-4cf8-b374-dc779b854834" (UID: "0ba9de25-ec88-4cf8-b374-dc779b854834"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:45:33.663128 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:33.663098 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ba9de25-ec88-4cf8-b374-dc779b854834-maas-api-xks-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:45:33.663128 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:33.663129 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p7qss\" (UniqueName: \"kubernetes.io/projected/0ba9de25-ec88-4cf8-b374-dc779b854834-kube-api-access-p7qss\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:45:34.256519 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:34.256425 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7df0891-6ppbs" Oct 30 12:45:34.257036 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:34.256459 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7df0891-6ppbs" event={"ID":"0ba9de25-ec88-4cf8-b374-dc779b854834","Type":"ContainerDied","Data":"40b4667dc92e19169fdf736f1ffbed16aac09365898ffff895666100a8c3741c"} Oct 30 12:45:34.257112 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:34.257073 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="40b4667dc92e19169fdf736f1ffbed16aac09365898ffff895666100a8c3741c" Oct 30 12:45:35.266138 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:35.266098 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-6ad1da/payload-processing-external-model-e2e-extmod-a-6ad1da-59c7f4475" event={"ID":"85b89442-972b-4fef-83cf-23875def30d1","Type":"ContainerStarted","Data":"2837c12538f6c0bc10512f31f83704371d7fa33b627d8a323ebaf5543a7ae722"} Oct 30 12:45:35.266509 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:35.266174 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-6ad1da/payload-processing-external-model-e2e-extmod-a-6ad1da-59c7f4475" Oct 30 12:45:35.325045 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:35.324993 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-6ad1da/payload-processing-external-model-e2e-extmod-a-6ad1da-59c7f4475" podStartSLOduration=2.190536568 podStartE2EDuration="4.324977317s" totalImagesPullingTime="2.134440749s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:45:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:45:35.320595053 +0000 UTC m=+3091.835023800" watchObservedRunningTime="2026-10-30 12:45:35.324977317 +0000 UTC m=+3091.839406115" Oct 30 12:45:37.364995 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:37.364963 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-6ad1da/payload-processing-external-model-e2e-extmod-a-6ad1da-59c7f4475"] Oct 30 12:45:42.365979 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:42.365938 4908 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-6ad1da/payload-processing-external-model-e2e-extmod-a-6ad1da-59c7f4475" podUID="85b89442-972b-4fef-83cf-23875def30d1" containerName="payload-processing" containerID="cri-o://2837c12538f6c0bc10512f31f83704371d7fa33b627d8a323ebaf5543a7ae722" gracePeriod=25 Oct 30 12:45:42.693396 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:42.693364 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6ad1da/payload-processing-external-model-e2e-extmod-a-6ad1da-59c7f4475" Oct 30 12:45:42.769277 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:42.769240 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85b89442-972b-4fef-83cf-23875def30d1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85b89442-972b-4fef-83cf-23875def30d1-plugins-config-volume\") pod \"85b89442-972b-4fef-83cf-23875def30d1\" (UID: \"85b89442-972b-4fef-83cf-23875def30d1\") " Oct 30 12:45:42.769469 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:42.769293 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85b89442-972b-4fef-83cf-23875def30d1-kube-api-access-x5lt9\" (UniqueName: \"kubernetes.io/projected/85b89442-972b-4fef-83cf-23875def30d1-kube-api-access-x5lt9\") pod \"85b89442-972b-4fef-83cf-23875def30d1\" (UID: \"85b89442-972b-4fef-83cf-23875def30d1\") " Oct 30 12:45:42.769469 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:42.769324 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85b89442-972b-4fef-83cf-23875def30d1-provider-credentials\" (UniqueName: \"kubernetes.io/projected/85b89442-972b-4fef-83cf-23875def30d1-provider-credentials\") pod \"85b89442-972b-4fef-83cf-23875def30d1\" (UID: \"85b89442-972b-4fef-83cf-23875def30d1\") " Oct 30 12:45:42.769469 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:42.769430 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85b89442-972b-4fef-83cf-23875def30d1-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/85b89442-972b-4fef-83cf-23875def30d1-routing-overlay-volume\") pod \"85b89442-972b-4fef-83cf-23875def30d1\" (UID: \"85b89442-972b-4fef-83cf-23875def30d1\") " Oct 30 12:45:42.770171 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:42.770139 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85b89442-972b-4fef-83cf-23875def30d1-routing-overlay-volume" pod "85b89442-972b-4fef-83cf-23875def30d1" (UID: "85b89442-972b-4fef-83cf-23875def30d1"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:45:42.770620 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:42.770591 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85b89442-972b-4fef-83cf-23875def30d1-plugins-config-volume" pod "85b89442-972b-4fef-83cf-23875def30d1" (UID: "85b89442-972b-4fef-83cf-23875def30d1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:45:42.770970 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:42.770949 4908 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85b89442-972b-4fef-83cf-23875def30d1-plugins-config-volume\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:45:42.771099 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:42.771085 4908 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/85b89442-972b-4fef-83cf-23875def30d1-routing-overlay-volume\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:45:42.772584 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:42.772553 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85b89442-972b-4fef-83cf-23875def30d1-kube-api-access-x5lt9" pod "85b89442-972b-4fef-83cf-23875def30d1" (UID: "85b89442-972b-4fef-83cf-23875def30d1"). InnerVolumeSpecName "kube-api-access-x5lt9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:45:42.773215 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:42.773188 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85b89442-972b-4fef-83cf-23875def30d1-provider-credentials" pod "85b89442-972b-4fef-83cf-23875def30d1" (UID: "85b89442-972b-4fef-83cf-23875def30d1"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:45:42.871882 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:42.871858 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5lt9\" (UniqueName: \"kubernetes.io/projected/85b89442-972b-4fef-83cf-23875def30d1-kube-api-access-x5lt9\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:45:42.871882 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:42.871880 4908 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/85b89442-972b-4fef-83cf-23875def30d1-provider-credentials\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:45:43.355549 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:43.355511 4908 generic.go:356] "Generic (PLEG): container finished" podID="85b89442-972b-4fef-83cf-23875def30d1" containerID="2837c12538f6c0bc10512f31f83704371d7fa33b627d8a323ebaf5543a7ae722" exitCode=0 Oct 30 12:45:43.355737 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:43.355570 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6ad1da/payload-processing-external-model-e2e-extmod-a-6ad1da-59c7f4475" Oct 30 12:45:43.355737 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:43.355602 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-6ad1da/payload-processing-external-model-e2e-extmod-a-6ad1da-59c7f4475" event={"ID":"85b89442-972b-4fef-83cf-23875def30d1","Type":"ContainerDied","Data":"2837c12538f6c0bc10512f31f83704371d7fa33b627d8a323ebaf5543a7ae722"} Oct 30 12:45:43.355737 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:43.355699 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-6ad1da/payload-processing-external-model-e2e-extmod-a-6ad1da-59c7f4475" event={"ID":"85b89442-972b-4fef-83cf-23875def30d1","Type":"ContainerDied","Data":"d83eb64387d4e050d33eb099f38fe83373b517bebca583792b676e3b55be050f"} Oct 30 12:45:43.355885 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:43.355737 4908 scope.go:122] "RemoveContainer" containerID="2837c12538f6c0bc10512f31f83704371d7fa33b627d8a323ebaf5543a7ae722" Oct 30 12:45:43.367762 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:43.367554 4908 scope.go:122] "RemoveContainer" containerID="2837c12538f6c0bc10512f31f83704371d7fa33b627d8a323ebaf5543a7ae722" Oct 30 12:45:43.368063 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:45:43.367913 4908 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2837c12538f6c0bc10512f31f83704371d7fa33b627d8a323ebaf5543a7ae722\": container with ID starting with 2837c12538f6c0bc10512f31f83704371d7fa33b627d8a323ebaf5543a7ae722 not found: ID does not exist" containerID="2837c12538f6c0bc10512f31f83704371d7fa33b627d8a323ebaf5543a7ae722" Oct 30 12:45:43.368063 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:43.367938 4908 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2837c12538f6c0bc10512f31f83704371d7fa33b627d8a323ebaf5543a7ae722"} err="failed to get container status \"2837c12538f6c0bc10512f31f83704371d7fa33b627d8a323ebaf5543a7ae722\": rpc error: code = NotFound desc = could not find container \"2837c12538f6c0bc10512f31f83704371d7fa33b627d8a323ebaf5543a7ae722\": container with ID starting with 2837c12538f6c0bc10512f31f83704371d7fa33b627d8a323ebaf5543a7ae722 not found: ID does not exist" Oct 30 12:45:43.409526 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:43.409499 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-6ad1da/payload-processing-external-model-e2e-extmod-a-6ad1da-59c7f4475"] Oct 30 12:45:43.418507 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:43.418482 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-6ad1da/payload-processing-external-model-e2e-extmod-a-6ad1da-59c7f4475"] Oct 30 12:45:43.606586 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:43.606503 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85b89442-972b-4fef-83cf-23875def30d1" path="/var/lib/kubelet/pods/85b89442-972b-4fef-83cf-23875def30d1/volumes" Oct 30 12:45:47.306973 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:47.306886 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9wwfc"] Oct 30 12:45:49.692523 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:45:49.692483 4908 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vn6vv" containerName="extractor" Oct 30 12:45:53.801319 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:53.801276 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e337e7b-qbvgq"] Oct 30 12:45:53.802182 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:53.802161 4908 cpu_manager.go:414] "containerMap: removing container" podUID="85b89442-972b-4fef-83cf-23875def30d1" containerName="payload-processing" Oct 30 12:45:53.802257 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:53.802184 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85b89442-972b-4fef-83cf-23875def30d1" containerName="payload-processing" Oct 30 12:45:53.802257 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:53.802209 4908 cpu_manager.go:414] "containerMap: removing container" podUID="0ba9de25-ec88-4cf8-b374-dc779b854834" containerName="revoke-keys" Oct 30 12:45:53.802257 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:53.802218 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ba9de25-ec88-4cf8-b374-dc779b854834" containerName="revoke-keys" Oct 30 12:45:53.802257 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:53.802234 4908 cpu_manager.go:414] "containerMap: removing container" podUID="ea780d26-11ed-426e-aef1-42f1779d5697" containerName="revoke-keys" Oct 30 12:45:53.802257 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:53.802242 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea780d26-11ed-426e-aef1-42f1779d5697" containerName="revoke-keys" Oct 30 12:45:53.802417 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:53.802363 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="85b89442-972b-4fef-83cf-23875def30d1" containerName="payload-processing" Oct 30 12:45:53.802417 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:53.802379 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ba9de25-ec88-4cf8-b374-dc779b854834" containerName="revoke-keys" Oct 30 12:45:53.802417 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:53.802391 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea780d26-11ed-426e-aef1-42f1779d5697" containerName="revoke-keys" Oct 30 12:45:53.807954 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:53.807931 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e337e7b-qbvgq" Oct 30 12:45:53.824418 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:53.824393 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e337e7b-qbvgq"] Oct 30 12:45:53.825865 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:53.825820 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 12:45:53.830615 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:53.830594 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s75hq\"" Oct 30 12:45:53.836871 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:53.836849 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 12:45:53.836871 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:53.836850 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 12:45:53.949522 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:53.949488 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvf2n\" (UniqueName: \"kubernetes.io/projected/99914eaa-84d9-4fc3-a4e2-7c951974eea1-kube-api-access-jvf2n\") pod \"maas-api-revoke-sub-1e337e7b-qbvgq\" (UID: \"99914eaa-84d9-4fc3-a4e2-7c951974eea1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e337e7b-qbvgq" Oct 30 12:45:53.949683 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:53.949560 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99914eaa-84d9-4fc3-a4e2-7c951974eea1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1e337e7b-qbvgq\" (UID: \"99914eaa-84d9-4fc3-a4e2-7c951974eea1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e337e7b-qbvgq" Oct 30 12:45:53.949862 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:53.949836 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99914eaa-84d9-4fc3-a4e2-7c951974eea1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1e337e7b-qbvgq\" (UID: \"99914eaa-84d9-4fc3-a4e2-7c951974eea1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e337e7b-qbvgq" Oct 30 12:45:54.051400 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:54.051324 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99914eaa-84d9-4fc3-a4e2-7c951974eea1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1e337e7b-qbvgq\" (UID: \"99914eaa-84d9-4fc3-a4e2-7c951974eea1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e337e7b-qbvgq" Oct 30 12:45:54.051554 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:54.051445 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99914eaa-84d9-4fc3-a4e2-7c951974eea1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1e337e7b-qbvgq\" (UID: \"99914eaa-84d9-4fc3-a4e2-7c951974eea1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e337e7b-qbvgq" Oct 30 12:45:54.051554 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:54.051545 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvf2n\" (UniqueName: \"kubernetes.io/projected/99914eaa-84d9-4fc3-a4e2-7c951974eea1-kube-api-access-jvf2n\") pod \"maas-api-revoke-sub-1e337e7b-qbvgq\" (UID: \"99914eaa-84d9-4fc3-a4e2-7c951974eea1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e337e7b-qbvgq" Oct 30 12:45:54.052197 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:54.052176 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99914eaa-84d9-4fc3-a4e2-7c951974eea1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1e337e7b-qbvgq\" (UID: \"99914eaa-84d9-4fc3-a4e2-7c951974eea1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e337e7b-qbvgq" Oct 30 12:45:54.053659 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:54.053640 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99914eaa-84d9-4fc3-a4e2-7c951974eea1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1e337e7b-qbvgq\" (UID: \"99914eaa-84d9-4fc3-a4e2-7c951974eea1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e337e7b-qbvgq" Oct 30 12:45:54.075175 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:54.075148 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvf2n\" (UniqueName: \"kubernetes.io/projected/99914eaa-84d9-4fc3-a4e2-7c951974eea1-kube-api-access-jvf2n\") pod \"maas-api-revoke-sub-1e337e7b-qbvgq\" (UID: \"99914eaa-84d9-4fc3-a4e2-7c951974eea1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e337e7b-qbvgq" Oct 30 12:45:54.121983 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:54.121959 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e337e7b-qbvgq" Oct 30 12:45:54.439474 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:54.439444 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e337e7b-qbvgq"] Oct 30 12:45:54.442028 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:45:54.441988 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99914eaa_84d9_4fc3_a4e2_7c951974eea1.slice/crio-7cdfc09a4386fb00d68f5c08032d3e7ece3b63a5cdf4190ee0f12b78509d6095 WatchSource:0}: Error finding container 7cdfc09a4386fb00d68f5c08032d3e7ece3b63a5cdf4190ee0f12b78509d6095: Status 404 returned error can't find the container with id 7cdfc09a4386fb00d68f5c08032d3e7ece3b63a5cdf4190ee0f12b78509d6095 Oct 30 12:45:54.486854 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:54.486801 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e337e7b-qbvgq" event={"ID":"99914eaa-84d9-4fc3-a4e2-7c951974eea1","Type":"ContainerStarted","Data":"7cdfc09a4386fb00d68f5c08032d3e7ece3b63a5cdf4190ee0f12b78509d6095"} Oct 30 12:45:55.528860 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:55.527880 4908 generic.go:356] "Generic (PLEG): container finished" podID="99914eaa-84d9-4fc3-a4e2-7c951974eea1" containerID="14b4618e3febde11d741191798be5640a19a7cb107ebc034d4f086f747847581" exitCode=0 Oct 30 12:45:55.528860 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:55.528155 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e337e7b-qbvgq" event={"ID":"99914eaa-84d9-4fc3-a4e2-7c951974eea1","Type":"ContainerDied","Data":"14b4618e3febde11d741191798be5640a19a7cb107ebc034d4f086f747847581"} Oct 30 12:45:56.709285 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:56.709261 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e337e7b-qbvgq" Oct 30 12:45:56.813142 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:56.813050 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99914eaa-84d9-4fc3-a4e2-7c951974eea1-kube-api-access-jvf2n\" (UniqueName: \"kubernetes.io/projected/99914eaa-84d9-4fc3-a4e2-7c951974eea1-kube-api-access-jvf2n\") pod \"99914eaa-84d9-4fc3-a4e2-7c951974eea1\" (UID: \"99914eaa-84d9-4fc3-a4e2-7c951974eea1\") " Oct 30 12:45:56.813345 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:56.813218 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/99914eaa-84d9-4fc3-a4e2-7c951974eea1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99914eaa-84d9-4fc3-a4e2-7c951974eea1-maas-api-xks-ca\") pod \"99914eaa-84d9-4fc3-a4e2-7c951974eea1\" (UID: \"99914eaa-84d9-4fc3-a4e2-7c951974eea1\") " Oct 30 12:45:56.813345 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:56.813298 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99914eaa-84d9-4fc3-a4e2-7c951974eea1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99914eaa-84d9-4fc3-a4e2-7c951974eea1-maas-api-openshift-ca\") pod \"99914eaa-84d9-4fc3-a4e2-7c951974eea1\" (UID: \"99914eaa-84d9-4fc3-a4e2-7c951974eea1\") " Oct 30 12:45:56.813885 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:56.813848 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99914eaa-84d9-4fc3-a4e2-7c951974eea1-maas-api-openshift-ca" pod "99914eaa-84d9-4fc3-a4e2-7c951974eea1" (UID: "99914eaa-84d9-4fc3-a4e2-7c951974eea1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:45:56.815370 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:56.815343 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99914eaa-84d9-4fc3-a4e2-7c951974eea1-kube-api-access-jvf2n" pod "99914eaa-84d9-4fc3-a4e2-7c951974eea1" (UID: "99914eaa-84d9-4fc3-a4e2-7c951974eea1"). InnerVolumeSpecName "kube-api-access-jvf2n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:45:56.815541 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:56.815514 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99914eaa-84d9-4fc3-a4e2-7c951974eea1-maas-api-xks-ca" pod "99914eaa-84d9-4fc3-a4e2-7c951974eea1" (UID: "99914eaa-84d9-4fc3-a4e2-7c951974eea1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:45:56.929806 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:56.929761 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99914eaa-84d9-4fc3-a4e2-7c951974eea1-maas-api-xks-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:45:56.929806 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:56.929802 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99914eaa-84d9-4fc3-a4e2-7c951974eea1-maas-api-openshift-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:45:56.930123 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:56.929819 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvf2n\" (UniqueName: \"kubernetes.io/projected/99914eaa-84d9-4fc3-a4e2-7c951974eea1-kube-api-access-jvf2n\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:45:57.577162 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:57.577122 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e337e7b-qbvgq" event={"ID":"99914eaa-84d9-4fc3-a4e2-7c951974eea1","Type":"ContainerDied","Data":"7cdfc09a4386fb00d68f5c08032d3e7ece3b63a5cdf4190ee0f12b78509d6095"} Oct 30 12:45:57.577413 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:57.577392 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7cdfc09a4386fb00d68f5c08032d3e7ece3b63a5cdf4190ee0f12b78509d6095" Oct 30 12:45:57.577555 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:45:57.577475 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e337e7b-qbvgq" Oct 30 12:46:02.058354 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:46:02.058322 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4afe31ef-bd98f"] Oct 30 12:46:02.067307 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:46:02.067283 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4afe31ef-bd98f"] Oct 30 12:46:03.609063 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:46:03.609024 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ce4096e-12a9-4293-9c47-6585f6ead95f" path="/var/lib/kubelet/pods/7ce4096e-12a9-4293-9c47-6585f6ead95f/volumes" Oct 30 12:46:05.033895 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:46:05.033860 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5021d04-xfvtm"] Oct 30 12:46:05.041778 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:46:05.041749 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5021d04-xfvtm"] Oct 30 12:46:05.608165 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:46:05.608128 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a437fd08-0b7b-47c2-a47a-36ce86fc62ff" path="/var/lib/kubelet/pods/a437fd08-0b7b-47c2-a47a-36ce86fc62ff/volumes" Oct 30 12:46:13.035362 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:46:13.035322 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f84120f4-vm445"] Oct 30 12:46:13.046121 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:46:13.046090 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f84120f4-vm445"] Oct 30 12:46:13.608002 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:46:13.607970 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37837576-7f1a-4341-b1b3-71a4e74afa81" path="/var/lib/kubelet/pods/37837576-7f1a-4341-b1b3-71a4e74afa81/volumes" Oct 30 12:46:16.035086 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:46:16.035050 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-751c30fd-lrtv2"] Oct 30 12:46:16.045530 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:46:16.045499 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-751c30fd-lrtv2"] Oct 30 12:46:17.609939 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:46:17.609901 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98e23727-b5b0-4c90-844a-d732b3e001de" path="/var/lib/kubelet/pods/98e23727-b5b0-4c90-844a-d732b3e001de/volumes" Oct 30 12:46:19.686478 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:46:19.686422 4908 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vn6vv" containerName="extractor" Oct 30 12:46:26.835567 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:46:26.835530 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-fddaa3-openshift-default-668fdb7bf-z4588"] Oct 30 12:46:26.837109 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:46:26.837079 4908 cpu_manager.go:414] "containerMap: removing container" podUID="99914eaa-84d9-4fc3-a4e2-7c951974eea1" containerName="revoke-keys" Oct 30 12:46:26.837271 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:46:26.837258 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99914eaa-84d9-4fc3-a4e2-7c951974eea1" containerName="revoke-keys" Oct 30 12:46:26.837553 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:46:26.837538 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="99914eaa-84d9-4fc3-a4e2-7c951974eea1" containerName="revoke-keys" Oct 30 12:46:26.848872 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:46:26.848710 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-fddaa3-openshift-default-668fdb7bf-z4588" Oct 30 12:46:26.895849 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:46:26.893109 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-fddaa3-openshift-default-dockercfg-qkljf\"" Oct 30 12:46:26.909253 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:46:26.909210 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-fddaa3-openshift-default-668fdb7bf-z4588"] Oct 30 12:46:26.953940 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:46:26.953910 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab9d645e-4df1-482d-9ee8-54947745d8b4-istio-podinfo\") pod \"e2e-extmod-b-fddaa3-openshift-default-668fdb7bf-z4588\" (UID: \"ab9d645e-4df1-482d-9ee8-54947745d8b4\") " pod="openshift-ingress/e2e-extmod-b-fddaa3-openshift-default-668fdb7bf-z4588" Oct 30 12:46:26.954192 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:46:26.954177 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab9d645e-4df1-482d-9ee8-54947745d8b4-workload-socket\") pod \"e2e-extmod-b-fddaa3-openshift-default-668fdb7bf-z4588\" (UID: \"ab9d645e-4df1-482d-9ee8-54947745d8b4\") " pod="openshift-ingress/e2e-extmod-b-fddaa3-openshift-default-668fdb7bf-z4588" Oct 30 12:46:26.954283 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:46:26.954274 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab9d645e-4df1-482d-9ee8-54947745d8b4-istiod-ca-cert\") pod \"e2e-extmod-b-fddaa3-openshift-default-668fdb7bf-z4588\" (UID: \"ab9d645e-4df1-482d-9ee8-54947745d8b4\") " pod="openshift-ingress/e2e-extmod-b-fddaa3-openshift-default-668fdb7bf-z4588" Oct 30 12:46:26.954441 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:46:26.954416 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dchfm\" (UniqueName: \"kubernetes.io/projected/ab9d645e-4df1-482d-9ee8-54947745d8b4-kube-api-access-dchfm\") pod \"e2e-extmod-b-fddaa3-openshift-default-668fdb7bf-z4588\" (UID: \"ab9d645e-4df1-482d-9ee8-54947745d8b4\") " pod="openshift-ingress/e2e-extmod-b-fddaa3-openshift-default-668fdb7bf-z4588" Oct 30 12:46:26.954508 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:46:26.954462 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab9d645e-4df1-482d-9ee8-54947745d8b4-istio-data\") pod \"e2e-extmod-b-fddaa3-openshift-default-668fdb7bf-z4588\" (UID: \"ab9d645e-4df1-482d-9ee8-54947745d8b4\") " pod="openshift-ingress/e2e-extmod-b-fddaa3-openshift-default-668fdb7bf-z4588" Oct 30 12:46:26.954545 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:46:26.954525 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab9d645e-4df1-482d-9ee8-54947745d8b4-workload-certs\") pod \"e2e-extmod-b-fddaa3-openshift-default-668fdb7bf-z4588\" (UID: \"ab9d645e-4df1-482d-9ee8-54947745d8b4\") " pod="openshift-ingress/e2e-extmod-b-fddaa3-openshift-default-668fdb7bf-z4588" Oct 30 12:46:26.954584 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:46:26.954563 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab9d645e-4df1-482d-9ee8-54947745d8b4-istio-token\") pod \"e2e-extmod-b-fddaa3-openshift-default-668fdb7bf-z4588\" (UID: \"ab9d645e-4df1-482d-9ee8-54947745d8b4\") " pod="openshift-ingress/e2e-extmod-b-fddaa3-openshift-default-668fdb7bf-z4588" Oct 30 12:46:26.954632 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:46:26.954620 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab9d645e-4df1-482d-9ee8-54947745d8b4-istio-envoy\") pod \"e2e-extmod-b-fddaa3-openshift-default-668fdb7bf-z4588\" (UID: \"ab9d645e-4df1-482d-9ee8-54947745d8b4\") " pod="openshift-ingress/e2e-extmod-b-fddaa3-openshift-default-668fdb7bf-z4588" Oct 30 12:46:26.954669 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:46:26.954644 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab9d645e-4df1-482d-9ee8-54947745d8b4-credential-socket\") pod \"e2e-extmod-b-fddaa3-openshift-default-668fdb7bf-z4588\" (UID: \"ab9d645e-4df1-482d-9ee8-54947745d8b4\") " pod="openshift-ingress/e2e-extmod-b-fddaa3-openshift-default-668fdb7bf-z4588" Oct 30 12:46:27.061851 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:46:27.057602 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab9d645e-4df1-482d-9ee8-54947745d8b4-credential-socket\") pod \"e2e-extmod-b-fddaa3-openshift-default-668fdb7bf-z4588\" (UID: \"ab9d645e-4df1-482d-9ee8-54947745d8b4\") " pod="openshift-ingress/e2e-extmod-b-fddaa3-openshift-default-668fdb7bf-z4588" Oct 30 12:46:27.061851 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:46:27.057771 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab9d645e-4df1-482d-9ee8-54947745d8b4-istio-podinfo\") pod \"e2e-extmod-b-fddaa3-openshift-default-668fdb7bf-z4588\" (UID: \"ab9d645e-4df1-482d-9ee8-54947745d8b4\") " pod="openshift-ingress/e2e-extmod-b-fddaa3-openshift-default-668fdb7bf-z4588" Oct 30 12:46:27.061851 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:46:27.057809 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab9d645e-4df1-482d-9ee8-54947745d8b4-workload-socket\") pod \"e2e-extmod-b-fddaa3-openshift-default-668fdb7bf-z4588\" (UID: \"ab9d645e-4df1-482d-9ee8-54947745d8b4\") " pod="openshift-ingress/e2e-extmod-b-fddaa3-openshift-default-668fdb7bf-z4588" Oct 30 12:46:27.061851 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:46:27.057866 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab9d645e-4df1-482d-9ee8-54947745d8b4-istiod-ca-cert\") pod \"e2e-extmod-b-fddaa3-openshift-default-668fdb7bf-z4588\" (UID: \"ab9d645e-4df1-482d-9ee8-54947745d8b4\") " pod="openshift-ingress/e2e-extmod-b-fddaa3-openshift-default-668fdb7bf-z4588" Oct 30 12:46:27.061851 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:46:27.057903 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dchfm\" (UniqueName: \"kubernetes.io/projected/ab9d645e-4df1-482d-9ee8-54947745d8b4-kube-api-access-dchfm\") pod \"e2e-extmod-b-fddaa3-openshift-default-668fdb7bf-z4588\" (UID: \"ab9d645e-4df1-482d-9ee8-54947745d8b4\") " pod="openshift-ingress/e2e-extmod-b-fddaa3-openshift-default-668fdb7bf-z4588" Oct 30 12:46:27.061851 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:46:27.057949 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab9d645e-4df1-482d-9ee8-54947745d8b4-istio-data\") pod \"e2e-extmod-b-fddaa3-openshift-default-668fdb7bf-z4588\" (UID: \"ab9d645e-4df1-482d-9ee8-54947745d8b4\") " pod="openshift-ingress/e2e-extmod-b-fddaa3-openshift-default-668fdb7bf-z4588" Oct 30 12:46:27.061851 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:46:27.058041 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab9d645e-4df1-482d-9ee8-54947745d8b4-workload-certs\") pod \"e2e-extmod-b-fddaa3-openshift-default-668fdb7bf-z4588\" (UID: \"ab9d645e-4df1-482d-9ee8-54947745d8b4\") " pod="openshift-ingress/e2e-extmod-b-fddaa3-openshift-default-668fdb7bf-z4588" Oct 30 12:46:27.061851 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:46:27.058105 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab9d645e-4df1-482d-9ee8-54947745d8b4-istio-token\") pod \"e2e-extmod-b-fddaa3-openshift-default-668fdb7bf-z4588\" (UID: \"ab9d645e-4df1-482d-9ee8-54947745d8b4\") " pod="openshift-ingress/e2e-extmod-b-fddaa3-openshift-default-668fdb7bf-z4588" Oct 30 12:46:27.061851 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:46:27.058172 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab9d645e-4df1-482d-9ee8-54947745d8b4-istio-envoy\") pod \"e2e-extmod-b-fddaa3-openshift-default-668fdb7bf-z4588\" (UID: \"ab9d645e-4df1-482d-9ee8-54947745d8b4\") " pod="openshift-ingress/e2e-extmod-b-fddaa3-openshift-default-668fdb7bf-z4588" Oct 30 12:46:27.061851 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:46:27.059434 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab9d645e-4df1-482d-9ee8-54947745d8b4-istiod-ca-cert\") pod \"e2e-extmod-b-fddaa3-openshift-default-668fdb7bf-z4588\" (UID: \"ab9d645e-4df1-482d-9ee8-54947745d8b4\") " pod="openshift-ingress/e2e-extmod-b-fddaa3-openshift-default-668fdb7bf-z4588" Oct 30 12:46:27.061851 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:46:27.059702 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab9d645e-4df1-482d-9ee8-54947745d8b4-credential-socket\") pod \"e2e-extmod-b-fddaa3-openshift-default-668fdb7bf-z4588\" (UID: \"ab9d645e-4df1-482d-9ee8-54947745d8b4\") " pod="openshift-ingress/e2e-extmod-b-fddaa3-openshift-default-668fdb7bf-z4588" Oct 30 12:46:27.061851 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:46:27.060312 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab9d645e-4df1-482d-9ee8-54947745d8b4-istio-data\") pod \"e2e-extmod-b-fddaa3-openshift-default-668fdb7bf-z4588\" (UID: \"ab9d645e-4df1-482d-9ee8-54947745d8b4\") " pod="openshift-ingress/e2e-extmod-b-fddaa3-openshift-default-668fdb7bf-z4588" Oct 30 12:46:27.081956 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:46:27.081915 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab9d645e-4df1-482d-9ee8-54947745d8b4-workload-certs\") pod \"e2e-extmod-b-fddaa3-openshift-default-668fdb7bf-z4588\" (UID: \"ab9d645e-4df1-482d-9ee8-54947745d8b4\") " pod="openshift-ingress/e2e-extmod-b-fddaa3-openshift-default-668fdb7bf-z4588" Oct 30 12:46:27.083371 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:46:27.083346 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab9d645e-4df1-482d-9ee8-54947745d8b4-workload-socket\") pod \"e2e-extmod-b-fddaa3-openshift-default-668fdb7bf-z4588\" (UID: \"ab9d645e-4df1-482d-9ee8-54947745d8b4\") " pod="openshift-ingress/e2e-extmod-b-fddaa3-openshift-default-668fdb7bf-z4588" Oct 30 12:46:27.095928 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:46:27.086873 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab9d645e-4df1-482d-9ee8-54947745d8b4-istio-podinfo\") pod \"e2e-extmod-b-fddaa3-openshift-default-668fdb7bf-z4588\" (UID: \"ab9d645e-4df1-482d-9ee8-54947745d8b4\") " pod="openshift-ingress/e2e-extmod-b-fddaa3-openshift-default-668fdb7bf-z4588" Oct 30 12:46:27.096130 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:46:27.088027 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dchfm\" (UniqueName: \"kubernetes.io/projected/ab9d645e-4df1-482d-9ee8-54947745d8b4-kube-api-access-dchfm\") pod \"e2e-extmod-b-fddaa3-openshift-default-668fdb7bf-z4588\" (UID: \"ab9d645e-4df1-482d-9ee8-54947745d8b4\") " pod="openshift-ingress/e2e-extmod-b-fddaa3-openshift-default-668fdb7bf-z4588" Oct 30 12:46:27.120399 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:46:27.120345 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab9d645e-4df1-482d-9ee8-54947745d8b4-istio-envoy\") pod \"e2e-extmod-b-fddaa3-openshift-default-668fdb7bf-z4588\" (UID: \"ab9d645e-4df1-482d-9ee8-54947745d8b4\") " pod="openshift-ingress/e2e-extmod-b-fddaa3-openshift-default-668fdb7bf-z4588" Oct 30 12:46:27.176615 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:46:27.176581 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab9d645e-4df1-482d-9ee8-54947745d8b4-istio-token\") pod \"e2e-extmod-b-fddaa3-openshift-default-668fdb7bf-z4588\" (UID: \"ab9d645e-4df1-482d-9ee8-54947745d8b4\") " pod="openshift-ingress/e2e-extmod-b-fddaa3-openshift-default-668fdb7bf-z4588" Oct 30 12:46:27.217619 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:46:27.217586 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-fddaa3-openshift-default-668fdb7bf-z4588" Oct 30 12:46:27.694469 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:46:27.694370 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-fddaa3-openshift-default-668fdb7bf-z4588"] Oct 30 12:46:27.725717 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:46:27.722311 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab9d645e_4df1_482d_9ee8_54947745d8b4.slice/crio-54162980ea7dc99721dcbc57c5fa3de5fc46b51a99e2f298f309cea9306fdccf WatchSource:0}: Error finding container 54162980ea7dc99721dcbc57c5fa3de5fc46b51a99e2f298f309cea9306fdccf: Status 404 returned error can't find the container with id 54162980ea7dc99721dcbc57c5fa3de5fc46b51a99e2f298f309cea9306fdccf Oct 30 12:46:27.736328 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:46:27.731130 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:46:27.736328 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:46:27.731230 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:46:27.736328 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:46:27.731275 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:46:28.046155 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:46:28.046113 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-fddaa3-openshift-default-668fdb7bf-z4588" event={"ID":"ab9d645e-4df1-482d-9ee8-54947745d8b4","Type":"ContainerStarted","Data":"b4b52188906f02662aa52c0d5c332567b35ee8b165722696d6808bcc37db32ec"} Oct 30 12:46:28.046808 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:46:28.046784 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-fddaa3-openshift-default-668fdb7bf-z4588" event={"ID":"ab9d645e-4df1-482d-9ee8-54947745d8b4","Type":"ContainerStarted","Data":"54162980ea7dc99721dcbc57c5fa3de5fc46b51a99e2f298f309cea9306fdccf"} Oct 30 12:46:28.094284 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:46:28.094221 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-fddaa3-openshift-default-668fdb7bf-z4588" podStartSLOduration=2.09420355 podStartE2EDuration="2.09420355s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:46:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:46:28.093031827 +0000 UTC m=+3144.607460599" watchObservedRunningTime="2026-10-30 12:46:28.09420355 +0000 UTC m=+3144.608632297" Oct 30 12:46:28.226588 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:46:28.222232 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-fddaa3-openshift-default-668fdb7bf-z4588" Oct 30 12:46:28.247857 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:46:28.245506 4908 patch_prober.go:29] interesting pod/e2e-extmod-b-fddaa3-openshift-default-668fdb7bf-z4588 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.4:15021/healthz/ready\": dial tcp 10.132.1.4:15021: connect: connection refused" start-of-body= Oct 30 12:46:28.247857 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:46:28.245576 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-fddaa3-openshift-default-668fdb7bf-z4588" podUID="ab9d645e-4df1-482d-9ee8-54947745d8b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.4:15021/healthz/ready\": dial tcp 10.132.1.4:15021: connect: connection refused" Oct 30 12:46:29.224237 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:46:29.224204 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-fddaa3-openshift-default-668fdb7bf-z4588" Oct 30 12:46:30.064432 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:46:30.064399 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-fddaa3-openshift-default-668fdb7bf-z4588" Oct 30 12:46:30.065515 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:46:30.065487 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-fddaa3-openshift-default-668fdb7bf-z4588" Oct 30 12:46:31.414250 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:46:31.414214 4908 patch_prober.go:29] interesting pod/e2e-extmod-a-10f364-openshift-default-965d4bfd5-4b57q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.239:15021/healthz/ready\": read tcp 10.132.0.2:42184->10.132.0.239:15021: read: connection reset by peer" start-of-body= Oct 30 12:46:31.414690 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:46:31.414286 4908 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-10f364-openshift-default-965d4bfd5-4b57q" podUID="aa19be03-336a-4af4-aba5-e089b4933bf9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.239:15021/healthz/ready\": read tcp 10.132.0.2:42184->10.132.0.239:15021: read: connection reset by peer" Oct 30 12:46:32.087547 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:46:32.087508 4908 generic.go:356] "Generic (PLEG): container finished" podID="aa19be03-336a-4af4-aba5-e089b4933bf9" containerID="240af9c35b77a33c0f8b0ff549be18a2e92490e84c801042f95eed7a94cdc084" exitCode=0 Oct 30 12:46:32.087803 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:46:32.087586 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-10f364-openshift-default-965d4bfd5-4b57q" event={"ID":"aa19be03-336a-4af4-aba5-e089b4933bf9","Type":"ContainerDied","Data":"240af9c35b77a33c0f8b0ff549be18a2e92490e84c801042f95eed7a94cdc084"} Oct 30 12:46:32.088149 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:46:32.088131 4908 scope.go:122] "RemoveContainer" containerID="240af9c35b77a33c0f8b0ff549be18a2e92490e84c801042f95eed7a94cdc084" Oct 30 12:46:32.090331 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:46:32.090297 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:46:32.090423 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:46:32.090384 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:46:32.090466 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:46:32.090433 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:46:33.098987 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:46:33.098949 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-10f364-openshift-default-965d4bfd5-4b57q" event={"ID":"aa19be03-336a-4af4-aba5-e089b4933bf9","Type":"ContainerStarted","Data":"0ee7336c32c90ca843b02e706f558154f455e05400d519702b8ca0f83f6b8058"} Oct 30 12:46:33.998801 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:46:33.998758 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-10f364-openshift-default-965d4bfd5-4b57q" Oct 30 12:46:33.999012 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:46:33.998802 4908 patch_prober.go:29] interesting pod/e2e-extmod-a-10f364-openshift-default-965d4bfd5-4b57q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" start-of-body= Oct 30 12:46:33.999012 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:46:33.998880 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-10f364-openshift-default-965d4bfd5-4b57q" podUID="aa19be03-336a-4af4-aba5-e089b4933bf9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" Oct 30 12:46:34.996363 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:46:34.996327 4908 patch_prober.go:29] interesting pod/e2e-extmod-a-10f364-openshift-default-965d4bfd5-4b57q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" start-of-body= Oct 30 12:46:34.996771 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:46:34.996416 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-10f364-openshift-default-965d4bfd5-4b57q" podUID="aa19be03-336a-4af4-aba5-e089b4933bf9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" Oct 30 12:46:35.995568 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:46:35.995534 4908 patch_prober.go:29] interesting pod/e2e-extmod-a-10f364-openshift-default-965d4bfd5-4b57q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" start-of-body= Oct 30 12:46:35.995808 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:46:35.995600 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-10f364-openshift-default-965d4bfd5-4b57q" podUID="aa19be03-336a-4af4-aba5-e089b4933bf9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" Oct 30 12:46:36.996400 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:46:36.996355 4908 patch_prober.go:29] interesting pod/e2e-extmod-a-10f364-openshift-default-965d4bfd5-4b57q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" start-of-body= Oct 30 12:46:36.996911 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:46:36.996426 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-10f364-openshift-default-965d4bfd5-4b57q" podUID="aa19be03-336a-4af4-aba5-e089b4933bf9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" Oct 30 12:46:37.996354 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:46:37.996308 4908 patch_prober.go:29] interesting pod/e2e-extmod-a-10f364-openshift-default-965d4bfd5-4b57q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" start-of-body= Oct 30 12:46:37.996566 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:46:37.996391 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-10f364-openshift-default-965d4bfd5-4b57q" podUID="aa19be03-336a-4af4-aba5-e089b4933bf9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" Oct 30 12:46:39.000939 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:46:39.000908 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-10f364-openshift-default-965d4bfd5-4b57q" Oct 30 12:46:39.001429 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:46:39.001303 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-10f364-openshift-default-965d4bfd5-4b57q" Oct 30 12:46:39.001926 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:46:39.001904 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-10f364-openshift-default-965d4bfd5-4b57q" Oct 30 12:46:49.684909 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:46:49.684867 4908 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vn6vv" containerName="extractor" Oct 30 12:46:52.632475 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:46:52.632440 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9wwfc"] Oct 30 12:47:06.140633 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:47:06.140558 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86b846ba-xwqcm"] Oct 30 12:47:06.144573 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:47:06.144554 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86b846ba-xwqcm" Oct 30 12:47:06.158327 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:47:06.158302 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 12:47:06.164476 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:47:06.164454 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86b846ba-xwqcm"] Oct 30 12:47:06.169608 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:47:06.169587 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 12:47:06.169747 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:47:06.169722 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s75hq\"" Oct 30 12:47:06.169747 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:47:06.169603 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 12:47:06.240312 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:47:06.240285 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcjvq\" (UniqueName: \"kubernetes.io/projected/97e643d2-7ef6-40ff-8299-ef78b8038b08-kube-api-access-qcjvq\") pod \"maas-api-revoke-sub-86b846ba-xwqcm\" (UID: \"97e643d2-7ef6-40ff-8299-ef78b8038b08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86b846ba-xwqcm" Oct 30 12:47:06.240480 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:47:06.240353 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97e643d2-7ef6-40ff-8299-ef78b8038b08-maas-api-xks-ca\") pod \"maas-api-revoke-sub-86b846ba-xwqcm\" (UID: \"97e643d2-7ef6-40ff-8299-ef78b8038b08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86b846ba-xwqcm" Oct 30 12:47:06.240561 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:47:06.240516 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97e643d2-7ef6-40ff-8299-ef78b8038b08-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-86b846ba-xwqcm\" (UID: \"97e643d2-7ef6-40ff-8299-ef78b8038b08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86b846ba-xwqcm" Oct 30 12:47:06.342024 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:47:06.341993 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97e643d2-7ef6-40ff-8299-ef78b8038b08-maas-api-xks-ca\") pod \"maas-api-revoke-sub-86b846ba-xwqcm\" (UID: \"97e643d2-7ef6-40ff-8299-ef78b8038b08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86b846ba-xwqcm" Oct 30 12:47:06.342188 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:47:06.342096 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97e643d2-7ef6-40ff-8299-ef78b8038b08-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-86b846ba-xwqcm\" (UID: \"97e643d2-7ef6-40ff-8299-ef78b8038b08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86b846ba-xwqcm" Oct 30 12:47:06.342259 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:47:06.342247 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcjvq\" (UniqueName: \"kubernetes.io/projected/97e643d2-7ef6-40ff-8299-ef78b8038b08-kube-api-access-qcjvq\") pod \"maas-api-revoke-sub-86b846ba-xwqcm\" (UID: \"97e643d2-7ef6-40ff-8299-ef78b8038b08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86b846ba-xwqcm" Oct 30 12:47:06.342800 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:47:06.342781 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97e643d2-7ef6-40ff-8299-ef78b8038b08-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-86b846ba-xwqcm\" (UID: \"97e643d2-7ef6-40ff-8299-ef78b8038b08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86b846ba-xwqcm" Oct 30 12:47:06.344366 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:47:06.344349 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97e643d2-7ef6-40ff-8299-ef78b8038b08-maas-api-xks-ca\") pod \"maas-api-revoke-sub-86b846ba-xwqcm\" (UID: \"97e643d2-7ef6-40ff-8299-ef78b8038b08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86b846ba-xwqcm" Oct 30 12:47:06.364145 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:47:06.364123 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcjvq\" (UniqueName: \"kubernetes.io/projected/97e643d2-7ef6-40ff-8299-ef78b8038b08-kube-api-access-qcjvq\") pod \"maas-api-revoke-sub-86b846ba-xwqcm\" (UID: \"97e643d2-7ef6-40ff-8299-ef78b8038b08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86b846ba-xwqcm" Oct 30 12:47:06.458982 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:47:06.458924 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86b846ba-xwqcm" Oct 30 12:47:06.773526 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:47:06.773435 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86b846ba-xwqcm"] Oct 30 12:47:07.419417 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:47:07.419335 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86b846ba-xwqcm" event={"ID":"97e643d2-7ef6-40ff-8299-ef78b8038b08","Type":"ContainerStarted","Data":"2000905ecd1bf85ac0362f6bc10ada04033504231cc4ebfdc922f3a35d42a28e"} Oct 30 12:47:07.419417 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:47:07.419391 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86b846ba-xwqcm" event={"ID":"97e643d2-7ef6-40ff-8299-ef78b8038b08","Type":"ContainerStarted","Data":"d2ddf66ee6fc04ab38a1273cd5b6cc6875058221958000ec33b89e25ed063688"} Oct 30 12:47:07.459987 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:47:07.459936 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86b846ba-xwqcm" podStartSLOduration=1.459917089 podStartE2EDuration="1.459917089s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:47:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:47:07.456539782 +0000 UTC m=+3183.970968522" watchObservedRunningTime="2026-10-30 12:47:07.459917089 +0000 UTC m=+3183.974345836" Oct 30 12:47:07.838163 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:47:07.838129 4908 scope.go:122] "RemoveContainer" containerID="37797f2216adf38d654723ffe76577870ba8031b30f35dc3ed0281fbafd3835a" Oct 30 12:47:07.849708 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:47:07.849681 4908 scope.go:122] "RemoveContainer" containerID="2ccf30533e97c1fb76834692c8a0ffa8df453842fafa38ec22e91d70d9bb2c66" Oct 30 12:47:07.858894 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:47:07.858816 4908 scope.go:122] "RemoveContainer" containerID="f5ef0ed8a525c358973d9cfabd5198a23117eaf13deb113916941bbea7be4a29" Oct 30 12:47:07.869905 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:47:07.869884 4908 scope.go:122] "RemoveContainer" containerID="3e9498d5c01d69e185e4252e6022d7ae3633c42042aea138817c6e520bd17a45" Oct 30 12:47:07.880861 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:47:07.880821 4908 scope.go:122] "RemoveContainer" containerID="3e52c5ada6cc9bd63d869e6c56413e7d0c7cca4feef736264bb277d2823cadce" Oct 30 12:47:07.891464 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:47:07.891447 4908 scope.go:122] "RemoveContainer" containerID="5033b8b333aa8c77bc2396964e434e9f681e968a557dd9d91142cd168718b50a" Oct 30 12:47:08.428075 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:47:08.428036 4908 generic.go:356] "Generic (PLEG): container finished" podID="97e643d2-7ef6-40ff-8299-ef78b8038b08" containerID="2000905ecd1bf85ac0362f6bc10ada04033504231cc4ebfdc922f3a35d42a28e" exitCode=0 Oct 30 12:47:08.428481 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:47:08.428117 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86b846ba-xwqcm" event={"ID":"97e643d2-7ef6-40ff-8299-ef78b8038b08","Type":"ContainerDied","Data":"2000905ecd1bf85ac0362f6bc10ada04033504231cc4ebfdc922f3a35d42a28e"} Oct 30 12:47:09.602253 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:47:09.602222 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86b846ba-xwqcm" Oct 30 12:47:09.693475 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:47:09.693391 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97e643d2-7ef6-40ff-8299-ef78b8038b08-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97e643d2-7ef6-40ff-8299-ef78b8038b08-maas-api-openshift-ca\") pod \"97e643d2-7ef6-40ff-8299-ef78b8038b08\" (UID: \"97e643d2-7ef6-40ff-8299-ef78b8038b08\") " Oct 30 12:47:09.693653 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:47:09.693484 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97e643d2-7ef6-40ff-8299-ef78b8038b08-kube-api-access-qcjvq\" (UniqueName: \"kubernetes.io/projected/97e643d2-7ef6-40ff-8299-ef78b8038b08-kube-api-access-qcjvq\") pod \"97e643d2-7ef6-40ff-8299-ef78b8038b08\" (UID: \"97e643d2-7ef6-40ff-8299-ef78b8038b08\") " Oct 30 12:47:09.693653 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:47:09.693560 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/97e643d2-7ef6-40ff-8299-ef78b8038b08-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97e643d2-7ef6-40ff-8299-ef78b8038b08-maas-api-xks-ca\") pod \"97e643d2-7ef6-40ff-8299-ef78b8038b08\" (UID: \"97e643d2-7ef6-40ff-8299-ef78b8038b08\") " Oct 30 12:47:09.693895 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:47:09.693869 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97e643d2-7ef6-40ff-8299-ef78b8038b08-maas-api-openshift-ca" pod "97e643d2-7ef6-40ff-8299-ef78b8038b08" (UID: "97e643d2-7ef6-40ff-8299-ef78b8038b08"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:47:09.694525 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:47:09.694498 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97e643d2-7ef6-40ff-8299-ef78b8038b08-maas-api-openshift-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:47:09.695890 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:47:09.695818 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97e643d2-7ef6-40ff-8299-ef78b8038b08-kube-api-access-qcjvq" pod "97e643d2-7ef6-40ff-8299-ef78b8038b08" (UID: "97e643d2-7ef6-40ff-8299-ef78b8038b08"). InnerVolumeSpecName "kube-api-access-qcjvq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:47:09.696610 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:47:09.696587 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97e643d2-7ef6-40ff-8299-ef78b8038b08-maas-api-xks-ca" pod "97e643d2-7ef6-40ff-8299-ef78b8038b08" (UID: "97e643d2-7ef6-40ff-8299-ef78b8038b08"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:47:09.796350 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:47:09.796318 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qcjvq\" (UniqueName: \"kubernetes.io/projected/97e643d2-7ef6-40ff-8299-ef78b8038b08-kube-api-access-qcjvq\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:47:09.796350 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:47:09.796349 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97e643d2-7ef6-40ff-8299-ef78b8038b08-maas-api-xks-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:47:10.443080 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:47:10.443049 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86b846ba-xwqcm" Oct 30 12:47:10.443280 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:47:10.443090 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86b846ba-xwqcm" event={"ID":"97e643d2-7ef6-40ff-8299-ef78b8038b08","Type":"ContainerDied","Data":"d2ddf66ee6fc04ab38a1273cd5b6cc6875058221958000ec33b89e25ed063688"} Oct 30 12:47:10.443280 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:47:10.443136 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d2ddf66ee6fc04ab38a1273cd5b6cc6875058221958000ec33b89e25ed063688" Oct 30 12:47:18.060713 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:47:18.060617 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89926aa6-dfz48"] Oct 30 12:47:18.071536 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:47:18.071498 4908 cpu_manager.go:414] "containerMap: removing container" podUID="97e643d2-7ef6-40ff-8299-ef78b8038b08" containerName="revoke-keys" Oct 30 12:47:18.071757 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:47:18.071741 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97e643d2-7ef6-40ff-8299-ef78b8038b08" containerName="revoke-keys" Oct 30 12:47:18.072059 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:47:18.072040 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="97e643d2-7ef6-40ff-8299-ef78b8038b08" containerName="revoke-keys" Oct 30 12:47:18.098070 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:47:18.098034 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89926aa6-dfz48"] Oct 30 12:47:18.098421 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:47:18.098356 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89926aa6-dfz48" Oct 30 12:47:18.130167 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:47:18.130102 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 12:47:18.130167 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:47:18.130145 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 12:47:18.130615 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:47:18.130592 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 12:47:18.138858 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:47:18.130698 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s75hq\"" Oct 30 12:47:18.202431 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:47:18.200680 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5cdee26-990c-4f16-8152-2921cf65972d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-89926aa6-dfz48\" (UID: \"a5cdee26-990c-4f16-8152-2921cf65972d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89926aa6-dfz48" Oct 30 12:47:18.202431 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:47:18.200791 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjjgs\" (UniqueName: \"kubernetes.io/projected/a5cdee26-990c-4f16-8152-2921cf65972d-kube-api-access-zjjgs\") pod \"maas-api-revoke-sub-89926aa6-dfz48\" (UID: \"a5cdee26-990c-4f16-8152-2921cf65972d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89926aa6-dfz48" Oct 30 12:47:18.202431 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:47:18.201040 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5cdee26-990c-4f16-8152-2921cf65972d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-89926aa6-dfz48\" (UID: \"a5cdee26-990c-4f16-8152-2921cf65972d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89926aa6-dfz48" Oct 30 12:47:18.302434 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:47:18.302375 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5cdee26-990c-4f16-8152-2921cf65972d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-89926aa6-dfz48\" (UID: \"a5cdee26-990c-4f16-8152-2921cf65972d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89926aa6-dfz48" Oct 30 12:47:18.302647 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:47:18.302528 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5cdee26-990c-4f16-8152-2921cf65972d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-89926aa6-dfz48\" (UID: \"a5cdee26-990c-4f16-8152-2921cf65972d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89926aa6-dfz48" Oct 30 12:47:18.302647 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:47:18.302603 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjjgs\" (UniqueName: \"kubernetes.io/projected/a5cdee26-990c-4f16-8152-2921cf65972d-kube-api-access-zjjgs\") pod \"maas-api-revoke-sub-89926aa6-dfz48\" (UID: \"a5cdee26-990c-4f16-8152-2921cf65972d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89926aa6-dfz48" Oct 30 12:47:18.303193 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:47:18.303171 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5cdee26-990c-4f16-8152-2921cf65972d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-89926aa6-dfz48\" (UID: \"a5cdee26-990c-4f16-8152-2921cf65972d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89926aa6-dfz48" Oct 30 12:47:18.304909 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:47:18.304887 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5cdee26-990c-4f16-8152-2921cf65972d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-89926aa6-dfz48\" (UID: \"a5cdee26-990c-4f16-8152-2921cf65972d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89926aa6-dfz48" Oct 30 12:47:18.325220 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:47:18.325157 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjjgs\" (UniqueName: \"kubernetes.io/projected/a5cdee26-990c-4f16-8152-2921cf65972d-kube-api-access-zjjgs\") pod \"maas-api-revoke-sub-89926aa6-dfz48\" (UID: \"a5cdee26-990c-4f16-8152-2921cf65972d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89926aa6-dfz48" Oct 30 12:47:18.505028 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:47:18.504995 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89926aa6-dfz48" Oct 30 12:47:18.809663 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:47:18.809629 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89926aa6-dfz48"] Oct 30 12:47:18.812000 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:47:18.811965 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5cdee26_990c_4f16_8152_2921cf65972d.slice/crio-3f561d9e7e0979f9f3115ceedaba2d68a1a57c884a6ee8cdb6c7db6123117ba9 WatchSource:0}: Error finding container 3f561d9e7e0979f9f3115ceedaba2d68a1a57c884a6ee8cdb6c7db6123117ba9: Status 404 returned error can't find the container with id 3f561d9e7e0979f9f3115ceedaba2d68a1a57c884a6ee8cdb6c7db6123117ba9 Oct 30 12:47:18.830163 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:47:18.830114 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-394fe43e-gvzjl"] Oct 30 12:47:18.838258 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:47:18.838234 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-394fe43e-gvzjl" Oct 30 12:47:18.851604 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:47:18.851571 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-394fe43e-gvzjl"] Oct 30 12:47:18.913176 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:47:18.913135 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9c8a505-c26d-4cc0-86f1-8f4ae6095e3e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-394fe43e-gvzjl\" (UID: \"d9c8a505-c26d-4cc0-86f1-8f4ae6095e3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-394fe43e-gvzjl" Oct 30 12:47:18.913391 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:47:18.913231 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9c8a505-c26d-4cc0-86f1-8f4ae6095e3e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-394fe43e-gvzjl\" (UID: \"d9c8a505-c26d-4cc0-86f1-8f4ae6095e3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-394fe43e-gvzjl" Oct 30 12:47:18.913391 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:47:18.913368 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fg7b\" (UniqueName: \"kubernetes.io/projected/d9c8a505-c26d-4cc0-86f1-8f4ae6095e3e-kube-api-access-5fg7b\") pod \"maas-api-revoke-sub-394fe43e-gvzjl\" (UID: \"d9c8a505-c26d-4cc0-86f1-8f4ae6095e3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-394fe43e-gvzjl" Oct 30 12:47:19.015048 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:47:19.015010 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9c8a505-c26d-4cc0-86f1-8f4ae6095e3e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-394fe43e-gvzjl\" (UID: \"d9c8a505-c26d-4cc0-86f1-8f4ae6095e3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-394fe43e-gvzjl" Oct 30 12:47:19.015252 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:47:19.015123 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9c8a505-c26d-4cc0-86f1-8f4ae6095e3e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-394fe43e-gvzjl\" (UID: \"d9c8a505-c26d-4cc0-86f1-8f4ae6095e3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-394fe43e-gvzjl" Oct 30 12:47:19.015252 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:47:19.015217 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fg7b\" (UniqueName: \"kubernetes.io/projected/d9c8a505-c26d-4cc0-86f1-8f4ae6095e3e-kube-api-access-5fg7b\") pod \"maas-api-revoke-sub-394fe43e-gvzjl\" (UID: \"d9c8a505-c26d-4cc0-86f1-8f4ae6095e3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-394fe43e-gvzjl" Oct 30 12:47:19.015761 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:47:19.015734 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9c8a505-c26d-4cc0-86f1-8f4ae6095e3e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-394fe43e-gvzjl\" (UID: \"d9c8a505-c26d-4cc0-86f1-8f4ae6095e3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-394fe43e-gvzjl" Oct 30 12:47:19.017428 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:47:19.017408 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9c8a505-c26d-4cc0-86f1-8f4ae6095e3e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-394fe43e-gvzjl\" (UID: \"d9c8a505-c26d-4cc0-86f1-8f4ae6095e3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-394fe43e-gvzjl" Oct 30 12:47:19.037343 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:47:19.037306 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fg7b\" (UniqueName: \"kubernetes.io/projected/d9c8a505-c26d-4cc0-86f1-8f4ae6095e3e-kube-api-access-5fg7b\") pod \"maas-api-revoke-sub-394fe43e-gvzjl\" (UID: \"d9c8a505-c26d-4cc0-86f1-8f4ae6095e3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-394fe43e-gvzjl" Oct 30 12:47:19.160813 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:47:19.160768 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-394fe43e-gvzjl" Oct 30 12:47:19.469125 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:47:19.469093 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-394fe43e-gvzjl"] Oct 30 12:47:19.471100 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:47:19.471065 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9c8a505_c26d_4cc0_86f1_8f4ae6095e3e.slice/crio-884bb440b9c9449f52e213d034bb771e3c03cb97a1b9daa83ebaa630d728e55b WatchSource:0}: Error finding container 884bb440b9c9449f52e213d034bb771e3c03cb97a1b9daa83ebaa630d728e55b: Status 404 returned error can't find the container with id 884bb440b9c9449f52e213d034bb771e3c03cb97a1b9daa83ebaa630d728e55b Oct 30 12:47:19.590017 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:47:19.589987 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-394fe43e-gvzjl" event={"ID":"d9c8a505-c26d-4cc0-86f1-8f4ae6095e3e","Type":"ContainerStarted","Data":"884bb440b9c9449f52e213d034bb771e3c03cb97a1b9daa83ebaa630d728e55b"} Oct 30 12:47:19.592360 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:47:19.592331 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89926aa6-dfz48" event={"ID":"a5cdee26-990c-4f16-8152-2921cf65972d","Type":"ContainerStarted","Data":"67709732d2be3672e0343e8633ba838d007e9dc9d9fef77ad04ebc8070ec370c"} Oct 30 12:47:19.592555 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:47:19.592536 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89926aa6-dfz48" event={"ID":"a5cdee26-990c-4f16-8152-2921cf65972d","Type":"ContainerStarted","Data":"3f561d9e7e0979f9f3115ceedaba2d68a1a57c884a6ee8cdb6c7db6123117ba9"} Oct 30 12:47:19.632726 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:47:19.632683 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89926aa6-dfz48" podStartSLOduration=1.632670039 podStartE2EDuration="1.632670039s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:47:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:47:19.630176573 +0000 UTC m=+3196.144605321" watchObservedRunningTime="2026-10-30 12:47:19.632670039 +0000 UTC m=+3196.147098784" Oct 30 12:47:19.690114 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:47:19.690082 4908 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vn6vv" containerName="extractor" Oct 30 12:47:20.603946 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:47:20.603910 4908 generic.go:356] "Generic (PLEG): container finished" podID="d9c8a505-c26d-4cc0-86f1-8f4ae6095e3e" containerID="05024102700c527f00819ad3b088af4475d0efaf19cb52dad023cf303b2b98cf" exitCode=0 Oct 30 12:47:20.604411 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:47:20.604005 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-394fe43e-gvzjl" event={"ID":"d9c8a505-c26d-4cc0-86f1-8f4ae6095e3e","Type":"ContainerDied","Data":"05024102700c527f00819ad3b088af4475d0efaf19cb52dad023cf303b2b98cf"} Oct 30 12:47:20.606224 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:47:20.606199 4908 generic.go:356] "Generic (PLEG): container finished" podID="a5cdee26-990c-4f16-8152-2921cf65972d" containerID="67709732d2be3672e0343e8633ba838d007e9dc9d9fef77ad04ebc8070ec370c" exitCode=0 Oct 30 12:47:20.606376 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:47:20.606226 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89926aa6-dfz48" event={"ID":"a5cdee26-990c-4f16-8152-2921cf65972d","Type":"ContainerDied","Data":"67709732d2be3672e0343e8633ba838d007e9dc9d9fef77ad04ebc8070ec370c"} Oct 30 12:47:21.783176 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:47:21.783100 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89926aa6-dfz48" Oct 30 12:47:21.814001 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:47:21.813976 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-394fe43e-gvzjl" Oct 30 12:47:21.861612 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:47:21.861580 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d9c8a505-c26d-4cc0-86f1-8f4ae6095e3e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9c8a505-c26d-4cc0-86f1-8f4ae6095e3e-maas-api-xks-ca\") pod \"d9c8a505-c26d-4cc0-86f1-8f4ae6095e3e\" (UID: \"d9c8a505-c26d-4cc0-86f1-8f4ae6095e3e\") " Oct 30 12:47:21.861901 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:47:21.861872 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9c8a505-c26d-4cc0-86f1-8f4ae6095e3e-kube-api-access-5fg7b\" (UniqueName: \"kubernetes.io/projected/d9c8a505-c26d-4cc0-86f1-8f4ae6095e3e-kube-api-access-5fg7b\") pod \"d9c8a505-c26d-4cc0-86f1-8f4ae6095e3e\" (UID: \"d9c8a505-c26d-4cc0-86f1-8f4ae6095e3e\") " Oct 30 12:47:21.862063 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:47:21.862047 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a5cdee26-990c-4f16-8152-2921cf65972d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5cdee26-990c-4f16-8152-2921cf65972d-maas-api-xks-ca\") pod \"a5cdee26-990c-4f16-8152-2921cf65972d\" (UID: \"a5cdee26-990c-4f16-8152-2921cf65972d\") " Oct 30 12:47:21.862246 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:47:21.862218 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5cdee26-990c-4f16-8152-2921cf65972d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5cdee26-990c-4f16-8152-2921cf65972d-maas-api-openshift-ca\") pod \"a5cdee26-990c-4f16-8152-2921cf65972d\" (UID: \"a5cdee26-990c-4f16-8152-2921cf65972d\") " Oct 30 12:47:21.862434 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:47:21.862418 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5cdee26-990c-4f16-8152-2921cf65972d-kube-api-access-zjjgs\" (UniqueName: \"kubernetes.io/projected/a5cdee26-990c-4f16-8152-2921cf65972d-kube-api-access-zjjgs\") pod \"a5cdee26-990c-4f16-8152-2921cf65972d\" (UID: \"a5cdee26-990c-4f16-8152-2921cf65972d\") " Oct 30 12:47:21.862873 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:47:21.862666 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5cdee26-990c-4f16-8152-2921cf65972d-maas-api-openshift-ca" pod "a5cdee26-990c-4f16-8152-2921cf65972d" (UID: "a5cdee26-990c-4f16-8152-2921cf65972d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:47:21.863145 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:47:21.863115 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9c8a505-c26d-4cc0-86f1-8f4ae6095e3e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9c8a505-c26d-4cc0-86f1-8f4ae6095e3e-maas-api-openshift-ca\") pod \"d9c8a505-c26d-4cc0-86f1-8f4ae6095e3e\" (UID: \"d9c8a505-c26d-4cc0-86f1-8f4ae6095e3e\") " Oct 30 12:47:21.863670 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:47:21.863634 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9c8a505-c26d-4cc0-86f1-8f4ae6095e3e-maas-api-openshift-ca" pod "d9c8a505-c26d-4cc0-86f1-8f4ae6095e3e" (UID: "d9c8a505-c26d-4cc0-86f1-8f4ae6095e3e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:47:21.863897 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:47:21.863874 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5cdee26-990c-4f16-8152-2921cf65972d-maas-api-openshift-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:47:21.864021 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:47:21.863903 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9c8a505-c26d-4cc0-86f1-8f4ae6095e3e-maas-api-openshift-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:47:21.864739 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:47:21.864531 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9c8a505-c26d-4cc0-86f1-8f4ae6095e3e-kube-api-access-5fg7b" pod "d9c8a505-c26d-4cc0-86f1-8f4ae6095e3e" (UID: "d9c8a505-c26d-4cc0-86f1-8f4ae6095e3e"). InnerVolumeSpecName "kube-api-access-5fg7b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:47:21.864739 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:47:21.864704 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5cdee26-990c-4f16-8152-2921cf65972d-maas-api-xks-ca" pod "a5cdee26-990c-4f16-8152-2921cf65972d" (UID: "a5cdee26-990c-4f16-8152-2921cf65972d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:47:21.865238 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:47:21.865210 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5cdee26-990c-4f16-8152-2921cf65972d-kube-api-access-zjjgs" pod "a5cdee26-990c-4f16-8152-2921cf65972d" (UID: "a5cdee26-990c-4f16-8152-2921cf65972d"). InnerVolumeSpecName "kube-api-access-zjjgs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:47:21.865901 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:47:21.865881 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9c8a505-c26d-4cc0-86f1-8f4ae6095e3e-maas-api-xks-ca" pod "d9c8a505-c26d-4cc0-86f1-8f4ae6095e3e" (UID: "d9c8a505-c26d-4cc0-86f1-8f4ae6095e3e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:47:21.965311 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:47:21.965278 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5fg7b\" (UniqueName: \"kubernetes.io/projected/d9c8a505-c26d-4cc0-86f1-8f4ae6095e3e-kube-api-access-5fg7b\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:47:21.965311 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:47:21.965308 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5cdee26-990c-4f16-8152-2921cf65972d-maas-api-xks-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:47:21.965311 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:47:21.965319 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjjgs\" (UniqueName: \"kubernetes.io/projected/a5cdee26-990c-4f16-8152-2921cf65972d-kube-api-access-zjjgs\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:47:21.965575 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:47:21.965328 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9c8a505-c26d-4cc0-86f1-8f4ae6095e3e-maas-api-xks-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:47:22.622608 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:47:22.622576 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-394fe43e-gvzjl" Oct 30 12:47:22.622804 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:47:22.622604 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-394fe43e-gvzjl" event={"ID":"d9c8a505-c26d-4cc0-86f1-8f4ae6095e3e","Type":"ContainerDied","Data":"884bb440b9c9449f52e213d034bb771e3c03cb97a1b9daa83ebaa630d728e55b"} Oct 30 12:47:22.622804 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:47:22.622656 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="884bb440b9c9449f52e213d034bb771e3c03cb97a1b9daa83ebaa630d728e55b" Oct 30 12:47:22.624105 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:47:22.624085 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89926aa6-dfz48" Oct 30 12:47:22.624260 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:47:22.624129 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89926aa6-dfz48" event={"ID":"a5cdee26-990c-4f16-8152-2921cf65972d","Type":"ContainerDied","Data":"3f561d9e7e0979f9f3115ceedaba2d68a1a57c884a6ee8cdb6c7db6123117ba9"} Oct 30 12:47:22.624260 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:47:22.624178 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3f561d9e7e0979f9f3115ceedaba2d68a1a57c884a6ee8cdb6c7db6123117ba9" Oct 30 12:47:30.165451 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:47:30.165405 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e350e85-tzb72"] Oct 30 12:47:30.166002 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:47:30.165987 4908 cpu_manager.go:414] "containerMap: removing container" podUID="a5cdee26-990c-4f16-8152-2921cf65972d" containerName="revoke-keys" Oct 30 12:47:30.166058 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:47:30.166003 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5cdee26-990c-4f16-8152-2921cf65972d" containerName="revoke-keys" Oct 30 12:47:30.166058 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:47:30.166018 4908 cpu_manager.go:414] "containerMap: removing container" podUID="d9c8a505-c26d-4cc0-86f1-8f4ae6095e3e" containerName="revoke-keys" Oct 30 12:47:30.166058 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:47:30.166023 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9c8a505-c26d-4cc0-86f1-8f4ae6095e3e" containerName="revoke-keys" Oct 30 12:47:30.166160 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:47:30.166124 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5cdee26-990c-4f16-8152-2921cf65972d" containerName="revoke-keys" Oct 30 12:47:30.166160 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:47:30.166133 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9c8a505-c26d-4cc0-86f1-8f4ae6095e3e" containerName="revoke-keys" Oct 30 12:47:30.170572 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:47:30.170552 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e350e85-tzb72" Oct 30 12:47:30.189660 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:47:30.189633 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e350e85-tzb72"] Oct 30 12:47:30.196895 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:47:30.196279 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 12:47:30.196895 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:47:30.196313 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 12:47:30.196895 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:47:30.196343 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 12:47:30.196895 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:47:30.196365 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s75hq\"" Oct 30 12:47:30.301666 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:47:30.301627 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a9055d5-d3ba-4a50-8f2d-a363bbb17ff8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e350e85-tzb72\" (UID: \"3a9055d5-d3ba-4a50-8f2d-a363bbb17ff8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e350e85-tzb72" Oct 30 12:47:30.301862 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:47:30.301696 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bm5t9\" (UniqueName: \"kubernetes.io/projected/3a9055d5-d3ba-4a50-8f2d-a363bbb17ff8-kube-api-access-bm5t9\") pod \"maas-api-revoke-sub-7e350e85-tzb72\" (UID: \"3a9055d5-d3ba-4a50-8f2d-a363bbb17ff8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e350e85-tzb72" Oct 30 12:47:30.301945 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:47:30.301871 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a9055d5-d3ba-4a50-8f2d-a363bbb17ff8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e350e85-tzb72\" (UID: \"3a9055d5-d3ba-4a50-8f2d-a363bbb17ff8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e350e85-tzb72" Oct 30 12:47:30.403154 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:47:30.403118 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a9055d5-d3ba-4a50-8f2d-a363bbb17ff8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e350e85-tzb72\" (UID: \"3a9055d5-d3ba-4a50-8f2d-a363bbb17ff8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e350e85-tzb72" Oct 30 12:47:30.403363 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:47:30.403281 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a9055d5-d3ba-4a50-8f2d-a363bbb17ff8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e350e85-tzb72\" (UID: \"3a9055d5-d3ba-4a50-8f2d-a363bbb17ff8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e350e85-tzb72" Oct 30 12:47:30.403363 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:47:30.403325 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bm5t9\" (UniqueName: \"kubernetes.io/projected/3a9055d5-d3ba-4a50-8f2d-a363bbb17ff8-kube-api-access-bm5t9\") pod \"maas-api-revoke-sub-7e350e85-tzb72\" (UID: \"3a9055d5-d3ba-4a50-8f2d-a363bbb17ff8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e350e85-tzb72" Oct 30 12:47:30.403945 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:47:30.403916 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a9055d5-d3ba-4a50-8f2d-a363bbb17ff8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e350e85-tzb72\" (UID: \"3a9055d5-d3ba-4a50-8f2d-a363bbb17ff8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e350e85-tzb72" Oct 30 12:47:30.405818 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:47:30.405794 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a9055d5-d3ba-4a50-8f2d-a363bbb17ff8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e350e85-tzb72\" (UID: \"3a9055d5-d3ba-4a50-8f2d-a363bbb17ff8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e350e85-tzb72" Oct 30 12:47:30.445136 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:47:30.444714 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bm5t9\" (UniqueName: \"kubernetes.io/projected/3a9055d5-d3ba-4a50-8f2d-a363bbb17ff8-kube-api-access-bm5t9\") pod \"maas-api-revoke-sub-7e350e85-tzb72\" (UID: \"3a9055d5-d3ba-4a50-8f2d-a363bbb17ff8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e350e85-tzb72" Oct 30 12:47:30.486853 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:47:30.485078 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e350e85-tzb72" Oct 30 12:47:31.079856 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:47:31.079813 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e350e85-tzb72"] Oct 30 12:47:31.081706 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:47:31.081674 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a9055d5_d3ba_4a50_8f2d_a363bbb17ff8.slice/crio-5de8d16eea72d8aafe2a089e37fdf3c5c4ac18c79a9d93368d5c3c968ec35b78 WatchSource:0}: Error finding container 5de8d16eea72d8aafe2a089e37fdf3c5c4ac18c79a9d93368d5c3c968ec35b78: Status 404 returned error can't find the container with id 5de8d16eea72d8aafe2a089e37fdf3c5c4ac18c79a9d93368d5c3c968ec35b78 Oct 30 12:47:32.000789 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:47:32.000750 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e350e85-tzb72" event={"ID":"3a9055d5-d3ba-4a50-8f2d-a363bbb17ff8","Type":"ContainerStarted","Data":"6d8542d163b47141f5536fe0f273dbf7bdbbd4aaac4bcf18ed1bb5988bf63a0e"} Oct 30 12:47:32.001324 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:47:32.000872 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e350e85-tzb72" event={"ID":"3a9055d5-d3ba-4a50-8f2d-a363bbb17ff8","Type":"ContainerStarted","Data":"5de8d16eea72d8aafe2a089e37fdf3c5c4ac18c79a9d93368d5c3c968ec35b78"} Oct 30 12:47:32.042073 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:47:32.042023 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e350e85-tzb72" podStartSLOduration=2.04201069 podStartE2EDuration="2.04201069s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:47:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:47:32.038224099 +0000 UTC m=+3208.552652844" watchObservedRunningTime="2026-10-30 12:47:32.04201069 +0000 UTC m=+3208.556439436" Oct 30 12:47:33.020868 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:47:33.016548 4908 generic.go:356] "Generic (PLEG): container finished" podID="3a9055d5-d3ba-4a50-8f2d-a363bbb17ff8" containerID="6d8542d163b47141f5536fe0f273dbf7bdbbd4aaac4bcf18ed1bb5988bf63a0e" exitCode=0 Oct 30 12:47:33.020868 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:47:33.016642 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e350e85-tzb72" event={"ID":"3a9055d5-d3ba-4a50-8f2d-a363bbb17ff8","Type":"ContainerDied","Data":"6d8542d163b47141f5536fe0f273dbf7bdbbd4aaac4bcf18ed1bb5988bf63a0e"} Oct 30 12:47:34.414567 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:47:34.414542 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e350e85-tzb72" Oct 30 12:47:34.455132 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:47:34.455103 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a9055d5-d3ba-4a50-8f2d-a363bbb17ff8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a9055d5-d3ba-4a50-8f2d-a363bbb17ff8-maas-api-xks-ca\") pod \"3a9055d5-d3ba-4a50-8f2d-a363bbb17ff8\" (UID: \"3a9055d5-d3ba-4a50-8f2d-a363bbb17ff8\") " Oct 30 12:47:34.455278 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:47:34.455152 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a9055d5-d3ba-4a50-8f2d-a363bbb17ff8-kube-api-access-bm5t9\" (UniqueName: \"kubernetes.io/projected/3a9055d5-d3ba-4a50-8f2d-a363bbb17ff8-kube-api-access-bm5t9\") pod \"3a9055d5-d3ba-4a50-8f2d-a363bbb17ff8\" (UID: \"3a9055d5-d3ba-4a50-8f2d-a363bbb17ff8\") " Oct 30 12:47:34.455278 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:47:34.455252 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a9055d5-d3ba-4a50-8f2d-a363bbb17ff8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a9055d5-d3ba-4a50-8f2d-a363bbb17ff8-maas-api-openshift-ca\") pod \"3a9055d5-d3ba-4a50-8f2d-a363bbb17ff8\" (UID: \"3a9055d5-d3ba-4a50-8f2d-a363bbb17ff8\") " Oct 30 12:47:34.455957 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:47:34.455922 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a9055d5-d3ba-4a50-8f2d-a363bbb17ff8-maas-api-openshift-ca" pod "3a9055d5-d3ba-4a50-8f2d-a363bbb17ff8" (UID: "3a9055d5-d3ba-4a50-8f2d-a363bbb17ff8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:47:34.456173 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:47:34.456149 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a9055d5-d3ba-4a50-8f2d-a363bbb17ff8-maas-api-openshift-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:47:34.457823 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:47:34.457786 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a9055d5-d3ba-4a50-8f2d-a363bbb17ff8-maas-api-xks-ca" pod "3a9055d5-d3ba-4a50-8f2d-a363bbb17ff8" (UID: "3a9055d5-d3ba-4a50-8f2d-a363bbb17ff8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:47:34.458529 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:47:34.458503 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a9055d5-d3ba-4a50-8f2d-a363bbb17ff8-kube-api-access-bm5t9" pod "3a9055d5-d3ba-4a50-8f2d-a363bbb17ff8" (UID: "3a9055d5-d3ba-4a50-8f2d-a363bbb17ff8"). InnerVolumeSpecName "kube-api-access-bm5t9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:47:34.557398 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:47:34.557370 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a9055d5-d3ba-4a50-8f2d-a363bbb17ff8-maas-api-xks-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:47:34.557398 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:47:34.557395 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bm5t9\" (UniqueName: \"kubernetes.io/projected/3a9055d5-d3ba-4a50-8f2d-a363bbb17ff8-kube-api-access-bm5t9\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:47:35.111046 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:47:35.111011 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e350e85-tzb72" Oct 30 12:47:35.111046 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:47:35.111019 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e350e85-tzb72" event={"ID":"3a9055d5-d3ba-4a50-8f2d-a363bbb17ff8","Type":"ContainerDied","Data":"5de8d16eea72d8aafe2a089e37fdf3c5c4ac18c79a9d93368d5c3c968ec35b78"} Oct 30 12:47:35.111272 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:47:35.111071 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5de8d16eea72d8aafe2a089e37fdf3c5c4ac18c79a9d93368d5c3c968ec35b78" Oct 30 12:47:49.689552 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:47:49.689513 4908 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vn6vv" containerName="extractor" Oct 30 12:47:52.106860 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:47:52.103865 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9wwfc"] Oct 30 12:48:01.106054 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:48:01.106009 4908 patch_prober.go:29] interesting pod/e2e-extmod-b-fddaa3-openshift-default-668fdb7bf-z4588 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.4:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 12:48:01.106787 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:48:01.106746 4908 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-fddaa3-openshift-default-668fdb7bf-z4588" podUID="ab9d645e-4df1-482d-9ee8-54947745d8b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.4:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 12:48:03.463526 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:48:03.463439 4908 generic.go:356] "Generic (PLEG): container finished" podID="ab9d645e-4df1-482d-9ee8-54947745d8b4" containerID="b4b52188906f02662aa52c0d5c332567b35ee8b165722696d6808bcc37db32ec" exitCode=0 Oct 30 12:48:03.463946 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:48:03.463525 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-fddaa3-openshift-default-668fdb7bf-z4588" event={"ID":"ab9d645e-4df1-482d-9ee8-54947745d8b4","Type":"ContainerDied","Data":"b4b52188906f02662aa52c0d5c332567b35ee8b165722696d6808bcc37db32ec"} Oct 30 12:48:03.464052 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:48:03.464035 4908 scope.go:122] "RemoveContainer" containerID="b4b52188906f02662aa52c0d5c332567b35ee8b165722696d6808bcc37db32ec" Oct 30 12:48:03.465923 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:48:03.465893 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:48:03.466056 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:48:03.465967 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:48:03.466056 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:48:03.466004 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:48:04.489851 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:48:04.489565 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-fddaa3-openshift-default-668fdb7bf-z4588" event={"ID":"ab9d645e-4df1-482d-9ee8-54947745d8b4","Type":"ContainerStarted","Data":"a949943904a4dac9925427a479fe7bae18f9fd78de78852aea359e7d10af2150"} Oct 30 12:48:05.218409 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:48:05.218375 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-fddaa3-openshift-default-668fdb7bf-z4588" Oct 30 12:48:05.218618 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:48:05.218471 4908 patch_prober.go:29] interesting pod/e2e-extmod-b-fddaa3-openshift-default-668fdb7bf-z4588 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.4:15021/healthz/ready\": dial tcp 10.132.1.4:15021: connect: connection refused" start-of-body= Oct 30 12:48:05.218618 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:48:05.218503 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-fddaa3-openshift-default-668fdb7bf-z4588" podUID="ab9d645e-4df1-482d-9ee8-54947745d8b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.4:15021/healthz/ready\": dial tcp 10.132.1.4:15021: connect: connection refused" Oct 30 12:48:06.218608 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:48:06.218572 4908 patch_prober.go:29] interesting pod/e2e-extmod-b-fddaa3-openshift-default-668fdb7bf-z4588 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.4:15021/healthz/ready\": dial tcp 10.132.1.4:15021: connect: connection refused" start-of-body= Oct 30 12:48:06.219057 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:48:06.218627 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-fddaa3-openshift-default-668fdb7bf-z4588" podUID="ab9d645e-4df1-482d-9ee8-54947745d8b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.4:15021/healthz/ready\": dial tcp 10.132.1.4:15021: connect: connection refused" Oct 30 12:48:07.218637 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:48:07.218599 4908 patch_prober.go:29] interesting pod/e2e-extmod-b-fddaa3-openshift-default-668fdb7bf-z4588 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.4:15021/healthz/ready\": dial tcp 10.132.1.4:15021: connect: connection refused" start-of-body= Oct 30 12:48:07.219064 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:48:07.218651 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-fddaa3-openshift-default-668fdb7bf-z4588" podUID="ab9d645e-4df1-482d-9ee8-54947745d8b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.4:15021/healthz/ready\": dial tcp 10.132.1.4:15021: connect: connection refused" Oct 30 12:48:07.943605 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:48:07.943574 4908 scope.go:122] "RemoveContainer" containerID="f4ba6a1670388f2dd2358a126cf01869973cf3a763392ac738e89ea6b95d7d6d" Oct 30 12:48:07.954155 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:48:07.954129 4908 scope.go:122] "RemoveContainer" containerID="e8cdb41e9670645c5c4439d943a2ed9183b0d209b61a6840c8c6eeab1f32e59f" Oct 30 12:48:08.218950 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:48:08.218862 4908 patch_prober.go:29] interesting pod/e2e-extmod-b-fddaa3-openshift-default-668fdb7bf-z4588 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.4:15021/healthz/ready\": dial tcp 10.132.1.4:15021: connect: connection refused" start-of-body= Oct 30 12:48:08.218950 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:48:08.218912 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-fddaa3-openshift-default-668fdb7bf-z4588" podUID="ab9d645e-4df1-482d-9ee8-54947745d8b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.4:15021/healthz/ready\": dial tcp 10.132.1.4:15021: connect: connection refused" Oct 30 12:48:09.218722 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:48:09.218684 4908 patch_prober.go:29] interesting pod/e2e-extmod-b-fddaa3-openshift-default-668fdb7bf-z4588 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.4:15021/healthz/ready\": dial tcp 10.132.1.4:15021: connect: connection refused" start-of-body= Oct 30 12:48:09.218925 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:48:09.218747 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-fddaa3-openshift-default-668fdb7bf-z4588" podUID="ab9d645e-4df1-482d-9ee8-54947745d8b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.4:15021/healthz/ready\": dial tcp 10.132.1.4:15021: connect: connection refused" Oct 30 12:48:10.223324 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:48:10.223291 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-fddaa3-openshift-default-668fdb7bf-z4588" Oct 30 12:48:10.223726 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:48:10.223667 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-fddaa3-openshift-default-668fdb7bf-z4588" Oct 30 12:48:10.224198 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:48:10.224176 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-fddaa3-openshift-default-668fdb7bf-z4588" Oct 30 12:48:19.708642 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:48:19.708597 4908 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vn6vv" containerName="extractor" Oct 30 12:48:46.727005 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:48:46.726929 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5smp6"] Oct 30 12:48:46.727499 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:48:46.727479 4908 cpu_manager.go:414] "containerMap: removing container" podUID="3a9055d5-d3ba-4a50-8f2d-a363bbb17ff8" containerName="revoke-keys" Oct 30 12:48:46.727499 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:48:46.727495 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a9055d5-d3ba-4a50-8f2d-a363bbb17ff8" containerName="revoke-keys" Oct 30 12:48:46.727661 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:48:46.727626 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a9055d5-d3ba-4a50-8f2d-a363bbb17ff8" containerName="revoke-keys" Oct 30 12:48:46.731245 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:48:46.731227 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5smp6" Oct 30 12:48:46.753556 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:48:46.753531 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5smp6"] Oct 30 12:48:46.825928 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:48:46.825901 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f090df9-3e00-4fec-9008-8088c2ae5f93-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-5smp6\" (UID: \"2f090df9-3e00-4fec-9008-8088c2ae5f93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5smp6" Oct 30 12:48:46.826089 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:48:46.825981 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f090df9-3e00-4fec-9008-8088c2ae5f93-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-5smp6\" (UID: \"2f090df9-3e00-4fec-9008-8088c2ae5f93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5smp6" Oct 30 12:48:46.826158 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:48:46.826136 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkpbp\" (UniqueName: \"kubernetes.io/projected/2f090df9-3e00-4fec-9008-8088c2ae5f93-kube-api-access-fkpbp\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-5smp6\" (UID: \"2f090df9-3e00-4fec-9008-8088c2ae5f93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5smp6" Oct 30 12:48:46.826218 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:48:46.826185 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f090df9-3e00-4fec-9008-8088c2ae5f93-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-5smp6\" (UID: \"2f090df9-3e00-4fec-9008-8088c2ae5f93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5smp6" Oct 30 12:48:46.826291 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:48:46.826265 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f090df9-3e00-4fec-9008-8088c2ae5f93-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-5smp6\" (UID: \"2f090df9-3e00-4fec-9008-8088c2ae5f93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5smp6" Oct 30 12:48:46.826391 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:48:46.826375 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f090df9-3e00-4fec-9008-8088c2ae5f93-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-5smp6\" (UID: \"2f090df9-3e00-4fec-9008-8088c2ae5f93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5smp6" Oct 30 12:48:46.826464 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:48:46.826451 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f090df9-3e00-4fec-9008-8088c2ae5f93-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-5smp6\" (UID: \"2f090df9-3e00-4fec-9008-8088c2ae5f93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5smp6" Oct 30 12:48:46.826508 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:48:46.826491 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f090df9-3e00-4fec-9008-8088c2ae5f93-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-5smp6\" (UID: \"2f090df9-3e00-4fec-9008-8088c2ae5f93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5smp6" Oct 30 12:48:46.826562 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:48:46.826520 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f090df9-3e00-4fec-9008-8088c2ae5f93-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-5smp6\" (UID: \"2f090df9-3e00-4fec-9008-8088c2ae5f93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5smp6" Oct 30 12:48:46.928000 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:48:46.927973 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f090df9-3e00-4fec-9008-8088c2ae5f93-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-5smp6\" (UID: \"2f090df9-3e00-4fec-9008-8088c2ae5f93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5smp6" Oct 30 12:48:46.928152 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:48:46.928008 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f090df9-3e00-4fec-9008-8088c2ae5f93-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-5smp6\" (UID: \"2f090df9-3e00-4fec-9008-8088c2ae5f93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5smp6" Oct 30 12:48:46.928152 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:48:46.928042 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f090df9-3e00-4fec-9008-8088c2ae5f93-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-5smp6\" (UID: \"2f090df9-3e00-4fec-9008-8088c2ae5f93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5smp6" Oct 30 12:48:46.928152 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:48:46.928091 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f090df9-3e00-4fec-9008-8088c2ae5f93-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-5smp6\" (UID: \"2f090df9-3e00-4fec-9008-8088c2ae5f93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5smp6" Oct 30 12:48:46.928337 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:48:46.928156 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkpbp\" (UniqueName: \"kubernetes.io/projected/2f090df9-3e00-4fec-9008-8088c2ae5f93-kube-api-access-fkpbp\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-5smp6\" (UID: \"2f090df9-3e00-4fec-9008-8088c2ae5f93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5smp6" Oct 30 12:48:46.928337 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:48:46.928181 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f090df9-3e00-4fec-9008-8088c2ae5f93-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-5smp6\" (UID: \"2f090df9-3e00-4fec-9008-8088c2ae5f93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5smp6" Oct 30 12:48:46.928452 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:48:46.928376 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f090df9-3e00-4fec-9008-8088c2ae5f93-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-5smp6\" (UID: \"2f090df9-3e00-4fec-9008-8088c2ae5f93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5smp6" Oct 30 12:48:46.928518 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:48:46.928473 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f090df9-3e00-4fec-9008-8088c2ae5f93-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-5smp6\" (UID: \"2f090df9-3e00-4fec-9008-8088c2ae5f93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5smp6" Oct 30 12:48:46.928576 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:48:46.928533 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f090df9-3e00-4fec-9008-8088c2ae5f93-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-5smp6\" (UID: \"2f090df9-3e00-4fec-9008-8088c2ae5f93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5smp6" Oct 30 12:48:46.928776 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:48:46.928737 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f090df9-3e00-4fec-9008-8088c2ae5f93-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-5smp6\" (UID: \"2f090df9-3e00-4fec-9008-8088c2ae5f93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5smp6" Oct 30 12:48:46.928902 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:48:46.928865 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f090df9-3e00-4fec-9008-8088c2ae5f93-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-5smp6\" (UID: \"2f090df9-3e00-4fec-9008-8088c2ae5f93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5smp6" Oct 30 12:48:46.928902 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:48:46.928886 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f090df9-3e00-4fec-9008-8088c2ae5f93-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-5smp6\" (UID: \"2f090df9-3e00-4fec-9008-8088c2ae5f93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5smp6" Oct 30 12:48:46.929020 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:48:46.928926 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f090df9-3e00-4fec-9008-8088c2ae5f93-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-5smp6\" (UID: \"2f090df9-3e00-4fec-9008-8088c2ae5f93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5smp6" Oct 30 12:48:46.929020 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:48:46.928927 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f090df9-3e00-4fec-9008-8088c2ae5f93-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-5smp6\" (UID: \"2f090df9-3e00-4fec-9008-8088c2ae5f93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5smp6" Oct 30 12:48:46.930442 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:48:46.930422 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f090df9-3e00-4fec-9008-8088c2ae5f93-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-5smp6\" (UID: \"2f090df9-3e00-4fec-9008-8088c2ae5f93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5smp6" Oct 30 12:48:46.930586 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:48:46.930567 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f090df9-3e00-4fec-9008-8088c2ae5f93-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-5smp6\" (UID: \"2f090df9-3e00-4fec-9008-8088c2ae5f93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5smp6" Oct 30 12:48:46.950534 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:48:46.950508 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkpbp\" (UniqueName: \"kubernetes.io/projected/2f090df9-3e00-4fec-9008-8088c2ae5f93-kube-api-access-fkpbp\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-5smp6\" (UID: \"2f090df9-3e00-4fec-9008-8088c2ae5f93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5smp6" Oct 30 12:48:46.950624 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:48:46.950564 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f090df9-3e00-4fec-9008-8088c2ae5f93-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-5smp6\" (UID: \"2f090df9-3e00-4fec-9008-8088c2ae5f93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5smp6" Oct 30 12:48:47.048718 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:48:47.048657 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5smp6" Oct 30 12:48:47.372892 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:48:47.372862 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5smp6"] Oct 30 12:48:47.373886 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:48:47.373859 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f090df9_3e00_4fec_9008_8088c2ae5f93.slice/crio-65b7eaf086f882bbe96093d3d15f54746225735cd1db2e5a91b76758ead26a86 WatchSource:0}: Error finding container 65b7eaf086f882bbe96093d3d15f54746225735cd1db2e5a91b76758ead26a86: Status 404 returned error can't find the container with id 65b7eaf086f882bbe96093d3d15f54746225735cd1db2e5a91b76758ead26a86 Oct 30 12:48:47.377041 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:48:47.377006 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:48:47.377127 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:48:47.377073 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:48:47.377127 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:48:47.377104 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:48:47.869558 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:48:47.869516 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5smp6" event={"ID":"2f090df9-3e00-4fec-9008-8088c2ae5f93","Type":"ContainerStarted","Data":"e8fceedfc1472209f34a8974d35c70a2372deaac130a5f368026bede217b1724"} Oct 30 12:48:47.870081 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:48:47.869586 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5smp6" event={"ID":"2f090df9-3e00-4fec-9008-8088c2ae5f93","Type":"ContainerStarted","Data":"65b7eaf086f882bbe96093d3d15f54746225735cd1db2e5a91b76758ead26a86"} Oct 30 12:48:47.917807 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:48:47.917746 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5smp6" podStartSLOduration=1.917729693 podStartE2EDuration="1.917729693s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:48:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:48:47.911930104 +0000 UTC m=+3284.426358875" watchObservedRunningTime="2026-10-30 12:48:47.917729693 +0000 UTC m=+3284.432158438" Oct 30 12:48:48.049439 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:48:48.049408 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5smp6" Oct 30 12:48:48.051104 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:48:48.051068 4908 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-5smp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.9:15021/healthz/ready\": dial tcp 10.132.1.9:15021: connect: connection refused" start-of-body= Oct 30 12:48:48.051256 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:48:48.051119 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5smp6" podUID="2f090df9-3e00-4fec-9008-8088c2ae5f93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.9:15021/healthz/ready\": dial tcp 10.132.1.9:15021: connect: connection refused" Oct 30 12:48:49.050013 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:48:49.049977 4908 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-5smp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.9:15021/healthz/ready\": dial tcp 10.132.1.9:15021: connect: connection refused" start-of-body= Oct 30 12:48:49.050403 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:48:49.050032 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5smp6" podUID="2f090df9-3e00-4fec-9008-8088c2ae5f93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.9:15021/healthz/ready\": dial tcp 10.132.1.9:15021: connect: connection refused" Oct 30 12:48:49.708150 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:48:49.708117 4908 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vn6vv" containerName="extractor" Oct 30 12:48:50.049900 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:48:50.049863 4908 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-5smp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.9:15021/healthz/ready\": dial tcp 10.132.1.9:15021: connect: connection refused" start-of-body= Oct 30 12:48:50.050136 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:48:50.049941 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5smp6" podUID="2f090df9-3e00-4fec-9008-8088c2ae5f93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.9:15021/healthz/ready\": dial tcp 10.132.1.9:15021: connect: connection refused" Oct 30 12:48:51.049913 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:48:51.049867 4908 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-5smp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.9:15021/healthz/ready\": dial tcp 10.132.1.9:15021: connect: connection refused" start-of-body= Oct 30 12:48:51.050106 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:48:51.049922 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5smp6" podUID="2f090df9-3e00-4fec-9008-8088c2ae5f93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.9:15021/healthz/ready\": dial tcp 10.132.1.9:15021: connect: connection refused" Oct 30 12:48:52.049475 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:48:52.049438 4908 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-5smp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.9:15021/healthz/ready\": dial tcp 10.132.1.9:15021: connect: connection refused" start-of-body= Oct 30 12:48:52.050014 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:48:52.049502 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5smp6" podUID="2f090df9-3e00-4fec-9008-8088c2ae5f93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.9:15021/healthz/ready\": dial tcp 10.132.1.9:15021: connect: connection refused" Oct 30 12:48:53.049989 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:48:53.049953 4908 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-5smp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.9:15021/healthz/ready\": dial tcp 10.132.1.9:15021: connect: connection refused" start-of-body= Oct 30 12:48:53.050393 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:48:53.050006 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5smp6" podUID="2f090df9-3e00-4fec-9008-8088c2ae5f93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.9:15021/healthz/ready\": dial tcp 10.132.1.9:15021: connect: connection refused" Oct 30 12:48:54.049138 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:48:54.049101 4908 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-5smp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.9:15021/healthz/ready\": dial tcp 10.132.1.9:15021: connect: connection refused" start-of-body= Oct 30 12:48:54.049323 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:48:54.049168 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5smp6" podUID="2f090df9-3e00-4fec-9008-8088c2ae5f93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.9:15021/healthz/ready\": dial tcp 10.132.1.9:15021: connect: connection refused" Oct 30 12:48:55.646426 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:48:55.646394 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5smp6" Oct 30 12:48:55.647000 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:48:55.646887 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5smp6" Oct 30 12:48:55.647456 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:48:55.647438 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5smp6" Oct 30 12:49:03.008758 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:03.008721 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9wwfc"] Oct 30 12:49:06.303506 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:06.303478 4908 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-plczt_a70ee11c-c20e-4a4f-8476-ac5eb02816d0/console-operator/3.log" Oct 30 12:49:06.314682 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:06.314656 4908 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b8bxz_644e19b3-10ad-4145-9ed3-595d19ed9b07/ovn-acl-logging/0.log" Oct 30 12:49:06.533947 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:06.530971 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fda34221-5srjn"] Oct 30 12:49:06.538083 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:06.538058 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fda34221-5srjn" Oct 30 12:49:06.549024 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:06.548996 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fda34221-5srjn"] Oct 30 12:49:06.556066 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:06.556002 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 12:49:06.556201 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:06.556104 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s75hq\"" Oct 30 12:49:06.556201 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:06.556168 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 12:49:06.562899 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:06.562879 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 12:49:06.708246 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:06.708208 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6384ae16-8d99-4c45-847c-fd5f20baf6e8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fda34221-5srjn\" (UID: \"6384ae16-8d99-4c45-847c-fd5f20baf6e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fda34221-5srjn" Oct 30 12:49:06.708582 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:06.708561 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqk7m\" (UniqueName: \"kubernetes.io/projected/6384ae16-8d99-4c45-847c-fd5f20baf6e8-kube-api-access-zqk7m\") pod \"maas-api-revoke-sub-fda34221-5srjn\" (UID: \"6384ae16-8d99-4c45-847c-fd5f20baf6e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fda34221-5srjn" Oct 30 12:49:06.708732 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:06.708714 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6384ae16-8d99-4c45-847c-fd5f20baf6e8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fda34221-5srjn\" (UID: \"6384ae16-8d99-4c45-847c-fd5f20baf6e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fda34221-5srjn" Oct 30 12:49:06.814379 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:06.814302 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6384ae16-8d99-4c45-847c-fd5f20baf6e8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fda34221-5srjn\" (UID: \"6384ae16-8d99-4c45-847c-fd5f20baf6e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fda34221-5srjn" Oct 30 12:49:06.819464 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:06.819441 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqk7m\" (UniqueName: \"kubernetes.io/projected/6384ae16-8d99-4c45-847c-fd5f20baf6e8-kube-api-access-zqk7m\") pod \"maas-api-revoke-sub-fda34221-5srjn\" (UID: \"6384ae16-8d99-4c45-847c-fd5f20baf6e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fda34221-5srjn" Oct 30 12:49:06.819953 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:06.819933 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6384ae16-8d99-4c45-847c-fd5f20baf6e8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fda34221-5srjn\" (UID: \"6384ae16-8d99-4c45-847c-fd5f20baf6e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fda34221-5srjn" Oct 30 12:49:06.820622 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:06.819352 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6384ae16-8d99-4c45-847c-fd5f20baf6e8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fda34221-5srjn\" (UID: \"6384ae16-8d99-4c45-847c-fd5f20baf6e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fda34221-5srjn" Oct 30 12:49:06.827657 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:06.827636 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6384ae16-8d99-4c45-847c-fd5f20baf6e8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fda34221-5srjn\" (UID: \"6384ae16-8d99-4c45-847c-fd5f20baf6e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fda34221-5srjn" Oct 30 12:49:06.841783 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:06.841760 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqk7m\" (UniqueName: \"kubernetes.io/projected/6384ae16-8d99-4c45-847c-fd5f20baf6e8-kube-api-access-zqk7m\") pod \"maas-api-revoke-sub-fda34221-5srjn\" (UID: \"6384ae16-8d99-4c45-847c-fd5f20baf6e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fda34221-5srjn" Oct 30 12:49:06.855609 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:06.855591 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fda34221-5srjn" Oct 30 12:49:06.949998 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:06.949972 4908 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-plczt_a70ee11c-c20e-4a4f-8476-ac5eb02816d0/console-operator/3.log" Oct 30 12:49:06.956238 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:06.956216 4908 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b8bxz_644e19b3-10ad-4145-9ed3-595d19ed9b07/ovn-acl-logging/0.log" Oct 30 12:49:07.225384 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:07.225311 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fda34221-5srjn"] Oct 30 12:49:07.229046 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:49:07.229003 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6384ae16_8d99_4c45_847c_fd5f20baf6e8.slice/crio-ea1eaa0ffeb1d34905ab271a844cfdbefec6ff1df8bbca0992cf19d784d00c25 WatchSource:0}: Error finding container ea1eaa0ffeb1d34905ab271a844cfdbefec6ff1df8bbca0992cf19d784d00c25: Status 404 returned error can't find the container with id ea1eaa0ffeb1d34905ab271a844cfdbefec6ff1df8bbca0992cf19d784d00c25 Oct 30 12:49:08.137679 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:08.137647 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fda34221-5srjn" event={"ID":"6384ae16-8d99-4c45-847c-fd5f20baf6e8","Type":"ContainerStarted","Data":"32cdef9b69fe7f3f27ce4bd2b6f8a19b1ea50ad608baa4553d94c9814835a86b"} Oct 30 12:49:08.138174 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:08.137702 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fda34221-5srjn" event={"ID":"6384ae16-8d99-4c45-847c-fd5f20baf6e8","Type":"ContainerStarted","Data":"ea1eaa0ffeb1d34905ab271a844cfdbefec6ff1df8bbca0992cf19d784d00c25"} Oct 30 12:49:08.177629 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:08.177573 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fda34221-5srjn" podStartSLOduration=2.177555843 podStartE2EDuration="2.177555843s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:49:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:49:08.174885153 +0000 UTC m=+3304.689313894" watchObservedRunningTime="2026-10-30 12:49:08.177555843 +0000 UTC m=+3304.691984589" Oct 30 12:49:09.145591 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:09.145552 4908 generic.go:356] "Generic (PLEG): container finished" podID="6384ae16-8d99-4c45-847c-fd5f20baf6e8" containerID="32cdef9b69fe7f3f27ce4bd2b6f8a19b1ea50ad608baa4553d94c9814835a86b" exitCode=0 Oct 30 12:49:09.146029 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:09.145633 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fda34221-5srjn" event={"ID":"6384ae16-8d99-4c45-847c-fd5f20baf6e8","Type":"ContainerDied","Data":"32cdef9b69fe7f3f27ce4bd2b6f8a19b1ea50ad608baa4553d94c9814835a86b"} Oct 30 12:49:10.318506 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:10.318477 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fda34221-5srjn" Oct 30 12:49:10.383680 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:10.383649 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6384ae16-8d99-4c45-847c-fd5f20baf6e8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6384ae16-8d99-4c45-847c-fd5f20baf6e8-maas-api-xks-ca\") pod \"6384ae16-8d99-4c45-847c-fd5f20baf6e8\" (UID: \"6384ae16-8d99-4c45-847c-fd5f20baf6e8\") " Oct 30 12:49:10.383917 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:10.383893 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6384ae16-8d99-4c45-847c-fd5f20baf6e8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6384ae16-8d99-4c45-847c-fd5f20baf6e8-maas-api-openshift-ca\") pod \"6384ae16-8d99-4c45-847c-fd5f20baf6e8\" (UID: \"6384ae16-8d99-4c45-847c-fd5f20baf6e8\") " Oct 30 12:49:10.384010 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:10.383960 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6384ae16-8d99-4c45-847c-fd5f20baf6e8-kube-api-access-zqk7m\" (UniqueName: \"kubernetes.io/projected/6384ae16-8d99-4c45-847c-fd5f20baf6e8-kube-api-access-zqk7m\") pod \"6384ae16-8d99-4c45-847c-fd5f20baf6e8\" (UID: \"6384ae16-8d99-4c45-847c-fd5f20baf6e8\") " Oct 30 12:49:10.384312 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:10.384285 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6384ae16-8d99-4c45-847c-fd5f20baf6e8-maas-api-openshift-ca" pod "6384ae16-8d99-4c45-847c-fd5f20baf6e8" (UID: "6384ae16-8d99-4c45-847c-fd5f20baf6e8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:49:10.384529 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:10.384505 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6384ae16-8d99-4c45-847c-fd5f20baf6e8-maas-api-openshift-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:49:10.386297 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:10.386257 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6384ae16-8d99-4c45-847c-fd5f20baf6e8-maas-api-xks-ca" pod "6384ae16-8d99-4c45-847c-fd5f20baf6e8" (UID: "6384ae16-8d99-4c45-847c-fd5f20baf6e8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:49:10.386918 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:10.386893 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6384ae16-8d99-4c45-847c-fd5f20baf6e8-kube-api-access-zqk7m" pod "6384ae16-8d99-4c45-847c-fd5f20baf6e8" (UID: "6384ae16-8d99-4c45-847c-fd5f20baf6e8"). InnerVolumeSpecName "kube-api-access-zqk7m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:49:10.485561 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:10.485539 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqk7m\" (UniqueName: \"kubernetes.io/projected/6384ae16-8d99-4c45-847c-fd5f20baf6e8-kube-api-access-zqk7m\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:49:10.485666 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:10.485562 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6384ae16-8d99-4c45-847c-fd5f20baf6e8-maas-api-xks-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:49:11.163549 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:11.163520 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fda34221-5srjn" Oct 30 12:49:11.163726 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:11.163521 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fda34221-5srjn" event={"ID":"6384ae16-8d99-4c45-847c-fd5f20baf6e8","Type":"ContainerDied","Data":"ea1eaa0ffeb1d34905ab271a844cfdbefec6ff1df8bbca0992cf19d784d00c25"} Oct 30 12:49:11.163726 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:11.163640 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ea1eaa0ffeb1d34905ab271a844cfdbefec6ff1df8bbca0992cf19d784d00c25" Oct 30 12:49:19.683373 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:49:19.683331 4908 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vn6vv" containerName="extractor" Oct 30 12:49:31.906853 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:31.903599 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xcqb4"] Oct 30 12:49:31.906853 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:31.904429 4908 cpu_manager.go:414] "containerMap: removing container" podUID="6384ae16-8d99-4c45-847c-fd5f20baf6e8" containerName="revoke-keys" Oct 30 12:49:31.906853 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:31.904451 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6384ae16-8d99-4c45-847c-fd5f20baf6e8" containerName="revoke-keys" Oct 30 12:49:31.906853 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:31.904654 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="6384ae16-8d99-4c45-847c-fd5f20baf6e8" containerName="revoke-keys" Oct 30 12:49:31.935248 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:31.935214 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xcqb4" Oct 30 12:49:31.954910 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:31.954877 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xcqb4"] Oct 30 12:49:31.984852 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:31.980897 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xnnfx"] Oct 30 12:49:32.030330 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:32.030299 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xnnfx"] Oct 30 12:49:32.030719 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:32.030643 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xnnfx" Oct 30 12:49:32.037019 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:32.036991 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2f439fb-154c-4997-b0f8-f18b275d262f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-xcqb4\" (UID: \"d2f439fb-154c-4997-b0f8-f18b275d262f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xcqb4" Oct 30 12:49:32.039065 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:32.039038 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2f439fb-154c-4997-b0f8-f18b275d262f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-xcqb4\" (UID: \"d2f439fb-154c-4997-b0f8-f18b275d262f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xcqb4" Oct 30 12:49:32.039235 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:32.039219 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2f439fb-154c-4997-b0f8-f18b275d262f-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-xcqb4\" (UID: \"d2f439fb-154c-4997-b0f8-f18b275d262f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xcqb4" Oct 30 12:49:32.039389 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:32.039373 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pgm7\" (UniqueName: \"kubernetes.io/projected/d2f439fb-154c-4997-b0f8-f18b275d262f-kube-api-access-4pgm7\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-xcqb4\" (UID: \"d2f439fb-154c-4997-b0f8-f18b275d262f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xcqb4" Oct 30 12:49:32.067542 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:32.067501 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2f439fb-154c-4997-b0f8-f18b275d262f-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-xcqb4\" (UID: \"d2f439fb-154c-4997-b0f8-f18b275d262f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xcqb4" Oct 30 12:49:32.067802 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:32.067777 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2f439fb-154c-4997-b0f8-f18b275d262f-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-xcqb4\" (UID: \"d2f439fb-154c-4997-b0f8-f18b275d262f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xcqb4" Oct 30 12:49:32.068052 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:32.068026 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2f439fb-154c-4997-b0f8-f18b275d262f-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-xcqb4\" (UID: \"d2f439fb-154c-4997-b0f8-f18b275d262f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xcqb4" Oct 30 12:49:32.068195 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:32.068178 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2f439fb-154c-4997-b0f8-f18b275d262f-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-xcqb4\" (UID: \"d2f439fb-154c-4997-b0f8-f18b275d262f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xcqb4" Oct 30 12:49:32.072951 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:32.072924 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2f439fb-154c-4997-b0f8-f18b275d262f-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-xcqb4\" (UID: \"d2f439fb-154c-4997-b0f8-f18b275d262f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xcqb4" Oct 30 12:49:32.174963 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:32.174867 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03eea3e0-0b0d-459d-b03b-3a2531f20185-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-xnnfx\" (UID: \"03eea3e0-0b0d-459d-b03b-3a2531f20185\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xnnfx" Oct 30 12:49:32.174963 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:32.174935 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03eea3e0-0b0d-459d-b03b-3a2531f20185-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-xnnfx\" (UID: \"03eea3e0-0b0d-459d-b03b-3a2531f20185\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xnnfx" Oct 30 12:49:32.175192 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:32.175052 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2f439fb-154c-4997-b0f8-f18b275d262f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-xcqb4\" (UID: \"d2f439fb-154c-4997-b0f8-f18b275d262f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xcqb4" Oct 30 12:49:32.175192 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:32.175098 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2f439fb-154c-4997-b0f8-f18b275d262f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-xcqb4\" (UID: \"d2f439fb-154c-4997-b0f8-f18b275d262f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xcqb4" Oct 30 12:49:32.175192 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:32.175132 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2f439fb-154c-4997-b0f8-f18b275d262f-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-xcqb4\" (UID: \"d2f439fb-154c-4997-b0f8-f18b275d262f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xcqb4" Oct 30 12:49:32.175423 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:32.175202 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pgm7\" (UniqueName: \"kubernetes.io/projected/d2f439fb-154c-4997-b0f8-f18b275d262f-kube-api-access-4pgm7\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-xcqb4\" (UID: \"d2f439fb-154c-4997-b0f8-f18b275d262f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xcqb4" Oct 30 12:49:32.175423 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:32.175244 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03eea3e0-0b0d-459d-b03b-3a2531f20185-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-xnnfx\" (UID: \"03eea3e0-0b0d-459d-b03b-3a2531f20185\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xnnfx" Oct 30 12:49:32.175423 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:32.175282 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03eea3e0-0b0d-459d-b03b-3a2531f20185-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-xnnfx\" (UID: \"03eea3e0-0b0d-459d-b03b-3a2531f20185\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xnnfx" Oct 30 12:49:32.175423 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:32.175309 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03eea3e0-0b0d-459d-b03b-3a2531f20185-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-xnnfx\" (UID: \"03eea3e0-0b0d-459d-b03b-3a2531f20185\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xnnfx" Oct 30 12:49:32.175423 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:32.175359 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2f439fb-154c-4997-b0f8-f18b275d262f-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-xcqb4\" (UID: \"d2f439fb-154c-4997-b0f8-f18b275d262f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xcqb4" Oct 30 12:49:32.175423 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:32.175395 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2f439fb-154c-4997-b0f8-f18b275d262f-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-xcqb4\" (UID: \"d2f439fb-154c-4997-b0f8-f18b275d262f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xcqb4" Oct 30 12:49:32.175755 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:32.175427 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2f439fb-154c-4997-b0f8-f18b275d262f-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-xcqb4\" (UID: \"d2f439fb-154c-4997-b0f8-f18b275d262f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xcqb4" Oct 30 12:49:32.175755 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:32.175455 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03eea3e0-0b0d-459d-b03b-3a2531f20185-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-xnnfx\" (UID: \"03eea3e0-0b0d-459d-b03b-3a2531f20185\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xnnfx" Oct 30 12:49:32.175755 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:32.175491 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03eea3e0-0b0d-459d-b03b-3a2531f20185-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-xnnfx\" (UID: \"03eea3e0-0b0d-459d-b03b-3a2531f20185\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xnnfx" Oct 30 12:49:32.175755 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:32.175531 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2f439fb-154c-4997-b0f8-f18b275d262f-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-xcqb4\" (UID: \"d2f439fb-154c-4997-b0f8-f18b275d262f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xcqb4" Oct 30 12:49:32.175755 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:32.175569 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03eea3e0-0b0d-459d-b03b-3a2531f20185-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-xnnfx\" (UID: \"03eea3e0-0b0d-459d-b03b-3a2531f20185\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xnnfx" Oct 30 12:49:32.175755 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:32.175630 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2f439fb-154c-4997-b0f8-f18b275d262f-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-xcqb4\" (UID: \"d2f439fb-154c-4997-b0f8-f18b275d262f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xcqb4" Oct 30 12:49:32.175755 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:32.175646 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2f439fb-154c-4997-b0f8-f18b275d262f-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-xcqb4\" (UID: \"d2f439fb-154c-4997-b0f8-f18b275d262f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xcqb4" Oct 30 12:49:32.176811 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:32.175822 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2f439fb-154c-4997-b0f8-f18b275d262f-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-xcqb4\" (UID: \"d2f439fb-154c-4997-b0f8-f18b275d262f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xcqb4" Oct 30 12:49:32.176811 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:32.175979 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2f439fb-154c-4997-b0f8-f18b275d262f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-xcqb4\" (UID: \"d2f439fb-154c-4997-b0f8-f18b275d262f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xcqb4" Oct 30 12:49:32.176811 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:32.176000 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2f439fb-154c-4997-b0f8-f18b275d262f-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-xcqb4\" (UID: \"d2f439fb-154c-4997-b0f8-f18b275d262f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xcqb4" Oct 30 12:49:32.176811 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:32.176150 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2f439fb-154c-4997-b0f8-f18b275d262f-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-xcqb4\" (UID: \"d2f439fb-154c-4997-b0f8-f18b275d262f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xcqb4" Oct 30 12:49:32.176811 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:32.176210 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pjf2\" (UniqueName: \"kubernetes.io/projected/03eea3e0-0b0d-459d-b03b-3a2531f20185-kube-api-access-7pjf2\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-xnnfx\" (UID: \"03eea3e0-0b0d-459d-b03b-3a2531f20185\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xnnfx" Oct 30 12:49:32.177937 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:32.177916 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2f439fb-154c-4997-b0f8-f18b275d262f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-xcqb4\" (UID: \"d2f439fb-154c-4997-b0f8-f18b275d262f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xcqb4" Oct 30 12:49:32.178435 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:32.178413 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2f439fb-154c-4997-b0f8-f18b275d262f-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-xcqb4\" (UID: \"d2f439fb-154c-4997-b0f8-f18b275d262f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xcqb4" Oct 30 12:49:32.197302 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:32.197277 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2f439fb-154c-4997-b0f8-f18b275d262f-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-xcqb4\" (UID: \"d2f439fb-154c-4997-b0f8-f18b275d262f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xcqb4" Oct 30 12:49:32.197691 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:32.197674 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pgm7\" (UniqueName: \"kubernetes.io/projected/d2f439fb-154c-4997-b0f8-f18b275d262f-kube-api-access-4pgm7\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-xcqb4\" (UID: \"d2f439fb-154c-4997-b0f8-f18b275d262f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xcqb4" Oct 30 12:49:32.270434 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:32.270408 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xcqb4" Oct 30 12:49:32.277456 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:32.277435 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03eea3e0-0b0d-459d-b03b-3a2531f20185-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-xnnfx\" (UID: \"03eea3e0-0b0d-459d-b03b-3a2531f20185\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xnnfx" Oct 30 12:49:32.277580 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:32.277480 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03eea3e0-0b0d-459d-b03b-3a2531f20185-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-xnnfx\" (UID: \"03eea3e0-0b0d-459d-b03b-3a2531f20185\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xnnfx" Oct 30 12:49:32.277580 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:32.277509 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03eea3e0-0b0d-459d-b03b-3a2531f20185-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-xnnfx\" (UID: \"03eea3e0-0b0d-459d-b03b-3a2531f20185\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xnnfx" Oct 30 12:49:32.277580 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:32.277569 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03eea3e0-0b0d-459d-b03b-3a2531f20185-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-xnnfx\" (UID: \"03eea3e0-0b0d-459d-b03b-3a2531f20185\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xnnfx" Oct 30 12:49:32.277742 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:32.277600 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03eea3e0-0b0d-459d-b03b-3a2531f20185-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-xnnfx\" (UID: \"03eea3e0-0b0d-459d-b03b-3a2531f20185\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xnnfx" Oct 30 12:49:32.277742 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:32.277643 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03eea3e0-0b0d-459d-b03b-3a2531f20185-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-xnnfx\" (UID: \"03eea3e0-0b0d-459d-b03b-3a2531f20185\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xnnfx" Oct 30 12:49:32.277889 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:32.277736 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pjf2\" (UniqueName: \"kubernetes.io/projected/03eea3e0-0b0d-459d-b03b-3a2531f20185-kube-api-access-7pjf2\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-xnnfx\" (UID: \"03eea3e0-0b0d-459d-b03b-3a2531f20185\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xnnfx" Oct 30 12:49:32.277889 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:32.277782 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03eea3e0-0b0d-459d-b03b-3a2531f20185-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-xnnfx\" (UID: \"03eea3e0-0b0d-459d-b03b-3a2531f20185\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xnnfx" Oct 30 12:49:32.277889 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:32.277848 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03eea3e0-0b0d-459d-b03b-3a2531f20185-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-xnnfx\" (UID: \"03eea3e0-0b0d-459d-b03b-3a2531f20185\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xnnfx" Oct 30 12:49:32.278410 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:32.278098 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03eea3e0-0b0d-459d-b03b-3a2531f20185-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-xnnfx\" (UID: \"03eea3e0-0b0d-459d-b03b-3a2531f20185\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xnnfx" Oct 30 12:49:32.278410 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:32.278262 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03eea3e0-0b0d-459d-b03b-3a2531f20185-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-xnnfx\" (UID: \"03eea3e0-0b0d-459d-b03b-3a2531f20185\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xnnfx" Oct 30 12:49:32.278410 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:32.278305 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03eea3e0-0b0d-459d-b03b-3a2531f20185-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-xnnfx\" (UID: \"03eea3e0-0b0d-459d-b03b-3a2531f20185\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xnnfx" Oct 30 12:49:32.278410 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:32.278397 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03eea3e0-0b0d-459d-b03b-3a2531f20185-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-xnnfx\" (UID: \"03eea3e0-0b0d-459d-b03b-3a2531f20185\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xnnfx" Oct 30 12:49:32.278682 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:32.278485 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03eea3e0-0b0d-459d-b03b-3a2531f20185-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-xnnfx\" (UID: \"03eea3e0-0b0d-459d-b03b-3a2531f20185\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xnnfx" Oct 30 12:49:32.280478 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:32.280454 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03eea3e0-0b0d-459d-b03b-3a2531f20185-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-xnnfx\" (UID: \"03eea3e0-0b0d-459d-b03b-3a2531f20185\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xnnfx" Oct 30 12:49:32.280478 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:32.280466 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03eea3e0-0b0d-459d-b03b-3a2531f20185-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-xnnfx\" (UID: \"03eea3e0-0b0d-459d-b03b-3a2531f20185\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xnnfx" Oct 30 12:49:32.300319 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:32.300289 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03eea3e0-0b0d-459d-b03b-3a2531f20185-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-xnnfx\" (UID: \"03eea3e0-0b0d-459d-b03b-3a2531f20185\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xnnfx" Oct 30 12:49:32.300610 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:32.300590 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pjf2\" (UniqueName: \"kubernetes.io/projected/03eea3e0-0b0d-459d-b03b-3a2531f20185-kube-api-access-7pjf2\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-xnnfx\" (UID: \"03eea3e0-0b0d-459d-b03b-3a2531f20185\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xnnfx" Oct 30 12:49:32.385669 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:32.385636 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xnnfx" Oct 30 12:49:32.598361 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:32.598259 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xcqb4"] Oct 30 12:49:32.601984 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:49:32.601936 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2f439fb_154c_4997_b0f8_f18b275d262f.slice/crio-ebefebca60fe69df83f85b5f7377dec7a5ef5ebdbc97af773e9c1729849890dc WatchSource:0}: Error finding container ebefebca60fe69df83f85b5f7377dec7a5ef5ebdbc97af773e9c1729849890dc: Status 404 returned error can't find the container with id ebefebca60fe69df83f85b5f7377dec7a5ef5ebdbc97af773e9c1729849890dc Oct 30 12:49:32.605274 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:32.605226 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:49:32.605391 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:32.605310 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:49:32.605391 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:32.605351 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:49:32.710642 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:32.710604 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xnnfx"] Oct 30 12:49:32.711472 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:49:32.711444 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03eea3e0_0b0d_459d_b03b_3a2531f20185.slice/crio-28bcaf273eb62279701b3936685f4e7fb548ff7cbe7b48c44f0677f4088de427 WatchSource:0}: Error finding container 28bcaf273eb62279701b3936685f4e7fb548ff7cbe7b48c44f0677f4088de427: Status 404 returned error can't find the container with id 28bcaf273eb62279701b3936685f4e7fb548ff7cbe7b48c44f0677f4088de427 Oct 30 12:49:32.714347 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:32.714317 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:49:32.714429 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:32.714379 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:49:32.714429 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:32.714405 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:49:33.362070 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:33.362030 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xnnfx" event={"ID":"03eea3e0-0b0d-459d-b03b-3a2531f20185","Type":"ContainerStarted","Data":"1cfa6e9bcd78f5e2646de2ee39cece3863dea75df13a71f952827ee871a002ef"} Oct 30 12:49:33.362545 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:33.362116 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xnnfx" event={"ID":"03eea3e0-0b0d-459d-b03b-3a2531f20185","Type":"ContainerStarted","Data":"28bcaf273eb62279701b3936685f4e7fb548ff7cbe7b48c44f0677f4088de427"} Oct 30 12:49:33.364038 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:33.364006 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xcqb4" event={"ID":"d2f439fb-154c-4997-b0f8-f18b275d262f","Type":"ContainerStarted","Data":"ef9aa7c8f24ef453f1e99c8a9ba70dd90e9e893a2982db650c44751916ce037d"} Oct 30 12:49:33.364165 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:33.364063 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xcqb4" event={"ID":"d2f439fb-154c-4997-b0f8-f18b275d262f","Type":"ContainerStarted","Data":"ebefebca60fe69df83f85b5f7377dec7a5ef5ebdbc97af773e9c1729849890dc"} Oct 30 12:49:33.385789 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:33.385762 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xnnfx" Oct 30 12:49:33.387350 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:33.387326 4908 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-xnnfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.12:15021/healthz/ready\": dial tcp 10.132.1.12:15021: connect: connection refused" start-of-body= Oct 30 12:49:33.387453 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:33.387370 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xnnfx" podUID="03eea3e0-0b0d-459d-b03b-3a2531f20185" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.12:15021/healthz/ready\": dial tcp 10.132.1.12:15021: connect: connection refused" Oct 30 12:49:33.420122 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:33.420046 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xnnfx" podStartSLOduration=2.420032705 podStartE2EDuration="2.420032705s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:49:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:49:33.417381274 +0000 UTC m=+3329.931810017" watchObservedRunningTime="2026-10-30 12:49:33.420032705 +0000 UTC m=+3329.934461452" Oct 30 12:49:33.471659 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:33.471602 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xcqb4" podStartSLOduration=2.47158462 podStartE2EDuration="2.47158462s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:49:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:49:33.465266131 +0000 UTC m=+3329.979694901" watchObservedRunningTime="2026-10-30 12:49:33.47158462 +0000 UTC m=+3329.986013389" Oct 30 12:49:34.272254 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:34.272223 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xcqb4" Oct 30 12:49:34.303539 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:34.303213 4908 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-xcqb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" start-of-body= Oct 30 12:49:34.303539 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:34.303276 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xcqb4" podUID="d2f439fb-154c-4997-b0f8-f18b275d262f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" Oct 30 12:49:34.404850 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:34.404291 4908 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-xnnfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.12:15021/healthz/ready\": dial tcp 10.132.1.12:15021: connect: connection refused" start-of-body= Oct 30 12:49:34.404850 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:34.404353 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xnnfx" podUID="03eea3e0-0b0d-459d-b03b-3a2531f20185" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.12:15021/healthz/ready\": dial tcp 10.132.1.12:15021: connect: connection refused" Oct 30 12:49:35.281854 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:35.279163 4908 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-xcqb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" start-of-body= Oct 30 12:49:35.281854 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:35.279222 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xcqb4" podUID="d2f439fb-154c-4997-b0f8-f18b275d262f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" Oct 30 12:49:35.386232 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:35.386194 4908 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-xnnfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.12:15021/healthz/ready\": dial tcp 10.132.1.12:15021: connect: connection refused" start-of-body= Oct 30 12:49:35.386506 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:35.386475 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xnnfx" podUID="03eea3e0-0b0d-459d-b03b-3a2531f20185" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.12:15021/healthz/ready\": dial tcp 10.132.1.12:15021: connect: connection refused" Oct 30 12:49:36.276856 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:36.271187 4908 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-xcqb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" start-of-body= Oct 30 12:49:36.276856 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:36.271249 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xcqb4" podUID="d2f439fb-154c-4997-b0f8-f18b275d262f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" Oct 30 12:49:36.395851 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:36.395357 4908 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-xnnfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.12:15021/healthz/ready\": dial tcp 10.132.1.12:15021: connect: connection refused" start-of-body= Oct 30 12:49:36.395851 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:36.395415 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xnnfx" podUID="03eea3e0-0b0d-459d-b03b-3a2531f20185" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.12:15021/healthz/ready\": dial tcp 10.132.1.12:15021: connect: connection refused" Oct 30 12:49:37.277853 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:37.277216 4908 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-xcqb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" start-of-body= Oct 30 12:49:37.277853 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:37.277276 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xcqb4" podUID="d2f439fb-154c-4997-b0f8-f18b275d262f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" Oct 30 12:49:37.390853 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:37.390117 4908 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-xnnfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.12:15021/healthz/ready\": dial tcp 10.132.1.12:15021: connect: connection refused" start-of-body= Oct 30 12:49:37.390853 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:37.390175 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xnnfx" podUID="03eea3e0-0b0d-459d-b03b-3a2531f20185" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.12:15021/healthz/ready\": dial tcp 10.132.1.12:15021: connect: connection refused" Oct 30 12:49:38.275241 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:38.275205 4908 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-xcqb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" start-of-body= Oct 30 12:49:38.275559 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:38.275520 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xcqb4" podUID="d2f439fb-154c-4997-b0f8-f18b275d262f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" Oct 30 12:49:38.386643 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:38.386608 4908 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-xnnfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.12:15021/healthz/ready\": dial tcp 10.132.1.12:15021: connect: connection refused" start-of-body= Oct 30 12:49:38.387265 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:38.387218 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xnnfx" podUID="03eea3e0-0b0d-459d-b03b-3a2531f20185" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.12:15021/healthz/ready\": dial tcp 10.132.1.12:15021: connect: connection refused" Oct 30 12:49:39.279853 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:39.279238 4908 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-xcqb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" start-of-body= Oct 30 12:49:39.279853 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:39.279315 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xcqb4" podUID="d2f439fb-154c-4997-b0f8-f18b275d262f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" Oct 30 12:49:39.393850 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:39.393746 4908 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-xnnfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.12:15021/healthz/ready\": dial tcp 10.132.1.12:15021: connect: connection refused" start-of-body= Oct 30 12:49:39.393850 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:39.393809 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xnnfx" podUID="03eea3e0-0b0d-459d-b03b-3a2531f20185" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.12:15021/healthz/ready\": dial tcp 10.132.1.12:15021: connect: connection refused" Oct 30 12:49:40.271114 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:40.271069 4908 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-xcqb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" start-of-body= Oct 30 12:49:40.271303 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:40.271150 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xcqb4" podUID="d2f439fb-154c-4997-b0f8-f18b275d262f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" Oct 30 12:49:40.388849 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:40.386295 4908 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-xnnfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.12:15021/healthz/ready\": dial tcp 10.132.1.12:15021: connect: connection refused" start-of-body= Oct 30 12:49:40.388849 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:40.386363 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xnnfx" podUID="03eea3e0-0b0d-459d-b03b-3a2531f20185" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.12:15021/healthz/ready\": dial tcp 10.132.1.12:15021: connect: connection refused" Oct 30 12:49:41.271878 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:41.271348 4908 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-xcqb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" start-of-body= Oct 30 12:49:41.271878 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:41.271432 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xcqb4" podUID="d2f439fb-154c-4997-b0f8-f18b275d262f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" Oct 30 12:49:41.389852 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:41.389229 4908 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-xnnfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.12:15021/healthz/ready\": dial tcp 10.132.1.12:15021: connect: connection refused" start-of-body= Oct 30 12:49:41.389852 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:41.389289 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xnnfx" podUID="03eea3e0-0b0d-459d-b03b-3a2531f20185" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.12:15021/healthz/ready\": dial tcp 10.132.1.12:15021: connect: connection refused" Oct 30 12:49:42.956939 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:42.956908 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xcqb4" Oct 30 12:49:42.957495 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:42.957437 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xcqb4" Oct 30 12:49:42.959632 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:42.959607 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xcqb4" Oct 30 12:49:43.315344 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:43.315304 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xnnfx" Oct 30 12:49:43.315772 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:43.315708 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xnnfx" Oct 30 12:49:43.316446 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:43.316427 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xnnfx" Oct 30 12:49:43.471491 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:43.471452 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9wwfc"] Oct 30 12:49:47.029769 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:47.029735 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e70aca9a-5wsmz"] Oct 30 12:49:47.034176 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:47.034156 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e70aca9a-5wsmz" Oct 30 12:49:47.057253 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:47.057226 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e70aca9a-5wsmz"] Oct 30 12:49:47.062230 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:47.062205 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 12:49:47.062356 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:47.062229 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s75hq\"" Oct 30 12:49:47.062356 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:47.062304 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 12:49:47.062574 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:47.062553 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 12:49:47.127109 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:47.127076 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/437b70b1-fea7-4b27-a34b-39a0acf5103d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e70aca9a-5wsmz\" (UID: \"437b70b1-fea7-4b27-a34b-39a0acf5103d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e70aca9a-5wsmz" Oct 30 12:49:47.127234 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:47.127203 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/437b70b1-fea7-4b27-a34b-39a0acf5103d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e70aca9a-5wsmz\" (UID: \"437b70b1-fea7-4b27-a34b-39a0acf5103d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e70aca9a-5wsmz" Oct 30 12:49:47.127288 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:47.127255 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8882\" (UniqueName: \"kubernetes.io/projected/437b70b1-fea7-4b27-a34b-39a0acf5103d-kube-api-access-r8882\") pod \"maas-api-revoke-sub-e70aca9a-5wsmz\" (UID: \"437b70b1-fea7-4b27-a34b-39a0acf5103d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e70aca9a-5wsmz" Oct 30 12:49:47.228958 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:47.228930 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/437b70b1-fea7-4b27-a34b-39a0acf5103d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e70aca9a-5wsmz\" (UID: \"437b70b1-fea7-4b27-a34b-39a0acf5103d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e70aca9a-5wsmz" Oct 30 12:49:47.229087 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:47.228987 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/437b70b1-fea7-4b27-a34b-39a0acf5103d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e70aca9a-5wsmz\" (UID: \"437b70b1-fea7-4b27-a34b-39a0acf5103d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e70aca9a-5wsmz" Oct 30 12:49:47.229087 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:47.229032 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8882\" (UniqueName: \"kubernetes.io/projected/437b70b1-fea7-4b27-a34b-39a0acf5103d-kube-api-access-r8882\") pod \"maas-api-revoke-sub-e70aca9a-5wsmz\" (UID: \"437b70b1-fea7-4b27-a34b-39a0acf5103d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e70aca9a-5wsmz" Oct 30 12:49:47.229692 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:47.229670 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/437b70b1-fea7-4b27-a34b-39a0acf5103d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e70aca9a-5wsmz\" (UID: \"437b70b1-fea7-4b27-a34b-39a0acf5103d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e70aca9a-5wsmz" Oct 30 12:49:47.231176 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:47.231158 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/437b70b1-fea7-4b27-a34b-39a0acf5103d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e70aca9a-5wsmz\" (UID: \"437b70b1-fea7-4b27-a34b-39a0acf5103d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e70aca9a-5wsmz" Oct 30 12:49:47.250886 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:47.250854 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8882\" (UniqueName: \"kubernetes.io/projected/437b70b1-fea7-4b27-a34b-39a0acf5103d-kube-api-access-r8882\") pod \"maas-api-revoke-sub-e70aca9a-5wsmz\" (UID: \"437b70b1-fea7-4b27-a34b-39a0acf5103d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e70aca9a-5wsmz" Oct 30 12:49:47.349892 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:47.348951 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e70aca9a-5wsmz" Oct 30 12:49:47.707735 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:47.707708 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e70aca9a-5wsmz"] Oct 30 12:49:47.710824 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:49:47.710793 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod437b70b1_fea7_4b27_a34b_39a0acf5103d.slice/crio-b47b3a50dfe96c89acecf6f3d1e8796dacc5fb63449cb8c904aa535945848022 WatchSource:0}: Error finding container b47b3a50dfe96c89acecf6f3d1e8796dacc5fb63449cb8c904aa535945848022: Status 404 returned error can't find the container with id b47b3a50dfe96c89acecf6f3d1e8796dacc5fb63449cb8c904aa535945848022 Oct 30 12:49:48.682482 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:48.682442 4908 generic.go:356] "Generic (PLEG): container finished" podID="437b70b1-fea7-4b27-a34b-39a0acf5103d" containerID="f061546f2cdff92d61853e84d54a5e8d548606aeb2ce06a0498adac38b769692" exitCode=0 Oct 30 12:49:48.682922 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:48.682488 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e70aca9a-5wsmz" event={"ID":"437b70b1-fea7-4b27-a34b-39a0acf5103d","Type":"ContainerDied","Data":"f061546f2cdff92d61853e84d54a5e8d548606aeb2ce06a0498adac38b769692"} Oct 30 12:49:48.682922 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:48.682551 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e70aca9a-5wsmz" event={"ID":"437b70b1-fea7-4b27-a34b-39a0acf5103d","Type":"ContainerStarted","Data":"b47b3a50dfe96c89acecf6f3d1e8796dacc5fb63449cb8c904aa535945848022"} Oct 30 12:49:50.039917 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:49:50.039233 4908 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vn6vv" containerName="extractor" Oct 30 12:49:50.161218 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:50.161195 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e70aca9a-5wsmz" Oct 30 12:49:50.286270 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:50.286239 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/437b70b1-fea7-4b27-a34b-39a0acf5103d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/437b70b1-fea7-4b27-a34b-39a0acf5103d-maas-api-openshift-ca\") pod \"437b70b1-fea7-4b27-a34b-39a0acf5103d\" (UID: \"437b70b1-fea7-4b27-a34b-39a0acf5103d\") " Oct 30 12:49:50.286421 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:50.286337 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/437b70b1-fea7-4b27-a34b-39a0acf5103d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/437b70b1-fea7-4b27-a34b-39a0acf5103d-maas-api-xks-ca\") pod \"437b70b1-fea7-4b27-a34b-39a0acf5103d\" (UID: \"437b70b1-fea7-4b27-a34b-39a0acf5103d\") " Oct 30 12:49:50.286421 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:50.286369 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/437b70b1-fea7-4b27-a34b-39a0acf5103d-kube-api-access-r8882\" (UniqueName: \"kubernetes.io/projected/437b70b1-fea7-4b27-a34b-39a0acf5103d-kube-api-access-r8882\") pod \"437b70b1-fea7-4b27-a34b-39a0acf5103d\" (UID: \"437b70b1-fea7-4b27-a34b-39a0acf5103d\") " Oct 30 12:49:50.286639 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:50.286617 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/437b70b1-fea7-4b27-a34b-39a0acf5103d-maas-api-openshift-ca" pod "437b70b1-fea7-4b27-a34b-39a0acf5103d" (UID: "437b70b1-fea7-4b27-a34b-39a0acf5103d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:49:50.287052 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:50.287028 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/437b70b1-fea7-4b27-a34b-39a0acf5103d-maas-api-openshift-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:49:50.288413 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:50.288387 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/437b70b1-fea7-4b27-a34b-39a0acf5103d-maas-api-xks-ca" pod "437b70b1-fea7-4b27-a34b-39a0acf5103d" (UID: "437b70b1-fea7-4b27-a34b-39a0acf5103d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:49:50.288413 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:50.288404 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/437b70b1-fea7-4b27-a34b-39a0acf5103d-kube-api-access-r8882" pod "437b70b1-fea7-4b27-a34b-39a0acf5103d" (UID: "437b70b1-fea7-4b27-a34b-39a0acf5103d"). InnerVolumeSpecName "kube-api-access-r8882". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:49:50.388085 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:50.388057 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/437b70b1-fea7-4b27-a34b-39a0acf5103d-maas-api-xks-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:49:50.388085 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:50.388083 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r8882\" (UniqueName: \"kubernetes.io/projected/437b70b1-fea7-4b27-a34b-39a0acf5103d-kube-api-access-r8882\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:49:50.752763 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:50.752735 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e70aca9a-5wsmz" Oct 30 12:49:50.752940 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:50.752764 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e70aca9a-5wsmz" event={"ID":"437b70b1-fea7-4b27-a34b-39a0acf5103d","Type":"ContainerDied","Data":"b47b3a50dfe96c89acecf6f3d1e8796dacc5fb63449cb8c904aa535945848022"} Oct 30 12:49:50.752940 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:50.752816 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b47b3a50dfe96c89acecf6f3d1e8796dacc5fb63449cb8c904aa535945848022" Oct 30 12:49:57.737777 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:57.737747 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7ccd5df-njfw8"] Oct 30 12:49:57.738350 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:57.738334 4908 cpu_manager.go:414] "containerMap: removing container" podUID="437b70b1-fea7-4b27-a34b-39a0acf5103d" containerName="revoke-keys" Oct 30 12:49:57.738402 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:57.738351 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="437b70b1-fea7-4b27-a34b-39a0acf5103d" containerName="revoke-keys" Oct 30 12:49:57.738478 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:57.738468 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="437b70b1-fea7-4b27-a34b-39a0acf5103d" containerName="revoke-keys" Oct 30 12:49:57.742285 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:57.742265 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7ccd5df-njfw8" Oct 30 12:49:57.761003 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:57.760980 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7ccd5df-njfw8"] Oct 30 12:49:57.766629 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:57.766608 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 12:49:57.766757 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:57.766670 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 12:49:57.766820 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:57.766760 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 12:49:57.766930 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:57.766889 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s75hq\"" Oct 30 12:49:57.807559 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:57.807522 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zm4p7\" (UniqueName: \"kubernetes.io/projected/09d371a1-43b5-4afd-b990-ed703b359d65-kube-api-access-zm4p7\") pod \"maas-api-revoke-sub-a7ccd5df-njfw8\" (UID: \"09d371a1-43b5-4afd-b990-ed703b359d65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7ccd5df-njfw8" Oct 30 12:49:57.807707 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:57.807667 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09d371a1-43b5-4afd-b990-ed703b359d65-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a7ccd5df-njfw8\" (UID: \"09d371a1-43b5-4afd-b990-ed703b359d65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7ccd5df-njfw8" Oct 30 12:49:57.807805 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:57.807791 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09d371a1-43b5-4afd-b990-ed703b359d65-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a7ccd5df-njfw8\" (UID: \"09d371a1-43b5-4afd-b990-ed703b359d65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7ccd5df-njfw8" Oct 30 12:49:57.891941 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:57.891908 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-4t45f"] Oct 30 12:49:57.892143 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:57.892121 4908 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-4t45f" podUID="866116c2-e247-4ac0-9af7-72fadb1d5c31" containerName="istio-proxy" containerID="cri-o://3acad7e3b6b9e387e2a22c8720e16f2e28039762db8bc1c8c76d7a0083cf4fe8" gracePeriod=30 Oct 30 12:49:57.909528 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:57.909485 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09d371a1-43b5-4afd-b990-ed703b359d65-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a7ccd5df-njfw8\" (UID: \"09d371a1-43b5-4afd-b990-ed703b359d65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7ccd5df-njfw8" Oct 30 12:49:57.909678 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:57.909617 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zm4p7\" (UniqueName: \"kubernetes.io/projected/09d371a1-43b5-4afd-b990-ed703b359d65-kube-api-access-zm4p7\") pod \"maas-api-revoke-sub-a7ccd5df-njfw8\" (UID: \"09d371a1-43b5-4afd-b990-ed703b359d65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7ccd5df-njfw8" Oct 30 12:49:57.909744 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:57.909720 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09d371a1-43b5-4afd-b990-ed703b359d65-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a7ccd5df-njfw8\" (UID: \"09d371a1-43b5-4afd-b990-ed703b359d65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7ccd5df-njfw8" Oct 30 12:49:57.910630 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:57.910604 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09d371a1-43b5-4afd-b990-ed703b359d65-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a7ccd5df-njfw8\" (UID: \"09d371a1-43b5-4afd-b990-ed703b359d65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7ccd5df-njfw8" Oct 30 12:49:57.912387 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:57.912362 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09d371a1-43b5-4afd-b990-ed703b359d65-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a7ccd5df-njfw8\" (UID: \"09d371a1-43b5-4afd-b990-ed703b359d65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7ccd5df-njfw8" Oct 30 12:49:57.932249 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:57.932194 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zm4p7\" (UniqueName: \"kubernetes.io/projected/09d371a1-43b5-4afd-b990-ed703b359d65-kube-api-access-zm4p7\") pod \"maas-api-revoke-sub-a7ccd5df-njfw8\" (UID: \"09d371a1-43b5-4afd-b990-ed703b359d65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7ccd5df-njfw8" Oct 30 12:49:58.056103 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:58.056029 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7ccd5df-njfw8" Oct 30 12:49:58.371488 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:58.371452 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7ccd5df-njfw8"] Oct 30 12:49:58.863972 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:58.863929 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7ccd5df-njfw8" event={"ID":"09d371a1-43b5-4afd-b990-ed703b359d65","Type":"ContainerStarted","Data":"c96370d1fb73ea1d884ef0b4b74f994a1a389e6f0f0f6a0c92c8f86eb74b5691"} Oct 30 12:49:58.864464 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:58.864027 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7ccd5df-njfw8" event={"ID":"09d371a1-43b5-4afd-b990-ed703b359d65","Type":"ContainerStarted","Data":"60918e2ebf23ef1ab9b9b998029ed5e90bd85aecea4395b2aef36e5bde9d035d"} Oct 30 12:49:58.914424 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:58.914372 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7ccd5df-njfw8" podStartSLOduration=1.9143586639999999 podStartE2EDuration="1.914358664s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:49:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:49:58.91278073 +0000 UTC m=+3355.427209476" watchObservedRunningTime="2026-10-30 12:49:58.914358664 +0000 UTC m=+3355.428787408" Oct 30 12:49:59.905342 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:59.905306 4908 generic.go:356] "Generic (PLEG): container finished" podID="09d371a1-43b5-4afd-b990-ed703b359d65" containerID="c96370d1fb73ea1d884ef0b4b74f994a1a389e6f0f0f6a0c92c8f86eb74b5691" exitCode=0 Oct 30 12:49:59.906258 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:49:59.906212 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7ccd5df-njfw8" event={"ID":"09d371a1-43b5-4afd-b990-ed703b359d65","Type":"ContainerDied","Data":"c96370d1fb73ea1d884ef0b4b74f994a1a389e6f0f0f6a0c92c8f86eb74b5691"} Oct 30 12:50:01.136024 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:01.135996 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7ccd5df-njfw8" Oct 30 12:50:01.374918 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:01.360784 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09d371a1-43b5-4afd-b990-ed703b359d65-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09d371a1-43b5-4afd-b990-ed703b359d65-maas-api-openshift-ca\") pod \"09d371a1-43b5-4afd-b990-ed703b359d65\" (UID: \"09d371a1-43b5-4afd-b990-ed703b359d65\") " Oct 30 12:50:01.374918 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:01.360927 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/09d371a1-43b5-4afd-b990-ed703b359d65-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09d371a1-43b5-4afd-b990-ed703b359d65-maas-api-xks-ca\") pod \"09d371a1-43b5-4afd-b990-ed703b359d65\" (UID: \"09d371a1-43b5-4afd-b990-ed703b359d65\") " Oct 30 12:50:01.374918 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:01.360979 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09d371a1-43b5-4afd-b990-ed703b359d65-kube-api-access-zm4p7\" (UniqueName: \"kubernetes.io/projected/09d371a1-43b5-4afd-b990-ed703b359d65-kube-api-access-zm4p7\") pod \"09d371a1-43b5-4afd-b990-ed703b359d65\" (UID: \"09d371a1-43b5-4afd-b990-ed703b359d65\") " Oct 30 12:50:01.374918 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:01.370249 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09d371a1-43b5-4afd-b990-ed703b359d65-maas-api-openshift-ca" pod "09d371a1-43b5-4afd-b990-ed703b359d65" (UID: "09d371a1-43b5-4afd-b990-ed703b359d65"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:50:01.401246 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:01.391649 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09d371a1-43b5-4afd-b990-ed703b359d65-kube-api-access-zm4p7" pod "09d371a1-43b5-4afd-b990-ed703b359d65" (UID: "09d371a1-43b5-4afd-b990-ed703b359d65"). InnerVolumeSpecName "kube-api-access-zm4p7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:50:01.403518 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:01.403484 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09d371a1-43b5-4afd-b990-ed703b359d65-maas-api-xks-ca" pod "09d371a1-43b5-4afd-b990-ed703b359d65" (UID: "09d371a1-43b5-4afd-b990-ed703b359d65"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:50:01.486854 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:01.482346 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09d371a1-43b5-4afd-b990-ed703b359d65-maas-api-openshift-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:50:01.486854 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:01.482391 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09d371a1-43b5-4afd-b990-ed703b359d65-maas-api-xks-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:50:01.486854 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:01.482406 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zm4p7\" (UniqueName: \"kubernetes.io/projected/09d371a1-43b5-4afd-b990-ed703b359d65-kube-api-access-zm4p7\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:50:01.929626 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:01.929590 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7ccd5df-njfw8" Oct 30 12:50:01.929626 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:01.929613 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7ccd5df-njfw8" event={"ID":"09d371a1-43b5-4afd-b990-ed703b359d65","Type":"ContainerDied","Data":"60918e2ebf23ef1ab9b9b998029ed5e90bd85aecea4395b2aef36e5bde9d035d"} Oct 30 12:50:01.929853 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:01.929683 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="60918e2ebf23ef1ab9b9b998029ed5e90bd85aecea4395b2aef36e5bde9d035d" Oct 30 12:50:02.935947 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:02.935915 4908 patch_prober.go:29] interesting pod/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-4t45f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body= Oct 30 12:50:02.936320 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:02.935981 4908 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-4t45f" podUID="866116c2-e247-4ac0-9af7-72fadb1d5c31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" Oct 30 12:50:03.170307 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:03.170252 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-4t45f" Oct 30 12:50:03.277452 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:03.277414 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/866116c2-e247-4ac0-9af7-72fadb1d5c31-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/866116c2-e247-4ac0-9af7-72fadb1d5c31-istio-data\") pod \"866116c2-e247-4ac0-9af7-72fadb1d5c31\" (UID: \"866116c2-e247-4ac0-9af7-72fadb1d5c31\") " Oct 30 12:50:03.277612 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:03.277503 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/866116c2-e247-4ac0-9af7-72fadb1d5c31-istio-token\" (UniqueName: \"kubernetes.io/projected/866116c2-e247-4ac0-9af7-72fadb1d5c31-istio-token\") pod \"866116c2-e247-4ac0-9af7-72fadb1d5c31\" (UID: \"866116c2-e247-4ac0-9af7-72fadb1d5c31\") " Oct 30 12:50:03.277612 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:03.277577 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/866116c2-e247-4ac0-9af7-72fadb1d5c31-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/866116c2-e247-4ac0-9af7-72fadb1d5c31-istio-envoy\") pod \"866116c2-e247-4ac0-9af7-72fadb1d5c31\" (UID: \"866116c2-e247-4ac0-9af7-72fadb1d5c31\") " Oct 30 12:50:03.277768 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:03.277737 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/866116c2-e247-4ac0-9af7-72fadb1d5c31-istio-data" pod "866116c2-e247-4ac0-9af7-72fadb1d5c31" (UID: "866116c2-e247-4ac0-9af7-72fadb1d5c31"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:50:03.277843 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:03.277755 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/866116c2-e247-4ac0-9af7-72fadb1d5c31-kube-api-access-znhkn\" (UniqueName: \"kubernetes.io/projected/866116c2-e247-4ac0-9af7-72fadb1d5c31-kube-api-access-znhkn\") pod \"866116c2-e247-4ac0-9af7-72fadb1d5c31\" (UID: \"866116c2-e247-4ac0-9af7-72fadb1d5c31\") " Oct 30 12:50:03.277911 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:03.277855 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/866116c2-e247-4ac0-9af7-72fadb1d5c31-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/866116c2-e247-4ac0-9af7-72fadb1d5c31-workload-certs\") pod \"866116c2-e247-4ac0-9af7-72fadb1d5c31\" (UID: \"866116c2-e247-4ac0-9af7-72fadb1d5c31\") " Oct 30 12:50:03.277911 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:03.277896 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/866116c2-e247-4ac0-9af7-72fadb1d5c31-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/866116c2-e247-4ac0-9af7-72fadb1d5c31-istiod-ca-cert\") pod \"866116c2-e247-4ac0-9af7-72fadb1d5c31\" (UID: \"866116c2-e247-4ac0-9af7-72fadb1d5c31\") " Oct 30 12:50:03.278024 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:03.277972 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/866116c2-e247-4ac0-9af7-72fadb1d5c31-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/866116c2-e247-4ac0-9af7-72fadb1d5c31-credential-socket\") pod \"866116c2-e247-4ac0-9af7-72fadb1d5c31\" (UID: \"866116c2-e247-4ac0-9af7-72fadb1d5c31\") " Oct 30 12:50:03.278024 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:03.278017 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/866116c2-e247-4ac0-9af7-72fadb1d5c31-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/866116c2-e247-4ac0-9af7-72fadb1d5c31-workload-socket\") pod \"866116c2-e247-4ac0-9af7-72fadb1d5c31\" (UID: \"866116c2-e247-4ac0-9af7-72fadb1d5c31\") " Oct 30 12:50:03.278137 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:03.278045 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/866116c2-e247-4ac0-9af7-72fadb1d5c31-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/866116c2-e247-4ac0-9af7-72fadb1d5c31-istio-podinfo\") pod \"866116c2-e247-4ac0-9af7-72fadb1d5c31\" (UID: \"866116c2-e247-4ac0-9af7-72fadb1d5c31\") " Oct 30 12:50:03.278358 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:03.278335 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/866116c2-e247-4ac0-9af7-72fadb1d5c31-istiod-ca-cert" pod "866116c2-e247-4ac0-9af7-72fadb1d5c31" (UID: "866116c2-e247-4ac0-9af7-72fadb1d5c31"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:50:03.278575 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:03.278555 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/866116c2-e247-4ac0-9af7-72fadb1d5c31-workload-certs" pod "866116c2-e247-4ac0-9af7-72fadb1d5c31" (UID: "866116c2-e247-4ac0-9af7-72fadb1d5c31"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:50:03.278853 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:03.278714 4908 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/866116c2-e247-4ac0-9af7-72fadb1d5c31-workload-certs\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:50:03.278853 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:03.278741 4908 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/866116c2-e247-4ac0-9af7-72fadb1d5c31-istiod-ca-cert\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:50:03.278853 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:03.278755 4908 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/866116c2-e247-4ac0-9af7-72fadb1d5c31-istio-data\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:50:03.278853 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:03.278722 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/866116c2-e247-4ac0-9af7-72fadb1d5c31-workload-socket" pod "866116c2-e247-4ac0-9af7-72fadb1d5c31" (UID: "866116c2-e247-4ac0-9af7-72fadb1d5c31"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:50:03.279151 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:03.278887 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/866116c2-e247-4ac0-9af7-72fadb1d5c31-credential-socket" pod "866116c2-e247-4ac0-9af7-72fadb1d5c31" (UID: "866116c2-e247-4ac0-9af7-72fadb1d5c31"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:50:03.281474 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:03.281440 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/866116c2-e247-4ac0-9af7-72fadb1d5c31-istio-token" pod "866116c2-e247-4ac0-9af7-72fadb1d5c31" (UID: "866116c2-e247-4ac0-9af7-72fadb1d5c31"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:50:03.282110 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:03.282073 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/866116c2-e247-4ac0-9af7-72fadb1d5c31-istio-podinfo" pod "866116c2-e247-4ac0-9af7-72fadb1d5c31" (UID: "866116c2-e247-4ac0-9af7-72fadb1d5c31"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 12:50:03.283507 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:03.283475 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/866116c2-e247-4ac0-9af7-72fadb1d5c31-kube-api-access-znhkn" pod "866116c2-e247-4ac0-9af7-72fadb1d5c31" (UID: "866116c2-e247-4ac0-9af7-72fadb1d5c31"). InnerVolumeSpecName "kube-api-access-znhkn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:50:03.283686 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:03.283663 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/866116c2-e247-4ac0-9af7-72fadb1d5c31-istio-envoy" pod "866116c2-e247-4ac0-9af7-72fadb1d5c31" (UID: "866116c2-e247-4ac0-9af7-72fadb1d5c31"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:50:03.380513 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:03.380480 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-znhkn\" (UniqueName: \"kubernetes.io/projected/866116c2-e247-4ac0-9af7-72fadb1d5c31-kube-api-access-znhkn\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:50:03.380513 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:03.380509 4908 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/866116c2-e247-4ac0-9af7-72fadb1d5c31-credential-socket\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:50:03.380513 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:03.380520 4908 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/866116c2-e247-4ac0-9af7-72fadb1d5c31-workload-socket\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:50:03.380817 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:03.380530 4908 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/866116c2-e247-4ac0-9af7-72fadb1d5c31-istio-podinfo\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:50:03.380817 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:03.380540 4908 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/866116c2-e247-4ac0-9af7-72fadb1d5c31-istio-token\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:50:03.380817 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:03.380548 4908 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/866116c2-e247-4ac0-9af7-72fadb1d5c31-istio-envoy\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:50:03.945993 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:03.945911 4908 generic.go:356] "Generic (PLEG): container finished" podID="866116c2-e247-4ac0-9af7-72fadb1d5c31" containerID="3acad7e3b6b9e387e2a22c8720e16f2e28039762db8bc1c8c76d7a0083cf4fe8" exitCode=0 Oct 30 12:50:03.945993 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:03.945956 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-4t45f" Oct 30 12:50:03.946422 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:03.945998 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-4t45f" event={"ID":"866116c2-e247-4ac0-9af7-72fadb1d5c31","Type":"ContainerDied","Data":"3acad7e3b6b9e387e2a22c8720e16f2e28039762db8bc1c8c76d7a0083cf4fe8"} Oct 30 12:50:03.946422 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:03.946057 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-4t45f" event={"ID":"866116c2-e247-4ac0-9af7-72fadb1d5c31","Type":"ContainerDied","Data":"9691bf79baf35af330c47ce01ebb6674e115c356c8690bd63e09e489d7a86bbe"} Oct 30 12:50:03.946422 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:03.946085 4908 scope.go:122] "RemoveContainer" containerID="3acad7e3b6b9e387e2a22c8720e16f2e28039762db8bc1c8c76d7a0083cf4fe8" Oct 30 12:50:03.957210 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:03.957191 4908 scope.go:122] "RemoveContainer" containerID="3acad7e3b6b9e387e2a22c8720e16f2e28039762db8bc1c8c76d7a0083cf4fe8" Oct 30 12:50:03.957477 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:50:03.957457 4908 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3acad7e3b6b9e387e2a22c8720e16f2e28039762db8bc1c8c76d7a0083cf4fe8\": container with ID starting with 3acad7e3b6b9e387e2a22c8720e16f2e28039762db8bc1c8c76d7a0083cf4fe8 not found: ID does not exist" containerID="3acad7e3b6b9e387e2a22c8720e16f2e28039762db8bc1c8c76d7a0083cf4fe8" Oct 30 12:50:03.957521 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:03.957486 4908 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3acad7e3b6b9e387e2a22c8720e16f2e28039762db8bc1c8c76d7a0083cf4fe8"} err="failed to get container status \"3acad7e3b6b9e387e2a22c8720e16f2e28039762db8bc1c8c76d7a0083cf4fe8\": rpc error: code = NotFound desc = could not find container \"3acad7e3b6b9e387e2a22c8720e16f2e28039762db8bc1c8c76d7a0083cf4fe8\": container with ID starting with 3acad7e3b6b9e387e2a22c8720e16f2e28039762db8bc1c8c76d7a0083cf4fe8 not found: ID does not exist" Oct 30 12:50:04.009601 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:04.009571 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-4t45f"] Oct 30 12:50:04.020034 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:04.020010 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-4t45f"] Oct 30 12:50:05.607444 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:05.607409 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="866116c2-e247-4ac0-9af7-72fadb1d5c31" path="/var/lib/kubelet/pods/866116c2-e247-4ac0-9af7-72fadb1d5c31/volumes" Oct 30 12:50:08.054852 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:08.047454 4908 scope.go:122] "RemoveContainer" containerID="2e4a993c889064abd572deb6699726c4c551100496810ade59ce8f88e2300604" Oct 30 12:50:08.209851 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:08.198925 4908 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-d9b94 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 12:50:08.209851 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:08.198994 4908 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-d9b94" podUID="a5b1685b-e185-4b12-9c20-3c87e9837fd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 12:50:08.611853 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:08.610899 4908 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f920f7/simulator-w0-f920f7-kserve-599b7ff45f-g79d7" podUID="a85e562f-d655-4c22-b4f4-a299a4e494a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:50:08.626849 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:08.616897 4908 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6a7148/premium-w3-6a7148-kserve-67b976c888-tgd6c" podUID="13de4441-e7f6-4cbd-97d9-0f079d11cb8e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:50:08.861852 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:08.848896 4908 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47fd99/premium-w2-47fd99-kserve-5d85f9bb95-hqkmf" podUID="fd2cbeda-dbc4-41f6-a730-e1c6f9343c5e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:50:09.049853 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:09.035919 4908 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-47fd99/premium-w2-47fd99-kserve-5d85f9bb95-hqkmf" podUID="fd2cbeda-dbc4-41f6-a730-e1c6f9343c5e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:50:10.340873 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:10.340811 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-74898s524"] Oct 30 12:50:10.341535 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:10.341506 4908 cpu_manager.go:414] "containerMap: removing container" podUID="09d371a1-43b5-4afd-b990-ed703b359d65" containerName="revoke-keys" Oct 30 12:50:10.341535 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:10.341527 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09d371a1-43b5-4afd-b990-ed703b359d65" containerName="revoke-keys" Oct 30 12:50:10.341694 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:10.341561 4908 cpu_manager.go:414] "containerMap: removing container" podUID="866116c2-e247-4ac0-9af7-72fadb1d5c31" containerName="istio-proxy" Oct 30 12:50:10.341694 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:10.341570 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="866116c2-e247-4ac0-9af7-72fadb1d5c31" containerName="istio-proxy" Oct 30 12:50:10.341799 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:10.341711 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="09d371a1-43b5-4afd-b990-ed703b359d65" containerName="revoke-keys" Oct 30 12:50:10.341799 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:10.341730 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="866116c2-e247-4ac0-9af7-72fadb1d5c31" containerName="istio-proxy" Oct 30 12:50:10.347091 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:10.347067 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-74898s524" Oct 30 12:50:10.354525 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:10.354495 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-74898s524"] Oct 30 12:50:10.377870 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:10.373550 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-f7464f\"/\"agc-cred-d510e13eefb7\"" Oct 30 12:50:10.377870 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:10.373626 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-f7464f\"/\"kube-root-ca.crt\"" Oct 30 12:50:10.377870 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:10.373637 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-f7464f\"/\"e2e-cfl-69b234-provider-cred\"" Oct 30 12:50:10.377870 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:10.373679 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-f7464f\"/\"openshift-service-ca.crt\"" Oct 30 12:50:10.377870 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:10.373758 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-f7464f\"/\"payload-processing-external-model-plugins-e2e-extmod-a-f7464f\"" Oct 30 12:50:10.377870 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:10.373765 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-f7464f\"/\"routing-overlay\"" Oct 30 12:50:10.377870 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:10.373907 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-f7464f\"/\"payload-processing-external-model-e2e--a9926bc2-dockercfg-fx898\"" Oct 30 12:50:10.462850 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:10.458918 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/03056eb5-8870-471d-95b2-e36c40bd0ca0-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-f7464f-74898s524\" (UID: \"03056eb5-8870-471d-95b2-e36c40bd0ca0\") " pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-74898s524" Oct 30 12:50:10.482712 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:10.476209 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qtlp\" (UniqueName: \"kubernetes.io/projected/03056eb5-8870-471d-95b2-e36c40bd0ca0-kube-api-access-7qtlp\") pod \"payload-processing-external-model-e2e-extmod-a-f7464f-74898s524\" (UID: \"03056eb5-8870-471d-95b2-e36c40bd0ca0\") " pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-74898s524" Oct 30 12:50:10.482712 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:10.476311 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03056eb5-8870-471d-95b2-e36c40bd0ca0-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-f7464f-74898s524\" (UID: \"03056eb5-8870-471d-95b2-e36c40bd0ca0\") " pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-74898s524" Oct 30 12:50:10.482712 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:10.476398 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/03056eb5-8870-471d-95b2-e36c40bd0ca0-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-f7464f-74898s524\" (UID: \"03056eb5-8870-471d-95b2-e36c40bd0ca0\") " pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-74898s524" Oct 30 12:50:10.579850 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:10.579030 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qtlp\" (UniqueName: \"kubernetes.io/projected/03056eb5-8870-471d-95b2-e36c40bd0ca0-kube-api-access-7qtlp\") pod \"payload-processing-external-model-e2e-extmod-a-f7464f-74898s524\" (UID: \"03056eb5-8870-471d-95b2-e36c40bd0ca0\") " pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-74898s524" Oct 30 12:50:10.579850 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:10.579107 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03056eb5-8870-471d-95b2-e36c40bd0ca0-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-f7464f-74898s524\" (UID: \"03056eb5-8870-471d-95b2-e36c40bd0ca0\") " pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-74898s524" Oct 30 12:50:10.579850 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:10.579178 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/03056eb5-8870-471d-95b2-e36c40bd0ca0-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-f7464f-74898s524\" (UID: \"03056eb5-8870-471d-95b2-e36c40bd0ca0\") " pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-74898s524" Oct 30 12:50:10.579850 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:10.579380 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/03056eb5-8870-471d-95b2-e36c40bd0ca0-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-f7464f-74898s524\" (UID: \"03056eb5-8870-471d-95b2-e36c40bd0ca0\") " pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-74898s524" Oct 30 12:50:10.579850 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:50:10.579615 4908 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-f7464f/agc-cred-d510e13eefb7: secret "agc-cred-d510e13eefb7" not found Oct 30 12:50:10.579850 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:50:10.579639 4908 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-74898s524: secret "agc-cred-d510e13eefb7" not found Oct 30 12:50:10.579850 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:50:10.579716 4908 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/03056eb5-8870-471d-95b2-e36c40bd0ca0-provider-credentials podName:03056eb5-8870-471d-95b2-e36c40bd0ca0 nodeName:}" failed. No retries permitted until 2026-10-30 12:50:11.079692326 +0000 UTC m=+3367.594121051 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/03056eb5-8870-471d-95b2-e36c40bd0ca0-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-f7464f-74898s524" (UID: "03056eb5-8870-471d-95b2-e36c40bd0ca0") : secret "agc-cred-d510e13eefb7" not found Oct 30 12:50:10.586853 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:10.584703 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03056eb5-8870-471d-95b2-e36c40bd0ca0-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-f7464f-74898s524\" (UID: \"03056eb5-8870-471d-95b2-e36c40bd0ca0\") " pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-74898s524" Oct 30 12:50:10.586853 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:10.585465 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/03056eb5-8870-471d-95b2-e36c40bd0ca0-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-f7464f-74898s524\" (UID: \"03056eb5-8870-471d-95b2-e36c40bd0ca0\") " pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-74898s524" Oct 30 12:50:10.603491 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:10.603413 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qtlp\" (UniqueName: \"kubernetes.io/projected/03056eb5-8870-471d-95b2-e36c40bd0ca0-kube-api-access-7qtlp\") pod \"payload-processing-external-model-e2e-extmod-a-f7464f-74898s524\" (UID: \"03056eb5-8870-471d-95b2-e36c40bd0ca0\") " pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-74898s524" Oct 30 12:50:10.963804 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:10.963731 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10629caa-2bx9z"] Oct 30 12:50:10.968037 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:10.968021 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10629caa-2bx9z" Oct 30 12:50:10.982192 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:10.982008 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 12:50:10.982192 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:10.982146 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s75hq\"" Oct 30 12:50:10.985735 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:10.985187 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vv67\" (UniqueName: \"kubernetes.io/projected/79714daf-eda5-4d2f-944d-385759f941ae-kube-api-access-7vv67\") pod \"maas-api-revoke-sub-10629caa-2bx9z\" (UID: \"79714daf-eda5-4d2f-944d-385759f941ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10629caa-2bx9z" Oct 30 12:50:10.985735 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:10.985271 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79714daf-eda5-4d2f-944d-385759f941ae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-10629caa-2bx9z\" (UID: \"79714daf-eda5-4d2f-944d-385759f941ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10629caa-2bx9z" Oct 30 12:50:10.985735 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:10.985418 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79714daf-eda5-4d2f-944d-385759f941ae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-10629caa-2bx9z\" (UID: \"79714daf-eda5-4d2f-944d-385759f941ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10629caa-2bx9z" Oct 30 12:50:10.988748 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:10.988725 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10629caa-2bx9z"] Oct 30 12:50:10.993051 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:10.993030 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 12:50:10.993143 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:10.993079 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 12:50:11.086618 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:11.086588 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vv67\" (UniqueName: \"kubernetes.io/projected/79714daf-eda5-4d2f-944d-385759f941ae-kube-api-access-7vv67\") pod \"maas-api-revoke-sub-10629caa-2bx9z\" (UID: \"79714daf-eda5-4d2f-944d-385759f941ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10629caa-2bx9z" Oct 30 12:50:11.086798 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:11.086627 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/03056eb5-8870-471d-95b2-e36c40bd0ca0-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-f7464f-74898s524\" (UID: \"03056eb5-8870-471d-95b2-e36c40bd0ca0\") " pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-74898s524" Oct 30 12:50:11.086798 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:11.086651 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79714daf-eda5-4d2f-944d-385759f941ae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-10629caa-2bx9z\" (UID: \"79714daf-eda5-4d2f-944d-385759f941ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10629caa-2bx9z" Oct 30 12:50:11.086798 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:50:11.086774 4908 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-f7464f/agc-cred-d510e13eefb7: secret "agc-cred-d510e13eefb7" not found Oct 30 12:50:11.086798 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:11.086786 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79714daf-eda5-4d2f-944d-385759f941ae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-10629caa-2bx9z\" (UID: \"79714daf-eda5-4d2f-944d-385759f941ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10629caa-2bx9z" Oct 30 12:50:11.087073 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:50:11.086802 4908 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-74898s524: secret "agc-cred-d510e13eefb7" not found Oct 30 12:50:11.087073 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:50:11.086875 4908 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/03056eb5-8870-471d-95b2-e36c40bd0ca0-provider-credentials podName:03056eb5-8870-471d-95b2-e36c40bd0ca0 nodeName:}" failed. No retries permitted until 2026-10-30 12:50:12.086856209 +0000 UTC m=+3368.601284960 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/03056eb5-8870-471d-95b2-e36c40bd0ca0-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-f7464f-74898s524" (UID: "03056eb5-8870-471d-95b2-e36c40bd0ca0") : secret "agc-cred-d510e13eefb7" not found Oct 30 12:50:11.087627 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:11.087594 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79714daf-eda5-4d2f-944d-385759f941ae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-10629caa-2bx9z\" (UID: \"79714daf-eda5-4d2f-944d-385759f941ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10629caa-2bx9z" Oct 30 12:50:11.089132 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:11.089102 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79714daf-eda5-4d2f-944d-385759f941ae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-10629caa-2bx9z\" (UID: \"79714daf-eda5-4d2f-944d-385759f941ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10629caa-2bx9z" Oct 30 12:50:11.109477 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:11.109446 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vv67\" (UniqueName: \"kubernetes.io/projected/79714daf-eda5-4d2f-944d-385759f941ae-kube-api-access-7vv67\") pod \"maas-api-revoke-sub-10629caa-2bx9z\" (UID: \"79714daf-eda5-4d2f-944d-385759f941ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10629caa-2bx9z" Oct 30 12:50:11.282313 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:11.282289 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10629caa-2bx9z" Oct 30 12:50:11.583328 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:11.583302 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10629caa-2bx9z"] Oct 30 12:50:11.586159 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:50:11.586131 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79714daf_eda5_4d2f_944d_385759f941ae.slice/crio-f8a11d6c9f6cdc18f1268f0f95fa6dbb72e58ca7a183873a1e2e1e94215d0d49 WatchSource:0}: Error finding container f8a11d6c9f6cdc18f1268f0f95fa6dbb72e58ca7a183873a1e2e1e94215d0d49: Status 404 returned error can't find the container with id f8a11d6c9f6cdc18f1268f0f95fa6dbb72e58ca7a183873a1e2e1e94215d0d49 Oct 30 12:50:11.866665 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:11.866636 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-74898s524"] Oct 30 12:50:11.867105 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:50:11.867078 4908 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[provider-credentials], unattached volumes=[], failed to process volumes=[]: context canceled" pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-74898s524" podUID="03056eb5-8870-471d-95b2-e36c40bd0ca0" Oct 30 12:50:12.101573 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:12.101539 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/03056eb5-8870-471d-95b2-e36c40bd0ca0-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-f7464f-74898s524\" (UID: \"03056eb5-8870-471d-95b2-e36c40bd0ca0\") " pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-74898s524" Oct 30 12:50:12.101775 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:50:12.101702 4908 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-f7464f/agc-cred-d510e13eefb7: secret "agc-cred-d510e13eefb7" not found Oct 30 12:50:12.101775 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:50:12.101727 4908 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-74898s524: secret "agc-cred-d510e13eefb7" not found Oct 30 12:50:12.101929 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:50:12.101790 4908 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/03056eb5-8870-471d-95b2-e36c40bd0ca0-provider-credentials podName:03056eb5-8870-471d-95b2-e36c40bd0ca0 nodeName:}" failed. No retries permitted until 2026-10-30 12:50:14.101770505 +0000 UTC m=+3370.616199241 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/03056eb5-8870-471d-95b2-e36c40bd0ca0-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-f7464f-74898s524" (UID: "03056eb5-8870-471d-95b2-e36c40bd0ca0") : secret "agc-cred-d510e13eefb7" not found Oct 30 12:50:12.145698 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:12.145623 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10629caa-2bx9z" event={"ID":"79714daf-eda5-4d2f-944d-385759f941ae","Type":"ContainerStarted","Data":"3306aa9d5fe035136e3d7e5ddc32c19e086eac9b8a14bd628f4996699619cc0f"} Oct 30 12:50:12.145698 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:12.145673 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10629caa-2bx9z" event={"ID":"79714daf-eda5-4d2f-944d-385759f941ae","Type":"ContainerStarted","Data":"f8a11d6c9f6cdc18f1268f0f95fa6dbb72e58ca7a183873a1e2e1e94215d0d49"} Oct 30 12:50:12.145933 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:12.145708 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-74898s524" Oct 30 12:50:12.153128 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:12.153107 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-74898s524" Oct 30 12:50:12.183692 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:12.183646 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10629caa-2bx9z" podStartSLOduration=2.183633308 podStartE2EDuration="2.183633308s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:50:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:50:12.182798538 +0000 UTC m=+3368.697227284" watchObservedRunningTime="2026-10-30 12:50:12.183633308 +0000 UTC m=+3368.698062052" Oct 30 12:50:12.202649 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:12.202628 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03056eb5-8870-471d-95b2-e36c40bd0ca0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03056eb5-8870-471d-95b2-e36c40bd0ca0-plugins-config-volume\") pod \"03056eb5-8870-471d-95b2-e36c40bd0ca0\" (UID: \"03056eb5-8870-471d-95b2-e36c40bd0ca0\") " Oct 30 12:50:12.202785 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:12.202772 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03056eb5-8870-471d-95b2-e36c40bd0ca0-kube-api-access-7qtlp\" (UniqueName: \"kubernetes.io/projected/03056eb5-8870-471d-95b2-e36c40bd0ca0-kube-api-access-7qtlp\") pod \"03056eb5-8870-471d-95b2-e36c40bd0ca0\" (UID: \"03056eb5-8870-471d-95b2-e36c40bd0ca0\") " Oct 30 12:50:12.202864 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:12.202855 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03056eb5-8870-471d-95b2-e36c40bd0ca0-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/03056eb5-8870-471d-95b2-e36c40bd0ca0-routing-overlay-volume\") pod \"03056eb5-8870-471d-95b2-e36c40bd0ca0\" (UID: \"03056eb5-8870-471d-95b2-e36c40bd0ca0\") " Oct 30 12:50:12.203170 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:12.203141 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03056eb5-8870-471d-95b2-e36c40bd0ca0-plugins-config-volume" pod "03056eb5-8870-471d-95b2-e36c40bd0ca0" (UID: "03056eb5-8870-471d-95b2-e36c40bd0ca0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:50:12.203170 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:12.203150 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03056eb5-8870-471d-95b2-e36c40bd0ca0-routing-overlay-volume" pod "03056eb5-8870-471d-95b2-e36c40bd0ca0" (UID: "03056eb5-8870-471d-95b2-e36c40bd0ca0"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:50:12.203440 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:12.203421 4908 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/03056eb5-8870-471d-95b2-e36c40bd0ca0-routing-overlay-volume\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:50:12.203503 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:12.203444 4908 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03056eb5-8870-471d-95b2-e36c40bd0ca0-plugins-config-volume\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:50:12.205030 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:12.205008 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03056eb5-8870-471d-95b2-e36c40bd0ca0-kube-api-access-7qtlp" pod "03056eb5-8870-471d-95b2-e36c40bd0ca0" (UID: "03056eb5-8870-471d-95b2-e36c40bd0ca0"). InnerVolumeSpecName "kube-api-access-7qtlp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:50:12.304631 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:12.304607 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7qtlp\" (UniqueName: \"kubernetes.io/projected/03056eb5-8870-471d-95b2-e36c40bd0ca0-kube-api-access-7qtlp\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:50:13.155056 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:13.155024 4908 generic.go:356] "Generic (PLEG): container finished" podID="79714daf-eda5-4d2f-944d-385759f941ae" containerID="3306aa9d5fe035136e3d7e5ddc32c19e086eac9b8a14bd628f4996699619cc0f" exitCode=0 Oct 30 12:50:13.155533 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:13.155115 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10629caa-2bx9z" event={"ID":"79714daf-eda5-4d2f-944d-385759f941ae","Type":"ContainerDied","Data":"3306aa9d5fe035136e3d7e5ddc32c19e086eac9b8a14bd628f4996699619cc0f"} Oct 30 12:50:13.155533 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:13.155181 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-74898s524" Oct 30 12:50:13.263118 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:13.263085 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-74898s524"] Oct 30 12:50:13.269414 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:13.269382 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-74898s524"] Oct 30 12:50:13.324625 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:13.324597 4908 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/03056eb5-8870-471d-95b2-e36c40bd0ca0-provider-credentials\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:50:13.606437 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:13.606405 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03056eb5-8870-471d-95b2-e36c40bd0ca0" path="/var/lib/kubelet/pods/03056eb5-8870-471d-95b2-e36c40bd0ca0/volumes" Oct 30 12:50:14.391890 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:14.391864 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10629caa-2bx9z" Oct 30 12:50:14.440402 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:14.440330 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79714daf-eda5-4d2f-944d-385759f941ae-kube-api-access-7vv67\" (UniqueName: \"kubernetes.io/projected/79714daf-eda5-4d2f-944d-385759f941ae-kube-api-access-7vv67\") pod \"79714daf-eda5-4d2f-944d-385759f941ae\" (UID: \"79714daf-eda5-4d2f-944d-385759f941ae\") " Oct 30 12:50:14.440554 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:14.440426 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79714daf-eda5-4d2f-944d-385759f941ae-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79714daf-eda5-4d2f-944d-385759f941ae-maas-api-openshift-ca\") pod \"79714daf-eda5-4d2f-944d-385759f941ae\" (UID: \"79714daf-eda5-4d2f-944d-385759f941ae\") " Oct 30 12:50:14.440624 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:14.440597 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/79714daf-eda5-4d2f-944d-385759f941ae-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79714daf-eda5-4d2f-944d-385759f941ae-maas-api-xks-ca\") pod \"79714daf-eda5-4d2f-944d-385759f941ae\" (UID: \"79714daf-eda5-4d2f-944d-385759f941ae\") " Oct 30 12:50:14.440882 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:14.440820 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79714daf-eda5-4d2f-944d-385759f941ae-maas-api-openshift-ca" pod "79714daf-eda5-4d2f-944d-385759f941ae" (UID: "79714daf-eda5-4d2f-944d-385759f941ae"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:50:14.441274 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:14.441249 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79714daf-eda5-4d2f-944d-385759f941ae-maas-api-openshift-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:50:14.443067 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:14.443036 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79714daf-eda5-4d2f-944d-385759f941ae-maas-api-xks-ca" pod "79714daf-eda5-4d2f-944d-385759f941ae" (UID: "79714daf-eda5-4d2f-944d-385759f941ae"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:50:14.443178 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:14.443047 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79714daf-eda5-4d2f-944d-385759f941ae-kube-api-access-7vv67" pod "79714daf-eda5-4d2f-944d-385759f941ae" (UID: "79714daf-eda5-4d2f-944d-385759f941ae"). InnerVolumeSpecName "kube-api-access-7vv67". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:50:14.542509 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:14.542475 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7vv67\" (UniqueName: \"kubernetes.io/projected/79714daf-eda5-4d2f-944d-385759f941ae-kube-api-access-7vv67\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:50:14.542509 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:14.542505 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79714daf-eda5-4d2f-944d-385759f941ae-maas-api-xks-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:50:15.175538 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:15.175498 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10629caa-2bx9z" Oct 30 12:50:15.175538 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:15.175532 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10629caa-2bx9z" event={"ID":"79714daf-eda5-4d2f-944d-385759f941ae","Type":"ContainerDied","Data":"f8a11d6c9f6cdc18f1268f0f95fa6dbb72e58ca7a183873a1e2e1e94215d0d49"} Oct 30 12:50:15.175752 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:15.175579 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f8a11d6c9f6cdc18f1268f0f95fa6dbb72e58ca7a183873a1e2e1e94215d0d49" Oct 30 12:50:19.688755 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:50:19.688643 4908 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vn6vv" containerName="extractor" Oct 30 12:50:23.395337 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:23.395299 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-6944bnb7n"] Oct 30 12:50:23.396238 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:23.396214 4908 cpu_manager.go:414] "containerMap: removing container" podUID="79714daf-eda5-4d2f-944d-385759f941ae" containerName="revoke-keys" Oct 30 12:50:23.396238 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:23.396239 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79714daf-eda5-4d2f-944d-385759f941ae" containerName="revoke-keys" Oct 30 12:50:23.396469 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:23.396455 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="79714daf-eda5-4d2f-944d-385759f941ae" containerName="revoke-keys" Oct 30 12:50:23.400558 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:23.400527 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-6944bnb7n" Oct 30 12:50:23.419755 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:23.419730 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-6944bnb7n"] Oct 30 12:50:23.424384 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:23.424361 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-f7464f\"/\"agc-cred-3cd06b9a0e1c\"" Oct 30 12:50:23.425488 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:23.425457 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-f7464f\"/\"openshift-service-ca.crt\"" Oct 30 12:50:23.425776 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:23.425718 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-f7464f\"/\"kube-root-ca.crt\"" Oct 30 12:50:23.426090 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:23.426069 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-f7464f\"/\"routing-overlay\"" Oct 30 12:50:23.426202 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:23.426110 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-f7464f\"/\"payload-processing-external-model-plugins-e2e-extmod-a-f7464f\"" Oct 30 12:50:23.426338 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:23.426320 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-f7464f\"/\"payload-processing-external-model-e2e--a9926bc2-dockercfg-fx898\"" Oct 30 12:50:23.488758 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:23.488245 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e49d3637-7899-4ea5-9544-7550b2ab8451-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-f7464f-6944bnb7n\" (UID: \"e49d3637-7899-4ea5-9544-7550b2ab8451\") " pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-6944bnb7n" Oct 30 12:50:23.488758 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:23.488397 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qt72v\" (UniqueName: \"kubernetes.io/projected/e49d3637-7899-4ea5-9544-7550b2ab8451-kube-api-access-qt72v\") pod \"payload-processing-external-model-e2e-extmod-a-f7464f-6944bnb7n\" (UID: \"e49d3637-7899-4ea5-9544-7550b2ab8451\") " pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-6944bnb7n" Oct 30 12:50:23.488758 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:23.488474 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e49d3637-7899-4ea5-9544-7550b2ab8451-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-f7464f-6944bnb7n\" (UID: \"e49d3637-7899-4ea5-9544-7550b2ab8451\") " pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-6944bnb7n" Oct 30 12:50:23.488758 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:23.488542 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e49d3637-7899-4ea5-9544-7550b2ab8451-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-f7464f-6944bnb7n\" (UID: \"e49d3637-7899-4ea5-9544-7550b2ab8451\") " pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-6944bnb7n" Oct 30 12:50:23.602892 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:23.593744 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qt72v\" (UniqueName: \"kubernetes.io/projected/e49d3637-7899-4ea5-9544-7550b2ab8451-kube-api-access-qt72v\") pod \"payload-processing-external-model-e2e-extmod-a-f7464f-6944bnb7n\" (UID: \"e49d3637-7899-4ea5-9544-7550b2ab8451\") " pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-6944bnb7n" Oct 30 12:50:23.650931 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:23.639220 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e49d3637-7899-4ea5-9544-7550b2ab8451-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-f7464f-6944bnb7n\" (UID: \"e49d3637-7899-4ea5-9544-7550b2ab8451\") " pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-6944bnb7n" Oct 30 12:50:23.650931 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:23.639595 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e49d3637-7899-4ea5-9544-7550b2ab8451-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-f7464f-6944bnb7n\" (UID: \"e49d3637-7899-4ea5-9544-7550b2ab8451\") " pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-6944bnb7n" Oct 30 12:50:23.650931 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:23.640031 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e49d3637-7899-4ea5-9544-7550b2ab8451-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-f7464f-6944bnb7n\" (UID: \"e49d3637-7899-4ea5-9544-7550b2ab8451\") " pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-6944bnb7n" Oct 30 12:50:23.650931 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:23.646547 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e49d3637-7899-4ea5-9544-7550b2ab8451-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-f7464f-6944bnb7n\" (UID: \"e49d3637-7899-4ea5-9544-7550b2ab8451\") " pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-6944bnb7n" Oct 30 12:50:23.651860 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:50:23.651699 4908 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-f7464f/agc-cred-3cd06b9a0e1c: secret "agc-cred-3cd06b9a0e1c" not found Oct 30 12:50:23.651860 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:50:23.651796 4908 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-6944bnb7n: secret "agc-cred-3cd06b9a0e1c" not found Oct 30 12:50:23.652753 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:50:23.652680 4908 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e49d3637-7899-4ea5-9544-7550b2ab8451-provider-credentials podName:e49d3637-7899-4ea5-9544-7550b2ab8451 nodeName:}" failed. No retries permitted until 2026-10-30 12:50:24.152651583 +0000 UTC m=+3380.667080330 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/e49d3637-7899-4ea5-9544-7550b2ab8451-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-f7464f-6944bnb7n" (UID: "e49d3637-7899-4ea5-9544-7550b2ab8451") : secret "agc-cred-3cd06b9a0e1c" not found Oct 30 12:50:23.695851 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:23.685622 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e49d3637-7899-4ea5-9544-7550b2ab8451-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-f7464f-6944bnb7n\" (UID: \"e49d3637-7899-4ea5-9544-7550b2ab8451\") " pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-6944bnb7n" Oct 30 12:50:23.706601 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:23.706568 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qt72v\" (UniqueName: \"kubernetes.io/projected/e49d3637-7899-4ea5-9544-7550b2ab8451-kube-api-access-qt72v\") pod \"payload-processing-external-model-e2e-extmod-a-f7464f-6944bnb7n\" (UID: \"e49d3637-7899-4ea5-9544-7550b2ab8451\") " pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-6944bnb7n" Oct 30 12:50:24.250332 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:24.250289 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e49d3637-7899-4ea5-9544-7550b2ab8451-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-f7464f-6944bnb7n\" (UID: \"e49d3637-7899-4ea5-9544-7550b2ab8451\") " pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-6944bnb7n" Oct 30 12:50:24.250544 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:50:24.250436 4908 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-f7464f/agc-cred-3cd06b9a0e1c: secret "agc-cred-3cd06b9a0e1c" not found Oct 30 12:50:24.250544 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:50:24.250455 4908 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-6944bnb7n: secret "agc-cred-3cd06b9a0e1c" not found Oct 30 12:50:24.250544 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:50:24.250517 4908 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e49d3637-7899-4ea5-9544-7550b2ab8451-provider-credentials podName:e49d3637-7899-4ea5-9544-7550b2ab8451 nodeName:}" failed. No retries permitted until 2026-10-30 12:50:25.250502211 +0000 UTC m=+3381.764930951 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/e49d3637-7899-4ea5-9544-7550b2ab8451-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-f7464f-6944bnb7n" (UID: "e49d3637-7899-4ea5-9544-7550b2ab8451") : secret "agc-cred-3cd06b9a0e1c" not found Oct 30 12:50:24.537675 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:24.537596 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-6944bnb7n"] Oct 30 12:50:24.538083 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:50:24.538002 4908 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[provider-credentials], unattached volumes=[], failed to process volumes=[]: context canceled" pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-6944bnb7n" podUID="e49d3637-7899-4ea5-9544-7550b2ab8451" Oct 30 12:50:25.269396 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:25.269360 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e49d3637-7899-4ea5-9544-7550b2ab8451-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-f7464f-6944bnb7n\" (UID: \"e49d3637-7899-4ea5-9544-7550b2ab8451\") " pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-6944bnb7n" Oct 30 12:50:25.269577 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:50:25.269522 4908 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-f7464f/agc-cred-3cd06b9a0e1c: secret "agc-cred-3cd06b9a0e1c" not found Oct 30 12:50:25.269577 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:50:25.269545 4908 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-6944bnb7n: secret "agc-cred-3cd06b9a0e1c" not found Oct 30 12:50:25.269658 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:50:25.269613 4908 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e49d3637-7899-4ea5-9544-7550b2ab8451-provider-credentials podName:e49d3637-7899-4ea5-9544-7550b2ab8451 nodeName:}" failed. No retries permitted until 2026-10-30 12:50:27.269594164 +0000 UTC m=+3383.784022890 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/e49d3637-7899-4ea5-9544-7550b2ab8451-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-f7464f-6944bnb7n" (UID: "e49d3637-7899-4ea5-9544-7550b2ab8451") : secret "agc-cred-3cd06b9a0e1c" not found Oct 30 12:50:25.285776 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:25.285744 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-6944bnb7n" Oct 30 12:50:25.295013 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:25.294992 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-6944bnb7n" Oct 30 12:50:25.369889 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:25.369865 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e49d3637-7899-4ea5-9544-7550b2ab8451-kube-api-access-qt72v\" (UniqueName: \"kubernetes.io/projected/e49d3637-7899-4ea5-9544-7550b2ab8451-kube-api-access-qt72v\") pod \"e49d3637-7899-4ea5-9544-7550b2ab8451\" (UID: \"e49d3637-7899-4ea5-9544-7550b2ab8451\") " Oct 30 12:50:25.370055 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:25.369971 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e49d3637-7899-4ea5-9544-7550b2ab8451-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e49d3637-7899-4ea5-9544-7550b2ab8451-plugins-config-volume\") pod \"e49d3637-7899-4ea5-9544-7550b2ab8451\" (UID: \"e49d3637-7899-4ea5-9544-7550b2ab8451\") " Oct 30 12:50:25.370055 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:25.370012 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e49d3637-7899-4ea5-9544-7550b2ab8451-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e49d3637-7899-4ea5-9544-7550b2ab8451-routing-overlay-volume\") pod \"e49d3637-7899-4ea5-9544-7550b2ab8451\" (UID: \"e49d3637-7899-4ea5-9544-7550b2ab8451\") " Oct 30 12:50:25.370448 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:25.370425 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e49d3637-7899-4ea5-9544-7550b2ab8451-routing-overlay-volume" pod "e49d3637-7899-4ea5-9544-7550b2ab8451" (UID: "e49d3637-7899-4ea5-9544-7550b2ab8451"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:50:25.370558 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:25.370423 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e49d3637-7899-4ea5-9544-7550b2ab8451-plugins-config-volume" pod "e49d3637-7899-4ea5-9544-7550b2ab8451" (UID: "e49d3637-7899-4ea5-9544-7550b2ab8451"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:50:25.372040 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:25.372016 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e49d3637-7899-4ea5-9544-7550b2ab8451-kube-api-access-qt72v" pod "e49d3637-7899-4ea5-9544-7550b2ab8451" (UID: "e49d3637-7899-4ea5-9544-7550b2ab8451"). InnerVolumeSpecName "kube-api-access-qt72v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:50:25.472095 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:25.472062 4908 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e49d3637-7899-4ea5-9544-7550b2ab8451-routing-overlay-volume\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:50:25.472095 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:25.472089 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qt72v\" (UniqueName: \"kubernetes.io/projected/e49d3637-7899-4ea5-9544-7550b2ab8451-kube-api-access-qt72v\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:50:25.472239 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:25.472103 4908 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e49d3637-7899-4ea5-9544-7550b2ab8451-plugins-config-volume\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:50:25.551927 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:25.551862 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9wwfc"] Oct 30 12:50:26.293149 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:26.293116 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-6944bnb7n" Oct 30 12:50:26.373508 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:26.373479 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-6944bnb7n"] Oct 30 12:50:26.385355 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:26.385327 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-6944bnb7n"] Oct 30 12:50:26.493005 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:26.492968 4908 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e49d3637-7899-4ea5-9544-7550b2ab8451-provider-credentials\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:50:27.689813 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:27.689765 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e49d3637-7899-4ea5-9544-7550b2ab8451" path="/var/lib/kubelet/pods/e49d3637-7899-4ea5-9544-7550b2ab8451/volumes" Oct 30 12:50:29.645881 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:29.645761 4908 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f920f7/simulator-w0-f920f7-kserve-599b7ff45f-g79d7" podUID="a85e562f-d655-4c22-b4f4-a299a4e494a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 12:50:30.286852 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:30.271147 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-163fb611-2kpzz"] Oct 30 12:50:30.297851 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:30.291563 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-163fb611-2kpzz" Oct 30 12:50:30.306883 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:30.304709 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-163fb611-2kpzz"] Oct 30 12:50:30.368056 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:30.368027 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3911335-4942-4c03-b240-06a51527ed22-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-163fb611-2kpzz\" (UID: \"a3911335-4942-4c03-b240-06a51527ed22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-163fb611-2kpzz" Oct 30 12:50:30.368446 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:30.368420 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3911335-4942-4c03-b240-06a51527ed22-maas-api-xks-ca\") pod \"maas-api-revoke-sub-163fb611-2kpzz\" (UID: \"a3911335-4942-4c03-b240-06a51527ed22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-163fb611-2kpzz" Oct 30 12:50:30.368793 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:30.368759 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bsmp4\" (UniqueName: \"kubernetes.io/projected/a3911335-4942-4c03-b240-06a51527ed22-kube-api-access-bsmp4\") pod \"maas-api-revoke-sub-163fb611-2kpzz\" (UID: \"a3911335-4942-4c03-b240-06a51527ed22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-163fb611-2kpzz" Oct 30 12:50:30.378849 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:30.371699 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 12:50:30.382901 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:30.382044 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 12:50:30.398885 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:30.382105 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 12:50:30.399028 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:30.382209 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s75hq\"" Oct 30 12:50:30.496848 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:30.496657 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3911335-4942-4c03-b240-06a51527ed22-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-163fb611-2kpzz\" (UID: \"a3911335-4942-4c03-b240-06a51527ed22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-163fb611-2kpzz" Oct 30 12:50:30.496848 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:30.496749 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3911335-4942-4c03-b240-06a51527ed22-maas-api-xks-ca\") pod \"maas-api-revoke-sub-163fb611-2kpzz\" (UID: \"a3911335-4942-4c03-b240-06a51527ed22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-163fb611-2kpzz" Oct 30 12:50:30.497099 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:30.496894 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsmp4\" (UniqueName: \"kubernetes.io/projected/a3911335-4942-4c03-b240-06a51527ed22-kube-api-access-bsmp4\") pod \"maas-api-revoke-sub-163fb611-2kpzz\" (UID: \"a3911335-4942-4c03-b240-06a51527ed22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-163fb611-2kpzz" Oct 30 12:50:30.521850 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:30.498094 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3911335-4942-4c03-b240-06a51527ed22-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-163fb611-2kpzz\" (UID: \"a3911335-4942-4c03-b240-06a51527ed22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-163fb611-2kpzz" Oct 30 12:50:30.683900 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:30.654999 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsmp4\" (UniqueName: \"kubernetes.io/projected/a3911335-4942-4c03-b240-06a51527ed22-kube-api-access-bsmp4\") pod \"maas-api-revoke-sub-163fb611-2kpzz\" (UID: \"a3911335-4942-4c03-b240-06a51527ed22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-163fb611-2kpzz" Oct 30 12:50:30.683900 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:30.655697 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3911335-4942-4c03-b240-06a51527ed22-maas-api-xks-ca\") pod \"maas-api-revoke-sub-163fb611-2kpzz\" (UID: \"a3911335-4942-4c03-b240-06a51527ed22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-163fb611-2kpzz" Oct 30 12:50:30.742874 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:30.738972 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-163fb611-2kpzz" Oct 30 12:50:30.838898 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:30.837044 4908 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47fd99/premium-w2-47fd99-kserve-5d85f9bb95-hqkmf" podUID="fd2cbeda-dbc4-41f6-a730-e1c6f9343c5e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:50:31.171451 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:31.171422 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-163fb611-2kpzz"] Oct 30 12:50:31.173957 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:50:31.173922 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3911335_4942_4c03_b240_06a51527ed22.slice/crio-f6962650d61f326ff4ed30dcafd02e8c517c859840aff46f80cba759119fdc5f WatchSource:0}: Error finding container f6962650d61f326ff4ed30dcafd02e8c517c859840aff46f80cba759119fdc5f: Status 404 returned error can't find the container with id f6962650d61f326ff4ed30dcafd02e8c517c859840aff46f80cba759119fdc5f Oct 30 12:50:31.444187 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:31.444046 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-163fb611-2kpzz" event={"ID":"a3911335-4942-4c03-b240-06a51527ed22","Type":"ContainerStarted","Data":"f6962650d61f326ff4ed30dcafd02e8c517c859840aff46f80cba759119fdc5f"} Oct 30 12:50:32.458428 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:32.458342 4908 generic.go:356] "Generic (PLEG): container finished" podID="a3911335-4942-4c03-b240-06a51527ed22" containerID="801e058264f289e9f6c7299977f678cfc70c752c923dc64f27926ca96d385a01" exitCode=0 Oct 30 12:50:32.458844 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:32.458437 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-163fb611-2kpzz" event={"ID":"a3911335-4942-4c03-b240-06a51527ed22","Type":"ContainerDied","Data":"801e058264f289e9f6c7299977f678cfc70c752c923dc64f27926ca96d385a01"} Oct 30 12:50:33.031332 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:33.031294 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44f58a36-r5vts"] Oct 30 12:50:33.042521 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:33.042492 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44f58a36-r5vts"] Oct 30 12:50:33.621302 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:33.621263 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea780d26-11ed-426e-aef1-42f1779d5697" path="/var/lib/kubelet/pods/ea780d26-11ed-426e-aef1-42f1779d5697/volumes" Oct 30 12:50:33.691558 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:33.691524 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-163fb611-2kpzz" Oct 30 12:50:33.758766 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:33.758735 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3911335-4942-4c03-b240-06a51527ed22-kube-api-access-bsmp4\" (UniqueName: \"kubernetes.io/projected/a3911335-4942-4c03-b240-06a51527ed22-kube-api-access-bsmp4\") pod \"a3911335-4942-4c03-b240-06a51527ed22\" (UID: \"a3911335-4942-4c03-b240-06a51527ed22\") " Oct 30 12:50:33.758959 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:33.758902 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a3911335-4942-4c03-b240-06a51527ed22-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3911335-4942-4c03-b240-06a51527ed22-maas-api-xks-ca\") pod \"a3911335-4942-4c03-b240-06a51527ed22\" (UID: \"a3911335-4942-4c03-b240-06a51527ed22\") " Oct 30 12:50:33.759087 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:33.759069 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3911335-4942-4c03-b240-06a51527ed22-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3911335-4942-4c03-b240-06a51527ed22-maas-api-openshift-ca\") pod \"a3911335-4942-4c03-b240-06a51527ed22\" (UID: \"a3911335-4942-4c03-b240-06a51527ed22\") " Oct 30 12:50:33.759484 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:33.759455 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3911335-4942-4c03-b240-06a51527ed22-maas-api-openshift-ca" pod "a3911335-4942-4c03-b240-06a51527ed22" (UID: "a3911335-4942-4c03-b240-06a51527ed22"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:50:33.759720 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:33.759700 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3911335-4942-4c03-b240-06a51527ed22-maas-api-openshift-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:50:33.760982 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:33.760945 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3911335-4942-4c03-b240-06a51527ed22-maas-api-xks-ca" pod "a3911335-4942-4c03-b240-06a51527ed22" (UID: "a3911335-4942-4c03-b240-06a51527ed22"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:50:33.761257 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:33.761232 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3911335-4942-4c03-b240-06a51527ed22-kube-api-access-bsmp4" pod "a3911335-4942-4c03-b240-06a51527ed22" (UID: "a3911335-4942-4c03-b240-06a51527ed22"). InnerVolumeSpecName "kube-api-access-bsmp4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:50:33.861135 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:33.861111 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3911335-4942-4c03-b240-06a51527ed22-maas-api-xks-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:50:33.861135 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:33.861135 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bsmp4\" (UniqueName: \"kubernetes.io/projected/a3911335-4942-4c03-b240-06a51527ed22-kube-api-access-bsmp4\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:50:34.030584 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:34.030515 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f7df0891-6ppbs"] Oct 30 12:50:34.034448 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:34.034424 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f7df0891-6ppbs"] Oct 30 12:50:34.475239 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:34.475160 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-163fb611-2kpzz" Oct 30 12:50:34.475239 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:34.475189 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-163fb611-2kpzz" event={"ID":"a3911335-4942-4c03-b240-06a51527ed22","Type":"ContainerDied","Data":"f6962650d61f326ff4ed30dcafd02e8c517c859840aff46f80cba759119fdc5f"} Oct 30 12:50:34.475424 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:34.475248 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f6962650d61f326ff4ed30dcafd02e8c517c859840aff46f80cba759119fdc5f" Oct 30 12:50:35.605621 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:35.605585 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ba9de25-ec88-4cf8-b374-dc779b854834" path="/var/lib/kubelet/pods/0ba9de25-ec88-4cf8-b374-dc779b854834/volumes" Oct 30 12:50:40.994417 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:40.994384 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-abbf0116-zplfz"] Oct 30 12:50:40.994985 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:40.994969 4908 cpu_manager.go:414] "containerMap: removing container" podUID="a3911335-4942-4c03-b240-06a51527ed22" containerName="revoke-keys" Oct 30 12:50:40.995032 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:40.994986 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3911335-4942-4c03-b240-06a51527ed22" containerName="revoke-keys" Oct 30 12:50:40.995113 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:40.995104 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3911335-4942-4c03-b240-06a51527ed22" containerName="revoke-keys" Oct 30 12:50:40.998398 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:40.998380 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abbf0116-zplfz" Oct 30 12:50:41.013457 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:41.013436 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 12:50:41.013578 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:41.013469 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 12:50:41.016840 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:41.016806 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-abbf0116-zplfz"] Oct 30 12:50:41.023136 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:41.023113 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 12:50:41.023293 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:41.023224 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s75hq\"" Oct 30 12:50:41.078999 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:41.078966 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r942f\" (UniqueName: \"kubernetes.io/projected/774163b0-110d-4239-80f9-941f9edd4351-kube-api-access-r942f\") pod \"maas-api-revoke-sub-abbf0116-zplfz\" (UID: \"774163b0-110d-4239-80f9-941f9edd4351\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abbf0116-zplfz" Oct 30 12:50:41.079149 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:41.079029 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/774163b0-110d-4239-80f9-941f9edd4351-maas-api-xks-ca\") pod \"maas-api-revoke-sub-abbf0116-zplfz\" (UID: \"774163b0-110d-4239-80f9-941f9edd4351\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abbf0116-zplfz" Oct 30 12:50:41.079194 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:41.079149 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/774163b0-110d-4239-80f9-941f9edd4351-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-abbf0116-zplfz\" (UID: \"774163b0-110d-4239-80f9-941f9edd4351\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abbf0116-zplfz" Oct 30 12:50:41.180739 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:41.180710 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/774163b0-110d-4239-80f9-941f9edd4351-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-abbf0116-zplfz\" (UID: \"774163b0-110d-4239-80f9-941f9edd4351\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abbf0116-zplfz" Oct 30 12:50:41.180980 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:41.180962 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r942f\" (UniqueName: \"kubernetes.io/projected/774163b0-110d-4239-80f9-941f9edd4351-kube-api-access-r942f\") pod \"maas-api-revoke-sub-abbf0116-zplfz\" (UID: \"774163b0-110d-4239-80f9-941f9edd4351\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abbf0116-zplfz" Oct 30 12:50:41.181089 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:41.181019 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/774163b0-110d-4239-80f9-941f9edd4351-maas-api-xks-ca\") pod \"maas-api-revoke-sub-abbf0116-zplfz\" (UID: \"774163b0-110d-4239-80f9-941f9edd4351\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abbf0116-zplfz" Oct 30 12:50:41.181571 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:41.181547 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/774163b0-110d-4239-80f9-941f9edd4351-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-abbf0116-zplfz\" (UID: \"774163b0-110d-4239-80f9-941f9edd4351\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abbf0116-zplfz" Oct 30 12:50:41.183393 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:41.183361 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/774163b0-110d-4239-80f9-941f9edd4351-maas-api-xks-ca\") pod \"maas-api-revoke-sub-abbf0116-zplfz\" (UID: \"774163b0-110d-4239-80f9-941f9edd4351\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abbf0116-zplfz" Oct 30 12:50:41.203992 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:41.203970 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r942f\" (UniqueName: \"kubernetes.io/projected/774163b0-110d-4239-80f9-941f9edd4351-kube-api-access-r942f\") pod \"maas-api-revoke-sub-abbf0116-zplfz\" (UID: \"774163b0-110d-4239-80f9-941f9edd4351\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abbf0116-zplfz" Oct 30 12:50:41.315989 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:41.315915 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abbf0116-zplfz" Oct 30 12:50:41.621554 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:41.621530 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-abbf0116-zplfz"] Oct 30 12:50:41.623893 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:50:41.623862 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod774163b0_110d_4239_80f9_941f9edd4351.slice/crio-cb4353a52b350fcab217f70b5b345abc8b9052b35e834528861c885ecdfddfc2 WatchSource:0}: Error finding container cb4353a52b350fcab217f70b5b345abc8b9052b35e834528861c885ecdfddfc2: Status 404 returned error can't find the container with id cb4353a52b350fcab217f70b5b345abc8b9052b35e834528861c885ecdfddfc2 Oct 30 12:50:42.579715 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:42.579669 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abbf0116-zplfz" event={"ID":"774163b0-110d-4239-80f9-941f9edd4351","Type":"ContainerStarted","Data":"263fd0abedbbb4197fcbed3dfd84e76599bc871ca89707c5314f141a76e8026d"} Oct 30 12:50:42.580192 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:42.579778 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abbf0116-zplfz" event={"ID":"774163b0-110d-4239-80f9-941f9edd4351","Type":"ContainerStarted","Data":"cb4353a52b350fcab217f70b5b345abc8b9052b35e834528861c885ecdfddfc2"} Oct 30 12:50:42.642964 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:42.642917 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abbf0116-zplfz" podStartSLOduration=2.642898358 podStartE2EDuration="2.642898358s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:50:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:50:42.638488281 +0000 UTC m=+3399.152917026" watchObservedRunningTime="2026-10-30 12:50:42.642898358 +0000 UTC m=+3399.157327133" Oct 30 12:50:43.589283 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:43.589241 4908 generic.go:356] "Generic (PLEG): container finished" podID="774163b0-110d-4239-80f9-941f9edd4351" containerID="263fd0abedbbb4197fcbed3dfd84e76599bc871ca89707c5314f141a76e8026d" exitCode=0 Oct 30 12:50:43.589669 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:43.589323 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abbf0116-zplfz" event={"ID":"774163b0-110d-4239-80f9-941f9edd4351","Type":"ContainerDied","Data":"263fd0abedbbb4197fcbed3dfd84e76599bc871ca89707c5314f141a76e8026d"} Oct 30 12:50:44.770117 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:44.770089 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abbf0116-zplfz" Oct 30 12:50:44.836045 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:44.836014 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/774163b0-110d-4239-80f9-941f9edd4351-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/774163b0-110d-4239-80f9-941f9edd4351-maas-api-xks-ca\") pod \"774163b0-110d-4239-80f9-941f9edd4351\" (UID: \"774163b0-110d-4239-80f9-941f9edd4351\") " Oct 30 12:50:44.836192 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:44.836174 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/774163b0-110d-4239-80f9-941f9edd4351-kube-api-access-r942f\" (UniqueName: \"kubernetes.io/projected/774163b0-110d-4239-80f9-941f9edd4351-kube-api-access-r942f\") pod \"774163b0-110d-4239-80f9-941f9edd4351\" (UID: \"774163b0-110d-4239-80f9-941f9edd4351\") " Oct 30 12:50:44.836315 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:44.836297 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/774163b0-110d-4239-80f9-941f9edd4351-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/774163b0-110d-4239-80f9-941f9edd4351-maas-api-openshift-ca\") pod \"774163b0-110d-4239-80f9-941f9edd4351\" (UID: \"774163b0-110d-4239-80f9-941f9edd4351\") " Oct 30 12:50:44.836757 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:44.836718 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/774163b0-110d-4239-80f9-941f9edd4351-maas-api-openshift-ca" pod "774163b0-110d-4239-80f9-941f9edd4351" (UID: "774163b0-110d-4239-80f9-941f9edd4351"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:50:44.836990 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:44.836966 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/774163b0-110d-4239-80f9-941f9edd4351-maas-api-openshift-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:50:44.838744 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:44.838715 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/774163b0-110d-4239-80f9-941f9edd4351-maas-api-xks-ca" pod "774163b0-110d-4239-80f9-941f9edd4351" (UID: "774163b0-110d-4239-80f9-941f9edd4351"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:50:44.839268 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:44.839245 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/774163b0-110d-4239-80f9-941f9edd4351-kube-api-access-r942f" pod "774163b0-110d-4239-80f9-941f9edd4351" (UID: "774163b0-110d-4239-80f9-941f9edd4351"). InnerVolumeSpecName "kube-api-access-r942f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:50:44.938561 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:44.938534 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r942f\" (UniqueName: \"kubernetes.io/projected/774163b0-110d-4239-80f9-941f9edd4351-kube-api-access-r942f\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:50:44.938561 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:44.938559 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/774163b0-110d-4239-80f9-941f9edd4351-maas-api-xks-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:50:45.612327 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:45.612286 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abbf0116-zplfz" Oct 30 12:50:45.612327 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:45.612317 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abbf0116-zplfz" event={"ID":"774163b0-110d-4239-80f9-941f9edd4351","Type":"ContainerDied","Data":"cb4353a52b350fcab217f70b5b345abc8b9052b35e834528861c885ecdfddfc2"} Oct 30 12:50:45.612613 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:45.612360 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cb4353a52b350fcab217f70b5b345abc8b9052b35e834528861c885ecdfddfc2" Oct 30 12:50:45.641390 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:50:45.641363 4908 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod774163b0_110d_4239_80f9_941f9edd4351.slice\": RecentStats: unable to find data in memory cache]" Oct 30 12:50:49.684733 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:50:49.684699 4908 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vn6vv" containerName="extractor" Oct 30 12:50:52.852988 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:52.852947 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-a786af/payload-processing-external-model-e2e-extmod-a-a786af-794c8h67h"] Oct 30 12:50:52.853767 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:52.853743 4908 cpu_manager.go:414] "containerMap: removing container" podUID="774163b0-110d-4239-80f9-941f9edd4351" containerName="revoke-keys" Oct 30 12:50:52.853767 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:52.853762 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="774163b0-110d-4239-80f9-941f9edd4351" containerName="revoke-keys" Oct 30 12:50:52.854045 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:52.854026 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="774163b0-110d-4239-80f9-941f9edd4351" containerName="revoke-keys" Oct 30 12:50:52.860273 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:52.860252 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a786af/payload-processing-external-model-e2e-extmod-a-a786af-794c8h67h" Oct 30 12:50:52.880743 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:52.879001 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-a786af/payload-processing-external-model-e2e-extmod-a-a786af-794c8h67h"] Oct 30 12:50:52.883850 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:52.882610 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93f3bfb1-48cf-450d-a283-11505f055e67-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a786af-794c8h67h\" (UID: \"93f3bfb1-48cf-450d-a283-11505f055e67\") " pod="ai-tenant-e2e-extmod-a-a786af/payload-processing-external-model-e2e-extmod-a-a786af-794c8h67h" Oct 30 12:50:52.883850 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:52.882733 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/93f3bfb1-48cf-450d-a283-11505f055e67-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a786af-794c8h67h\" (UID: \"93f3bfb1-48cf-450d-a283-11505f055e67\") " pod="ai-tenant-e2e-extmod-a-a786af/payload-processing-external-model-e2e-extmod-a-a786af-794c8h67h" Oct 30 12:50:52.883850 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:52.882881 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/93f3bfb1-48cf-450d-a283-11505f055e67-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a786af-794c8h67h\" (UID: \"93f3bfb1-48cf-450d-a283-11505f055e67\") " pod="ai-tenant-e2e-extmod-a-a786af/payload-processing-external-model-e2e-extmod-a-a786af-794c8h67h" Oct 30 12:50:52.883850 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:52.883097 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfwnl\" (UniqueName: \"kubernetes.io/projected/93f3bfb1-48cf-450d-a283-11505f055e67-kube-api-access-bfwnl\") pod \"payload-processing-external-model-e2e-extmod-a-a786af-794c8h67h\" (UID: \"93f3bfb1-48cf-450d-a283-11505f055e67\") " pod="ai-tenant-e2e-extmod-a-a786af/payload-processing-external-model-e2e-extmod-a-a786af-794c8h67h" Oct 30 12:50:52.885960 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:52.885939 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-a786af\"/\"payload-processing-external-model-plugins-e2e-extmod-a-a786af\"" Oct 30 12:50:52.886206 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:52.886189 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-a786af\"/\"openshift-service-ca.crt\"" Oct 30 12:50:52.886465 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:52.886448 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-a786af\"/\"payload-processing-external-model-e2e--3f99fe6c-dockercfg-npqld\"" Oct 30 12:50:52.886656 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:52.886640 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-a786af\"/\"agc-cred-5791f6dab64a\"" Oct 30 12:50:52.891707 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:52.891679 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-a786af\"/\"kube-root-ca.crt\"" Oct 30 12:50:52.891969 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:52.891948 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-a786af\"/\"routing-overlay\"" Oct 30 12:50:52.984277 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:52.984251 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93f3bfb1-48cf-450d-a283-11505f055e67-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a786af-794c8h67h\" (UID: \"93f3bfb1-48cf-450d-a283-11505f055e67\") " pod="ai-tenant-e2e-extmod-a-a786af/payload-processing-external-model-e2e-extmod-a-a786af-794c8h67h" Oct 30 12:50:52.984406 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:52.984299 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/93f3bfb1-48cf-450d-a283-11505f055e67-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a786af-794c8h67h\" (UID: \"93f3bfb1-48cf-450d-a283-11505f055e67\") " pod="ai-tenant-e2e-extmod-a-a786af/payload-processing-external-model-e2e-extmod-a-a786af-794c8h67h" Oct 30 12:50:52.984406 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:52.984348 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/93f3bfb1-48cf-450d-a283-11505f055e67-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a786af-794c8h67h\" (UID: \"93f3bfb1-48cf-450d-a283-11505f055e67\") " pod="ai-tenant-e2e-extmod-a-a786af/payload-processing-external-model-e2e-extmod-a-a786af-794c8h67h" Oct 30 12:50:52.984497 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:52.984477 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfwnl\" (UniqueName: \"kubernetes.io/projected/93f3bfb1-48cf-450d-a283-11505f055e67-kube-api-access-bfwnl\") pod \"payload-processing-external-model-e2e-extmod-a-a786af-794c8h67h\" (UID: \"93f3bfb1-48cf-450d-a283-11505f055e67\") " pod="ai-tenant-e2e-extmod-a-a786af/payload-processing-external-model-e2e-extmod-a-a786af-794c8h67h" Oct 30 12:50:52.985097 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:52.985072 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/93f3bfb1-48cf-450d-a283-11505f055e67-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a786af-794c8h67h\" (UID: \"93f3bfb1-48cf-450d-a283-11505f055e67\") " pod="ai-tenant-e2e-extmod-a-a786af/payload-processing-external-model-e2e-extmod-a-a786af-794c8h67h" Oct 30 12:50:52.985227 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:52.985084 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93f3bfb1-48cf-450d-a283-11505f055e67-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a786af-794c8h67h\" (UID: \"93f3bfb1-48cf-450d-a283-11505f055e67\") " pod="ai-tenant-e2e-extmod-a-a786af/payload-processing-external-model-e2e-extmod-a-a786af-794c8h67h" Oct 30 12:50:52.986821 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:52.986803 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/93f3bfb1-48cf-450d-a283-11505f055e67-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a786af-794c8h67h\" (UID: \"93f3bfb1-48cf-450d-a283-11505f055e67\") " pod="ai-tenant-e2e-extmod-a-a786af/payload-processing-external-model-e2e-extmod-a-a786af-794c8h67h" Oct 30 12:50:53.006752 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:53.006728 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfwnl\" (UniqueName: \"kubernetes.io/projected/93f3bfb1-48cf-450d-a283-11505f055e67-kube-api-access-bfwnl\") pod \"payload-processing-external-model-e2e-extmod-a-a786af-794c8h67h\" (UID: \"93f3bfb1-48cf-450d-a283-11505f055e67\") " pod="ai-tenant-e2e-extmod-a-a786af/payload-processing-external-model-e2e-extmod-a-a786af-794c8h67h" Oct 30 12:50:53.190127 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:53.190062 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a786af/payload-processing-external-model-e2e-extmod-a-a786af-794c8h67h" Oct 30 12:50:53.500352 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:53.500325 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-a786af/payload-processing-external-model-e2e-extmod-a-a786af-794c8h67h"] Oct 30 12:50:53.503287 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:50:53.503256 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93f3bfb1_48cf_450d_a283_11505f055e67.slice/crio-c0edb0c93e9b23046724e459388fc4e786c0e45c7ca9410d884f817391cca1c0 WatchSource:0}: Error finding container c0edb0c93e9b23046724e459388fc4e786c0e45c7ca9410d884f817391cca1c0: Status 404 returned error can't find the container with id c0edb0c93e9b23046724e459388fc4e786c0e45c7ca9410d884f817391cca1c0 Oct 30 12:50:53.654432 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:53.654400 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-a786af/payload-processing-external-model-e2e-extmod-a-a786af-794c8h67h"] Oct 30 12:50:53.768687 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:53.768597 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a786af/payload-processing-external-model-e2e-extmod-a-a786af-794c8h67h" event={"ID":"93f3bfb1-48cf-450d-a283-11505f055e67","Type":"ContainerStarted","Data":"6d5271ab1788aa47e93440adeae87c89fa475480933f144441d4dc33449f8207"} Oct 30 12:50:53.768687 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:53.768654 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a786af/payload-processing-external-model-e2e-extmod-a-a786af-794c8h67h" event={"ID":"93f3bfb1-48cf-450d-a283-11505f055e67","Type":"ContainerStarted","Data":"c0edb0c93e9b23046724e459388fc4e786c0e45c7ca9410d884f817391cca1c0"} Oct 30 12:50:53.768909 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:53.768689 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-a786af/payload-processing-external-model-e2e-extmod-a-a786af-794c8h67h" Oct 30 12:50:53.768909 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:53.768769 4908 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="ai-tenant-e2e-extmod-a-a786af/payload-processing-external-model-e2e-extmod-a-a786af-794c8h67h" secret="" err="secret \"payload-processing-external-model-e2e--3f99fe6c-dockercfg-npqld\" not found" Oct 30 12:50:53.797275 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:50:53.797251 4908 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-a786af/payload-processing-external-model-plugins-e2e-extmod-a-a786af: configmap "payload-processing-external-model-plugins-e2e-extmod-a-a786af" not found Oct 30 12:50:53.797414 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:50:53.797307 4908 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/93f3bfb1-48cf-450d-a283-11505f055e67-plugins-config-volume podName:93f3bfb1-48cf-450d-a283-11505f055e67 nodeName:}" failed. No retries permitted until 2026-10-30 12:50:54.297291439 +0000 UTC m=+3410.811720162 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/93f3bfb1-48cf-450d-a283-11505f055e67-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-a786af-794c8h67h" (UID: "93f3bfb1-48cf-450d-a283-11505f055e67") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-a786af" not found Oct 30 12:50:53.797414 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:50:53.797309 4908 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-a786af/agc-cred-5791f6dab64a: secret "agc-cred-5791f6dab64a" not found Oct 30 12:50:53.797414 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:50:53.797331 4908 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-a786af/payload-processing-external-model-e2e-extmod-a-a786af-794c8h67h: secret "agc-cred-5791f6dab64a" not found Oct 30 12:50:53.797414 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:50:53.797386 4908 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/93f3bfb1-48cf-450d-a283-11505f055e67-provider-credentials podName:93f3bfb1-48cf-450d-a283-11505f055e67 nodeName:}" failed. No retries permitted until 2026-10-30 12:50:54.297369019 +0000 UTC m=+3410.811797760 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/93f3bfb1-48cf-450d-a283-11505f055e67-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-a786af-794c8h67h" (UID: "93f3bfb1-48cf-450d-a283-11505f055e67") : secret "agc-cred-5791f6dab64a" not found Oct 30 12:50:53.813367 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:53.813327 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-a786af/payload-processing-external-model-e2e-extmod-a-a786af-794c8h67h" podStartSLOduration=1.813310781 podStartE2EDuration="1.813310781s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:50:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:50:53.810147457 +0000 UTC m=+3410.324576202" watchObservedRunningTime="2026-10-30 12:50:53.813310781 +0000 UTC m=+3410.327739525" Oct 30 12:50:54.340252 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:50:54.340140 4908 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-a786af/payload-processing-external-model-plugins-e2e-extmod-a-a786af: configmap "payload-processing-external-model-plugins-e2e-extmod-a-a786af" not found Oct 30 12:50:54.340252 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:50:54.340222 4908 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/93f3bfb1-48cf-450d-a283-11505f055e67-plugins-config-volume podName:93f3bfb1-48cf-450d-a283-11505f055e67 nodeName:}" failed. No retries permitted until 2026-10-30 12:50:55.3402027 +0000 UTC m=+3411.854631439 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/93f3bfb1-48cf-450d-a283-11505f055e67-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-a786af-794c8h67h" (UID: "93f3bfb1-48cf-450d-a283-11505f055e67") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-a786af" not found Oct 30 12:50:54.340937 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:50:54.340786 4908 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-a786af/agc-cred-5791f6dab64a: secret "agc-cred-5791f6dab64a" not found Oct 30 12:50:54.340937 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:50:54.340805 4908 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-a786af/payload-processing-external-model-e2e-extmod-a-a786af-794c8h67h: secret "agc-cred-5791f6dab64a" not found Oct 30 12:50:54.340937 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:50:54.340868 4908 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/93f3bfb1-48cf-450d-a283-11505f055e67-provider-credentials podName:93f3bfb1-48cf-450d-a283-11505f055e67 nodeName:}" failed. No retries permitted until 2026-10-30 12:50:55.340853872 +0000 UTC m=+3411.855282596 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/93f3bfb1-48cf-450d-a283-11505f055e67-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-a786af-794c8h67h" (UID: "93f3bfb1-48cf-450d-a283-11505f055e67") : secret "agc-cred-5791f6dab64a" not found Oct 30 12:50:55.355820 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:50:55.355792 4908 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-a786af/payload-processing-external-model-plugins-e2e-extmod-a-a786af: configmap "payload-processing-external-model-plugins-e2e-extmod-a-a786af" not found Oct 30 12:50:55.356325 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:50:55.355898 4908 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/93f3bfb1-48cf-450d-a283-11505f055e67-plugins-config-volume podName:93f3bfb1-48cf-450d-a283-11505f055e67 nodeName:}" failed. No retries permitted until 2026-10-30 12:50:57.355878254 +0000 UTC m=+3413.870306994 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/93f3bfb1-48cf-450d-a283-11505f055e67-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-a786af-794c8h67h" (UID: "93f3bfb1-48cf-450d-a283-11505f055e67") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-a786af" not found Oct 30 12:50:55.356325 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:50:55.355923 4908 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-a786af/agc-cred-5791f6dab64a: secret "agc-cred-5791f6dab64a" not found Oct 30 12:50:55.356325 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:50:55.355945 4908 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-a786af/payload-processing-external-model-e2e-extmod-a-a786af-794c8h67h: secret "agc-cred-5791f6dab64a" not found Oct 30 12:50:55.356325 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:50:55.356008 4908 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/93f3bfb1-48cf-450d-a283-11505f055e67-provider-credentials podName:93f3bfb1-48cf-450d-a283-11505f055e67 nodeName:}" failed. No retries permitted until 2026-10-30 12:50:57.355992593 +0000 UTC m=+3413.870421316 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/93f3bfb1-48cf-450d-a283-11505f055e67-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-a786af-794c8h67h" (UID: "93f3bfb1-48cf-450d-a283-11505f055e67") : secret "agc-cred-5791f6dab64a" not found Oct 30 12:50:57.042336 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:57.042299 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e337e7b-qbvgq"] Oct 30 12:50:57.053056 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:57.053019 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e337e7b-qbvgq"] Oct 30 12:50:57.382341 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:50:57.382265 4908 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-a786af/agc-cred-5791f6dab64a: secret "agc-cred-5791f6dab64a" not found Oct 30 12:50:57.382341 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:50:57.382291 4908 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-a786af/payload-processing-external-model-e2e-extmod-a-a786af-794c8h67h: secret "agc-cred-5791f6dab64a" not found Oct 30 12:50:57.382341 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:50:57.382317 4908 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-a786af/payload-processing-external-model-plugins-e2e-extmod-a-a786af: configmap "payload-processing-external-model-plugins-e2e-extmod-a-a786af" not found Oct 30 12:50:57.382559 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:50:57.382352 4908 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/93f3bfb1-48cf-450d-a283-11505f055e67-provider-credentials podName:93f3bfb1-48cf-450d-a283-11505f055e67 nodeName:}" failed. No retries permitted until 2026-10-30 12:51:01.382336674 +0000 UTC m=+3417.896765414 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/93f3bfb1-48cf-450d-a283-11505f055e67-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-a786af-794c8h67h" (UID: "93f3bfb1-48cf-450d-a283-11505f055e67") : secret "agc-cred-5791f6dab64a" not found Oct 30 12:50:57.382559 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:50:57.382367 4908 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/93f3bfb1-48cf-450d-a283-11505f055e67-plugins-config-volume podName:93f3bfb1-48cf-450d-a283-11505f055e67 nodeName:}" failed. No retries permitted until 2026-10-30 12:51:01.382360803 +0000 UTC m=+3417.896789526 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/93f3bfb1-48cf-450d-a283-11505f055e67-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-a786af-794c8h67h" (UID: "93f3bfb1-48cf-450d-a283-11505f055e67") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-a786af" not found Oct 30 12:50:57.606908 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:57.606870 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99914eaa-84d9-4fc3-a4e2-7c951974eea1" path="/var/lib/kubelet/pods/99914eaa-84d9-4fc3-a4e2-7c951974eea1/volumes" Oct 30 12:50:59.776030 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:50:59.775991 4908 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-a786af/payload-processing-external-model-e2e-extmod-a-a786af-794c8h67h" podUID="93f3bfb1-48cf-450d-a283-11505f055e67" containerName="payload-processing" containerID="cri-o://6d5271ab1788aa47e93440adeae87c89fa475480933f144441d4dc33449f8207" gracePeriod=25 Oct 30 12:51:00.118470 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:51:00.118441 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a786af/payload-processing-external-model-e2e-extmod-a-a786af-794c8h67h" Oct 30 12:51:00.224311 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:51:00.224273 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93f3bfb1-48cf-450d-a283-11505f055e67-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/93f3bfb1-48cf-450d-a283-11505f055e67-routing-overlay-volume\") pod \"93f3bfb1-48cf-450d-a283-11505f055e67\" (UID: \"93f3bfb1-48cf-450d-a283-11505f055e67\") " Oct 30 12:51:00.224466 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:51:00.224437 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93f3bfb1-48cf-450d-a283-11505f055e67-kube-api-access-bfwnl\" (UniqueName: \"kubernetes.io/projected/93f3bfb1-48cf-450d-a283-11505f055e67-kube-api-access-bfwnl\") pod \"93f3bfb1-48cf-450d-a283-11505f055e67\" (UID: \"93f3bfb1-48cf-450d-a283-11505f055e67\") " Oct 30 12:51:00.224626 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:51:00.224600 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93f3bfb1-48cf-450d-a283-11505f055e67-routing-overlay-volume" pod "93f3bfb1-48cf-450d-a283-11505f055e67" (UID: "93f3bfb1-48cf-450d-a283-11505f055e67"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:51:00.224693 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:51:00.224622 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93f3bfb1-48cf-450d-a283-11505f055e67-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93f3bfb1-48cf-450d-a283-11505f055e67-plugins-config-volume\") pod \"93f3bfb1-48cf-450d-a283-11505f055e67\" (UID: \"93f3bfb1-48cf-450d-a283-11505f055e67\") " Oct 30 12:51:00.224751 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:51:00.224731 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93f3bfb1-48cf-450d-a283-11505f055e67-provider-credentials\" (UniqueName: \"kubernetes.io/projected/93f3bfb1-48cf-450d-a283-11505f055e67-provider-credentials\") pod \"93f3bfb1-48cf-450d-a283-11505f055e67\" (UID: \"93f3bfb1-48cf-450d-a283-11505f055e67\") " Oct 30 12:51:00.224992 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:51:00.224969 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93f3bfb1-48cf-450d-a283-11505f055e67-plugins-config-volume" pod "93f3bfb1-48cf-450d-a283-11505f055e67" (UID: "93f3bfb1-48cf-450d-a283-11505f055e67"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:51:00.225229 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:51:00.225208 4908 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93f3bfb1-48cf-450d-a283-11505f055e67-plugins-config-volume\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:51:00.225229 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:51:00.225226 4908 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/93f3bfb1-48cf-450d-a283-11505f055e67-routing-overlay-volume\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:51:00.226768 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:51:00.226738 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93f3bfb1-48cf-450d-a283-11505f055e67-kube-api-access-bfwnl" pod "93f3bfb1-48cf-450d-a283-11505f055e67" (UID: "93f3bfb1-48cf-450d-a283-11505f055e67"). InnerVolumeSpecName "kube-api-access-bfwnl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:51:00.226882 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:51:00.226772 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93f3bfb1-48cf-450d-a283-11505f055e67-provider-credentials" pod "93f3bfb1-48cf-450d-a283-11505f055e67" (UID: "93f3bfb1-48cf-450d-a283-11505f055e67"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:51:00.325978 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:51:00.325955 4908 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/93f3bfb1-48cf-450d-a283-11505f055e67-provider-credentials\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:51:00.325978 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:51:00.325977 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfwnl\" (UniqueName: \"kubernetes.io/projected/93f3bfb1-48cf-450d-a283-11505f055e67-kube-api-access-bfwnl\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:51:00.822819 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:51:00.822788 4908 generic.go:356] "Generic (PLEG): container finished" podID="93f3bfb1-48cf-450d-a283-11505f055e67" containerID="6d5271ab1788aa47e93440adeae87c89fa475480933f144441d4dc33449f8207" exitCode=0 Oct 30 12:51:00.823259 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:51:00.822853 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a786af/payload-processing-external-model-e2e-extmod-a-a786af-794c8h67h" Oct 30 12:51:00.823259 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:51:00.822863 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a786af/payload-processing-external-model-e2e-extmod-a-a786af-794c8h67h" event={"ID":"93f3bfb1-48cf-450d-a283-11505f055e67","Type":"ContainerDied","Data":"6d5271ab1788aa47e93440adeae87c89fa475480933f144441d4dc33449f8207"} Oct 30 12:51:00.823259 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:51:00.822919 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a786af/payload-processing-external-model-e2e-extmod-a-a786af-794c8h67h" event={"ID":"93f3bfb1-48cf-450d-a283-11505f055e67","Type":"ContainerDied","Data":"c0edb0c93e9b23046724e459388fc4e786c0e45c7ca9410d884f817391cca1c0"} Oct 30 12:51:00.823259 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:51:00.822940 4908 scope.go:122] "RemoveContainer" containerID="6d5271ab1788aa47e93440adeae87c89fa475480933f144441d4dc33449f8207" Oct 30 12:51:00.834002 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:51:00.833978 4908 scope.go:122] "RemoveContainer" containerID="6d5271ab1788aa47e93440adeae87c89fa475480933f144441d4dc33449f8207" Oct 30 12:51:00.834294 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:51:00.834270 4908 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d5271ab1788aa47e93440adeae87c89fa475480933f144441d4dc33449f8207\": container with ID starting with 6d5271ab1788aa47e93440adeae87c89fa475480933f144441d4dc33449f8207 not found: ID does not exist" containerID="6d5271ab1788aa47e93440adeae87c89fa475480933f144441d4dc33449f8207" Oct 30 12:51:00.834381 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:51:00.834300 4908 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d5271ab1788aa47e93440adeae87c89fa475480933f144441d4dc33449f8207"} err="failed to get container status \"6d5271ab1788aa47e93440adeae87c89fa475480933f144441d4dc33449f8207\": rpc error: code = NotFound desc = could not find container \"6d5271ab1788aa47e93440adeae87c89fa475480933f144441d4dc33449f8207\": container with ID starting with 6d5271ab1788aa47e93440adeae87c89fa475480933f144441d4dc33449f8207 not found: ID does not exist" Oct 30 12:51:00.876514 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:51:00.876491 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-a786af/payload-processing-external-model-e2e-extmod-a-a786af-794c8h67h"] Oct 30 12:51:00.889490 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:51:00.889462 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-a786af/payload-processing-external-model-e2e-extmod-a-a786af-794c8h67h"] Oct 30 12:51:01.608909 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:51:01.608877 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93f3bfb1-48cf-450d-a283-11505f055e67" path="/var/lib/kubelet/pods/93f3bfb1-48cf-450d-a283-11505f055e67/volumes" Oct 30 12:51:07.012189 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:51:07.012151 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9wwfc"] Oct 30 12:51:09.838103 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:51:09.838066 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-extmod-shr-user-9f38e1/e2e-usage-sim-0f45af-55f5696f49-j9l6t"] Oct 30 12:51:09.838980 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:51:09.838955 4908 cpu_manager.go:414] "containerMap: removing container" podUID="93f3bfb1-48cf-450d-a283-11505f055e67" containerName="payload-processing" Oct 30 12:51:09.838980 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:51:09.838980 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93f3bfb1-48cf-450d-a283-11505f055e67" containerName="payload-processing" Oct 30 12:51:09.839181 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:51:09.839154 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="93f3bfb1-48cf-450d-a283-11505f055e67" containerName="payload-processing" Oct 30 12:51:09.844504 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:51:09.844480 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-9f38e1/e2e-usage-sim-0f45af-55f5696f49-j9l6t" Oct 30 12:51:09.861150 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:51:09.861126 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-extmod-shr-user-9f38e1\"/\"openshift-service-ca.crt\"" Oct 30 12:51:09.869480 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:51:09.869456 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-extmod-shr-user-9f38e1\"/\"kube-root-ca.crt\"" Oct 30 12:51:09.873569 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:51:09.873537 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-extmod-shr-user-9f38e1/e2e-usage-sim-0f45af-55f5696f49-j9l6t"] Oct 30 12:51:09.919438 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:51:09.919389 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-extmod-shr-user-9f38e1\"/\"default-dockercfg-wrml7\"" Oct 30 12:51:09.920944 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:51:09.920877 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-extmod-shr-user-9f38e1\"/\"e2e-usage-sim-0f45af-cert\"" Oct 30 12:51:10.011101 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:51:10.011062 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/48c3e203-4fe7-4c38-a65e-e24f5605c69d-tls\") pod \"e2e-usage-sim-0f45af-55f5696f49-j9l6t\" (UID: \"48c3e203-4fe7-4c38-a65e-e24f5605c69d\") " pod="e2e-extmod-shr-user-9f38e1/e2e-usage-sim-0f45af-55f5696f49-j9l6t" Oct 30 12:51:10.011428 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:51:10.011397 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fv7mb\" (UniqueName: \"kubernetes.io/projected/48c3e203-4fe7-4c38-a65e-e24f5605c69d-kube-api-access-fv7mb\") pod \"e2e-usage-sim-0f45af-55f5696f49-j9l6t\" (UID: \"48c3e203-4fe7-4c38-a65e-e24f5605c69d\") " pod="e2e-extmod-shr-user-9f38e1/e2e-usage-sim-0f45af-55f5696f49-j9l6t" Oct 30 12:51:10.113190 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:51:10.113054 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/48c3e203-4fe7-4c38-a65e-e24f5605c69d-tls\") pod \"e2e-usage-sim-0f45af-55f5696f49-j9l6t\" (UID: \"48c3e203-4fe7-4c38-a65e-e24f5605c69d\") " pod="e2e-extmod-shr-user-9f38e1/e2e-usage-sim-0f45af-55f5696f49-j9l6t" Oct 30 12:51:10.113352 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:51:10.113246 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fv7mb\" (UniqueName: \"kubernetes.io/projected/48c3e203-4fe7-4c38-a65e-e24f5605c69d-kube-api-access-fv7mb\") pod \"e2e-usage-sim-0f45af-55f5696f49-j9l6t\" (UID: \"48c3e203-4fe7-4c38-a65e-e24f5605c69d\") " pod="e2e-extmod-shr-user-9f38e1/e2e-usage-sim-0f45af-55f5696f49-j9l6t" Oct 30 12:51:10.117198 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:51:10.117171 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/48c3e203-4fe7-4c38-a65e-e24f5605c69d-tls\") pod \"e2e-usage-sim-0f45af-55f5696f49-j9l6t\" (UID: \"48c3e203-4fe7-4c38-a65e-e24f5605c69d\") " pod="e2e-extmod-shr-user-9f38e1/e2e-usage-sim-0f45af-55f5696f49-j9l6t" Oct 30 12:51:10.189787 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:51:10.189751 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fv7mb\" (UniqueName: \"kubernetes.io/projected/48c3e203-4fe7-4c38-a65e-e24f5605c69d-kube-api-access-fv7mb\") pod \"e2e-usage-sim-0f45af-55f5696f49-j9l6t\" (UID: \"48c3e203-4fe7-4c38-a65e-e24f5605c69d\") " pod="e2e-extmod-shr-user-9f38e1/e2e-usage-sim-0f45af-55f5696f49-j9l6t" Oct 30 12:51:10.476915 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:51:10.473762 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-9f38e1/e2e-usage-sim-0f45af-55f5696f49-j9l6t" Oct 30 12:51:11.522721 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:51:11.522656 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-extmod-shr-user-9f38e1/e2e-usage-sim-0f45af-55f5696f49-j9l6t"] Oct 30 12:51:11.614335 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:51:11.613353 4908 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f920f7/simulator-w0-f920f7-kserve-599b7ff45f-g79d7" podUID="a85e562f-d655-4c22-b4f4-a299a4e494a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:51:11.618856 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:51:11.614677 4908 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6a7148/premium-w3-6a7148-kserve-67b976c888-tgd6c" podUID="13de4441-e7f6-4cbd-97d9-0f079d11cb8e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 30 12:51:11.715909 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:51:11.715597 4908 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-5smp6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.9:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 12:51:11.715909 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:51:11.715650 4908 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5smp6" podUID="2f090df9-3e00-4fec-9008-8088c2ae5f93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.9:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 12:51:11.827159 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:51:11.826323 4908 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47fd99/premium-w2-47fd99-kserve-5d85f9bb95-hqkmf" podUID="fd2cbeda-dbc4-41f6-a730-e1c6f9343c5e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Oct 30 12:51:12.303854 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:51:12.296849 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-9f38e1/e2e-usage-sim-0f45af-55f5696f49-j9l6t" event={"ID":"48c3e203-4fe7-4c38-a65e-e24f5605c69d","Type":"ContainerStarted","Data":"c02c476ff7d0133d1df1d637a2d3313e47abe2c103808ee7823121610e484d97"} Oct 30 12:51:12.303854 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:51:12.296908 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-9f38e1/e2e-usage-sim-0f45af-55f5696f49-j9l6t" event={"ID":"48c3e203-4fe7-4c38-a65e-e24f5605c69d","Type":"ContainerStarted","Data":"250768cb2b7b1349cad3d5f5ad847ff698d9e85243a88b5f3d2ba3292665ecfe"} Oct 30 12:51:12.303854 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:51:12.299378 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-extmod-shr-user-9f38e1/e2e-usage-sim-0f45af-55f5696f49-j9l6t" Oct 30 12:51:12.307159 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:51:12.305492 4908 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-extmod-shr-user-9f38e1/e2e-usage-sim-0f45af-55f5696f49-j9l6t" podUID="48c3e203-4fe7-4c38-a65e-e24f5605c69d" containerName="sim" probeResult="failure" output="Get \"https://10.132.1.21:8000/ready\": dial tcp 10.132.1.21:8000: connect: connection refused" Oct 30 12:51:12.355125 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:51:12.354461 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-extmod-shr-user-9f38e1/e2e-usage-sim-0f45af-55f5696f49-j9l6t" podStartSLOduration=3.354443999 podStartE2EDuration="3.354443999s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:51:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:51:12.342023023 +0000 UTC m=+3428.856451769" watchObservedRunningTime="2026-10-30 12:51:12.354443999 +0000 UTC m=+3428.868872747" Oct 30 12:51:13.308719 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:51:13.308688 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-extmod-shr-user-9f38e1/e2e-usage-sim-0f45af-55f5696f49-j9l6t" Oct 30 12:51:19.684880 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:51:19.684843 4908 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vn6vv" containerName="extractor" Oct 30 12:51:49.687601 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:51:49.687513 4908 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vn6vv" containerName="extractor" Oct 30 12:51:57.715440 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:51:57.715406 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9wwfc"] Oct 30 12:52:08.241954 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:52:08.241915 4908 scope.go:122] "RemoveContainer" containerID="a939d98d58a7c943a93eeaa4f8f63483f5fbd67d33cabcbaa9eceacad46e323a" Oct 30 12:52:08.252490 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:52:08.252472 4908 scope.go:122] "RemoveContainer" containerID="14b4618e3febde11d741191798be5640a19a7cb107ebc034d4f086f747847581" Oct 30 12:52:08.262444 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:52:08.262422 4908 scope.go:122] "RemoveContainer" containerID="6bb731e41d13376b40da40e3940bbfe21c236a896d7adc0260fdc71e13262b2e" Oct 30 12:52:10.047765 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:52:10.047730 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86b846ba-xwqcm"] Oct 30 12:52:10.058106 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:52:10.058080 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86b846ba-xwqcm"] Oct 30 12:52:11.608868 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:52:11.608816 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97e643d2-7ef6-40ff-8299-ef78b8038b08" path="/var/lib/kubelet/pods/97e643d2-7ef6-40ff-8299-ef78b8038b08/volumes" Oct 30 12:52:19.684546 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:52:19.684503 4908 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vn6vv" containerName="extractor" Oct 30 12:52:22.037678 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:52:22.037645 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-394fe43e-gvzjl"] Oct 30 12:52:22.047819 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:52:22.047791 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89926aa6-dfz48"] Oct 30 12:52:22.051604 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:52:22.051582 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-394fe43e-gvzjl"] Oct 30 12:52:22.054918 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:52:22.054898 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89926aa6-dfz48"] Oct 30 12:52:23.609012 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:52:23.608978 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5cdee26-990c-4f16-8152-2921cf65972d" path="/var/lib/kubelet/pods/a5cdee26-990c-4f16-8152-2921cf65972d/volumes" Oct 30 12:52:23.609441 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:52:23.609427 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9c8a505-c26d-4cc0-86f1-8f4ae6095e3e" path="/var/lib/kubelet/pods/d9c8a505-c26d-4cc0-86f1-8f4ae6095e3e/volumes" Oct 30 12:52:32.431817 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:52:32.431772 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9wwfc"] Oct 30 12:52:35.032517 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:52:35.032483 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e350e85-tzb72"] Oct 30 12:52:35.043031 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:52:35.043004 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e350e85-tzb72"] Oct 30 12:52:35.607295 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:52:35.607255 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a9055d5-d3ba-4a50-8f2d-a363bbb17ff8" path="/var/lib/kubelet/pods/3a9055d5-d3ba-4a50-8f2d-a363bbb17ff8/volumes" Oct 30 12:52:49.683967 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:52:49.683928 4908 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vn6vv" containerName="extractor" Oct 30 12:53:08.303978 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:08.303944 4908 scope.go:122] "RemoveContainer" containerID="2000905ecd1bf85ac0362f6bc10ada04033504231cc4ebfdc922f3a35d42a28e" Oct 30 12:53:13.463008 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:13.462968 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-pzxkx"] Oct 30 12:53:13.463510 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:13.463226 4908 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-pzxkx" podUID="8b4543f5-ebb4-4ebc-94f8-fa451750afa6" containerName="istio-proxy" containerID="cri-o://fe633ac50288577e8ff75c53d3a4ce7a91bce30d42c38ed3407dd57504d013d9" gracePeriod=30 Oct 30 12:53:17.803940 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:17.803856 4908 patch_prober.go:29] interesting pod/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-pzxkx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 12:53:17.804426 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:17.803928 4908 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-pzxkx" podUID="8b4543f5-ebb4-4ebc-94f8-fa451750afa6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 12:53:18.751042 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:18.751016 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-pzxkx" Oct 30 12:53:18.836148 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:18.836114 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b4543f5-ebb4-4ebc-94f8-fa451750afa6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b4543f5-ebb4-4ebc-94f8-fa451750afa6-workload-certs\") pod \"8b4543f5-ebb4-4ebc-94f8-fa451750afa6\" (UID: \"8b4543f5-ebb4-4ebc-94f8-fa451750afa6\") " Oct 30 12:53:18.836624 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:18.836201 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b4543f5-ebb4-4ebc-94f8-fa451750afa6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b4543f5-ebb4-4ebc-94f8-fa451750afa6-workload-socket\") pod \"8b4543f5-ebb4-4ebc-94f8-fa451750afa6\" (UID: \"8b4543f5-ebb4-4ebc-94f8-fa451750afa6\") " Oct 30 12:53:18.836624 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:18.836306 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b4543f5-ebb4-4ebc-94f8-fa451750afa6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b4543f5-ebb4-4ebc-94f8-fa451750afa6-credential-socket\") pod \"8b4543f5-ebb4-4ebc-94f8-fa451750afa6\" (UID: \"8b4543f5-ebb4-4ebc-94f8-fa451750afa6\") " Oct 30 12:53:18.836624 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:18.836352 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b4543f5-ebb4-4ebc-94f8-fa451750afa6-istio-token\" (UniqueName: \"kubernetes.io/projected/8b4543f5-ebb4-4ebc-94f8-fa451750afa6-istio-token\") pod \"8b4543f5-ebb4-4ebc-94f8-fa451750afa6\" (UID: \"8b4543f5-ebb4-4ebc-94f8-fa451750afa6\") " Oct 30 12:53:18.836624 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:18.836396 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8b4543f5-ebb4-4ebc-94f8-fa451750afa6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b4543f5-ebb4-4ebc-94f8-fa451750afa6-istio-podinfo\") pod \"8b4543f5-ebb4-4ebc-94f8-fa451750afa6\" (UID: \"8b4543f5-ebb4-4ebc-94f8-fa451750afa6\") " Oct 30 12:53:18.836624 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:18.836407 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b4543f5-ebb4-4ebc-94f8-fa451750afa6-workload-certs" pod "8b4543f5-ebb4-4ebc-94f8-fa451750afa6" (UID: "8b4543f5-ebb4-4ebc-94f8-fa451750afa6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:53:18.836624 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:18.836460 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b4543f5-ebb4-4ebc-94f8-fa451750afa6-kube-api-access-lfvth\" (UniqueName: \"kubernetes.io/projected/8b4543f5-ebb4-4ebc-94f8-fa451750afa6-kube-api-access-lfvth\") pod \"8b4543f5-ebb4-4ebc-94f8-fa451750afa6\" (UID: \"8b4543f5-ebb4-4ebc-94f8-fa451750afa6\") " Oct 30 12:53:18.836624 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:18.836503 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b4543f5-ebb4-4ebc-94f8-fa451750afa6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b4543f5-ebb4-4ebc-94f8-fa451750afa6-istio-envoy\") pod \"8b4543f5-ebb4-4ebc-94f8-fa451750afa6\" (UID: \"8b4543f5-ebb4-4ebc-94f8-fa451750afa6\") " Oct 30 12:53:18.836624 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:18.836535 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b4543f5-ebb4-4ebc-94f8-fa451750afa6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b4543f5-ebb4-4ebc-94f8-fa451750afa6-istio-data\") pod \"8b4543f5-ebb4-4ebc-94f8-fa451750afa6\" (UID: \"8b4543f5-ebb4-4ebc-94f8-fa451750afa6\") " Oct 30 12:53:18.836624 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:18.836530 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b4543f5-ebb4-4ebc-94f8-fa451750afa6-workload-socket" pod "8b4543f5-ebb4-4ebc-94f8-fa451750afa6" (UID: "8b4543f5-ebb4-4ebc-94f8-fa451750afa6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:53:18.838045 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:18.836663 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b4543f5-ebb4-4ebc-94f8-fa451750afa6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b4543f5-ebb4-4ebc-94f8-fa451750afa6-istiod-ca-cert\") pod \"8b4543f5-ebb4-4ebc-94f8-fa451750afa6\" (UID: \"8b4543f5-ebb4-4ebc-94f8-fa451750afa6\") " Oct 30 12:53:18.838045 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:18.837254 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b4543f5-ebb4-4ebc-94f8-fa451750afa6-istio-data" pod "8b4543f5-ebb4-4ebc-94f8-fa451750afa6" (UID: "8b4543f5-ebb4-4ebc-94f8-fa451750afa6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:53:18.838045 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:18.837287 4908 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b4543f5-ebb4-4ebc-94f8-fa451750afa6-workload-certs\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:53:18.838045 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:18.837324 4908 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b4543f5-ebb4-4ebc-94f8-fa451750afa6-workload-socket\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:53:18.838045 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:18.837344 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b4543f5-ebb4-4ebc-94f8-fa451750afa6-credential-socket" pod "8b4543f5-ebb4-4ebc-94f8-fa451750afa6" (UID: "8b4543f5-ebb4-4ebc-94f8-fa451750afa6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:53:18.838045 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:18.837602 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b4543f5-ebb4-4ebc-94f8-fa451750afa6-istiod-ca-cert" pod "8b4543f5-ebb4-4ebc-94f8-fa451750afa6" (UID: "8b4543f5-ebb4-4ebc-94f8-fa451750afa6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:53:18.839232 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:18.839190 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b4543f5-ebb4-4ebc-94f8-fa451750afa6-kube-api-access-lfvth" pod "8b4543f5-ebb4-4ebc-94f8-fa451750afa6" (UID: "8b4543f5-ebb4-4ebc-94f8-fa451750afa6"). InnerVolumeSpecName "kube-api-access-lfvth". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:53:18.840038 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:18.839995 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8b4543f5-ebb4-4ebc-94f8-fa451750afa6-istio-podinfo" pod "8b4543f5-ebb4-4ebc-94f8-fa451750afa6" (UID: "8b4543f5-ebb4-4ebc-94f8-fa451750afa6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 12:53:18.840161 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:18.840075 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b4543f5-ebb4-4ebc-94f8-fa451750afa6-istio-token" pod "8b4543f5-ebb4-4ebc-94f8-fa451750afa6" (UID: "8b4543f5-ebb4-4ebc-94f8-fa451750afa6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:53:18.840161 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:18.840133 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b4543f5-ebb4-4ebc-94f8-fa451750afa6-istio-envoy" pod "8b4543f5-ebb4-4ebc-94f8-fa451750afa6" (UID: "8b4543f5-ebb4-4ebc-94f8-fa451750afa6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:53:18.939053 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:18.939016 4908 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b4543f5-ebb4-4ebc-94f8-fa451750afa6-istio-envoy\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:53:18.939053 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:18.939050 4908 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b4543f5-ebb4-4ebc-94f8-fa451750afa6-istio-data\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:53:18.939053 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:18.939060 4908 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b4543f5-ebb4-4ebc-94f8-fa451750afa6-istiod-ca-cert\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:53:18.939312 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:18.939068 4908 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b4543f5-ebb4-4ebc-94f8-fa451750afa6-credential-socket\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:53:18.939312 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:18.939079 4908 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b4543f5-ebb4-4ebc-94f8-fa451750afa6-istio-token\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:53:18.939312 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:18.939087 4908 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b4543f5-ebb4-4ebc-94f8-fa451750afa6-istio-podinfo\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:53:18.939312 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:18.939097 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lfvth\" (UniqueName: \"kubernetes.io/projected/8b4543f5-ebb4-4ebc-94f8-fa451750afa6-kube-api-access-lfvth\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:53:19.544423 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:19.544383 4908 generic.go:356] "Generic (PLEG): container finished" podID="8b4543f5-ebb4-4ebc-94f8-fa451750afa6" containerID="fe633ac50288577e8ff75c53d3a4ce7a91bce30d42c38ed3407dd57504d013d9" exitCode=0 Oct 30 12:53:19.544600 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:19.544435 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-pzxkx" Oct 30 12:53:19.544600 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:19.544464 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-pzxkx" event={"ID":"8b4543f5-ebb4-4ebc-94f8-fa451750afa6","Type":"ContainerDied","Data":"fe633ac50288577e8ff75c53d3a4ce7a91bce30d42c38ed3407dd57504d013d9"} Oct 30 12:53:19.544600 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:19.544522 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-pzxkx" event={"ID":"8b4543f5-ebb4-4ebc-94f8-fa451750afa6","Type":"ContainerDied","Data":"2a2269ede944158044bc0b2f6fdb63daf2080094f8075741188772671269e8db"} Oct 30 12:53:19.544600 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:19.544556 4908 scope.go:122] "RemoveContainer" containerID="fe633ac50288577e8ff75c53d3a4ce7a91bce30d42c38ed3407dd57504d013d9" Oct 30 12:53:19.557962 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:19.557931 4908 scope.go:122] "RemoveContainer" containerID="fe633ac50288577e8ff75c53d3a4ce7a91bce30d42c38ed3407dd57504d013d9" Oct 30 12:53:19.558221 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:53:19.558198 4908 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe633ac50288577e8ff75c53d3a4ce7a91bce30d42c38ed3407dd57504d013d9\": container with ID starting with fe633ac50288577e8ff75c53d3a4ce7a91bce30d42c38ed3407dd57504d013d9 not found: ID does not exist" containerID="fe633ac50288577e8ff75c53d3a4ce7a91bce30d42c38ed3407dd57504d013d9" Oct 30 12:53:19.558276 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:19.558231 4908 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe633ac50288577e8ff75c53d3a4ce7a91bce30d42c38ed3407dd57504d013d9"} err="failed to get container status \"fe633ac50288577e8ff75c53d3a4ce7a91bce30d42c38ed3407dd57504d013d9\": rpc error: code = NotFound desc = could not find container \"fe633ac50288577e8ff75c53d3a4ce7a91bce30d42c38ed3407dd57504d013d9\": container with ID starting with fe633ac50288577e8ff75c53d3a4ce7a91bce30d42c38ed3407dd57504d013d9 not found: ID does not exist" Oct 30 12:53:19.611521 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:19.611494 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-pzxkx"] Oct 30 12:53:19.620918 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:19.620892 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-pzxkx"] Oct 30 12:53:19.687176 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:53:19.687139 4908 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vn6vv" containerName="extractor" Oct 30 12:53:21.076195 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:21.076154 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9wwfc"] Oct 30 12:53:21.609208 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:21.609172 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b4543f5-ebb4-4ebc-94f8-fa451750afa6" path="/var/lib/kubelet/pods/8b4543f5-ebb4-4ebc-94f8-fa451750afa6/volumes" Oct 30 12:53:33.823282 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:33.823249 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0cc48ca4-f8pxw"] Oct 30 12:53:33.823865 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:33.823842 4908 cpu_manager.go:414] "containerMap: removing container" podUID="8b4543f5-ebb4-4ebc-94f8-fa451750afa6" containerName="istio-proxy" Oct 30 12:53:33.823865 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:33.823857 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b4543f5-ebb4-4ebc-94f8-fa451750afa6" containerName="istio-proxy" Oct 30 12:53:33.823980 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:33.823969 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b4543f5-ebb4-4ebc-94f8-fa451750afa6" containerName="istio-proxy" Oct 30 12:53:33.827332 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:33.827313 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cc48ca4-f8pxw" Oct 30 12:53:33.866873 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:33.853695 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 12:53:33.866873 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:33.853891 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 12:53:33.866873 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:33.854085 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ccb8306-200d-492a-86d6-c381f6e7b34d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0cc48ca4-f8pxw\" (UID: \"7ccb8306-200d-492a-86d6-c381f6e7b34d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cc48ca4-f8pxw" Oct 30 12:53:33.866873 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:33.854095 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s75hq\"" Oct 30 12:53:33.866873 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:33.854257 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z95dx\" (UniqueName: \"kubernetes.io/projected/7ccb8306-200d-492a-86d6-c381f6e7b34d-kube-api-access-z95dx\") pod \"maas-api-revoke-sub-0cc48ca4-f8pxw\" (UID: \"7ccb8306-200d-492a-86d6-c381f6e7b34d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cc48ca4-f8pxw" Oct 30 12:53:33.866873 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:33.854564 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ccb8306-200d-492a-86d6-c381f6e7b34d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0cc48ca4-f8pxw\" (UID: \"7ccb8306-200d-492a-86d6-c381f6e7b34d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cc48ca4-f8pxw" Oct 30 12:53:33.866873 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:33.855605 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 12:53:33.866873 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:33.865724 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0cc48ca4-f8pxw"] Oct 30 12:53:33.956135 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:33.956092 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ccb8306-200d-492a-86d6-c381f6e7b34d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0cc48ca4-f8pxw\" (UID: \"7ccb8306-200d-492a-86d6-c381f6e7b34d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cc48ca4-f8pxw" Oct 30 12:53:33.956330 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:33.956139 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ccb8306-200d-492a-86d6-c381f6e7b34d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0cc48ca4-f8pxw\" (UID: \"7ccb8306-200d-492a-86d6-c381f6e7b34d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cc48ca4-f8pxw" Oct 30 12:53:33.956330 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:33.956238 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z95dx\" (UniqueName: \"kubernetes.io/projected/7ccb8306-200d-492a-86d6-c381f6e7b34d-kube-api-access-z95dx\") pod \"maas-api-revoke-sub-0cc48ca4-f8pxw\" (UID: \"7ccb8306-200d-492a-86d6-c381f6e7b34d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cc48ca4-f8pxw" Oct 30 12:53:33.956767 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:33.956743 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ccb8306-200d-492a-86d6-c381f6e7b34d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0cc48ca4-f8pxw\" (UID: \"7ccb8306-200d-492a-86d6-c381f6e7b34d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cc48ca4-f8pxw" Oct 30 12:53:33.958516 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:33.958492 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ccb8306-200d-492a-86d6-c381f6e7b34d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0cc48ca4-f8pxw\" (UID: \"7ccb8306-200d-492a-86d6-c381f6e7b34d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cc48ca4-f8pxw" Oct 30 12:53:33.978549 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:33.978520 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z95dx\" (UniqueName: \"kubernetes.io/projected/7ccb8306-200d-492a-86d6-c381f6e7b34d-kube-api-access-z95dx\") pod \"maas-api-revoke-sub-0cc48ca4-f8pxw\" (UID: \"7ccb8306-200d-492a-86d6-c381f6e7b34d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cc48ca4-f8pxw" Oct 30 12:53:34.140886 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:34.140770 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cc48ca4-f8pxw" Oct 30 12:53:34.519071 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:34.518942 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0cc48ca4-f8pxw"] Oct 30 12:53:34.522258 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:53:34.522227 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ccb8306_200d_492a_86d6_c381f6e7b34d.slice/crio-9bfce477c3577edb9bcd95f1854e6cc072e87a1f57af9dea55fad8493dbb3b80 WatchSource:0}: Error finding container 9bfce477c3577edb9bcd95f1854e6cc072e87a1f57af9dea55fad8493dbb3b80: Status 404 returned error can't find the container with id 9bfce477c3577edb9bcd95f1854e6cc072e87a1f57af9dea55fad8493dbb3b80 Oct 30 12:53:34.674045 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:34.674008 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cc48ca4-f8pxw" event={"ID":"7ccb8306-200d-492a-86d6-c381f6e7b34d","Type":"ContainerStarted","Data":"9bfce477c3577edb9bcd95f1854e6cc072e87a1f57af9dea55fad8493dbb3b80"} Oct 30 12:53:35.691098 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:35.691063 4908 generic.go:356] "Generic (PLEG): container finished" podID="7ccb8306-200d-492a-86d6-c381f6e7b34d" containerID="57969536ae6c3f0b27edfd023b2089bea00b2166168c54c70c09447e1b12a857" exitCode=0 Oct 30 12:53:35.691543 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:35.691147 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cc48ca4-f8pxw" event={"ID":"7ccb8306-200d-492a-86d6-c381f6e7b34d","Type":"ContainerDied","Data":"57969536ae6c3f0b27edfd023b2089bea00b2166168c54c70c09447e1b12a857"} Oct 30 12:53:36.874608 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:36.874582 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cc48ca4-f8pxw" Oct 30 12:53:36.998908 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:36.998870 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ccb8306-200d-492a-86d6-c381f6e7b34d-kube-api-access-z95dx\" (UniqueName: \"kubernetes.io/projected/7ccb8306-200d-492a-86d6-c381f6e7b34d-kube-api-access-z95dx\") pod \"7ccb8306-200d-492a-86d6-c381f6e7b34d\" (UID: \"7ccb8306-200d-492a-86d6-c381f6e7b34d\") " Oct 30 12:53:36.999111 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:36.998939 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7ccb8306-200d-492a-86d6-c381f6e7b34d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ccb8306-200d-492a-86d6-c381f6e7b34d-maas-api-xks-ca\") pod \"7ccb8306-200d-492a-86d6-c381f6e7b34d\" (UID: \"7ccb8306-200d-492a-86d6-c381f6e7b34d\") " Oct 30 12:53:36.999111 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:36.999094 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ccb8306-200d-492a-86d6-c381f6e7b34d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ccb8306-200d-492a-86d6-c381f6e7b34d-maas-api-openshift-ca\") pod \"7ccb8306-200d-492a-86d6-c381f6e7b34d\" (UID: \"7ccb8306-200d-492a-86d6-c381f6e7b34d\") " Oct 30 12:53:36.999528 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:36.999495 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ccb8306-200d-492a-86d6-c381f6e7b34d-maas-api-openshift-ca" pod "7ccb8306-200d-492a-86d6-c381f6e7b34d" (UID: "7ccb8306-200d-492a-86d6-c381f6e7b34d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:53:36.999671 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:36.999645 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ccb8306-200d-492a-86d6-c381f6e7b34d-maas-api-openshift-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:53:37.001210 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:37.001182 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ccb8306-200d-492a-86d6-c381f6e7b34d-maas-api-xks-ca" pod "7ccb8306-200d-492a-86d6-c381f6e7b34d" (UID: "7ccb8306-200d-492a-86d6-c381f6e7b34d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:53:37.001340 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:37.001206 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ccb8306-200d-492a-86d6-c381f6e7b34d-kube-api-access-z95dx" pod "7ccb8306-200d-492a-86d6-c381f6e7b34d" (UID: "7ccb8306-200d-492a-86d6-c381f6e7b34d"). InnerVolumeSpecName "kube-api-access-z95dx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:53:37.100926 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:37.100893 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z95dx\" (UniqueName: \"kubernetes.io/projected/7ccb8306-200d-492a-86d6-c381f6e7b34d-kube-api-access-z95dx\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:53:37.100926 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:37.100920 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ccb8306-200d-492a-86d6-c381f6e7b34d-maas-api-xks-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:53:37.709638 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:37.709605 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cc48ca4-f8pxw" event={"ID":"7ccb8306-200d-492a-86d6-c381f6e7b34d","Type":"ContainerDied","Data":"9bfce477c3577edb9bcd95f1854e6cc072e87a1f57af9dea55fad8493dbb3b80"} Oct 30 12:53:37.709808 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:37.709669 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9bfce477c3577edb9bcd95f1854e6cc072e87a1f57af9dea55fad8493dbb3b80" Oct 30 12:53:37.709808 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:37.709616 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cc48ca4-f8pxw" Oct 30 12:53:45.875234 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:45.875197 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-94eab2bb-hzvrh"] Oct 30 12:53:45.876077 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:45.876055 4908 cpu_manager.go:414] "containerMap: removing container" podUID="7ccb8306-200d-492a-86d6-c381f6e7b34d" containerName="revoke-keys" Oct 30 12:53:45.876153 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:45.876079 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ccb8306-200d-492a-86d6-c381f6e7b34d" containerName="revoke-keys" Oct 30 12:53:45.876285 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:45.876273 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ccb8306-200d-492a-86d6-c381f6e7b34d" containerName="revoke-keys" Oct 30 12:53:45.880392 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:45.880373 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94eab2bb-hzvrh" Oct 30 12:53:45.900282 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:45.900255 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-94eab2bb-hzvrh"] Oct 30 12:53:45.904148 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:45.904127 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 12:53:45.907038 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:45.907016 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 12:53:45.907232 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:45.907048 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s75hq\"" Oct 30 12:53:45.907232 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:45.907077 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 12:53:46.002769 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:46.002735 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5d09275-8c9d-4118-ac23-c7452ecee0c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-94eab2bb-hzvrh\" (UID: \"b5d09275-8c9d-4118-ac23-c7452ecee0c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94eab2bb-hzvrh" Oct 30 12:53:46.002937 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:46.002779 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5d09275-8c9d-4118-ac23-c7452ecee0c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-94eab2bb-hzvrh\" (UID: \"b5d09275-8c9d-4118-ac23-c7452ecee0c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94eab2bb-hzvrh" Oct 30 12:53:46.003056 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:46.003039 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmmgt\" (UniqueName: \"kubernetes.io/projected/b5d09275-8c9d-4118-ac23-c7452ecee0c6-kube-api-access-rmmgt\") pod \"maas-api-revoke-sub-94eab2bb-hzvrh\" (UID: \"b5d09275-8c9d-4118-ac23-c7452ecee0c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94eab2bb-hzvrh" Oct 30 12:53:46.104582 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:46.104549 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmmgt\" (UniqueName: \"kubernetes.io/projected/b5d09275-8c9d-4118-ac23-c7452ecee0c6-kube-api-access-rmmgt\") pod \"maas-api-revoke-sub-94eab2bb-hzvrh\" (UID: \"b5d09275-8c9d-4118-ac23-c7452ecee0c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94eab2bb-hzvrh" Oct 30 12:53:46.104780 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:46.104605 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5d09275-8c9d-4118-ac23-c7452ecee0c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-94eab2bb-hzvrh\" (UID: \"b5d09275-8c9d-4118-ac23-c7452ecee0c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94eab2bb-hzvrh" Oct 30 12:53:46.104780 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:46.104646 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5d09275-8c9d-4118-ac23-c7452ecee0c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-94eab2bb-hzvrh\" (UID: \"b5d09275-8c9d-4118-ac23-c7452ecee0c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94eab2bb-hzvrh" Oct 30 12:53:46.105467 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:46.105443 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5d09275-8c9d-4118-ac23-c7452ecee0c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-94eab2bb-hzvrh\" (UID: \"b5d09275-8c9d-4118-ac23-c7452ecee0c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94eab2bb-hzvrh" Oct 30 12:53:46.107232 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:46.107202 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5d09275-8c9d-4118-ac23-c7452ecee0c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-94eab2bb-hzvrh\" (UID: \"b5d09275-8c9d-4118-ac23-c7452ecee0c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94eab2bb-hzvrh" Oct 30 12:53:46.126302 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:46.126252 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmmgt\" (UniqueName: \"kubernetes.io/projected/b5d09275-8c9d-4118-ac23-c7452ecee0c6-kube-api-access-rmmgt\") pod \"maas-api-revoke-sub-94eab2bb-hzvrh\" (UID: \"b5d09275-8c9d-4118-ac23-c7452ecee0c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94eab2bb-hzvrh" Oct 30 12:53:46.193897 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:46.193868 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94eab2bb-hzvrh" Oct 30 12:53:46.564663 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:46.564621 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-94eab2bb-hzvrh"] Oct 30 12:53:46.777319 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:46.777284 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94eab2bb-hzvrh" event={"ID":"b5d09275-8c9d-4118-ac23-c7452ecee0c6","Type":"ContainerStarted","Data":"63afea26bef664e0f626426b5d6cdd2a9e7f75de9abf70e5ab013425fa1a83c8"} Oct 30 12:53:47.576191 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:47.576152 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-eab799-openshift-default-574ddfd59c-tfw7h"] Oct 30 12:53:47.583239 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:47.582081 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-eab799-openshift-default-574ddfd59c-tfw7h" Oct 30 12:53:47.620817 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:47.620761 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-eab799-openshift-default-574ddfd59c-tfw7h"] Oct 30 12:53:47.622200 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:47.622156 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-eab799-openshift-default-dockercfg-4rb87\"" Oct 30 12:53:47.669884 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:47.669507 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61f8676a-8552-4066-a292-26bbd8b64d6b-workload-certs\") pod \"e2e-inf-a-eab799-openshift-default-574ddfd59c-tfw7h\" (UID: \"61f8676a-8552-4066-a292-26bbd8b64d6b\") " pod="openshift-ingress/e2e-inf-a-eab799-openshift-default-574ddfd59c-tfw7h" Oct 30 12:53:47.690861 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:47.678999 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61f8676a-8552-4066-a292-26bbd8b64d6b-istio-podinfo\") pod \"e2e-inf-a-eab799-openshift-default-574ddfd59c-tfw7h\" (UID: \"61f8676a-8552-4066-a292-26bbd8b64d6b\") " pod="openshift-ingress/e2e-inf-a-eab799-openshift-default-574ddfd59c-tfw7h" Oct 30 12:53:47.690861 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:47.679053 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61f8676a-8552-4066-a292-26bbd8b64d6b-istio-envoy\") pod \"e2e-inf-a-eab799-openshift-default-574ddfd59c-tfw7h\" (UID: \"61f8676a-8552-4066-a292-26bbd8b64d6b\") " pod="openshift-ingress/e2e-inf-a-eab799-openshift-default-574ddfd59c-tfw7h" Oct 30 12:53:47.690861 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:47.679179 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61f8676a-8552-4066-a292-26bbd8b64d6b-istiod-ca-cert\") pod \"e2e-inf-a-eab799-openshift-default-574ddfd59c-tfw7h\" (UID: \"61f8676a-8552-4066-a292-26bbd8b64d6b\") " pod="openshift-ingress/e2e-inf-a-eab799-openshift-default-574ddfd59c-tfw7h" Oct 30 12:53:47.690861 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:47.679208 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61f8676a-8552-4066-a292-26bbd8b64d6b-istio-data\") pod \"e2e-inf-a-eab799-openshift-default-574ddfd59c-tfw7h\" (UID: \"61f8676a-8552-4066-a292-26bbd8b64d6b\") " pod="openshift-ingress/e2e-inf-a-eab799-openshift-default-574ddfd59c-tfw7h" Oct 30 12:53:47.690861 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:47.679256 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-425jt\" (UniqueName: \"kubernetes.io/projected/61f8676a-8552-4066-a292-26bbd8b64d6b-kube-api-access-425jt\") pod \"e2e-inf-a-eab799-openshift-default-574ddfd59c-tfw7h\" (UID: \"61f8676a-8552-4066-a292-26bbd8b64d6b\") " pod="openshift-ingress/e2e-inf-a-eab799-openshift-default-574ddfd59c-tfw7h" Oct 30 12:53:47.690861 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:47.679310 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61f8676a-8552-4066-a292-26bbd8b64d6b-istio-token\") pod \"e2e-inf-a-eab799-openshift-default-574ddfd59c-tfw7h\" (UID: \"61f8676a-8552-4066-a292-26bbd8b64d6b\") " pod="openshift-ingress/e2e-inf-a-eab799-openshift-default-574ddfd59c-tfw7h" Oct 30 12:53:47.690861 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:47.679334 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61f8676a-8552-4066-a292-26bbd8b64d6b-credential-socket\") pod \"e2e-inf-a-eab799-openshift-default-574ddfd59c-tfw7h\" (UID: \"61f8676a-8552-4066-a292-26bbd8b64d6b\") " pod="openshift-ingress/e2e-inf-a-eab799-openshift-default-574ddfd59c-tfw7h" Oct 30 12:53:47.690861 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:47.679411 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61f8676a-8552-4066-a292-26bbd8b64d6b-workload-socket\") pod \"e2e-inf-a-eab799-openshift-default-574ddfd59c-tfw7h\" (UID: \"61f8676a-8552-4066-a292-26bbd8b64d6b\") " pod="openshift-ingress/e2e-inf-a-eab799-openshift-default-574ddfd59c-tfw7h" Oct 30 12:53:47.790755 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:47.788736 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61f8676a-8552-4066-a292-26bbd8b64d6b-workload-socket\") pod \"e2e-inf-a-eab799-openshift-default-574ddfd59c-tfw7h\" (UID: \"61f8676a-8552-4066-a292-26bbd8b64d6b\") " pod="openshift-ingress/e2e-inf-a-eab799-openshift-default-574ddfd59c-tfw7h" Oct 30 12:53:47.790755 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:47.788917 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61f8676a-8552-4066-a292-26bbd8b64d6b-workload-certs\") pod \"e2e-inf-a-eab799-openshift-default-574ddfd59c-tfw7h\" (UID: \"61f8676a-8552-4066-a292-26bbd8b64d6b\") " pod="openshift-ingress/e2e-inf-a-eab799-openshift-default-574ddfd59c-tfw7h" Oct 30 12:53:47.790755 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:47.789000 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61f8676a-8552-4066-a292-26bbd8b64d6b-istio-podinfo\") pod \"e2e-inf-a-eab799-openshift-default-574ddfd59c-tfw7h\" (UID: \"61f8676a-8552-4066-a292-26bbd8b64d6b\") " pod="openshift-ingress/e2e-inf-a-eab799-openshift-default-574ddfd59c-tfw7h" Oct 30 12:53:47.790755 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:47.789034 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61f8676a-8552-4066-a292-26bbd8b64d6b-istio-envoy\") pod \"e2e-inf-a-eab799-openshift-default-574ddfd59c-tfw7h\" (UID: \"61f8676a-8552-4066-a292-26bbd8b64d6b\") " pod="openshift-ingress/e2e-inf-a-eab799-openshift-default-574ddfd59c-tfw7h" Oct 30 12:53:47.790755 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:47.789144 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61f8676a-8552-4066-a292-26bbd8b64d6b-istiod-ca-cert\") pod \"e2e-inf-a-eab799-openshift-default-574ddfd59c-tfw7h\" (UID: \"61f8676a-8552-4066-a292-26bbd8b64d6b\") " pod="openshift-ingress/e2e-inf-a-eab799-openshift-default-574ddfd59c-tfw7h" Oct 30 12:53:47.790755 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:47.789176 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61f8676a-8552-4066-a292-26bbd8b64d6b-istio-data\") pod \"e2e-inf-a-eab799-openshift-default-574ddfd59c-tfw7h\" (UID: \"61f8676a-8552-4066-a292-26bbd8b64d6b\") " pod="openshift-ingress/e2e-inf-a-eab799-openshift-default-574ddfd59c-tfw7h" Oct 30 12:53:47.790755 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:47.789217 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-425jt\" (UniqueName: \"kubernetes.io/projected/61f8676a-8552-4066-a292-26bbd8b64d6b-kube-api-access-425jt\") pod \"e2e-inf-a-eab799-openshift-default-574ddfd59c-tfw7h\" (UID: \"61f8676a-8552-4066-a292-26bbd8b64d6b\") " pod="openshift-ingress/e2e-inf-a-eab799-openshift-default-574ddfd59c-tfw7h" Oct 30 12:53:47.790755 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:47.789261 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61f8676a-8552-4066-a292-26bbd8b64d6b-istio-token\") pod \"e2e-inf-a-eab799-openshift-default-574ddfd59c-tfw7h\" (UID: \"61f8676a-8552-4066-a292-26bbd8b64d6b\") " pod="openshift-ingress/e2e-inf-a-eab799-openshift-default-574ddfd59c-tfw7h" Oct 30 12:53:47.790755 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:47.789303 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61f8676a-8552-4066-a292-26bbd8b64d6b-credential-socket\") pod \"e2e-inf-a-eab799-openshift-default-574ddfd59c-tfw7h\" (UID: \"61f8676a-8552-4066-a292-26bbd8b64d6b\") " pod="openshift-ingress/e2e-inf-a-eab799-openshift-default-574ddfd59c-tfw7h" Oct 30 12:53:47.790755 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:47.789786 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61f8676a-8552-4066-a292-26bbd8b64d6b-credential-socket\") pod \"e2e-inf-a-eab799-openshift-default-574ddfd59c-tfw7h\" (UID: \"61f8676a-8552-4066-a292-26bbd8b64d6b\") " pod="openshift-ingress/e2e-inf-a-eab799-openshift-default-574ddfd59c-tfw7h" Oct 30 12:53:47.790755 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:47.790068 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61f8676a-8552-4066-a292-26bbd8b64d6b-workload-socket\") pod \"e2e-inf-a-eab799-openshift-default-574ddfd59c-tfw7h\" (UID: \"61f8676a-8552-4066-a292-26bbd8b64d6b\") " pod="openshift-ingress/e2e-inf-a-eab799-openshift-default-574ddfd59c-tfw7h" Oct 30 12:53:47.790755 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:47.790299 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61f8676a-8552-4066-a292-26bbd8b64d6b-workload-certs\") pod \"e2e-inf-a-eab799-openshift-default-574ddfd59c-tfw7h\" (UID: \"61f8676a-8552-4066-a292-26bbd8b64d6b\") " pod="openshift-ingress/e2e-inf-a-eab799-openshift-default-574ddfd59c-tfw7h" Oct 30 12:53:47.791617 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:47.791184 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61f8676a-8552-4066-a292-26bbd8b64d6b-istio-data\") pod \"e2e-inf-a-eab799-openshift-default-574ddfd59c-tfw7h\" (UID: \"61f8676a-8552-4066-a292-26bbd8b64d6b\") " pod="openshift-ingress/e2e-inf-a-eab799-openshift-default-574ddfd59c-tfw7h" Oct 30 12:53:47.792179 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:47.792158 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61f8676a-8552-4066-a292-26bbd8b64d6b-istiod-ca-cert\") pod \"e2e-inf-a-eab799-openshift-default-574ddfd59c-tfw7h\" (UID: \"61f8676a-8552-4066-a292-26bbd8b64d6b\") " pod="openshift-ingress/e2e-inf-a-eab799-openshift-default-574ddfd59c-tfw7h" Oct 30 12:53:47.844298 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:47.844259 4908 generic.go:356] "Generic (PLEG): container finished" podID="b5d09275-8c9d-4118-ac23-c7452ecee0c6" containerID="f4d3843e83c76011c2d7b6c8292837396ce1b8e29643f00fc2d16c80afec8d95" exitCode=0 Oct 30 12:53:47.856850 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:47.848010 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94eab2bb-hzvrh" event={"ID":"b5d09275-8c9d-4118-ac23-c7452ecee0c6","Type":"ContainerDied","Data":"f4d3843e83c76011c2d7b6c8292837396ce1b8e29643f00fc2d16c80afec8d95"} Oct 30 12:53:47.881656 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:47.881612 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61f8676a-8552-4066-a292-26bbd8b64d6b-istio-token\") pod \"e2e-inf-a-eab799-openshift-default-574ddfd59c-tfw7h\" (UID: \"61f8676a-8552-4066-a292-26bbd8b64d6b\") " pod="openshift-ingress/e2e-inf-a-eab799-openshift-default-574ddfd59c-tfw7h" Oct 30 12:53:47.891922 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:47.888796 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61f8676a-8552-4066-a292-26bbd8b64d6b-istio-envoy\") pod \"e2e-inf-a-eab799-openshift-default-574ddfd59c-tfw7h\" (UID: \"61f8676a-8552-4066-a292-26bbd8b64d6b\") " pod="openshift-ingress/e2e-inf-a-eab799-openshift-default-574ddfd59c-tfw7h" Oct 30 12:53:47.891922 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:47.891307 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61f8676a-8552-4066-a292-26bbd8b64d6b-istio-podinfo\") pod \"e2e-inf-a-eab799-openshift-default-574ddfd59c-tfw7h\" (UID: \"61f8676a-8552-4066-a292-26bbd8b64d6b\") " pod="openshift-ingress/e2e-inf-a-eab799-openshift-default-574ddfd59c-tfw7h" Oct 30 12:53:47.895249 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:47.895167 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-425jt\" (UniqueName: \"kubernetes.io/projected/61f8676a-8552-4066-a292-26bbd8b64d6b-kube-api-access-425jt\") pod \"e2e-inf-a-eab799-openshift-default-574ddfd59c-tfw7h\" (UID: \"61f8676a-8552-4066-a292-26bbd8b64d6b\") " pod="openshift-ingress/e2e-inf-a-eab799-openshift-default-574ddfd59c-tfw7h" Oct 30 12:53:47.917124 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:47.917085 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-eab799-openshift-default-574ddfd59c-tfw7h" Oct 30 12:53:48.450222 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:48.450192 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-eab799-openshift-default-574ddfd59c-tfw7h"] Oct 30 12:53:48.453465 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:53:48.453436 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61f8676a_8552_4066_a292_26bbd8b64d6b.slice/crio-69087d3e4b72b8ba3236f8d4e449a182aa9faf5deabb91fc0e102b8485645496 WatchSource:0}: Error finding container 69087d3e4b72b8ba3236f8d4e449a182aa9faf5deabb91fc0e102b8485645496: Status 404 returned error can't find the container with id 69087d3e4b72b8ba3236f8d4e449a182aa9faf5deabb91fc0e102b8485645496 Oct 30 12:53:48.456926 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:48.456892 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:53:48.457017 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:48.456952 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:53:48.457017 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:48.456984 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:53:48.860258 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:48.860207 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-eab799-openshift-default-574ddfd59c-tfw7h" event={"ID":"61f8676a-8552-4066-a292-26bbd8b64d6b","Type":"ContainerStarted","Data":"13fa47b6b1bc9c45ac4920d0db4a61a189ed3bd195fc322079723c64eb6e4f0f"} Oct 30 12:53:48.860796 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:48.860273 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-eab799-openshift-default-574ddfd59c-tfw7h" event={"ID":"61f8676a-8552-4066-a292-26bbd8b64d6b","Type":"ContainerStarted","Data":"69087d3e4b72b8ba3236f8d4e449a182aa9faf5deabb91fc0e102b8485645496"} Oct 30 12:53:48.918990 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:48.918956 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-eab799-openshift-default-574ddfd59c-tfw7h" Oct 30 12:53:48.934682 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:48.934243 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-eab799-openshift-default-574ddfd59c-tfw7h" podStartSLOduration=1.934222933 podStartE2EDuration="1.934222933s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:53:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:53:48.918581607 +0000 UTC m=+3585.433010353" watchObservedRunningTime="2026-10-30 12:53:48.934222933 +0000 UTC m=+3585.448651680" Oct 30 12:53:49.217315 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:49.217282 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94eab2bb-hzvrh" Oct 30 12:53:49.258220 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:49.258187 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5d09275-8c9d-4118-ac23-c7452ecee0c6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5d09275-8c9d-4118-ac23-c7452ecee0c6-maas-api-openshift-ca\") pod \"b5d09275-8c9d-4118-ac23-c7452ecee0c6\" (UID: \"b5d09275-8c9d-4118-ac23-c7452ecee0c6\") " Oct 30 12:53:49.258442 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:49.258421 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5d09275-8c9d-4118-ac23-c7452ecee0c6-kube-api-access-rmmgt\" (UniqueName: \"kubernetes.io/projected/b5d09275-8c9d-4118-ac23-c7452ecee0c6-kube-api-access-rmmgt\") pod \"b5d09275-8c9d-4118-ac23-c7452ecee0c6\" (UID: \"b5d09275-8c9d-4118-ac23-c7452ecee0c6\") " Oct 30 12:53:49.258622 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:49.258604 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5d09275-8c9d-4118-ac23-c7452ecee0c6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5d09275-8c9d-4118-ac23-c7452ecee0c6-maas-api-xks-ca\") pod \"b5d09275-8c9d-4118-ac23-c7452ecee0c6\" (UID: \"b5d09275-8c9d-4118-ac23-c7452ecee0c6\") " Oct 30 12:53:49.258720 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:49.258619 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5d09275-8c9d-4118-ac23-c7452ecee0c6-maas-api-openshift-ca" pod "b5d09275-8c9d-4118-ac23-c7452ecee0c6" (UID: "b5d09275-8c9d-4118-ac23-c7452ecee0c6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:53:49.262320 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:49.259178 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5d09275-8c9d-4118-ac23-c7452ecee0c6-maas-api-openshift-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:53:49.267436 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:49.267395 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5d09275-8c9d-4118-ac23-c7452ecee0c6-kube-api-access-rmmgt" pod "b5d09275-8c9d-4118-ac23-c7452ecee0c6" (UID: "b5d09275-8c9d-4118-ac23-c7452ecee0c6"). InnerVolumeSpecName "kube-api-access-rmmgt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:53:49.359475 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:49.359436 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5d09275-8c9d-4118-ac23-c7452ecee0c6-maas-api-xks-ca" pod "b5d09275-8c9d-4118-ac23-c7452ecee0c6" (UID: "b5d09275-8c9d-4118-ac23-c7452ecee0c6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:53:49.365122 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:49.365091 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rmmgt\" (UniqueName: \"kubernetes.io/projected/b5d09275-8c9d-4118-ac23-c7452ecee0c6-kube-api-access-rmmgt\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:53:49.365286 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:49.365272 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5d09275-8c9d-4118-ac23-c7452ecee0c6-maas-api-xks-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:53:49.689671 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:53:49.689627 4908 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vn6vv" containerName="extractor" Oct 30 12:53:49.870460 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:49.870370 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94eab2bb-hzvrh" event={"ID":"b5d09275-8c9d-4118-ac23-c7452ecee0c6","Type":"ContainerDied","Data":"63afea26bef664e0f626426b5d6cdd2a9e7f75de9abf70e5ab013425fa1a83c8"} Oct 30 12:53:49.870460 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:49.870438 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="63afea26bef664e0f626426b5d6cdd2a9e7f75de9abf70e5ab013425fa1a83c8" Oct 30 12:53:49.871026 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:49.870562 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94eab2bb-hzvrh" Oct 30 12:53:49.924027 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:49.923998 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-eab799-openshift-default-574ddfd59c-tfw7h" Oct 30 12:53:50.880487 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:50.880447 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-eab799-openshift-default-574ddfd59c-tfw7h" Oct 30 12:53:50.881358 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:50.881338 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-eab799-openshift-default-574ddfd59c-tfw7h" Oct 30 12:53:56.510557 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:56.510523 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5424a42b-knz9w"] Oct 30 12:53:56.511183 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:56.511165 4908 cpu_manager.go:414] "containerMap: removing container" podUID="b5d09275-8c9d-4118-ac23-c7452ecee0c6" containerName="revoke-keys" Oct 30 12:53:56.511235 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:56.511184 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5d09275-8c9d-4118-ac23-c7452ecee0c6" containerName="revoke-keys" Oct 30 12:53:56.511302 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:56.511292 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5d09275-8c9d-4118-ac23-c7452ecee0c6" containerName="revoke-keys" Oct 30 12:53:56.515050 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:56.515034 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5424a42b-knz9w" Oct 30 12:53:56.542896 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:56.542865 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5424a42b-knz9w"] Oct 30 12:53:56.547026 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:56.546999 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 12:53:56.547227 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:56.547187 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 12:53:56.547608 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:56.547314 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s75hq\"" Oct 30 12:53:56.547608 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:56.547401 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 12:53:56.629921 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:56.629887 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5abc3d06-43b1-421b-95d2-be659f9668c4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5424a42b-knz9w\" (UID: \"5abc3d06-43b1-421b-95d2-be659f9668c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5424a42b-knz9w" Oct 30 12:53:56.630093 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:56.630016 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8cln\" (UniqueName: \"kubernetes.io/projected/5abc3d06-43b1-421b-95d2-be659f9668c4-kube-api-access-w8cln\") pod \"maas-api-revoke-sub-5424a42b-knz9w\" (UID: \"5abc3d06-43b1-421b-95d2-be659f9668c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5424a42b-knz9w" Oct 30 12:53:56.630152 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:56.630103 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5abc3d06-43b1-421b-95d2-be659f9668c4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5424a42b-knz9w\" (UID: \"5abc3d06-43b1-421b-95d2-be659f9668c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5424a42b-knz9w" Oct 30 12:53:56.730984 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:56.730957 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5abc3d06-43b1-421b-95d2-be659f9668c4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5424a42b-knz9w\" (UID: \"5abc3d06-43b1-421b-95d2-be659f9668c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5424a42b-knz9w" Oct 30 12:53:56.731123 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:56.731034 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8cln\" (UniqueName: \"kubernetes.io/projected/5abc3d06-43b1-421b-95d2-be659f9668c4-kube-api-access-w8cln\") pod \"maas-api-revoke-sub-5424a42b-knz9w\" (UID: \"5abc3d06-43b1-421b-95d2-be659f9668c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5424a42b-knz9w" Oct 30 12:53:56.731123 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:56.731100 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5abc3d06-43b1-421b-95d2-be659f9668c4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5424a42b-knz9w\" (UID: \"5abc3d06-43b1-421b-95d2-be659f9668c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5424a42b-knz9w" Oct 30 12:53:56.731788 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:56.731767 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5abc3d06-43b1-421b-95d2-be659f9668c4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5424a42b-knz9w\" (UID: \"5abc3d06-43b1-421b-95d2-be659f9668c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5424a42b-knz9w" Oct 30 12:53:56.733495 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:56.733468 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5abc3d06-43b1-421b-95d2-be659f9668c4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5424a42b-knz9w\" (UID: \"5abc3d06-43b1-421b-95d2-be659f9668c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5424a42b-knz9w" Oct 30 12:53:56.753935 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:56.753905 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8cln\" (UniqueName: \"kubernetes.io/projected/5abc3d06-43b1-421b-95d2-be659f9668c4-kube-api-access-w8cln\") pod \"maas-api-revoke-sub-5424a42b-knz9w\" (UID: \"5abc3d06-43b1-421b-95d2-be659f9668c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5424a42b-knz9w" Oct 30 12:53:56.828584 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:56.828527 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5424a42b-knz9w" Oct 30 12:53:57.132038 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:57.132016 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5424a42b-knz9w"] Oct 30 12:53:57.134461 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:53:57.134420 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5abc3d06_43b1_421b_95d2_be659f9668c4.slice/crio-ddfe6a0951928974b17e9c8881dc64d0bb80757351df91c05c20713b33eec1db WatchSource:0}: Error finding container ddfe6a0951928974b17e9c8881dc64d0bb80757351df91c05c20713b33eec1db: Status 404 returned error can't find the container with id ddfe6a0951928974b17e9c8881dc64d0bb80757351df91c05c20713b33eec1db Oct 30 12:53:58.019716 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:58.019680 4908 generic.go:356] "Generic (PLEG): container finished" podID="5abc3d06-43b1-421b-95d2-be659f9668c4" containerID="d741cbe455d300b615158f42d49ea382c5c1d260941cc2511902141b5d8166d7" exitCode=0 Oct 30 12:53:58.020147 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:58.019771 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5424a42b-knz9w" event={"ID":"5abc3d06-43b1-421b-95d2-be659f9668c4","Type":"ContainerDied","Data":"d741cbe455d300b615158f42d49ea382c5c1d260941cc2511902141b5d8166d7"} Oct 30 12:53:58.020147 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:58.019823 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5424a42b-knz9w" event={"ID":"5abc3d06-43b1-421b-95d2-be659f9668c4","Type":"ContainerStarted","Data":"ddfe6a0951928974b17e9c8881dc64d0bb80757351df91c05c20713b33eec1db"} Oct 30 12:53:59.202776 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:59.202746 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5424a42b-knz9w" Oct 30 12:53:59.266417 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:59.266384 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5abc3d06-43b1-421b-95d2-be659f9668c4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5abc3d06-43b1-421b-95d2-be659f9668c4-maas-api-xks-ca\") pod \"5abc3d06-43b1-421b-95d2-be659f9668c4\" (UID: \"5abc3d06-43b1-421b-95d2-be659f9668c4\") " Oct 30 12:53:59.266576 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:59.266448 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5abc3d06-43b1-421b-95d2-be659f9668c4-kube-api-access-w8cln\" (UniqueName: \"kubernetes.io/projected/5abc3d06-43b1-421b-95d2-be659f9668c4-kube-api-access-w8cln\") pod \"5abc3d06-43b1-421b-95d2-be659f9668c4\" (UID: \"5abc3d06-43b1-421b-95d2-be659f9668c4\") " Oct 30 12:53:59.266576 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:59.266493 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5abc3d06-43b1-421b-95d2-be659f9668c4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5abc3d06-43b1-421b-95d2-be659f9668c4-maas-api-openshift-ca\") pod \"5abc3d06-43b1-421b-95d2-be659f9668c4\" (UID: \"5abc3d06-43b1-421b-95d2-be659f9668c4\") " Oct 30 12:53:59.267001 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:59.266970 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5abc3d06-43b1-421b-95d2-be659f9668c4-maas-api-openshift-ca" pod "5abc3d06-43b1-421b-95d2-be659f9668c4" (UID: "5abc3d06-43b1-421b-95d2-be659f9668c4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:53:59.267267 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:59.267244 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5abc3d06-43b1-421b-95d2-be659f9668c4-maas-api-openshift-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:53:59.269086 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:59.269043 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5abc3d06-43b1-421b-95d2-be659f9668c4-maas-api-xks-ca" pod "5abc3d06-43b1-421b-95d2-be659f9668c4" (UID: "5abc3d06-43b1-421b-95d2-be659f9668c4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:53:59.272994 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:59.272962 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5abc3d06-43b1-421b-95d2-be659f9668c4-kube-api-access-w8cln" pod "5abc3d06-43b1-421b-95d2-be659f9668c4" (UID: "5abc3d06-43b1-421b-95d2-be659f9668c4"). InnerVolumeSpecName "kube-api-access-w8cln". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:53:59.368709 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:59.368673 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5abc3d06-43b1-421b-95d2-be659f9668c4-maas-api-xks-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:53:59.368891 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:53:59.368712 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8cln\" (UniqueName: \"kubernetes.io/projected/5abc3d06-43b1-421b-95d2-be659f9668c4-kube-api-access-w8cln\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:54:00.037649 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:00.037619 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5424a42b-knz9w" Oct 30 12:54:00.037649 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:00.037640 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5424a42b-knz9w" event={"ID":"5abc3d06-43b1-421b-95d2-be659f9668c4","Type":"ContainerDied","Data":"ddfe6a0951928974b17e9c8881dc64d0bb80757351df91c05c20713b33eec1db"} Oct 30 12:54:00.037898 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:00.037685 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ddfe6a0951928974b17e9c8881dc64d0bb80757351df91c05c20713b33eec1db" Oct 30 12:54:06.378559 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:06.378527 4908 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-plczt_a70ee11c-c20e-4a4f-8476-ac5eb02816d0/console-operator/3.log" Oct 30 12:54:06.384407 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:06.384385 4908 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b8bxz_644e19b3-10ad-4145-9ed3-595d19ed9b07/ovn-acl-logging/0.log" Oct 30 12:54:07.178270 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:07.178241 4908 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-plczt_a70ee11c-c20e-4a4f-8476-ac5eb02816d0/console-operator/3.log" Oct 30 12:54:07.185905 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:07.185877 4908 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b8bxz_644e19b3-10ad-4145-9ed3-595d19ed9b07/ovn-acl-logging/0.log" Oct 30 12:54:08.343858 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:08.342955 4908 scope.go:122] "RemoveContainer" containerID="05024102700c527f00819ad3b088af4475d0efaf19cb52dad023cf303b2b98cf" Oct 30 12:54:08.426848 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:08.422757 4908 scope.go:122] "RemoveContainer" containerID="6d8542d163b47141f5536fe0f273dbf7bdbbd4aaac4bcf18ed1bb5988bf63a0e" Oct 30 12:54:08.439546 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:08.439524 4908 scope.go:122] "RemoveContainer" containerID="67709732d2be3672e0343e8633ba838d007e9dc9d9fef77ad04ebc8070ec370c" Oct 30 12:54:08.806214 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:08.803113 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0294fd6-d9hh2"] Oct 30 12:54:08.806214 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:08.804093 4908 cpu_manager.go:414] "containerMap: removing container" podUID="5abc3d06-43b1-421b-95d2-be659f9668c4" containerName="revoke-keys" Oct 30 12:54:08.806214 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:08.804116 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5abc3d06-43b1-421b-95d2-be659f9668c4" containerName="revoke-keys" Oct 30 12:54:08.806214 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:08.804312 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="5abc3d06-43b1-421b-95d2-be659f9668c4" containerName="revoke-keys" Oct 30 12:54:08.833248 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:08.833216 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0294fd6-d9hh2" Oct 30 12:54:08.883791 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:08.878207 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s75hq\"" Oct 30 12:54:08.883791 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:08.878281 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 12:54:08.883791 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:08.880730 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0294fd6-d9hh2"] Oct 30 12:54:08.893131 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:08.893102 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 12:54:08.917851 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:08.900573 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 12:54:09.018278 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:08.989481 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4cb971e-145e-403f-8118-9d2e0cde2396-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a0294fd6-d9hh2\" (UID: \"f4cb971e-145e-403f-8118-9d2e0cde2396\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0294fd6-d9hh2" Oct 30 12:54:09.018278 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:08.993283 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7r54d\" (UniqueName: \"kubernetes.io/projected/f4cb971e-145e-403f-8118-9d2e0cde2396-kube-api-access-7r54d\") pod \"maas-api-revoke-sub-a0294fd6-d9hh2\" (UID: \"f4cb971e-145e-403f-8118-9d2e0cde2396\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0294fd6-d9hh2" Oct 30 12:54:09.018278 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:08.993373 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4cb971e-145e-403f-8118-9d2e0cde2396-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a0294fd6-d9hh2\" (UID: \"f4cb971e-145e-403f-8118-9d2e0cde2396\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0294fd6-d9hh2" Oct 30 12:54:09.094737 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:09.094644 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7r54d\" (UniqueName: \"kubernetes.io/projected/f4cb971e-145e-403f-8118-9d2e0cde2396-kube-api-access-7r54d\") pod \"maas-api-revoke-sub-a0294fd6-d9hh2\" (UID: \"f4cb971e-145e-403f-8118-9d2e0cde2396\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0294fd6-d9hh2" Oct 30 12:54:09.094970 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:09.094744 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4cb971e-145e-403f-8118-9d2e0cde2396-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a0294fd6-d9hh2\" (UID: \"f4cb971e-145e-403f-8118-9d2e0cde2396\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0294fd6-d9hh2" Oct 30 12:54:09.094970 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:09.094954 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4cb971e-145e-403f-8118-9d2e0cde2396-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a0294fd6-d9hh2\" (UID: \"f4cb971e-145e-403f-8118-9d2e0cde2396\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0294fd6-d9hh2" Oct 30 12:54:09.095616 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:09.095583 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4cb971e-145e-403f-8118-9d2e0cde2396-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a0294fd6-d9hh2\" (UID: \"f4cb971e-145e-403f-8118-9d2e0cde2396\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0294fd6-d9hh2" Oct 30 12:54:09.097416 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:09.097388 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4cb971e-145e-403f-8118-9d2e0cde2396-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a0294fd6-d9hh2\" (UID: \"f4cb971e-145e-403f-8118-9d2e0cde2396\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0294fd6-d9hh2" Oct 30 12:54:09.117345 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:09.117318 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7r54d\" (UniqueName: \"kubernetes.io/projected/f4cb971e-145e-403f-8118-9d2e0cde2396-kube-api-access-7r54d\") pod \"maas-api-revoke-sub-a0294fd6-d9hh2\" (UID: \"f4cb971e-145e-403f-8118-9d2e0cde2396\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0294fd6-d9hh2" Oct 30 12:54:09.349740 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:09.349643 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0294fd6-d9hh2" Oct 30 12:54:09.746685 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:09.746660 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0294fd6-d9hh2"] Oct 30 12:54:09.748434 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:54:09.748401 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4cb971e_145e_403f_8118_9d2e0cde2396.slice/crio-3f25ad8063922c1fa934b5b777cab9acc340c817593d44b05ada5c926ac818d7 WatchSource:0}: Error finding container 3f25ad8063922c1fa934b5b777cab9acc340c817593d44b05ada5c926ac818d7: Status 404 returned error can't find the container with id 3f25ad8063922c1fa934b5b777cab9acc340c817593d44b05ada5c926ac818d7 Oct 30 12:54:10.171204 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:10.171175 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0294fd6-d9hh2" event={"ID":"f4cb971e-145e-403f-8118-9d2e0cde2396","Type":"ContainerStarted","Data":"03586bfef5de1e35c8f5ba0bc84d8d26d39255b0ec477ec6bbf1fca09e36c93e"} Oct 30 12:54:10.171371 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:10.171236 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0294fd6-d9hh2" event={"ID":"f4cb971e-145e-403f-8118-9d2e0cde2396","Type":"ContainerStarted","Data":"3f25ad8063922c1fa934b5b777cab9acc340c817593d44b05ada5c926ac818d7"} Oct 30 12:54:10.212975 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:10.212920 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0294fd6-d9hh2" podStartSLOduration=2.212906331 podStartE2EDuration="2.212906331s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:54:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:54:10.212652823 +0000 UTC m=+3606.727081568" watchObservedRunningTime="2026-10-30 12:54:10.212906331 +0000 UTC m=+3606.727335076" Oct 30 12:54:11.049461 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:11.049378 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fda34221-5srjn"] Oct 30 12:54:11.059819 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:11.059788 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fda34221-5srjn"] Oct 30 12:54:11.181435 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:11.181398 4908 generic.go:356] "Generic (PLEG): container finished" podID="f4cb971e-145e-403f-8118-9d2e0cde2396" containerID="03586bfef5de1e35c8f5ba0bc84d8d26d39255b0ec477ec6bbf1fca09e36c93e" exitCode=0 Oct 30 12:54:11.181633 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:11.181611 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0294fd6-d9hh2" event={"ID":"f4cb971e-145e-403f-8118-9d2e0cde2396","Type":"ContainerDied","Data":"03586bfef5de1e35c8f5ba0bc84d8d26d39255b0ec477ec6bbf1fca09e36c93e"} Oct 30 12:54:11.609820 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:11.609787 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6384ae16-8d99-4c45-847c-fd5f20baf6e8" path="/var/lib/kubelet/pods/6384ae16-8d99-4c45-847c-fd5f20baf6e8/volumes" Oct 30 12:54:12.387146 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:12.387119 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0294fd6-d9hh2" Oct 30 12:54:12.595728 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:12.595684 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4cb971e-145e-403f-8118-9d2e0cde2396-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4cb971e-145e-403f-8118-9d2e0cde2396-maas-api-openshift-ca\") pod \"f4cb971e-145e-403f-8118-9d2e0cde2396\" (UID: \"f4cb971e-145e-403f-8118-9d2e0cde2396\") " Oct 30 12:54:12.595956 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:12.595799 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f4cb971e-145e-403f-8118-9d2e0cde2396-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4cb971e-145e-403f-8118-9d2e0cde2396-maas-api-xks-ca\") pod \"f4cb971e-145e-403f-8118-9d2e0cde2396\" (UID: \"f4cb971e-145e-403f-8118-9d2e0cde2396\") " Oct 30 12:54:12.596042 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:12.595985 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4cb971e-145e-403f-8118-9d2e0cde2396-kube-api-access-7r54d\" (UniqueName: \"kubernetes.io/projected/f4cb971e-145e-403f-8118-9d2e0cde2396-kube-api-access-7r54d\") pod \"f4cb971e-145e-403f-8118-9d2e0cde2396\" (UID: \"f4cb971e-145e-403f-8118-9d2e0cde2396\") " Oct 30 12:54:12.597253 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:12.597157 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4cb971e-145e-403f-8118-9d2e0cde2396-maas-api-openshift-ca" pod "f4cb971e-145e-403f-8118-9d2e0cde2396" (UID: "f4cb971e-145e-403f-8118-9d2e0cde2396"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:54:12.598585 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:12.598507 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4cb971e-145e-403f-8118-9d2e0cde2396-maas-api-xks-ca" pod "f4cb971e-145e-403f-8118-9d2e0cde2396" (UID: "f4cb971e-145e-403f-8118-9d2e0cde2396"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:54:12.603794 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:12.603765 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4cb971e-145e-403f-8118-9d2e0cde2396-kube-api-access-7r54d" pod "f4cb971e-145e-403f-8118-9d2e0cde2396" (UID: "f4cb971e-145e-403f-8118-9d2e0cde2396"). InnerVolumeSpecName "kube-api-access-7r54d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:54:12.698129 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:12.698095 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7r54d\" (UniqueName: \"kubernetes.io/projected/f4cb971e-145e-403f-8118-9d2e0cde2396-kube-api-access-7r54d\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:54:12.698129 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:12.698129 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4cb971e-145e-403f-8118-9d2e0cde2396-maas-api-openshift-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:54:12.698310 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:12.698144 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4cb971e-145e-403f-8118-9d2e0cde2396-maas-api-xks-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:54:13.201660 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:13.201619 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0294fd6-d9hh2" Oct 30 12:54:13.201869 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:13.201655 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0294fd6-d9hh2" event={"ID":"f4cb971e-145e-403f-8118-9d2e0cde2396","Type":"ContainerDied","Data":"3f25ad8063922c1fa934b5b777cab9acc340c817593d44b05ada5c926ac818d7"} Oct 30 12:54:13.201869 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:13.201712 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3f25ad8063922c1fa934b5b777cab9acc340c817593d44b05ada5c926ac818d7" Oct 30 12:54:15.529905 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:15.529875 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9wwfc"] Oct 30 12:54:18.167203 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:18.167158 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51f1d960-jtrp5"] Oct 30 12:54:18.168866 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:18.168814 4908 cpu_manager.go:414] "containerMap: removing container" podUID="f4cb971e-145e-403f-8118-9d2e0cde2396" containerName="revoke-keys" Oct 30 12:54:18.169038 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:18.169022 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4cb971e-145e-403f-8118-9d2e0cde2396" containerName="revoke-keys" Oct 30 12:54:18.169315 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:18.169285 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4cb971e-145e-403f-8118-9d2e0cde2396" containerName="revoke-keys" Oct 30 12:54:18.173797 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:18.173769 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51f1d960-jtrp5" Oct 30 12:54:18.202437 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:18.202409 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51f1d960-jtrp5"] Oct 30 12:54:18.204125 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:18.204102 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s75hq\"" Oct 30 12:54:18.204345 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:18.204328 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 12:54:18.205774 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:18.204558 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 12:54:18.205980 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:18.205962 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 12:54:18.269819 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:18.259254 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abf51df1-f815-498c-964a-18b7db09014b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-51f1d960-jtrp5\" (UID: \"abf51df1-f815-498c-964a-18b7db09014b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51f1d960-jtrp5" Oct 30 12:54:18.269819 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:18.259327 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abf51df1-f815-498c-964a-18b7db09014b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-51f1d960-jtrp5\" (UID: \"abf51df1-f815-498c-964a-18b7db09014b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51f1d960-jtrp5" Oct 30 12:54:18.269819 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:18.259490 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d244l\" (UniqueName: \"kubernetes.io/projected/abf51df1-f815-498c-964a-18b7db09014b-kube-api-access-d244l\") pod \"maas-api-revoke-sub-51f1d960-jtrp5\" (UID: \"abf51df1-f815-498c-964a-18b7db09014b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51f1d960-jtrp5" Oct 30 12:54:18.365550 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:18.365510 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abf51df1-f815-498c-964a-18b7db09014b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-51f1d960-jtrp5\" (UID: \"abf51df1-f815-498c-964a-18b7db09014b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51f1d960-jtrp5" Oct 30 12:54:18.366722 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:18.366696 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abf51df1-f815-498c-964a-18b7db09014b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-51f1d960-jtrp5\" (UID: \"abf51df1-f815-498c-964a-18b7db09014b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51f1d960-jtrp5" Oct 30 12:54:18.367550 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:18.367523 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d244l\" (UniqueName: \"kubernetes.io/projected/abf51df1-f815-498c-964a-18b7db09014b-kube-api-access-d244l\") pod \"maas-api-revoke-sub-51f1d960-jtrp5\" (UID: \"abf51df1-f815-498c-964a-18b7db09014b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51f1d960-jtrp5" Oct 30 12:54:18.368204 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:18.366611 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abf51df1-f815-498c-964a-18b7db09014b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-51f1d960-jtrp5\" (UID: \"abf51df1-f815-498c-964a-18b7db09014b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51f1d960-jtrp5" Oct 30 12:54:18.401848 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:18.385561 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abf51df1-f815-498c-964a-18b7db09014b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-51f1d960-jtrp5\" (UID: \"abf51df1-f815-498c-964a-18b7db09014b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51f1d960-jtrp5" Oct 30 12:54:18.422794 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:18.422719 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d244l\" (UniqueName: \"kubernetes.io/projected/abf51df1-f815-498c-964a-18b7db09014b-kube-api-access-d244l\") pod \"maas-api-revoke-sub-51f1d960-jtrp5\" (UID: \"abf51df1-f815-498c-964a-18b7db09014b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51f1d960-jtrp5" Oct 30 12:54:18.507847 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:18.507075 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51f1d960-jtrp5" Oct 30 12:54:19.620852 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:19.620074 4908 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f920f7/simulator-w0-f920f7-kserve-599b7ff45f-g79d7" podUID="a85e562f-d655-4c22-b4f4-a299a4e494a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Oct 30 12:54:19.660852 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:19.657056 4908 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6a7148/premium-w3-6a7148-kserve-67b976c888-tgd6c" podUID="13de4441-e7f6-4cbd-97d9-0f079d11cb8e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 12:54:19.780566 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:19.778850 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51f1d960-jtrp5"] Oct 30 12:54:20.115543 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:54:19.801963 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabf51df1_f815_498c_964a_18b7db09014b.slice/crio-ad5f60f6e92af946eff5702516344a06c286724354f33221005bd6b649372ae8 WatchSource:0}: Error finding container ad5f60f6e92af946eff5702516344a06c286724354f33221005bd6b649372ae8: Status 404 returned error can't find the container with id ad5f60f6e92af946eff5702516344a06c286724354f33221005bd6b649372ae8 Oct 30 12:54:20.115543 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:19.810904 4908 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47fd99/premium-w2-47fd99-kserve-5d85f9bb95-hqkmf" podUID="fd2cbeda-dbc4-41f6-a730-e1c6f9343c5e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:54:20.115543 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:20.037373 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5da5dee2-x8p4m"] Oct 30 12:54:20.217560 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:20.217534 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5da5dee2-x8p4m" Oct 30 12:54:20.261243 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:20.261205 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5da5dee2-x8p4m"] Oct 30 12:54:20.265577 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:20.262309 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e518fd27-b28e-47da-9365-52cf1c7c5e05-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5da5dee2-x8p4m\" (UID: \"e518fd27-b28e-47da-9365-52cf1c7c5e05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5da5dee2-x8p4m" Oct 30 12:54:20.265577 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:20.262421 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzf57\" (UniqueName: \"kubernetes.io/projected/e518fd27-b28e-47da-9365-52cf1c7c5e05-kube-api-access-tzf57\") pod \"maas-api-revoke-sub-5da5dee2-x8p4m\" (UID: \"e518fd27-b28e-47da-9365-52cf1c7c5e05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5da5dee2-x8p4m" Oct 30 12:54:20.289737 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:20.262455 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e518fd27-b28e-47da-9365-52cf1c7c5e05-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5da5dee2-x8p4m\" (UID: \"e518fd27-b28e-47da-9365-52cf1c7c5e05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5da5dee2-x8p4m" Oct 30 12:54:20.315388 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:54:20.315319 4908 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vn6vv" containerName="extractor" Oct 30 12:54:20.399910 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:20.398874 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e518fd27-b28e-47da-9365-52cf1c7c5e05-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5da5dee2-x8p4m\" (UID: \"e518fd27-b28e-47da-9365-52cf1c7c5e05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5da5dee2-x8p4m" Oct 30 12:54:20.399910 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:20.399020 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzf57\" (UniqueName: \"kubernetes.io/projected/e518fd27-b28e-47da-9365-52cf1c7c5e05-kube-api-access-tzf57\") pod \"maas-api-revoke-sub-5da5dee2-x8p4m\" (UID: \"e518fd27-b28e-47da-9365-52cf1c7c5e05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5da5dee2-x8p4m" Oct 30 12:54:20.399910 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:20.399063 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e518fd27-b28e-47da-9365-52cf1c7c5e05-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5da5dee2-x8p4m\" (UID: \"e518fd27-b28e-47da-9365-52cf1c7c5e05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5da5dee2-x8p4m" Oct 30 12:54:20.505981 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:20.457248 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e518fd27-b28e-47da-9365-52cf1c7c5e05-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5da5dee2-x8p4m\" (UID: \"e518fd27-b28e-47da-9365-52cf1c7c5e05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5da5dee2-x8p4m" Oct 30 12:54:20.505981 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:20.473415 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e518fd27-b28e-47da-9365-52cf1c7c5e05-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5da5dee2-x8p4m\" (UID: \"e518fd27-b28e-47da-9365-52cf1c7c5e05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5da5dee2-x8p4m" Oct 30 12:54:20.507527 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:20.507496 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzf57\" (UniqueName: \"kubernetes.io/projected/e518fd27-b28e-47da-9365-52cf1c7c5e05-kube-api-access-tzf57\") pod \"maas-api-revoke-sub-5da5dee2-x8p4m\" (UID: \"e518fd27-b28e-47da-9365-52cf1c7c5e05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5da5dee2-x8p4m" Oct 30 12:54:20.600678 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:20.600646 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5da5dee2-x8p4m" Oct 30 12:54:21.131606 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:21.131566 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51f1d960-jtrp5" event={"ID":"abf51df1-f815-498c-964a-18b7db09014b","Type":"ContainerStarted","Data":"15ca7ceefb401dfd5c367a43bebcd9bb81e686557cef9c5fe612fe52003ae561"} Oct 30 12:54:21.132038 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:21.131625 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51f1d960-jtrp5" event={"ID":"abf51df1-f815-498c-964a-18b7db09014b","Type":"ContainerStarted","Data":"ad5f60f6e92af946eff5702516344a06c286724354f33221005bd6b649372ae8"} Oct 30 12:54:21.162307 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:21.162276 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5da5dee2-x8p4m"] Oct 30 12:54:21.163521 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:54:21.163470 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode518fd27_b28e_47da_9365_52cf1c7c5e05.slice/crio-d962a0f5ccb8bae5a4aed2113a99150a96322233befbcefbec547eb8e2681096 WatchSource:0}: Error finding container d962a0f5ccb8bae5a4aed2113a99150a96322233befbcefbec547eb8e2681096: Status 404 returned error can't find the container with id d962a0f5ccb8bae5a4aed2113a99150a96322233befbcefbec547eb8e2681096 Oct 30 12:54:21.237255 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:21.237212 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51f1d960-jtrp5" podStartSLOduration=3.237198412 podStartE2EDuration="3.237198412s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:54:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:54:21.235071997 +0000 UTC m=+3617.749500744" watchObservedRunningTime="2026-10-30 12:54:21.237198412 +0000 UTC m=+3617.751627158" Oct 30 12:54:22.140679 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:22.140638 4908 generic.go:356] "Generic (PLEG): container finished" podID="e518fd27-b28e-47da-9365-52cf1c7c5e05" containerID="e2b110faabc6ddcec4e788fc67643af0c14e892227da86bb30e0d9905b31f103" exitCode=0 Oct 30 12:54:22.141131 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:22.140712 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5da5dee2-x8p4m" event={"ID":"e518fd27-b28e-47da-9365-52cf1c7c5e05","Type":"ContainerDied","Data":"e2b110faabc6ddcec4e788fc67643af0c14e892227da86bb30e0d9905b31f103"} Oct 30 12:54:22.141131 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:22.140768 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5da5dee2-x8p4m" event={"ID":"e518fd27-b28e-47da-9365-52cf1c7c5e05","Type":"ContainerStarted","Data":"d962a0f5ccb8bae5a4aed2113a99150a96322233befbcefbec547eb8e2681096"} Oct 30 12:54:22.142887 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:22.142864 4908 generic.go:356] "Generic (PLEG): container finished" podID="abf51df1-f815-498c-964a-18b7db09014b" containerID="15ca7ceefb401dfd5c367a43bebcd9bb81e686557cef9c5fe612fe52003ae561" exitCode=0 Oct 30 12:54:22.143004 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:22.142896 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51f1d960-jtrp5" event={"ID":"abf51df1-f815-498c-964a-18b7db09014b","Type":"ContainerDied","Data":"15ca7ceefb401dfd5c367a43bebcd9bb81e686557cef9c5fe612fe52003ae561"} Oct 30 12:54:23.321240 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:23.321208 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51f1d960-jtrp5" Oct 30 12:54:23.366974 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:23.366931 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abf51df1-f815-498c-964a-18b7db09014b-kube-api-access-d244l\" (UniqueName: \"kubernetes.io/projected/abf51df1-f815-498c-964a-18b7db09014b-kube-api-access-d244l\") pod \"abf51df1-f815-498c-964a-18b7db09014b\" (UID: \"abf51df1-f815-498c-964a-18b7db09014b\") " Oct 30 12:54:23.367181 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:23.367090 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/abf51df1-f815-498c-964a-18b7db09014b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abf51df1-f815-498c-964a-18b7db09014b-maas-api-xks-ca\") pod \"abf51df1-f815-498c-964a-18b7db09014b\" (UID: \"abf51df1-f815-498c-964a-18b7db09014b\") " Oct 30 12:54:23.367257 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:23.367235 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/abf51df1-f815-498c-964a-18b7db09014b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abf51df1-f815-498c-964a-18b7db09014b-maas-api-openshift-ca\") pod \"abf51df1-f815-498c-964a-18b7db09014b\" (UID: \"abf51df1-f815-498c-964a-18b7db09014b\") " Oct 30 12:54:23.377056 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:23.372256 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5da5dee2-x8p4m" Oct 30 12:54:23.383270 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:23.383179 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abf51df1-f815-498c-964a-18b7db09014b-maas-api-openshift-ca" pod "abf51df1-f815-498c-964a-18b7db09014b" (UID: "abf51df1-f815-498c-964a-18b7db09014b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:54:23.389154 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:23.389116 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abf51df1-f815-498c-964a-18b7db09014b-maas-api-openshift-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:54:23.409852 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:23.409738 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abf51df1-f815-498c-964a-18b7db09014b-maas-api-xks-ca" pod "abf51df1-f815-498c-964a-18b7db09014b" (UID: "abf51df1-f815-498c-964a-18b7db09014b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:54:23.410441 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:23.410401 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abf51df1-f815-498c-964a-18b7db09014b-kube-api-access-d244l" pod "abf51df1-f815-498c-964a-18b7db09014b" (UID: "abf51df1-f815-498c-964a-18b7db09014b"). InnerVolumeSpecName "kube-api-access-d244l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:54:23.490543 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:23.490511 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e518fd27-b28e-47da-9365-52cf1c7c5e05-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e518fd27-b28e-47da-9365-52cf1c7c5e05-maas-api-openshift-ca\") pod \"e518fd27-b28e-47da-9365-52cf1c7c5e05\" (UID: \"e518fd27-b28e-47da-9365-52cf1c7c5e05\") " Oct 30 12:54:23.490741 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:23.490558 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e518fd27-b28e-47da-9365-52cf1c7c5e05-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e518fd27-b28e-47da-9365-52cf1c7c5e05-maas-api-xks-ca\") pod \"e518fd27-b28e-47da-9365-52cf1c7c5e05\" (UID: \"e518fd27-b28e-47da-9365-52cf1c7c5e05\") " Oct 30 12:54:23.490741 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:23.490616 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e518fd27-b28e-47da-9365-52cf1c7c5e05-kube-api-access-tzf57\" (UniqueName: \"kubernetes.io/projected/e518fd27-b28e-47da-9365-52cf1c7c5e05-kube-api-access-tzf57\") pod \"e518fd27-b28e-47da-9365-52cf1c7c5e05\" (UID: \"e518fd27-b28e-47da-9365-52cf1c7c5e05\") " Oct 30 12:54:23.491036 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:23.491009 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e518fd27-b28e-47da-9365-52cf1c7c5e05-maas-api-openshift-ca" pod "e518fd27-b28e-47da-9365-52cf1c7c5e05" (UID: "e518fd27-b28e-47da-9365-52cf1c7c5e05"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:54:23.491116 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:23.491033 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d244l\" (UniqueName: \"kubernetes.io/projected/abf51df1-f815-498c-964a-18b7db09014b-kube-api-access-d244l\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:54:23.491116 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:23.491051 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abf51df1-f815-498c-964a-18b7db09014b-maas-api-xks-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:54:23.492817 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:23.492793 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e518fd27-b28e-47da-9365-52cf1c7c5e05-maas-api-xks-ca" pod "e518fd27-b28e-47da-9365-52cf1c7c5e05" (UID: "e518fd27-b28e-47da-9365-52cf1c7c5e05"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:54:23.492817 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:23.492809 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e518fd27-b28e-47da-9365-52cf1c7c5e05-kube-api-access-tzf57" pod "e518fd27-b28e-47da-9365-52cf1c7c5e05" (UID: "e518fd27-b28e-47da-9365-52cf1c7c5e05"). InnerVolumeSpecName "kube-api-access-tzf57". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:54:23.591913 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:23.591887 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e518fd27-b28e-47da-9365-52cf1c7c5e05-maas-api-openshift-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:54:23.591913 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:23.591912 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e518fd27-b28e-47da-9365-52cf1c7c5e05-maas-api-xks-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:54:23.592065 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:23.591922 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzf57\" (UniqueName: \"kubernetes.io/projected/e518fd27-b28e-47da-9365-52cf1c7c5e05-kube-api-access-tzf57\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:54:24.158503 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:24.158473 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5da5dee2-x8p4m" Oct 30 12:54:24.158706 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:24.158516 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5da5dee2-x8p4m" event={"ID":"e518fd27-b28e-47da-9365-52cf1c7c5e05","Type":"ContainerDied","Data":"d962a0f5ccb8bae5a4aed2113a99150a96322233befbcefbec547eb8e2681096"} Oct 30 12:54:24.158706 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:24.158614 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d962a0f5ccb8bae5a4aed2113a99150a96322233befbcefbec547eb8e2681096" Oct 30 12:54:24.161323 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:24.161295 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51f1d960-jtrp5" Oct 30 12:54:24.161323 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:24.161324 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51f1d960-jtrp5" event={"ID":"abf51df1-f815-498c-964a-18b7db09014b","Type":"ContainerDied","Data":"ad5f60f6e92af946eff5702516344a06c286724354f33221005bd6b649372ae8"} Oct 30 12:54:24.161525 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:24.161362 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ad5f60f6e92af946eff5702516344a06c286724354f33221005bd6b649372ae8" Oct 30 12:54:29.181513 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:29.181476 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-908f43fe-75ft7"] Oct 30 12:54:29.182114 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:29.182097 4908 cpu_manager.go:414] "containerMap: removing container" podUID="e518fd27-b28e-47da-9365-52cf1c7c5e05" containerName="revoke-keys" Oct 30 12:54:29.182114 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:29.182114 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e518fd27-b28e-47da-9365-52cf1c7c5e05" containerName="revoke-keys" Oct 30 12:54:29.182261 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:29.182135 4908 cpu_manager.go:414] "containerMap: removing container" podUID="abf51df1-f815-498c-964a-18b7db09014b" containerName="revoke-keys" Oct 30 12:54:29.182261 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:29.182140 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abf51df1-f815-498c-964a-18b7db09014b" containerName="revoke-keys" Oct 30 12:54:29.182261 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:29.182231 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="e518fd27-b28e-47da-9365-52cf1c7c5e05" containerName="revoke-keys" Oct 30 12:54:29.182261 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:29.182243 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="abf51df1-f815-498c-964a-18b7db09014b" containerName="revoke-keys" Oct 30 12:54:29.186975 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:29.186952 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-908f43fe-75ft7" Oct 30 12:54:29.205517 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:29.205490 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 12:54:29.216578 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:29.216555 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s75hq\"" Oct 30 12:54:29.216726 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:29.216658 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 12:54:29.216794 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:29.216764 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 12:54:29.223250 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:29.223222 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-908f43fe-75ft7"] Oct 30 12:54:29.271213 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:29.271177 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4664497f-4066-4a36-874f-ad6bd275944e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-908f43fe-75ft7\" (UID: \"4664497f-4066-4a36-874f-ad6bd275944e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-908f43fe-75ft7" Oct 30 12:54:29.271358 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:29.271266 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9452b\" (UniqueName: \"kubernetes.io/projected/4664497f-4066-4a36-874f-ad6bd275944e-kube-api-access-9452b\") pod \"maas-api-revoke-sub-908f43fe-75ft7\" (UID: \"4664497f-4066-4a36-874f-ad6bd275944e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-908f43fe-75ft7" Oct 30 12:54:29.271358 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:29.271331 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4664497f-4066-4a36-874f-ad6bd275944e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-908f43fe-75ft7\" (UID: \"4664497f-4066-4a36-874f-ad6bd275944e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-908f43fe-75ft7" Oct 30 12:54:29.372079 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:29.372049 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4664497f-4066-4a36-874f-ad6bd275944e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-908f43fe-75ft7\" (UID: \"4664497f-4066-4a36-874f-ad6bd275944e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-908f43fe-75ft7" Oct 30 12:54:29.372229 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:29.372102 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9452b\" (UniqueName: \"kubernetes.io/projected/4664497f-4066-4a36-874f-ad6bd275944e-kube-api-access-9452b\") pod \"maas-api-revoke-sub-908f43fe-75ft7\" (UID: \"4664497f-4066-4a36-874f-ad6bd275944e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-908f43fe-75ft7" Oct 30 12:54:29.372229 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:29.372142 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4664497f-4066-4a36-874f-ad6bd275944e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-908f43fe-75ft7\" (UID: \"4664497f-4066-4a36-874f-ad6bd275944e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-908f43fe-75ft7" Oct 30 12:54:29.372761 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:29.372740 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4664497f-4066-4a36-874f-ad6bd275944e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-908f43fe-75ft7\" (UID: \"4664497f-4066-4a36-874f-ad6bd275944e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-908f43fe-75ft7" Oct 30 12:54:29.374573 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:29.374550 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4664497f-4066-4a36-874f-ad6bd275944e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-908f43fe-75ft7\" (UID: \"4664497f-4066-4a36-874f-ad6bd275944e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-908f43fe-75ft7" Oct 30 12:54:29.395172 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:29.395142 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9452b\" (UniqueName: \"kubernetes.io/projected/4664497f-4066-4a36-874f-ad6bd275944e-kube-api-access-9452b\") pod \"maas-api-revoke-sub-908f43fe-75ft7\" (UID: \"4664497f-4066-4a36-874f-ad6bd275944e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-908f43fe-75ft7" Oct 30 12:54:29.501392 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:29.501356 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-908f43fe-75ft7" Oct 30 12:54:29.812955 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:29.812915 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-908f43fe-75ft7"] Oct 30 12:54:29.813691 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:54:29.813663 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4664497f_4066_4a36_874f_ad6bd275944e.slice/crio-bdf9190d30b743f65ff1dff74e0c03692cc41e3ca034108a7e5178d8a840d2b0 WatchSource:0}: Error finding container bdf9190d30b743f65ff1dff74e0c03692cc41e3ca034108a7e5178d8a840d2b0: Status 404 returned error can't find the container with id bdf9190d30b743f65ff1dff74e0c03692cc41e3ca034108a7e5178d8a840d2b0 Oct 30 12:54:30.281253 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:30.281217 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-908f43fe-75ft7" event={"ID":"4664497f-4066-4a36-874f-ad6bd275944e","Type":"ContainerStarted","Data":"aef7cea00feba7d1bf708a6ca9a5ba0b5a15efebe16ef79d4272e56ddb2cd87a"} Oct 30 12:54:30.282895 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:30.282863 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-908f43fe-75ft7" event={"ID":"4664497f-4066-4a36-874f-ad6bd275944e","Type":"ContainerStarted","Data":"bdf9190d30b743f65ff1dff74e0c03692cc41e3ca034108a7e5178d8a840d2b0"} Oct 30 12:54:30.325393 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:30.325300 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-908f43fe-75ft7" podStartSLOduration=1.325285259 podStartE2EDuration="1.325285259s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:54:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:54:30.324185355 +0000 UTC m=+3626.838614101" watchObservedRunningTime="2026-10-30 12:54:30.325285259 +0000 UTC m=+3626.839714007" Oct 30 12:54:31.290734 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:31.290695 4908 generic.go:356] "Generic (PLEG): container finished" podID="4664497f-4066-4a36-874f-ad6bd275944e" containerID="aef7cea00feba7d1bf708a6ca9a5ba0b5a15efebe16ef79d4272e56ddb2cd87a" exitCode=0 Oct 30 12:54:31.291159 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:31.290757 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-908f43fe-75ft7" event={"ID":"4664497f-4066-4a36-874f-ad6bd275944e","Type":"ContainerDied","Data":"aef7cea00feba7d1bf708a6ca9a5ba0b5a15efebe16ef79d4272e56ddb2cd87a"} Oct 30 12:54:32.468560 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:32.468530 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-908f43fe-75ft7" Oct 30 12:54:32.537619 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:32.537583 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4664497f-4066-4a36-874f-ad6bd275944e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4664497f-4066-4a36-874f-ad6bd275944e-maas-api-xks-ca\") pod \"4664497f-4066-4a36-874f-ad6bd275944e\" (UID: \"4664497f-4066-4a36-874f-ad6bd275944e\") " Oct 30 12:54:32.537807 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:32.537651 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4664497f-4066-4a36-874f-ad6bd275944e-kube-api-access-9452b\" (UniqueName: \"kubernetes.io/projected/4664497f-4066-4a36-874f-ad6bd275944e-kube-api-access-9452b\") pod \"4664497f-4066-4a36-874f-ad6bd275944e\" (UID: \"4664497f-4066-4a36-874f-ad6bd275944e\") " Oct 30 12:54:32.537807 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:32.537699 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4664497f-4066-4a36-874f-ad6bd275944e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4664497f-4066-4a36-874f-ad6bd275944e-maas-api-openshift-ca\") pod \"4664497f-4066-4a36-874f-ad6bd275944e\" (UID: \"4664497f-4066-4a36-874f-ad6bd275944e\") " Oct 30 12:54:32.538182 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:32.538152 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4664497f-4066-4a36-874f-ad6bd275944e-maas-api-openshift-ca" pod "4664497f-4066-4a36-874f-ad6bd275944e" (UID: "4664497f-4066-4a36-874f-ad6bd275944e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:54:32.538442 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:32.538420 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4664497f-4066-4a36-874f-ad6bd275944e-maas-api-openshift-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:54:32.540061 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:32.540033 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4664497f-4066-4a36-874f-ad6bd275944e-kube-api-access-9452b" pod "4664497f-4066-4a36-874f-ad6bd275944e" (UID: "4664497f-4066-4a36-874f-ad6bd275944e"). InnerVolumeSpecName "kube-api-access-9452b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:54:32.540164 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:32.540063 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4664497f-4066-4a36-874f-ad6bd275944e-maas-api-xks-ca" pod "4664497f-4066-4a36-874f-ad6bd275944e" (UID: "4664497f-4066-4a36-874f-ad6bd275944e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:54:32.640176 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:32.640103 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4664497f-4066-4a36-874f-ad6bd275944e-maas-api-xks-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:54:32.640176 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:32.640132 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9452b\" (UniqueName: \"kubernetes.io/projected/4664497f-4066-4a36-874f-ad6bd275944e-kube-api-access-9452b\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:54:33.312551 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:33.312521 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-908f43fe-75ft7" Oct 30 12:54:33.312730 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:33.312554 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-908f43fe-75ft7" event={"ID":"4664497f-4066-4a36-874f-ad6bd275944e","Type":"ContainerDied","Data":"bdf9190d30b743f65ff1dff74e0c03692cc41e3ca034108a7e5178d8a840d2b0"} Oct 30 12:54:33.312730 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:33.312625 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bdf9190d30b743f65ff1dff74e0c03692cc41e3ca034108a7e5178d8a840d2b0" Oct 30 12:54:41.681425 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:41.681394 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-63f4e4-openshift-default-786cd844c9-lxsbx"] Oct 30 12:54:41.684720 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:41.683561 4908 cpu_manager.go:414] "containerMap: removing container" podUID="4664497f-4066-4a36-874f-ad6bd275944e" containerName="revoke-keys" Oct 30 12:54:41.684971 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:41.684929 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4664497f-4066-4a36-874f-ad6bd275944e" containerName="revoke-keys" Oct 30 12:54:41.685344 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:41.685310 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="4664497f-4066-4a36-874f-ad6bd275944e" containerName="revoke-keys" Oct 30 12:54:41.739514 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:41.730089 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-63f4e4-openshift-default-786cd844c9-lxsbx"] Oct 30 12:54:41.747429 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:41.730650 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-63f4e4-openshift-default-786cd844c9-lxsbx" Oct 30 12:54:41.747672 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:41.747485 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-extmod-shr-user-9f38e1/e2e-usage-sim-0f45af-55f5696f49-j9l6t"] Oct 30 12:54:41.748006 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:41.747975 4908 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-extmod-shr-user-9f38e1/e2e-usage-sim-0f45af-55f5696f49-j9l6t" podUID="48c3e203-4fe7-4c38-a65e-e24f5605c69d" containerName="sim" containerID="cri-o://c02c476ff7d0133d1df1d637a2d3313e47abe2c103808ee7823121610e484d97" gracePeriod=30 Oct 30 12:54:41.763206 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:41.763181 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-63f4e4-openshift-default-dockercfg-k8jx4\"" Oct 30 12:54:41.888409 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:41.888369 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1ca551b-2beb-407b-9e9c-e410199f6f33-istiod-ca-cert\") pod \"e2e-inf-a-63f4e4-openshift-default-786cd844c9-lxsbx\" (UID: \"f1ca551b-2beb-407b-9e9c-e410199f6f33\") " pod="openshift-ingress/e2e-inf-a-63f4e4-openshift-default-786cd844c9-lxsbx" Oct 30 12:54:41.988907 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:41.975415 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nb9tb\" (UniqueName: \"kubernetes.io/projected/f1ca551b-2beb-407b-9e9c-e410199f6f33-kube-api-access-nb9tb\") pod \"e2e-inf-a-63f4e4-openshift-default-786cd844c9-lxsbx\" (UID: \"f1ca551b-2beb-407b-9e9c-e410199f6f33\") " pod="openshift-ingress/e2e-inf-a-63f4e4-openshift-default-786cd844c9-lxsbx" Oct 30 12:54:41.992847 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:41.989195 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1ca551b-2beb-407b-9e9c-e410199f6f33-credential-socket\") pod \"e2e-inf-a-63f4e4-openshift-default-786cd844c9-lxsbx\" (UID: \"f1ca551b-2beb-407b-9e9c-e410199f6f33\") " pod="openshift-ingress/e2e-inf-a-63f4e4-openshift-default-786cd844c9-lxsbx" Oct 30 12:54:42.006138 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:42.005981 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1ca551b-2beb-407b-9e9c-e410199f6f33-workload-socket\") pod \"e2e-inf-a-63f4e4-openshift-default-786cd844c9-lxsbx\" (UID: \"f1ca551b-2beb-407b-9e9c-e410199f6f33\") " pod="openshift-ingress/e2e-inf-a-63f4e4-openshift-default-786cd844c9-lxsbx" Oct 30 12:54:42.006389 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:42.006373 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1ca551b-2beb-407b-9e9c-e410199f6f33-istio-token\") pod \"e2e-inf-a-63f4e4-openshift-default-786cd844c9-lxsbx\" (UID: \"f1ca551b-2beb-407b-9e9c-e410199f6f33\") " pod="openshift-ingress/e2e-inf-a-63f4e4-openshift-default-786cd844c9-lxsbx" Oct 30 12:54:42.006710 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:42.006693 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1ca551b-2beb-407b-9e9c-e410199f6f33-istio-data\") pod \"e2e-inf-a-63f4e4-openshift-default-786cd844c9-lxsbx\" (UID: \"f1ca551b-2beb-407b-9e9c-e410199f6f33\") " pod="openshift-ingress/e2e-inf-a-63f4e4-openshift-default-786cd844c9-lxsbx" Oct 30 12:54:42.007110 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:42.007091 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1ca551b-2beb-407b-9e9c-e410199f6f33-workload-certs\") pod \"e2e-inf-a-63f4e4-openshift-default-786cd844c9-lxsbx\" (UID: \"f1ca551b-2beb-407b-9e9c-e410199f6f33\") " pod="openshift-ingress/e2e-inf-a-63f4e4-openshift-default-786cd844c9-lxsbx" Oct 30 12:54:42.007509 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:42.007302 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1ca551b-2beb-407b-9e9c-e410199f6f33-istio-podinfo\") pod \"e2e-inf-a-63f4e4-openshift-default-786cd844c9-lxsbx\" (UID: \"f1ca551b-2beb-407b-9e9c-e410199f6f33\") " pod="openshift-ingress/e2e-inf-a-63f4e4-openshift-default-786cd844c9-lxsbx" Oct 30 12:54:42.007646 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:42.007632 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1ca551b-2beb-407b-9e9c-e410199f6f33-istio-envoy\") pod \"e2e-inf-a-63f4e4-openshift-default-786cd844c9-lxsbx\" (UID: \"f1ca551b-2beb-407b-9e9c-e410199f6f33\") " pod="openshift-ingress/e2e-inf-a-63f4e4-openshift-default-786cd844c9-lxsbx" Oct 30 12:54:42.316255 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:42.316203 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1ca551b-2beb-407b-9e9c-e410199f6f33-istiod-ca-cert\") pod \"e2e-inf-a-63f4e4-openshift-default-786cd844c9-lxsbx\" (UID: \"f1ca551b-2beb-407b-9e9c-e410199f6f33\") " pod="openshift-ingress/e2e-inf-a-63f4e4-openshift-default-786cd844c9-lxsbx" Oct 30 12:54:42.316572 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:42.316551 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nb9tb\" (UniqueName: \"kubernetes.io/projected/f1ca551b-2beb-407b-9e9c-e410199f6f33-kube-api-access-nb9tb\") pod \"e2e-inf-a-63f4e4-openshift-default-786cd844c9-lxsbx\" (UID: \"f1ca551b-2beb-407b-9e9c-e410199f6f33\") " pod="openshift-ingress/e2e-inf-a-63f4e4-openshift-default-786cd844c9-lxsbx" Oct 30 12:54:42.316750 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:42.316722 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1ca551b-2beb-407b-9e9c-e410199f6f33-credential-socket\") pod \"e2e-inf-a-63f4e4-openshift-default-786cd844c9-lxsbx\" (UID: \"f1ca551b-2beb-407b-9e9c-e410199f6f33\") " pod="openshift-ingress/e2e-inf-a-63f4e4-openshift-default-786cd844c9-lxsbx" Oct 30 12:54:42.316917 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:42.316902 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1ca551b-2beb-407b-9e9c-e410199f6f33-workload-socket\") pod \"e2e-inf-a-63f4e4-openshift-default-786cd844c9-lxsbx\" (UID: \"f1ca551b-2beb-407b-9e9c-e410199f6f33\") " pod="openshift-ingress/e2e-inf-a-63f4e4-openshift-default-786cd844c9-lxsbx" Oct 30 12:54:42.317051 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:42.317036 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1ca551b-2beb-407b-9e9c-e410199f6f33-istio-token\") pod \"e2e-inf-a-63f4e4-openshift-default-786cd844c9-lxsbx\" (UID: \"f1ca551b-2beb-407b-9e9c-e410199f6f33\") " pod="openshift-ingress/e2e-inf-a-63f4e4-openshift-default-786cd844c9-lxsbx" Oct 30 12:54:42.317201 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:42.317180 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1ca551b-2beb-407b-9e9c-e410199f6f33-istio-data\") pod \"e2e-inf-a-63f4e4-openshift-default-786cd844c9-lxsbx\" (UID: \"f1ca551b-2beb-407b-9e9c-e410199f6f33\") " pod="openshift-ingress/e2e-inf-a-63f4e4-openshift-default-786cd844c9-lxsbx" Oct 30 12:54:42.317436 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:42.317407 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1ca551b-2beb-407b-9e9c-e410199f6f33-workload-certs\") pod \"e2e-inf-a-63f4e4-openshift-default-786cd844c9-lxsbx\" (UID: \"f1ca551b-2beb-407b-9e9c-e410199f6f33\") " pod="openshift-ingress/e2e-inf-a-63f4e4-openshift-default-786cd844c9-lxsbx" Oct 30 12:54:42.317666 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:42.317649 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1ca551b-2beb-407b-9e9c-e410199f6f33-istio-podinfo\") pod \"e2e-inf-a-63f4e4-openshift-default-786cd844c9-lxsbx\" (UID: \"f1ca551b-2beb-407b-9e9c-e410199f6f33\") " pod="openshift-ingress/e2e-inf-a-63f4e4-openshift-default-786cd844c9-lxsbx" Oct 30 12:54:42.320012 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:42.319980 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1ca551b-2beb-407b-9e9c-e410199f6f33-istio-envoy\") pod \"e2e-inf-a-63f4e4-openshift-default-786cd844c9-lxsbx\" (UID: \"f1ca551b-2beb-407b-9e9c-e410199f6f33\") " pod="openshift-ingress/e2e-inf-a-63f4e4-openshift-default-786cd844c9-lxsbx" Oct 30 12:54:42.320344 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:42.318846 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1ca551b-2beb-407b-9e9c-e410199f6f33-istiod-ca-cert\") pod \"e2e-inf-a-63f4e4-openshift-default-786cd844c9-lxsbx\" (UID: \"f1ca551b-2beb-407b-9e9c-e410199f6f33\") " pod="openshift-ingress/e2e-inf-a-63f4e4-openshift-default-786cd844c9-lxsbx" Oct 30 12:54:42.320493 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:42.319419 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1ca551b-2beb-407b-9e9c-e410199f6f33-credential-socket\") pod \"e2e-inf-a-63f4e4-openshift-default-786cd844c9-lxsbx\" (UID: \"f1ca551b-2beb-407b-9e9c-e410199f6f33\") " pod="openshift-ingress/e2e-inf-a-63f4e4-openshift-default-786cd844c9-lxsbx" Oct 30 12:54:42.321555 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:42.321183 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1ca551b-2beb-407b-9e9c-e410199f6f33-istio-data\") pod \"e2e-inf-a-63f4e4-openshift-default-786cd844c9-lxsbx\" (UID: \"f1ca551b-2beb-407b-9e9c-e410199f6f33\") " pod="openshift-ingress/e2e-inf-a-63f4e4-openshift-default-786cd844c9-lxsbx" Oct 30 12:54:42.322017 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:42.321973 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1ca551b-2beb-407b-9e9c-e410199f6f33-workload-certs\") pod \"e2e-inf-a-63f4e4-openshift-default-786cd844c9-lxsbx\" (UID: \"f1ca551b-2beb-407b-9e9c-e410199f6f33\") " pod="openshift-ingress/e2e-inf-a-63f4e4-openshift-default-786cd844c9-lxsbx" Oct 30 12:54:42.322677 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:42.318208 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1ca551b-2beb-407b-9e9c-e410199f6f33-workload-socket\") pod \"e2e-inf-a-63f4e4-openshift-default-786cd844c9-lxsbx\" (UID: \"f1ca551b-2beb-407b-9e9c-e410199f6f33\") " pod="openshift-ingress/e2e-inf-a-63f4e4-openshift-default-786cd844c9-lxsbx" Oct 30 12:54:42.354179 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:42.353179 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1ca551b-2beb-407b-9e9c-e410199f6f33-istio-envoy\") pod \"e2e-inf-a-63f4e4-openshift-default-786cd844c9-lxsbx\" (UID: \"f1ca551b-2beb-407b-9e9c-e410199f6f33\") " pod="openshift-ingress/e2e-inf-a-63f4e4-openshift-default-786cd844c9-lxsbx" Oct 30 12:54:42.363519 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:42.363485 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1ca551b-2beb-407b-9e9c-e410199f6f33-istio-podinfo\") pod \"e2e-inf-a-63f4e4-openshift-default-786cd844c9-lxsbx\" (UID: \"f1ca551b-2beb-407b-9e9c-e410199f6f33\") " pod="openshift-ingress/e2e-inf-a-63f4e4-openshift-default-786cd844c9-lxsbx" Oct 30 12:54:42.364345 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:42.364309 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1ca551b-2beb-407b-9e9c-e410199f6f33-istio-token\") pod \"e2e-inf-a-63f4e4-openshift-default-786cd844c9-lxsbx\" (UID: \"f1ca551b-2beb-407b-9e9c-e410199f6f33\") " pod="openshift-ingress/e2e-inf-a-63f4e4-openshift-default-786cd844c9-lxsbx" Oct 30 12:54:42.372946 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:42.372917 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nb9tb\" (UniqueName: \"kubernetes.io/projected/f1ca551b-2beb-407b-9e9c-e410199f6f33-kube-api-access-nb9tb\") pod \"e2e-inf-a-63f4e4-openshift-default-786cd844c9-lxsbx\" (UID: \"f1ca551b-2beb-407b-9e9c-e410199f6f33\") " pod="openshift-ingress/e2e-inf-a-63f4e4-openshift-default-786cd844c9-lxsbx" Oct 30 12:54:42.394940 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:42.394767 4908 generic.go:356] "Generic (PLEG): container finished" podID="48c3e203-4fe7-4c38-a65e-e24f5605c69d" containerID="c02c476ff7d0133d1df1d637a2d3313e47abe2c103808ee7823121610e484d97" exitCode=0 Oct 30 12:54:42.394940 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:42.394887 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-9f38e1/e2e-usage-sim-0f45af-55f5696f49-j9l6t" event={"ID":"48c3e203-4fe7-4c38-a65e-e24f5605c69d","Type":"ContainerDied","Data":"c02c476ff7d0133d1df1d637a2d3313e47abe2c103808ee7823121610e484d97"} Oct 30 12:54:42.519811 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:42.519513 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-63f4e4-openshift-default-786cd844c9-lxsbx" Oct 30 12:54:42.881361 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:42.881048 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-9f38e1/e2e-usage-sim-0f45af-55f5696f49-j9l6t" Oct 30 12:54:43.004213 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:43.004180 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48c3e203-4fe7-4c38-a65e-e24f5605c69d-kube-api-access-fv7mb\" (UniqueName: \"kubernetes.io/projected/48c3e203-4fe7-4c38-a65e-e24f5605c69d-kube-api-access-fv7mb\") pod \"48c3e203-4fe7-4c38-a65e-e24f5605c69d\" (UID: \"48c3e203-4fe7-4c38-a65e-e24f5605c69d\") " Oct 30 12:54:43.004382 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:43.004245 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/48c3e203-4fe7-4c38-a65e-e24f5605c69d-tls\" (UniqueName: \"kubernetes.io/secret/48c3e203-4fe7-4c38-a65e-e24f5605c69d-tls\") pod \"48c3e203-4fe7-4c38-a65e-e24f5605c69d\" (UID: \"48c3e203-4fe7-4c38-a65e-e24f5605c69d\") " Oct 30 12:54:43.006541 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:43.006504 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48c3e203-4fe7-4c38-a65e-e24f5605c69d-tls" pod "48c3e203-4fe7-4c38-a65e-e24f5605c69d" (UID: "48c3e203-4fe7-4c38-a65e-e24f5605c69d"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:54:43.006692 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:43.006664 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48c3e203-4fe7-4c38-a65e-e24f5605c69d-kube-api-access-fv7mb" pod "48c3e203-4fe7-4c38-a65e-e24f5605c69d" (UID: "48c3e203-4fe7-4c38-a65e-e24f5605c69d"). InnerVolumeSpecName "kube-api-access-fv7mb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:54:43.106234 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:43.106149 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fv7mb\" (UniqueName: \"kubernetes.io/projected/48c3e203-4fe7-4c38-a65e-e24f5605c69d-kube-api-access-fv7mb\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:54:43.106234 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:43.106191 4908 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/48c3e203-4fe7-4c38-a65e-e24f5605c69d-tls\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:54:43.121965 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:43.121940 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-63f4e4-openshift-default-786cd844c9-lxsbx"] Oct 30 12:54:43.122603 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:54:43.122582 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1ca551b_2beb_407b_9e9c_e410199f6f33.slice/crio-38aa741646389e8e45b5488d0d81d94c6f925cee0cfb9856a0ad52cfdabebcff WatchSource:0}: Error finding container 38aa741646389e8e45b5488d0d81d94c6f925cee0cfb9856a0ad52cfdabebcff: Status 404 returned error can't find the container with id 38aa741646389e8e45b5488d0d81d94c6f925cee0cfb9856a0ad52cfdabebcff Oct 30 12:54:43.125802 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:43.125770 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:54:43.125931 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:43.125872 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:54:43.125931 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:43.125905 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:54:43.417899 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:43.414735 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-9f38e1/e2e-usage-sim-0f45af-55f5696f49-j9l6t" event={"ID":"48c3e203-4fe7-4c38-a65e-e24f5605c69d","Type":"ContainerDied","Data":"250768cb2b7b1349cad3d5f5ad847ff698d9e85243a88b5f3d2ba3292665ecfe"} Oct 30 12:54:43.417899 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:43.414802 4908 scope.go:122] "RemoveContainer" containerID="c02c476ff7d0133d1df1d637a2d3313e47abe2c103808ee7823121610e484d97" Oct 30 12:54:43.417899 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:43.414982 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-9f38e1/e2e-usage-sim-0f45af-55f5696f49-j9l6t" Oct 30 12:54:43.489098 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:43.489056 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-63f4e4-openshift-default-786cd844c9-lxsbx" event={"ID":"f1ca551b-2beb-407b-9e9c-e410199f6f33","Type":"ContainerStarted","Data":"48dc6ab44528d0f5a5162fc8b89d173676851668e94c506a2ab0d5f16d0cb238"} Oct 30 12:54:43.489268 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:43.489132 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-63f4e4-openshift-default-786cd844c9-lxsbx" event={"ID":"f1ca551b-2beb-407b-9e9c-e410199f6f33","Type":"ContainerStarted","Data":"38aa741646389e8e45b5488d0d81d94c6f925cee0cfb9856a0ad52cfdabebcff"} Oct 30 12:54:43.523277 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:43.523248 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-extmod-shr-user-9f38e1/e2e-usage-sim-0f45af-55f5696f49-j9l6t"] Oct 30 12:54:43.523422 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:43.523317 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-63f4e4-openshift-default-786cd844c9-lxsbx" Oct 30 12:54:43.541190 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:43.541164 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-extmod-shr-user-9f38e1/e2e-usage-sim-0f45af-55f5696f49-j9l6t"] Oct 30 12:54:43.588133 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:43.588076 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-63f4e4-openshift-default-786cd844c9-lxsbx" podStartSLOduration=2.58805865 podStartE2EDuration="2.58805865s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:54:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:54:43.580933559 +0000 UTC m=+3640.095362305" watchObservedRunningTime="2026-10-30 12:54:43.58805865 +0000 UTC m=+3640.102487397" Oct 30 12:54:43.610067 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:43.610022 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48c3e203-4fe7-4c38-a65e-e24f5605c69d" path="/var/lib/kubelet/pods/48c3e203-4fe7-4c38-a65e-e24f5605c69d/volumes" Oct 30 12:54:44.526568 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:44.526538 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-63f4e4-openshift-default-786cd844c9-lxsbx" Oct 30 12:54:45.505620 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:45.505584 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-63f4e4-openshift-default-786cd844c9-lxsbx" Oct 30 12:54:45.506420 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:45.506394 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-63f4e4-openshift-default-786cd844c9-lxsbx" Oct 30 12:54:49.688359 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:54:49.688310 4908 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vn6vv" containerName="extractor" Oct 30 12:54:50.944317 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:50.944269 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-6b67f8-openshift-default-696ff8b6f-6t5q2"] Oct 30 12:54:50.945810 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:50.945772 4908 cpu_manager.go:414] "containerMap: removing container" podUID="48c3e203-4fe7-4c38-a65e-e24f5605c69d" containerName="sim" Oct 30 12:54:50.945810 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:50.945808 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48c3e203-4fe7-4c38-a65e-e24f5605c69d" containerName="sim" Oct 30 12:54:50.946043 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:50.946012 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="48c3e203-4fe7-4c38-a65e-e24f5605c69d" containerName="sim" Oct 30 12:54:50.952035 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:50.952014 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6b67f8-openshift-default-696ff8b6f-6t5q2" Oct 30 12:54:50.974508 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:50.974483 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-6b67f8-openshift-default-696ff8b6f-6t5q2"] Oct 30 12:54:50.978310 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:50.978276 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ab626b9-027f-4799-ba80-e47276070fdb-istio-envoy\") pod \"e2e-inf-a-6b67f8-openshift-default-696ff8b6f-6t5q2\" (UID: \"0ab626b9-027f-4799-ba80-e47276070fdb\") " pod="openshift-ingress/e2e-inf-a-6b67f8-openshift-default-696ff8b6f-6t5q2" Oct 30 12:54:50.978444 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:50.978333 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ab626b9-027f-4799-ba80-e47276070fdb-workload-socket\") pod \"e2e-inf-a-6b67f8-openshift-default-696ff8b6f-6t5q2\" (UID: \"0ab626b9-027f-4799-ba80-e47276070fdb\") " pod="openshift-ingress/e2e-inf-a-6b67f8-openshift-default-696ff8b6f-6t5q2" Oct 30 12:54:50.978513 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:50.978454 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ab626b9-027f-4799-ba80-e47276070fdb-credential-socket\") pod \"e2e-inf-a-6b67f8-openshift-default-696ff8b6f-6t5q2\" (UID: \"0ab626b9-027f-4799-ba80-e47276070fdb\") " pod="openshift-ingress/e2e-inf-a-6b67f8-openshift-default-696ff8b6f-6t5q2" Oct 30 12:54:50.978576 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:50.978511 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bq5hv\" (UniqueName: \"kubernetes.io/projected/0ab626b9-027f-4799-ba80-e47276070fdb-kube-api-access-bq5hv\") pod \"e2e-inf-a-6b67f8-openshift-default-696ff8b6f-6t5q2\" (UID: \"0ab626b9-027f-4799-ba80-e47276070fdb\") " pod="openshift-ingress/e2e-inf-a-6b67f8-openshift-default-696ff8b6f-6t5q2" Oct 30 12:54:50.978576 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:50.978544 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ab626b9-027f-4799-ba80-e47276070fdb-workload-certs\") pod \"e2e-inf-a-6b67f8-openshift-default-696ff8b6f-6t5q2\" (UID: \"0ab626b9-027f-4799-ba80-e47276070fdb\") " pod="openshift-ingress/e2e-inf-a-6b67f8-openshift-default-696ff8b6f-6t5q2" Oct 30 12:54:50.979390 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:50.978614 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ab626b9-027f-4799-ba80-e47276070fdb-istiod-ca-cert\") pod \"e2e-inf-a-6b67f8-openshift-default-696ff8b6f-6t5q2\" (UID: \"0ab626b9-027f-4799-ba80-e47276070fdb\") " pod="openshift-ingress/e2e-inf-a-6b67f8-openshift-default-696ff8b6f-6t5q2" Oct 30 12:54:50.979390 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:50.978725 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ab626b9-027f-4799-ba80-e47276070fdb-istio-data\") pod \"e2e-inf-a-6b67f8-openshift-default-696ff8b6f-6t5q2\" (UID: \"0ab626b9-027f-4799-ba80-e47276070fdb\") " pod="openshift-ingress/e2e-inf-a-6b67f8-openshift-default-696ff8b6f-6t5q2" Oct 30 12:54:50.979390 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:50.978750 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ab626b9-027f-4799-ba80-e47276070fdb-istio-podinfo\") pod \"e2e-inf-a-6b67f8-openshift-default-696ff8b6f-6t5q2\" (UID: \"0ab626b9-027f-4799-ba80-e47276070fdb\") " pod="openshift-ingress/e2e-inf-a-6b67f8-openshift-default-696ff8b6f-6t5q2" Oct 30 12:54:50.979390 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:50.978794 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ab626b9-027f-4799-ba80-e47276070fdb-istio-token\") pod \"e2e-inf-a-6b67f8-openshift-default-696ff8b6f-6t5q2\" (UID: \"0ab626b9-027f-4799-ba80-e47276070fdb\") " pod="openshift-ingress/e2e-inf-a-6b67f8-openshift-default-696ff8b6f-6t5q2" Oct 30 12:54:50.997604 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:50.997578 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-6b67f8-openshift-default-dockercfg-zws2t\"" Oct 30 12:54:51.054710 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:51.054663 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e70aca9a-5wsmz"] Oct 30 12:54:51.059621 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:51.059591 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e70aca9a-5wsmz"] Oct 30 12:54:51.080863 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:51.080815 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ab626b9-027f-4799-ba80-e47276070fdb-istio-data\") pod \"e2e-inf-a-6b67f8-openshift-default-696ff8b6f-6t5q2\" (UID: \"0ab626b9-027f-4799-ba80-e47276070fdb\") " pod="openshift-ingress/e2e-inf-a-6b67f8-openshift-default-696ff8b6f-6t5q2" Oct 30 12:54:51.081028 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:51.080873 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ab626b9-027f-4799-ba80-e47276070fdb-istio-podinfo\") pod \"e2e-inf-a-6b67f8-openshift-default-696ff8b6f-6t5q2\" (UID: \"0ab626b9-027f-4799-ba80-e47276070fdb\") " pod="openshift-ingress/e2e-inf-a-6b67f8-openshift-default-696ff8b6f-6t5q2" Oct 30 12:54:51.081028 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:51.080914 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ab626b9-027f-4799-ba80-e47276070fdb-istio-token\") pod \"e2e-inf-a-6b67f8-openshift-default-696ff8b6f-6t5q2\" (UID: \"0ab626b9-027f-4799-ba80-e47276070fdb\") " pod="openshift-ingress/e2e-inf-a-6b67f8-openshift-default-696ff8b6f-6t5q2" Oct 30 12:54:51.081167 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:51.081085 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ab626b9-027f-4799-ba80-e47276070fdb-istio-envoy\") pod \"e2e-inf-a-6b67f8-openshift-default-696ff8b6f-6t5q2\" (UID: \"0ab626b9-027f-4799-ba80-e47276070fdb\") " pod="openshift-ingress/e2e-inf-a-6b67f8-openshift-default-696ff8b6f-6t5q2" Oct 30 12:54:51.081167 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:51.081118 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ab626b9-027f-4799-ba80-e47276070fdb-workload-socket\") pod \"e2e-inf-a-6b67f8-openshift-default-696ff8b6f-6t5q2\" (UID: \"0ab626b9-027f-4799-ba80-e47276070fdb\") " pod="openshift-ingress/e2e-inf-a-6b67f8-openshift-default-696ff8b6f-6t5q2" Oct 30 12:54:51.081295 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:51.081196 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ab626b9-027f-4799-ba80-e47276070fdb-credential-socket\") pod \"e2e-inf-a-6b67f8-openshift-default-696ff8b6f-6t5q2\" (UID: \"0ab626b9-027f-4799-ba80-e47276070fdb\") " pod="openshift-ingress/e2e-inf-a-6b67f8-openshift-default-696ff8b6f-6t5q2" Oct 30 12:54:51.081295 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:51.081221 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bq5hv\" (UniqueName: \"kubernetes.io/projected/0ab626b9-027f-4799-ba80-e47276070fdb-kube-api-access-bq5hv\") pod \"e2e-inf-a-6b67f8-openshift-default-696ff8b6f-6t5q2\" (UID: \"0ab626b9-027f-4799-ba80-e47276070fdb\") " pod="openshift-ingress/e2e-inf-a-6b67f8-openshift-default-696ff8b6f-6t5q2" Oct 30 12:54:51.081295 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:51.081247 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ab626b9-027f-4799-ba80-e47276070fdb-workload-certs\") pod \"e2e-inf-a-6b67f8-openshift-default-696ff8b6f-6t5q2\" (UID: \"0ab626b9-027f-4799-ba80-e47276070fdb\") " pod="openshift-ingress/e2e-inf-a-6b67f8-openshift-default-696ff8b6f-6t5q2" Oct 30 12:54:51.081295 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:51.081278 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ab626b9-027f-4799-ba80-e47276070fdb-istio-data\") pod \"e2e-inf-a-6b67f8-openshift-default-696ff8b6f-6t5q2\" (UID: \"0ab626b9-027f-4799-ba80-e47276070fdb\") " pod="openshift-ingress/e2e-inf-a-6b67f8-openshift-default-696ff8b6f-6t5q2" Oct 30 12:54:51.081525 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:51.081311 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ab626b9-027f-4799-ba80-e47276070fdb-istiod-ca-cert\") pod \"e2e-inf-a-6b67f8-openshift-default-696ff8b6f-6t5q2\" (UID: \"0ab626b9-027f-4799-ba80-e47276070fdb\") " pod="openshift-ingress/e2e-inf-a-6b67f8-openshift-default-696ff8b6f-6t5q2" Oct 30 12:54:51.081525 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:51.081512 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ab626b9-027f-4799-ba80-e47276070fdb-workload-socket\") pod \"e2e-inf-a-6b67f8-openshift-default-696ff8b6f-6t5q2\" (UID: \"0ab626b9-027f-4799-ba80-e47276070fdb\") " pod="openshift-ingress/e2e-inf-a-6b67f8-openshift-default-696ff8b6f-6t5q2" Oct 30 12:54:51.081765 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:51.081734 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ab626b9-027f-4799-ba80-e47276070fdb-workload-certs\") pod \"e2e-inf-a-6b67f8-openshift-default-696ff8b6f-6t5q2\" (UID: \"0ab626b9-027f-4799-ba80-e47276070fdb\") " pod="openshift-ingress/e2e-inf-a-6b67f8-openshift-default-696ff8b6f-6t5q2" Oct 30 12:54:51.081873 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:51.081800 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ab626b9-027f-4799-ba80-e47276070fdb-credential-socket\") pod \"e2e-inf-a-6b67f8-openshift-default-696ff8b6f-6t5q2\" (UID: \"0ab626b9-027f-4799-ba80-e47276070fdb\") " pod="openshift-ingress/e2e-inf-a-6b67f8-openshift-default-696ff8b6f-6t5q2" Oct 30 12:54:51.082340 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:51.082312 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ab626b9-027f-4799-ba80-e47276070fdb-istiod-ca-cert\") pod \"e2e-inf-a-6b67f8-openshift-default-696ff8b6f-6t5q2\" (UID: \"0ab626b9-027f-4799-ba80-e47276070fdb\") " pod="openshift-ingress/e2e-inf-a-6b67f8-openshift-default-696ff8b6f-6t5q2" Oct 30 12:54:51.083936 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:51.083904 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ab626b9-027f-4799-ba80-e47276070fdb-istio-envoy\") pod \"e2e-inf-a-6b67f8-openshift-default-696ff8b6f-6t5q2\" (UID: \"0ab626b9-027f-4799-ba80-e47276070fdb\") " pod="openshift-ingress/e2e-inf-a-6b67f8-openshift-default-696ff8b6f-6t5q2" Oct 30 12:54:51.084045 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:51.083937 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ab626b9-027f-4799-ba80-e47276070fdb-istio-podinfo\") pod \"e2e-inf-a-6b67f8-openshift-default-696ff8b6f-6t5q2\" (UID: \"0ab626b9-027f-4799-ba80-e47276070fdb\") " pod="openshift-ingress/e2e-inf-a-6b67f8-openshift-default-696ff8b6f-6t5q2" Oct 30 12:54:51.148814 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:51.148771 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ab626b9-027f-4799-ba80-e47276070fdb-istio-token\") pod \"e2e-inf-a-6b67f8-openshift-default-696ff8b6f-6t5q2\" (UID: \"0ab626b9-027f-4799-ba80-e47276070fdb\") " pod="openshift-ingress/e2e-inf-a-6b67f8-openshift-default-696ff8b6f-6t5q2" Oct 30 12:54:51.185737 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:51.185701 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bq5hv\" (UniqueName: \"kubernetes.io/projected/0ab626b9-027f-4799-ba80-e47276070fdb-kube-api-access-bq5hv\") pod \"e2e-inf-a-6b67f8-openshift-default-696ff8b6f-6t5q2\" (UID: \"0ab626b9-027f-4799-ba80-e47276070fdb\") " pod="openshift-ingress/e2e-inf-a-6b67f8-openshift-default-696ff8b6f-6t5q2" Oct 30 12:54:51.294144 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:51.294111 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6b67f8-openshift-default-696ff8b6f-6t5q2" Oct 30 12:54:51.615784 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:51.615696 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="437b70b1-fea7-4b27-a34b-39a0acf5103d" path="/var/lib/kubelet/pods/437b70b1-fea7-4b27-a34b-39a0acf5103d/volumes" Oct 30 12:54:51.791090 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:51.791053 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-6b67f8-openshift-default-696ff8b6f-6t5q2"] Oct 30 12:54:51.791458 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:54:51.791430 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ab626b9_027f_4799_ba80_e47276070fdb.slice/crio-d9893206e43269a68e21332fd11dc6e67b2895f2e6c3236422617ec94e8bab60 WatchSource:0}: Error finding container d9893206e43269a68e21332fd11dc6e67b2895f2e6c3236422617ec94e8bab60: Status 404 returned error can't find the container with id d9893206e43269a68e21332fd11dc6e67b2895f2e6c3236422617ec94e8bab60 Oct 30 12:54:51.794881 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:51.794840 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:54:51.794986 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:51.794916 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:54:51.794986 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:51.794948 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:54:52.604512 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:52.604471 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6b67f8-openshift-default-696ff8b6f-6t5q2" event={"ID":"0ab626b9-027f-4799-ba80-e47276070fdb","Type":"ContainerStarted","Data":"3c174bcf714e7781bff416a5de8a3c29a0a190b20f792f857b4e573918540ef7"} Oct 30 12:54:52.605412 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:52.605384 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6b67f8-openshift-default-696ff8b6f-6t5q2" event={"ID":"0ab626b9-027f-4799-ba80-e47276070fdb","Type":"ContainerStarted","Data":"d9893206e43269a68e21332fd11dc6e67b2895f2e6c3236422617ec94e8bab60"} Oct 30 12:54:52.664227 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:52.664169 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-6b67f8-openshift-default-696ff8b6f-6t5q2" podStartSLOduration=2.6641544550000003 podStartE2EDuration="2.664154455s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:54:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:54:52.660721133 +0000 UTC m=+3649.175149879" watchObservedRunningTime="2026-10-30 12:54:52.664154455 +0000 UTC m=+3649.178583201" Oct 30 12:54:53.295571 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:53.295532 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-6b67f8-openshift-default-696ff8b6f-6t5q2" Oct 30 12:54:53.302363 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:53.302338 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-6b67f8-openshift-default-696ff8b6f-6t5q2" Oct 30 12:54:53.614619 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:53.614538 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-6b67f8-openshift-default-696ff8b6f-6t5q2" Oct 30 12:54:53.615377 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:53.615356 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-6b67f8-openshift-default-696ff8b6f-6t5q2" Oct 30 12:54:59.035853 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:54:59.031881 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9wwfc"] Oct 30 12:55:01.056585 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:01.056485 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7ccd5df-njfw8"] Oct 30 12:55:01.085462 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:01.085414 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7ccd5df-njfw8"] Oct 30 12:55:01.609064 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:01.609030 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09d371a1-43b5-4afd-b990-ed703b359d65" path="/var/lib/kubelet/pods/09d371a1-43b5-4afd-b990-ed703b359d65/volumes" Oct 30 12:55:08.492745 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:08.492705 4908 scope.go:122] "RemoveContainer" containerID="32cdef9b69fe7f3f27ce4bd2b6f8a19b1ea50ad608baa4553d94c9814835a86b" Oct 30 12:55:14.453927 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:14.453887 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98ccc618-wz48b"] Oct 30 12:55:14.458189 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:14.458170 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98ccc618-wz48b" Oct 30 12:55:14.477200 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:14.477176 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98ccc618-wz48b"] Oct 30 12:55:14.492865 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:14.482960 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 12:55:14.492865 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:14.483032 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 12:55:14.492865 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:14.483099 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 12:55:14.492865 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:14.483169 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s75hq\"" Oct 30 12:55:14.555971 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:14.555943 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9208b4e7-5b24-45ad-a6ba-faea07eae2b6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-98ccc618-wz48b\" (UID: \"9208b4e7-5b24-45ad-a6ba-faea07eae2b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98ccc618-wz48b" Oct 30 12:55:14.556132 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:14.556088 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9208b4e7-5b24-45ad-a6ba-faea07eae2b6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-98ccc618-wz48b\" (UID: \"9208b4e7-5b24-45ad-a6ba-faea07eae2b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98ccc618-wz48b" Oct 30 12:55:14.556208 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:14.556142 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hv4j4\" (UniqueName: \"kubernetes.io/projected/9208b4e7-5b24-45ad-a6ba-faea07eae2b6-kube-api-access-hv4j4\") pod \"maas-api-revoke-sub-98ccc618-wz48b\" (UID: \"9208b4e7-5b24-45ad-a6ba-faea07eae2b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98ccc618-wz48b" Oct 30 12:55:14.657288 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:14.657256 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9208b4e7-5b24-45ad-a6ba-faea07eae2b6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-98ccc618-wz48b\" (UID: \"9208b4e7-5b24-45ad-a6ba-faea07eae2b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98ccc618-wz48b" Oct 30 12:55:14.657460 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:14.657348 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9208b4e7-5b24-45ad-a6ba-faea07eae2b6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-98ccc618-wz48b\" (UID: \"9208b4e7-5b24-45ad-a6ba-faea07eae2b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98ccc618-wz48b" Oct 30 12:55:14.657460 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:14.657375 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hv4j4\" (UniqueName: \"kubernetes.io/projected/9208b4e7-5b24-45ad-a6ba-faea07eae2b6-kube-api-access-hv4j4\") pod \"maas-api-revoke-sub-98ccc618-wz48b\" (UID: \"9208b4e7-5b24-45ad-a6ba-faea07eae2b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98ccc618-wz48b" Oct 30 12:55:14.658102 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:14.658077 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9208b4e7-5b24-45ad-a6ba-faea07eae2b6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-98ccc618-wz48b\" (UID: \"9208b4e7-5b24-45ad-a6ba-faea07eae2b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98ccc618-wz48b" Oct 30 12:55:14.659909 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:14.659884 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9208b4e7-5b24-45ad-a6ba-faea07eae2b6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-98ccc618-wz48b\" (UID: \"9208b4e7-5b24-45ad-a6ba-faea07eae2b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98ccc618-wz48b" Oct 30 12:55:14.679990 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:14.679964 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hv4j4\" (UniqueName: \"kubernetes.io/projected/9208b4e7-5b24-45ad-a6ba-faea07eae2b6-kube-api-access-hv4j4\") pod \"maas-api-revoke-sub-98ccc618-wz48b\" (UID: \"9208b4e7-5b24-45ad-a6ba-faea07eae2b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98ccc618-wz48b" Oct 30 12:55:14.775499 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:14.775460 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98ccc618-wz48b" Oct 30 12:55:15.053845 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:15.053762 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10629caa-2bx9z"] Oct 30 12:55:15.066587 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:15.066560 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10629caa-2bx9z"] Oct 30 12:55:15.096304 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:15.096234 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98ccc618-wz48b"] Oct 30 12:55:15.099582 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:55:15.099545 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9208b4e7_5b24_45ad_a6ba_faea07eae2b6.slice/crio-4c1f245041448aa33809911df4d1d2c4c1d812b0c5ff4481d5288eb031bcea13 WatchSource:0}: Error finding container 4c1f245041448aa33809911df4d1d2c4c1d812b0c5ff4481d5288eb031bcea13: Status 404 returned error can't find the container with id 4c1f245041448aa33809911df4d1d2c4c1d812b0c5ff4481d5288eb031bcea13 Oct 30 12:55:15.611144 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:15.611100 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79714daf-eda5-4d2f-944d-385759f941ae" path="/var/lib/kubelet/pods/79714daf-eda5-4d2f-944d-385759f941ae/volumes" Oct 30 12:55:15.817568 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:15.817531 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98ccc618-wz48b" event={"ID":"9208b4e7-5b24-45ad-a6ba-faea07eae2b6","Type":"ContainerStarted","Data":"81a38e655caf32be04534d1f53a6f06d7226d8839289864f5d3a2e71f3d6a55e"} Oct 30 12:55:15.817771 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:15.817590 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98ccc618-wz48b" event={"ID":"9208b4e7-5b24-45ad-a6ba-faea07eae2b6","Type":"ContainerStarted","Data":"4c1f245041448aa33809911df4d1d2c4c1d812b0c5ff4481d5288eb031bcea13"} Oct 30 12:55:15.862120 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:15.862061 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98ccc618-wz48b" podStartSLOduration=1.862042585 podStartE2EDuration="1.862042585s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:55:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:55:15.856890358 +0000 UTC m=+3672.371319104" watchObservedRunningTime="2026-10-30 12:55:15.862042585 +0000 UTC m=+3672.376471332" Oct 30 12:55:16.826377 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:16.826326 4908 generic.go:356] "Generic (PLEG): container finished" podID="9208b4e7-5b24-45ad-a6ba-faea07eae2b6" containerID="81a38e655caf32be04534d1f53a6f06d7226d8839289864f5d3a2e71f3d6a55e" exitCode=0 Oct 30 12:55:16.826791 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:16.826402 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98ccc618-wz48b" event={"ID":"9208b4e7-5b24-45ad-a6ba-faea07eae2b6","Type":"ContainerDied","Data":"81a38e655caf32be04534d1f53a6f06d7226d8839289864f5d3a2e71f3d6a55e"} Oct 30 12:55:18.000860 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:18.000814 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98ccc618-wz48b" Oct 30 12:55:18.128816 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:18.128726 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9208b4e7-5b24-45ad-a6ba-faea07eae2b6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9208b4e7-5b24-45ad-a6ba-faea07eae2b6-maas-api-xks-ca\") pod \"9208b4e7-5b24-45ad-a6ba-faea07eae2b6\" (UID: \"9208b4e7-5b24-45ad-a6ba-faea07eae2b6\") " Oct 30 12:55:18.128816 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:18.128808 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9208b4e7-5b24-45ad-a6ba-faea07eae2b6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9208b4e7-5b24-45ad-a6ba-faea07eae2b6-maas-api-openshift-ca\") pod \"9208b4e7-5b24-45ad-a6ba-faea07eae2b6\" (UID: \"9208b4e7-5b24-45ad-a6ba-faea07eae2b6\") " Oct 30 12:55:18.129089 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:18.128930 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9208b4e7-5b24-45ad-a6ba-faea07eae2b6-kube-api-access-hv4j4\" (UniqueName: \"kubernetes.io/projected/9208b4e7-5b24-45ad-a6ba-faea07eae2b6-kube-api-access-hv4j4\") pod \"9208b4e7-5b24-45ad-a6ba-faea07eae2b6\" (UID: \"9208b4e7-5b24-45ad-a6ba-faea07eae2b6\") " Oct 30 12:55:18.129192 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:18.129165 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9208b4e7-5b24-45ad-a6ba-faea07eae2b6-maas-api-openshift-ca" pod "9208b4e7-5b24-45ad-a6ba-faea07eae2b6" (UID: "9208b4e7-5b24-45ad-a6ba-faea07eae2b6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:55:18.129568 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:18.129545 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9208b4e7-5b24-45ad-a6ba-faea07eae2b6-maas-api-openshift-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:55:18.130994 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:18.130970 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9208b4e7-5b24-45ad-a6ba-faea07eae2b6-maas-api-xks-ca" pod "9208b4e7-5b24-45ad-a6ba-faea07eae2b6" (UID: "9208b4e7-5b24-45ad-a6ba-faea07eae2b6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:55:18.131125 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:18.131107 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9208b4e7-5b24-45ad-a6ba-faea07eae2b6-kube-api-access-hv4j4" pod "9208b4e7-5b24-45ad-a6ba-faea07eae2b6" (UID: "9208b4e7-5b24-45ad-a6ba-faea07eae2b6"). InnerVolumeSpecName "kube-api-access-hv4j4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:55:18.231363 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:18.231337 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hv4j4\" (UniqueName: \"kubernetes.io/projected/9208b4e7-5b24-45ad-a6ba-faea07eae2b6-kube-api-access-hv4j4\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:55:18.231363 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:18.231363 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9208b4e7-5b24-45ad-a6ba-faea07eae2b6-maas-api-xks-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:55:18.843388 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:18.843359 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98ccc618-wz48b" Oct 30 12:55:18.843579 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:18.843355 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98ccc618-wz48b" event={"ID":"9208b4e7-5b24-45ad-a6ba-faea07eae2b6","Type":"ContainerDied","Data":"4c1f245041448aa33809911df4d1d2c4c1d812b0c5ff4481d5288eb031bcea13"} Oct 30 12:55:18.843579 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:18.843483 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4c1f245041448aa33809911df4d1d2c4c1d812b0c5ff4481d5288eb031bcea13" Oct 30 12:55:19.685377 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:55:19.685342 4908 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vn6vv" containerName="extractor" Oct 30 12:55:22.934770 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:22.934691 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-2s2dq"] Oct 30 12:55:22.935249 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:22.934909 4908 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-2s2dq" podUID="c3729561-3035-459d-a451-290595a76f5d" containerName="istio-proxy" containerID="cri-o://99bb35caa0eb730f715e9c61b8621115a44b26a4d229d207dd2110d6c2ecd2f4" gracePeriod=30 Oct 30 12:55:27.152854 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:27.151946 4908 patch_prober.go:29] interesting pod/e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-2s2dq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 12:55:27.152854 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:27.152025 4908 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-2s2dq" podUID="c3729561-3035-459d-a451-290595a76f5d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 12:55:28.176999 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:28.176939 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-1333c8-openshift-default-847ccc6485-tp95k"] Oct 30 12:55:28.221095 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:28.221011 4908 cpu_manager.go:414] "containerMap: removing container" podUID="9208b4e7-5b24-45ad-a6ba-faea07eae2b6" containerName="revoke-keys" Oct 30 12:55:28.221294 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:28.221277 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9208b4e7-5b24-45ad-a6ba-faea07eae2b6" containerName="revoke-keys" Oct 30 12:55:28.221615 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:28.221600 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="9208b4e7-5b24-45ad-a6ba-faea07eae2b6" containerName="revoke-keys" Oct 30 12:55:28.233059 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:28.232386 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-1333c8-openshift-default-847ccc6485-tp95k"] Oct 30 12:55:28.233059 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:28.232500 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1333c8-openshift-default-847ccc6485-tp95k" Oct 30 12:55:28.247626 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:28.247579 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-1333c8-openshift-default-dockercfg-gdjml\"" Oct 30 12:55:28.390593 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:28.371561 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75b90e54-7d8b-44cf-9e8d-06fe342acc3b-istiod-ca-cert\") pod \"e2e-inf-a-1333c8-openshift-default-847ccc6485-tp95k\" (UID: \"75b90e54-7d8b-44cf-9e8d-06fe342acc3b\") " pod="openshift-ingress/e2e-inf-a-1333c8-openshift-default-847ccc6485-tp95k" Oct 30 12:55:28.390593 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:28.371635 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75b90e54-7d8b-44cf-9e8d-06fe342acc3b-workload-socket\") pod \"e2e-inf-a-1333c8-openshift-default-847ccc6485-tp95k\" (UID: \"75b90e54-7d8b-44cf-9e8d-06fe342acc3b\") " pod="openshift-ingress/e2e-inf-a-1333c8-openshift-default-847ccc6485-tp95k" Oct 30 12:55:28.390593 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:28.371710 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75b90e54-7d8b-44cf-9e8d-06fe342acc3b-istio-data\") pod \"e2e-inf-a-1333c8-openshift-default-847ccc6485-tp95k\" (UID: \"75b90e54-7d8b-44cf-9e8d-06fe342acc3b\") " pod="openshift-ingress/e2e-inf-a-1333c8-openshift-default-847ccc6485-tp95k" Oct 30 12:55:28.390593 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:28.371746 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75b90e54-7d8b-44cf-9e8d-06fe342acc3b-istio-token\") pod \"e2e-inf-a-1333c8-openshift-default-847ccc6485-tp95k\" (UID: \"75b90e54-7d8b-44cf-9e8d-06fe342acc3b\") " pod="openshift-ingress/e2e-inf-a-1333c8-openshift-default-847ccc6485-tp95k" Oct 30 12:55:28.390593 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:28.371778 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75b90e54-7d8b-44cf-9e8d-06fe342acc3b-istio-podinfo\") pod \"e2e-inf-a-1333c8-openshift-default-847ccc6485-tp95k\" (UID: \"75b90e54-7d8b-44cf-9e8d-06fe342acc3b\") " pod="openshift-ingress/e2e-inf-a-1333c8-openshift-default-847ccc6485-tp95k" Oct 30 12:55:28.390593 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:28.371853 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75b90e54-7d8b-44cf-9e8d-06fe342acc3b-istio-envoy\") pod \"e2e-inf-a-1333c8-openshift-default-847ccc6485-tp95k\" (UID: \"75b90e54-7d8b-44cf-9e8d-06fe342acc3b\") " pod="openshift-ingress/e2e-inf-a-1333c8-openshift-default-847ccc6485-tp95k" Oct 30 12:55:28.390593 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:28.371923 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gx9n\" (UniqueName: \"kubernetes.io/projected/75b90e54-7d8b-44cf-9e8d-06fe342acc3b-kube-api-access-4gx9n\") pod \"e2e-inf-a-1333c8-openshift-default-847ccc6485-tp95k\" (UID: \"75b90e54-7d8b-44cf-9e8d-06fe342acc3b\") " pod="openshift-ingress/e2e-inf-a-1333c8-openshift-default-847ccc6485-tp95k" Oct 30 12:55:28.390593 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:28.372116 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75b90e54-7d8b-44cf-9e8d-06fe342acc3b-credential-socket\") pod \"e2e-inf-a-1333c8-openshift-default-847ccc6485-tp95k\" (UID: \"75b90e54-7d8b-44cf-9e8d-06fe342acc3b\") " pod="openshift-ingress/e2e-inf-a-1333c8-openshift-default-847ccc6485-tp95k" Oct 30 12:55:28.390593 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:28.372187 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75b90e54-7d8b-44cf-9e8d-06fe342acc3b-workload-certs\") pod \"e2e-inf-a-1333c8-openshift-default-847ccc6485-tp95k\" (UID: \"75b90e54-7d8b-44cf-9e8d-06fe342acc3b\") " pod="openshift-ingress/e2e-inf-a-1333c8-openshift-default-847ccc6485-tp95k" Oct 30 12:55:28.550852 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:28.548803 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75b90e54-7d8b-44cf-9e8d-06fe342acc3b-istio-data\") pod \"e2e-inf-a-1333c8-openshift-default-847ccc6485-tp95k\" (UID: \"75b90e54-7d8b-44cf-9e8d-06fe342acc3b\") " pod="openshift-ingress/e2e-inf-a-1333c8-openshift-default-847ccc6485-tp95k" Oct 30 12:55:28.635877 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:28.635761 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75b90e54-7d8b-44cf-9e8d-06fe342acc3b-istio-token\") pod \"e2e-inf-a-1333c8-openshift-default-847ccc6485-tp95k\" (UID: \"75b90e54-7d8b-44cf-9e8d-06fe342acc3b\") " pod="openshift-ingress/e2e-inf-a-1333c8-openshift-default-847ccc6485-tp95k" Oct 30 12:55:28.635877 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:28.635817 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75b90e54-7d8b-44cf-9e8d-06fe342acc3b-istio-podinfo\") pod \"e2e-inf-a-1333c8-openshift-default-847ccc6485-tp95k\" (UID: \"75b90e54-7d8b-44cf-9e8d-06fe342acc3b\") " pod="openshift-ingress/e2e-inf-a-1333c8-openshift-default-847ccc6485-tp95k" Oct 30 12:55:28.636176 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:28.556957 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75b90e54-7d8b-44cf-9e8d-06fe342acc3b-istio-data\") pod \"e2e-inf-a-1333c8-openshift-default-847ccc6485-tp95k\" (UID: \"75b90e54-7d8b-44cf-9e8d-06fe342acc3b\") " pod="openshift-ingress/e2e-inf-a-1333c8-openshift-default-847ccc6485-tp95k" Oct 30 12:55:28.646881 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:28.636274 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75b90e54-7d8b-44cf-9e8d-06fe342acc3b-istio-envoy\") pod \"e2e-inf-a-1333c8-openshift-default-847ccc6485-tp95k\" (UID: \"75b90e54-7d8b-44cf-9e8d-06fe342acc3b\") " pod="openshift-ingress/e2e-inf-a-1333c8-openshift-default-847ccc6485-tp95k" Oct 30 12:55:28.646881 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:28.636359 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gx9n\" (UniqueName: \"kubernetes.io/projected/75b90e54-7d8b-44cf-9e8d-06fe342acc3b-kube-api-access-4gx9n\") pod \"e2e-inf-a-1333c8-openshift-default-847ccc6485-tp95k\" (UID: \"75b90e54-7d8b-44cf-9e8d-06fe342acc3b\") " pod="openshift-ingress/e2e-inf-a-1333c8-openshift-default-847ccc6485-tp95k" Oct 30 12:55:28.646881 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:28.636566 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75b90e54-7d8b-44cf-9e8d-06fe342acc3b-credential-socket\") pod \"e2e-inf-a-1333c8-openshift-default-847ccc6485-tp95k\" (UID: \"75b90e54-7d8b-44cf-9e8d-06fe342acc3b\") " pod="openshift-ingress/e2e-inf-a-1333c8-openshift-default-847ccc6485-tp95k" Oct 30 12:55:28.646881 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:28.636639 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75b90e54-7d8b-44cf-9e8d-06fe342acc3b-workload-certs\") pod \"e2e-inf-a-1333c8-openshift-default-847ccc6485-tp95k\" (UID: \"75b90e54-7d8b-44cf-9e8d-06fe342acc3b\") " pod="openshift-ingress/e2e-inf-a-1333c8-openshift-default-847ccc6485-tp95k" Oct 30 12:55:28.646881 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:28.636683 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75b90e54-7d8b-44cf-9e8d-06fe342acc3b-istiod-ca-cert\") pod \"e2e-inf-a-1333c8-openshift-default-847ccc6485-tp95k\" (UID: \"75b90e54-7d8b-44cf-9e8d-06fe342acc3b\") " pod="openshift-ingress/e2e-inf-a-1333c8-openshift-default-847ccc6485-tp95k" Oct 30 12:55:28.646881 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:28.636728 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75b90e54-7d8b-44cf-9e8d-06fe342acc3b-workload-socket\") pod \"e2e-inf-a-1333c8-openshift-default-847ccc6485-tp95k\" (UID: \"75b90e54-7d8b-44cf-9e8d-06fe342acc3b\") " pod="openshift-ingress/e2e-inf-a-1333c8-openshift-default-847ccc6485-tp95k" Oct 30 12:55:28.692276 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:28.692151 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75b90e54-7d8b-44cf-9e8d-06fe342acc3b-istio-podinfo\") pod \"e2e-inf-a-1333c8-openshift-default-847ccc6485-tp95k\" (UID: \"75b90e54-7d8b-44cf-9e8d-06fe342acc3b\") " pod="openshift-ingress/e2e-inf-a-1333c8-openshift-default-847ccc6485-tp95k" Oct 30 12:55:28.692592 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:28.692562 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-2s2dq" Oct 30 12:55:28.693815 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:28.693756 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75b90e54-7d8b-44cf-9e8d-06fe342acc3b-istiod-ca-cert\") pod \"e2e-inf-a-1333c8-openshift-default-847ccc6485-tp95k\" (UID: \"75b90e54-7d8b-44cf-9e8d-06fe342acc3b\") " pod="openshift-ingress/e2e-inf-a-1333c8-openshift-default-847ccc6485-tp95k" Oct 30 12:55:28.694083 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:28.694065 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75b90e54-7d8b-44cf-9e8d-06fe342acc3b-workload-socket\") pod \"e2e-inf-a-1333c8-openshift-default-847ccc6485-tp95k\" (UID: \"75b90e54-7d8b-44cf-9e8d-06fe342acc3b\") " pod="openshift-ingress/e2e-inf-a-1333c8-openshift-default-847ccc6485-tp95k" Oct 30 12:55:28.696135 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:28.696083 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75b90e54-7d8b-44cf-9e8d-06fe342acc3b-credential-socket\") pod \"e2e-inf-a-1333c8-openshift-default-847ccc6485-tp95k\" (UID: \"75b90e54-7d8b-44cf-9e8d-06fe342acc3b\") " pod="openshift-ingress/e2e-inf-a-1333c8-openshift-default-847ccc6485-tp95k" Oct 30 12:55:28.696232 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:28.696136 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75b90e54-7d8b-44cf-9e8d-06fe342acc3b-workload-certs\") pod \"e2e-inf-a-1333c8-openshift-default-847ccc6485-tp95k\" (UID: \"75b90e54-7d8b-44cf-9e8d-06fe342acc3b\") " pod="openshift-ingress/e2e-inf-a-1333c8-openshift-default-847ccc6485-tp95k" Oct 30 12:55:28.702581 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:28.702556 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75b90e54-7d8b-44cf-9e8d-06fe342acc3b-istio-token\") pod \"e2e-inf-a-1333c8-openshift-default-847ccc6485-tp95k\" (UID: \"75b90e54-7d8b-44cf-9e8d-06fe342acc3b\") " pod="openshift-ingress/e2e-inf-a-1333c8-openshift-default-847ccc6485-tp95k" Oct 30 12:55:28.732802 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:28.731482 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gx9n\" (UniqueName: \"kubernetes.io/projected/75b90e54-7d8b-44cf-9e8d-06fe342acc3b-kube-api-access-4gx9n\") pod \"e2e-inf-a-1333c8-openshift-default-847ccc6485-tp95k\" (UID: \"75b90e54-7d8b-44cf-9e8d-06fe342acc3b\") " pod="openshift-ingress/e2e-inf-a-1333c8-openshift-default-847ccc6485-tp95k" Oct 30 12:55:28.836906 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:28.835503 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75b90e54-7d8b-44cf-9e8d-06fe342acc3b-istio-envoy\") pod \"e2e-inf-a-1333c8-openshift-default-847ccc6485-tp95k\" (UID: \"75b90e54-7d8b-44cf-9e8d-06fe342acc3b\") " pod="openshift-ingress/e2e-inf-a-1333c8-openshift-default-847ccc6485-tp95k" Oct 30 12:55:28.909699 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:28.889877 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3729561-3035-459d-a451-290595a76f5d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3729561-3035-459d-a451-290595a76f5d-workload-socket\") pod \"c3729561-3035-459d-a451-290595a76f5d\" (UID: \"c3729561-3035-459d-a451-290595a76f5d\") " Oct 30 12:55:28.910459 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:28.910428 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3729561-3035-459d-a451-290595a76f5d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3729561-3035-459d-a451-290595a76f5d-credential-socket\") pod \"c3729561-3035-459d-a451-290595a76f5d\" (UID: \"c3729561-3035-459d-a451-290595a76f5d\") " Oct 30 12:55:28.910711 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:28.910668 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c3729561-3035-459d-a451-290595a76f5d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3729561-3035-459d-a451-290595a76f5d-istio-podinfo\") pod \"c3729561-3035-459d-a451-290595a76f5d\" (UID: \"c3729561-3035-459d-a451-290595a76f5d\") " Oct 30 12:55:28.910976 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:28.910951 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3729561-3035-459d-a451-290595a76f5d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3729561-3035-459d-a451-290595a76f5d-istio-envoy\") pod \"c3729561-3035-459d-a451-290595a76f5d\" (UID: \"c3729561-3035-459d-a451-290595a76f5d\") " Oct 30 12:55:28.911185 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:28.911168 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3729561-3035-459d-a451-290595a76f5d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3729561-3035-459d-a451-290595a76f5d-istiod-ca-cert\") pod \"c3729561-3035-459d-a451-290595a76f5d\" (UID: \"c3729561-3035-459d-a451-290595a76f5d\") " Oct 30 12:55:28.911326 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:28.911309 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3729561-3035-459d-a451-290595a76f5d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3729561-3035-459d-a451-290595a76f5d-workload-certs\") pod \"c3729561-3035-459d-a451-290595a76f5d\" (UID: \"c3729561-3035-459d-a451-290595a76f5d\") " Oct 30 12:55:28.911455 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:28.911441 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3729561-3035-459d-a451-290595a76f5d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3729561-3035-459d-a451-290595a76f5d-istio-data\") pod \"c3729561-3035-459d-a451-290595a76f5d\" (UID: \"c3729561-3035-459d-a451-290595a76f5d\") " Oct 30 12:55:28.911705 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:28.911688 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3729561-3035-459d-a451-290595a76f5d-istio-token\" (UniqueName: \"kubernetes.io/projected/c3729561-3035-459d-a451-290595a76f5d-istio-token\") pod \"c3729561-3035-459d-a451-290595a76f5d\" (UID: \"c3729561-3035-459d-a451-290595a76f5d\") " Oct 30 12:55:28.912670 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:28.912626 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3729561-3035-459d-a451-290595a76f5d-kube-api-access-6wkv7\" (UniqueName: \"kubernetes.io/projected/c3729561-3035-459d-a451-290595a76f5d-kube-api-access-6wkv7\") pod \"c3729561-3035-459d-a451-290595a76f5d\" (UID: \"c3729561-3035-459d-a451-290595a76f5d\") " Oct 30 12:55:28.915050 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:28.891224 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3729561-3035-459d-a451-290595a76f5d-workload-socket" pod "c3729561-3035-459d-a451-290595a76f5d" (UID: "c3729561-3035-459d-a451-290595a76f5d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:55:28.915184 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:28.914746 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3729561-3035-459d-a451-290595a76f5d-istiod-ca-cert" pod "c3729561-3035-459d-a451-290595a76f5d" (UID: "c3729561-3035-459d-a451-290595a76f5d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:55:28.915268 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:28.915023 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3729561-3035-459d-a451-290595a76f5d-credential-socket" pod "c3729561-3035-459d-a451-290595a76f5d" (UID: "c3729561-3035-459d-a451-290595a76f5d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:55:28.915544 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:28.915526 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3729561-3035-459d-a451-290595a76f5d-istio-data" pod "c3729561-3035-459d-a451-290595a76f5d" (UID: "c3729561-3035-459d-a451-290595a76f5d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:55:28.915662 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:28.910260 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1333c8-openshift-default-847ccc6485-tp95k" Oct 30 12:55:28.921129 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:28.920973 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c3729561-3035-459d-a451-290595a76f5d-istio-podinfo" pod "c3729561-3035-459d-a451-290595a76f5d" (UID: "c3729561-3035-459d-a451-290595a76f5d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 12:55:28.921129 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:28.921097 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3729561-3035-459d-a451-290595a76f5d-kube-api-access-6wkv7" pod "c3729561-3035-459d-a451-290595a76f5d" (UID: "c3729561-3035-459d-a451-290595a76f5d"). InnerVolumeSpecName "kube-api-access-6wkv7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:55:28.922125 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:28.922105 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3729561-3035-459d-a451-290595a76f5d-workload-certs" pod "c3729561-3035-459d-a451-290595a76f5d" (UID: "c3729561-3035-459d-a451-290595a76f5d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:55:28.923398 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:28.923358 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3729561-3035-459d-a451-290595a76f5d-istio-token" pod "c3729561-3035-459d-a451-290595a76f5d" (UID: "c3729561-3035-459d-a451-290595a76f5d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:55:28.948879 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:28.938927 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3729561-3035-459d-a451-290595a76f5d-istio-envoy" pod "c3729561-3035-459d-a451-290595a76f5d" (UID: "c3729561-3035-459d-a451-290595a76f5d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:55:29.015309 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:29.015089 4908 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3729561-3035-459d-a451-290595a76f5d-istio-token\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:55:29.015309 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:29.015130 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6wkv7\" (UniqueName: \"kubernetes.io/projected/c3729561-3035-459d-a451-290595a76f5d-kube-api-access-6wkv7\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:55:29.015309 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:29.015146 4908 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3729561-3035-459d-a451-290595a76f5d-workload-socket\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:55:29.015309 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:29.015160 4908 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3729561-3035-459d-a451-290595a76f5d-credential-socket\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:55:29.015309 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:29.015175 4908 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3729561-3035-459d-a451-290595a76f5d-istio-podinfo\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:55:29.015309 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:29.015187 4908 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3729561-3035-459d-a451-290595a76f5d-istio-envoy\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:55:29.015309 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:29.015200 4908 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3729561-3035-459d-a451-290595a76f5d-istiod-ca-cert\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:55:29.015309 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:29.015213 4908 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3729561-3035-459d-a451-290595a76f5d-workload-certs\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:55:29.015309 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:29.015224 4908 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3729561-3035-459d-a451-290595a76f5d-istio-data\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:55:29.023904 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:29.023868 4908 generic.go:356] "Generic (PLEG): container finished" podID="c3729561-3035-459d-a451-290595a76f5d" containerID="99bb35caa0eb730f715e9c61b8621115a44b26a4d229d207dd2110d6c2ecd2f4" exitCode=0 Oct 30 12:55:29.024022 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:29.023985 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-2s2dq" event={"ID":"c3729561-3035-459d-a451-290595a76f5d","Type":"ContainerDied","Data":"99bb35caa0eb730f715e9c61b8621115a44b26a4d229d207dd2110d6c2ecd2f4"} Oct 30 12:55:29.024080 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:29.024055 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-2s2dq" event={"ID":"c3729561-3035-459d-a451-290595a76f5d","Type":"ContainerDied","Data":"1647e26cab206e76c296dd423ceb9b823471400fe90ef364391a09b367e279a4"} Oct 30 12:55:29.024128 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:29.024086 4908 scope.go:122] "RemoveContainer" containerID="99bb35caa0eb730f715e9c61b8621115a44b26a4d229d207dd2110d6c2ecd2f4" Oct 30 12:55:29.026106 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:29.024270 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-2s2dq" Oct 30 12:55:29.039348 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:29.039320 4908 scope.go:122] "RemoveContainer" containerID="99bb35caa0eb730f715e9c61b8621115a44b26a4d229d207dd2110d6c2ecd2f4" Oct 30 12:55:29.039964 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:55:29.039941 4908 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99bb35caa0eb730f715e9c61b8621115a44b26a4d229d207dd2110d6c2ecd2f4\": container with ID starting with 99bb35caa0eb730f715e9c61b8621115a44b26a4d229d207dd2110d6c2ecd2f4 not found: ID does not exist" containerID="99bb35caa0eb730f715e9c61b8621115a44b26a4d229d207dd2110d6c2ecd2f4" Oct 30 12:55:29.040051 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:29.039972 4908 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99bb35caa0eb730f715e9c61b8621115a44b26a4d229d207dd2110d6c2ecd2f4"} err="failed to get container status \"99bb35caa0eb730f715e9c61b8621115a44b26a4d229d207dd2110d6c2ecd2f4\": rpc error: code = NotFound desc = could not find container \"99bb35caa0eb730f715e9c61b8621115a44b26a4d229d207dd2110d6c2ecd2f4\": container with ID starting with 99bb35caa0eb730f715e9c61b8621115a44b26a4d229d207dd2110d6c2ecd2f4 not found: ID does not exist" Oct 30 12:55:29.126630 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:29.126576 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-2s2dq"] Oct 30 12:55:29.131699 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:29.131665 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-2s2dq"] Oct 30 12:55:29.379455 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:29.377219 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-1333c8-openshift-default-847ccc6485-tp95k"] Oct 30 12:55:29.399643 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:55:29.399614 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75b90e54_7d8b_44cf_9e8d_06fe342acc3b.slice/crio-f18e2945f6c50788f0e357058220fdcc1b3f3cd11c762b40a9b275c169e0c5b9 WatchSource:0}: Error finding container f18e2945f6c50788f0e357058220fdcc1b3f3cd11c762b40a9b275c169e0c5b9: Status 404 returned error can't find the container with id f18e2945f6c50788f0e357058220fdcc1b3f3cd11c762b40a9b275c169e0c5b9 Oct 30 12:55:29.408733 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:29.408694 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:55:29.408840 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:29.408787 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:55:29.408901 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:29.408875 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:55:29.511760 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:29.509668 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-e9f184-openshift-default-55dd875b9f-jjt74"] Oct 30 12:55:29.511760 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:29.510271 4908 cpu_manager.go:414] "containerMap: removing container" podUID="c3729561-3035-459d-a451-290595a76f5d" containerName="istio-proxy" Oct 30 12:55:29.511760 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:29.510285 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3729561-3035-459d-a451-290595a76f5d" containerName="istio-proxy" Oct 30 12:55:29.511760 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:29.510407 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3729561-3035-459d-a451-290595a76f5d" containerName="istio-proxy" Oct 30 12:55:29.516242 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:29.516221 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e9f184-openshift-default-55dd875b9f-jjt74" Oct 30 12:55:29.531883 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:29.530790 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-e9f184-openshift-default-55dd875b9f-jjt74"] Oct 30 12:55:29.536369 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:29.536347 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-e9f184-openshift-default-dockercfg-rkzqq\"" Oct 30 12:55:29.854348 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:29.854288 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3729561-3035-459d-a451-290595a76f5d" path="/var/lib/kubelet/pods/c3729561-3035-459d-a451-290595a76f5d/volumes" Oct 30 12:55:29.864856 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:29.856643 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1b34ded-198e-48a2-bd18-303865ac83f3-workload-certs\") pod \"e2e-inf-b-e9f184-openshift-default-55dd875b9f-jjt74\" (UID: \"b1b34ded-198e-48a2-bd18-303865ac83f3\") " pod="openshift-ingress/e2e-inf-b-e9f184-openshift-default-55dd875b9f-jjt74" Oct 30 12:55:29.865197 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:29.865167 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1b34ded-198e-48a2-bd18-303865ac83f3-istio-envoy\") pod \"e2e-inf-b-e9f184-openshift-default-55dd875b9f-jjt74\" (UID: \"b1b34ded-198e-48a2-bd18-303865ac83f3\") " pod="openshift-ingress/e2e-inf-b-e9f184-openshift-default-55dd875b9f-jjt74" Oct 30 12:55:29.865439 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:29.865421 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1b34ded-198e-48a2-bd18-303865ac83f3-istiod-ca-cert\") pod \"e2e-inf-b-e9f184-openshift-default-55dd875b9f-jjt74\" (UID: \"b1b34ded-198e-48a2-bd18-303865ac83f3\") " pod="openshift-ingress/e2e-inf-b-e9f184-openshift-default-55dd875b9f-jjt74" Oct 30 12:55:29.865629 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:29.865612 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1b34ded-198e-48a2-bd18-303865ac83f3-workload-socket\") pod \"e2e-inf-b-e9f184-openshift-default-55dd875b9f-jjt74\" (UID: \"b1b34ded-198e-48a2-bd18-303865ac83f3\") " pod="openshift-ingress/e2e-inf-b-e9f184-openshift-default-55dd875b9f-jjt74" Oct 30 12:55:29.865758 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:29.865743 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1b34ded-198e-48a2-bd18-303865ac83f3-istio-podinfo\") pod \"e2e-inf-b-e9f184-openshift-default-55dd875b9f-jjt74\" (UID: \"b1b34ded-198e-48a2-bd18-303865ac83f3\") " pod="openshift-ingress/e2e-inf-b-e9f184-openshift-default-55dd875b9f-jjt74" Oct 30 12:55:29.871848 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:29.865973 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1b34ded-198e-48a2-bd18-303865ac83f3-istio-data\") pod \"e2e-inf-b-e9f184-openshift-default-55dd875b9f-jjt74\" (UID: \"b1b34ded-198e-48a2-bd18-303865ac83f3\") " pod="openshift-ingress/e2e-inf-b-e9f184-openshift-default-55dd875b9f-jjt74" Oct 30 12:55:29.918994 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:29.915983 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1b34ded-198e-48a2-bd18-303865ac83f3-istio-token\") pod \"e2e-inf-b-e9f184-openshift-default-55dd875b9f-jjt74\" (UID: \"b1b34ded-198e-48a2-bd18-303865ac83f3\") " pod="openshift-ingress/e2e-inf-b-e9f184-openshift-default-55dd875b9f-jjt74" Oct 30 12:55:29.918994 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:29.916085 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45dts\" (UniqueName: \"kubernetes.io/projected/b1b34ded-198e-48a2-bd18-303865ac83f3-kube-api-access-45dts\") pod \"e2e-inf-b-e9f184-openshift-default-55dd875b9f-jjt74\" (UID: \"b1b34ded-198e-48a2-bd18-303865ac83f3\") " pod="openshift-ingress/e2e-inf-b-e9f184-openshift-default-55dd875b9f-jjt74" Oct 30 12:55:29.918994 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:29.916238 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1b34ded-198e-48a2-bd18-303865ac83f3-credential-socket\") pod \"e2e-inf-b-e9f184-openshift-default-55dd875b9f-jjt74\" (UID: \"b1b34ded-198e-48a2-bd18-303865ac83f3\") " pod="openshift-ingress/e2e-inf-b-e9f184-openshift-default-55dd875b9f-jjt74" Oct 30 12:55:30.026524 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:30.017490 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1b34ded-198e-48a2-bd18-303865ac83f3-credential-socket\") pod \"e2e-inf-b-e9f184-openshift-default-55dd875b9f-jjt74\" (UID: \"b1b34ded-198e-48a2-bd18-303865ac83f3\") " pod="openshift-ingress/e2e-inf-b-e9f184-openshift-default-55dd875b9f-jjt74" Oct 30 12:55:30.026524 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:30.017592 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1b34ded-198e-48a2-bd18-303865ac83f3-workload-certs\") pod \"e2e-inf-b-e9f184-openshift-default-55dd875b9f-jjt74\" (UID: \"b1b34ded-198e-48a2-bd18-303865ac83f3\") " pod="openshift-ingress/e2e-inf-b-e9f184-openshift-default-55dd875b9f-jjt74" Oct 30 12:55:30.026524 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:30.017649 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1b34ded-198e-48a2-bd18-303865ac83f3-istio-envoy\") pod \"e2e-inf-b-e9f184-openshift-default-55dd875b9f-jjt74\" (UID: \"b1b34ded-198e-48a2-bd18-303865ac83f3\") " pod="openshift-ingress/e2e-inf-b-e9f184-openshift-default-55dd875b9f-jjt74" Oct 30 12:55:30.026524 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:30.017717 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1b34ded-198e-48a2-bd18-303865ac83f3-istiod-ca-cert\") pod \"e2e-inf-b-e9f184-openshift-default-55dd875b9f-jjt74\" (UID: \"b1b34ded-198e-48a2-bd18-303865ac83f3\") " pod="openshift-ingress/e2e-inf-b-e9f184-openshift-default-55dd875b9f-jjt74" Oct 30 12:55:30.026524 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:30.017790 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1b34ded-198e-48a2-bd18-303865ac83f3-workload-socket\") pod \"e2e-inf-b-e9f184-openshift-default-55dd875b9f-jjt74\" (UID: \"b1b34ded-198e-48a2-bd18-303865ac83f3\") " pod="openshift-ingress/e2e-inf-b-e9f184-openshift-default-55dd875b9f-jjt74" Oct 30 12:55:30.026524 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:30.017824 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1b34ded-198e-48a2-bd18-303865ac83f3-istio-podinfo\") pod \"e2e-inf-b-e9f184-openshift-default-55dd875b9f-jjt74\" (UID: \"b1b34ded-198e-48a2-bd18-303865ac83f3\") " pod="openshift-ingress/e2e-inf-b-e9f184-openshift-default-55dd875b9f-jjt74" Oct 30 12:55:30.026524 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:30.017932 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1b34ded-198e-48a2-bd18-303865ac83f3-istio-data\") pod \"e2e-inf-b-e9f184-openshift-default-55dd875b9f-jjt74\" (UID: \"b1b34ded-198e-48a2-bd18-303865ac83f3\") " pod="openshift-ingress/e2e-inf-b-e9f184-openshift-default-55dd875b9f-jjt74" Oct 30 12:55:30.026524 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:30.017968 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1b34ded-198e-48a2-bd18-303865ac83f3-istio-token\") pod \"e2e-inf-b-e9f184-openshift-default-55dd875b9f-jjt74\" (UID: \"b1b34ded-198e-48a2-bd18-303865ac83f3\") " pod="openshift-ingress/e2e-inf-b-e9f184-openshift-default-55dd875b9f-jjt74" Oct 30 12:55:30.026524 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:30.018033 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45dts\" (UniqueName: \"kubernetes.io/projected/b1b34ded-198e-48a2-bd18-303865ac83f3-kube-api-access-45dts\") pod \"e2e-inf-b-e9f184-openshift-default-55dd875b9f-jjt74\" (UID: \"b1b34ded-198e-48a2-bd18-303865ac83f3\") " pod="openshift-ingress/e2e-inf-b-e9f184-openshift-default-55dd875b9f-jjt74" Oct 30 12:55:30.026524 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:30.018853 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1b34ded-198e-48a2-bd18-303865ac83f3-credential-socket\") pod \"e2e-inf-b-e9f184-openshift-default-55dd875b9f-jjt74\" (UID: \"b1b34ded-198e-48a2-bd18-303865ac83f3\") " pod="openshift-ingress/e2e-inf-b-e9f184-openshift-default-55dd875b9f-jjt74" Oct 30 12:55:30.026524 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:30.019119 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1b34ded-198e-48a2-bd18-303865ac83f3-workload-certs\") pod \"e2e-inf-b-e9f184-openshift-default-55dd875b9f-jjt74\" (UID: \"b1b34ded-198e-48a2-bd18-303865ac83f3\") " pod="openshift-ingress/e2e-inf-b-e9f184-openshift-default-55dd875b9f-jjt74" Oct 30 12:55:30.035538 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:30.035508 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1b34ded-198e-48a2-bd18-303865ac83f3-istiod-ca-cert\") pod \"e2e-inf-b-e9f184-openshift-default-55dd875b9f-jjt74\" (UID: \"b1b34ded-198e-48a2-bd18-303865ac83f3\") " pod="openshift-ingress/e2e-inf-b-e9f184-openshift-default-55dd875b9f-jjt74" Oct 30 12:55:30.036124 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:30.036096 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1b34ded-198e-48a2-bd18-303865ac83f3-workload-socket\") pod \"e2e-inf-b-e9f184-openshift-default-55dd875b9f-jjt74\" (UID: \"b1b34ded-198e-48a2-bd18-303865ac83f3\") " pod="openshift-ingress/e2e-inf-b-e9f184-openshift-default-55dd875b9f-jjt74" Oct 30 12:55:30.045174 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:30.043805 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1b34ded-198e-48a2-bd18-303865ac83f3-istio-data\") pod \"e2e-inf-b-e9f184-openshift-default-55dd875b9f-jjt74\" (UID: \"b1b34ded-198e-48a2-bd18-303865ac83f3\") " pod="openshift-ingress/e2e-inf-b-e9f184-openshift-default-55dd875b9f-jjt74" Oct 30 12:55:30.061556 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:30.050332 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1b34ded-198e-48a2-bd18-303865ac83f3-istio-podinfo\") pod \"e2e-inf-b-e9f184-openshift-default-55dd875b9f-jjt74\" (UID: \"b1b34ded-198e-48a2-bd18-303865ac83f3\") " pod="openshift-ingress/e2e-inf-b-e9f184-openshift-default-55dd875b9f-jjt74" Oct 30 12:55:30.075373 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:30.067358 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1b34ded-198e-48a2-bd18-303865ac83f3-istio-envoy\") pod \"e2e-inf-b-e9f184-openshift-default-55dd875b9f-jjt74\" (UID: \"b1b34ded-198e-48a2-bd18-303865ac83f3\") " pod="openshift-ingress/e2e-inf-b-e9f184-openshift-default-55dd875b9f-jjt74" Oct 30 12:55:30.101852 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:30.097811 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1333c8-openshift-default-847ccc6485-tp95k" event={"ID":"75b90e54-7d8b-44cf-9e8d-06fe342acc3b","Type":"ContainerStarted","Data":"aeb670d117bd2951334cae801bb431a0a884adf89d173a4d11efdaa3c2610696"} Oct 30 12:55:30.101852 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:30.097953 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1333c8-openshift-default-847ccc6485-tp95k" event={"ID":"75b90e54-7d8b-44cf-9e8d-06fe342acc3b","Type":"ContainerStarted","Data":"f18e2945f6c50788f0e357058220fdcc1b3f3cd11c762b40a9b275c169e0c5b9"} Oct 30 12:55:30.113868 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:30.111813 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45dts\" (UniqueName: \"kubernetes.io/projected/b1b34ded-198e-48a2-bd18-303865ac83f3-kube-api-access-45dts\") pod \"e2e-inf-b-e9f184-openshift-default-55dd875b9f-jjt74\" (UID: \"b1b34ded-198e-48a2-bd18-303865ac83f3\") " pod="openshift-ingress/e2e-inf-b-e9f184-openshift-default-55dd875b9f-jjt74" Oct 30 12:55:30.118846 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:30.115435 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1b34ded-198e-48a2-bd18-303865ac83f3-istio-token\") pod \"e2e-inf-b-e9f184-openshift-default-55dd875b9f-jjt74\" (UID: \"b1b34ded-198e-48a2-bd18-303865ac83f3\") " pod="openshift-ingress/e2e-inf-b-e9f184-openshift-default-55dd875b9f-jjt74" Oct 30 12:55:30.138204 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:30.137934 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e9f184-openshift-default-55dd875b9f-jjt74" Oct 30 12:55:30.146948 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:30.145708 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-1333c8-openshift-default-847ccc6485-tp95k" podStartSLOduration=2.145689312 podStartE2EDuration="2.145689312s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:55:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:55:30.142018634 +0000 UTC m=+3686.656447381" watchObservedRunningTime="2026-10-30 12:55:30.145689312 +0000 UTC m=+3686.660118060" Oct 30 12:55:30.810919 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:30.809585 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-e9f184-openshift-default-55dd875b9f-jjt74"] Oct 30 12:55:30.813749 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:55:30.813057 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1b34ded_198e_48a2_bd18_303865ac83f3.slice/crio-7e4dd14598c2e6c2b6775b150517e904160928faf5c67b22c7859665112ff9e5 WatchSource:0}: Error finding container 7e4dd14598c2e6c2b6775b150517e904160928faf5c67b22c7859665112ff9e5: Status 404 returned error can't find the container with id 7e4dd14598c2e6c2b6775b150517e904160928faf5c67b22c7859665112ff9e5 Oct 30 12:55:30.817326 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:30.817284 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:55:30.817549 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:30.817526 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:55:30.817683 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:30.817665 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:55:30.911898 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:30.911866 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-1333c8-openshift-default-847ccc6485-tp95k" Oct 30 12:55:30.918163 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:30.918131 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-1333c8-openshift-default-847ccc6485-tp95k" Oct 30 12:55:31.126159 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:31.126039 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e9f184-openshift-default-55dd875b9f-jjt74" event={"ID":"b1b34ded-198e-48a2-bd18-303865ac83f3","Type":"ContainerStarted","Data":"bc985fb1bf4ea590b514fce128c27da55759701718af051ba7efcee15a3e13a8"} Oct 30 12:55:31.126159 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:31.126105 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e9f184-openshift-default-55dd875b9f-jjt74" event={"ID":"b1b34ded-198e-48a2-bd18-303865ac83f3","Type":"ContainerStarted","Data":"7e4dd14598c2e6c2b6775b150517e904160928faf5c67b22c7859665112ff9e5"} Oct 30 12:55:31.126493 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:31.126469 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-1333c8-openshift-default-847ccc6485-tp95k" Oct 30 12:55:31.127922 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:31.127894 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-1333c8-openshift-default-847ccc6485-tp95k" Oct 30 12:55:31.138872 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:31.138809 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-e9f184-openshift-default-55dd875b9f-jjt74" Oct 30 12:55:31.139864 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:31.139823 4908 patch_prober.go:29] interesting pod/e2e-inf-b-e9f184-openshift-default-55dd875b9f-jjt74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.34:15021/healthz/ready\": dial tcp 10.132.1.34:15021: connect: connection refused" start-of-body= Oct 30 12:55:31.139961 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:31.139881 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e9f184-openshift-default-55dd875b9f-jjt74" podUID="b1b34ded-198e-48a2-bd18-303865ac83f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.34:15021/healthz/ready\": dial tcp 10.132.1.34:15021: connect: connection refused" Oct 30 12:55:31.193686 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:31.193603 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-e9f184-openshift-default-55dd875b9f-jjt74" podStartSLOduration=2.193583967 podStartE2EDuration="2.193583967s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:55:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:55:31.186404988 +0000 UTC m=+3687.700833734" watchObservedRunningTime="2026-10-30 12:55:31.193583967 +0000 UTC m=+3687.708012713" Oct 30 12:55:32.144936 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:32.144907 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-e9f184-openshift-default-55dd875b9f-jjt74" Oct 30 12:55:33.142318 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:33.142281 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-e9f184-openshift-default-55dd875b9f-jjt74" Oct 30 12:55:33.143229 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:33.143207 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-e9f184-openshift-default-55dd875b9f-jjt74" Oct 30 12:55:34.033627 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:34.033593 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-163fb611-2kpzz"] Oct 30 12:55:34.044367 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:34.044341 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-163fb611-2kpzz"] Oct 30 12:55:35.634850 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:35.626613 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3911335-4942-4c03-b240-06a51527ed22" path="/var/lib/kubelet/pods/a3911335-4942-4c03-b240-06a51527ed22/volumes" Oct 30 12:55:36.923028 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:36.922985 4908 patch_prober.go:29] interesting pod/e2e-inf-a-eab799-openshift-default-574ddfd59c-tfw7h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.24:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 12:55:36.923547 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:36.923046 4908 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-eab799-openshift-default-574ddfd59c-tfw7h" podUID="61f8676a-8552-4066-a292-26bbd8b64d6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.24:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 12:55:39.329848 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:39.329599 4908 generic.go:356] "Generic (PLEG): container finished" podID="61f8676a-8552-4066-a292-26bbd8b64d6b" containerID="13fa47b6b1bc9c45ac4920d0db4a61a189ed3bd195fc322079723c64eb6e4f0f" exitCode=0 Oct 30 12:55:39.329848 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:39.329791 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-eab799-openshift-default-574ddfd59c-tfw7h" event={"ID":"61f8676a-8552-4066-a292-26bbd8b64d6b","Type":"ContainerDied","Data":"13fa47b6b1bc9c45ac4920d0db4a61a189ed3bd195fc322079723c64eb6e4f0f"} Oct 30 12:55:39.330506 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:39.330475 4908 scope.go:122] "RemoveContainer" containerID="13fa47b6b1bc9c45ac4920d0db4a61a189ed3bd195fc322079723c64eb6e4f0f" Oct 30 12:55:39.335359 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:39.334574 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:55:39.335359 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:39.334716 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:55:39.335359 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:39.334767 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:55:40.339722 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:40.339683 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-eab799-openshift-default-574ddfd59c-tfw7h" event={"ID":"61f8676a-8552-4066-a292-26bbd8b64d6b","Type":"ContainerStarted","Data":"84fee92755c2dd385d20823190e7b6344f9f725ba5dd89e3ad8630a2419be57d"} Oct 30 12:55:40.918084 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:40.918045 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-eab799-openshift-default-574ddfd59c-tfw7h" Oct 30 12:55:40.918296 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:40.918216 4908 patch_prober.go:29] interesting pod/e2e-inf-a-eab799-openshift-default-574ddfd59c-tfw7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.24:15021/healthz/ready\": dial tcp 10.132.1.24:15021: connect: connection refused" start-of-body= Oct 30 12:55:40.918296 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:40.918272 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-eab799-openshift-default-574ddfd59c-tfw7h" podUID="61f8676a-8552-4066-a292-26bbd8b64d6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.24:15021/healthz/ready\": dial tcp 10.132.1.24:15021: connect: connection refused" Oct 30 12:55:41.918728 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:41.918693 4908 patch_prober.go:29] interesting pod/e2e-inf-a-eab799-openshift-default-574ddfd59c-tfw7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.24:15021/healthz/ready\": dial tcp 10.132.1.24:15021: connect: connection refused" start-of-body= Oct 30 12:55:41.919171 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:41.918749 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-eab799-openshift-default-574ddfd59c-tfw7h" podUID="61f8676a-8552-4066-a292-26bbd8b64d6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.24:15021/healthz/ready\": dial tcp 10.132.1.24:15021: connect: connection refused" Oct 30 12:55:42.918873 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:42.918807 4908 patch_prober.go:29] interesting pod/e2e-inf-a-eab799-openshift-default-574ddfd59c-tfw7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.24:15021/healthz/ready\": dial tcp 10.132.1.24:15021: connect: connection refused" start-of-body= Oct 30 12:55:42.919289 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:42.918894 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-eab799-openshift-default-574ddfd59c-tfw7h" podUID="61f8676a-8552-4066-a292-26bbd8b64d6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.24:15021/healthz/ready\": dial tcp 10.132.1.24:15021: connect: connection refused" Oct 30 12:55:43.918339 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:43.918304 4908 patch_prober.go:29] interesting pod/e2e-inf-a-eab799-openshift-default-574ddfd59c-tfw7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.24:15021/healthz/ready\": dial tcp 10.132.1.24:15021: connect: connection refused" start-of-body= Oct 30 12:55:43.918564 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:43.918365 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-eab799-openshift-default-574ddfd59c-tfw7h" podUID="61f8676a-8552-4066-a292-26bbd8b64d6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.24:15021/healthz/ready\": dial tcp 10.132.1.24:15021: connect: connection refused" Oct 30 12:55:44.918713 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:44.918677 4908 patch_prober.go:29] interesting pod/e2e-inf-a-eab799-openshift-default-574ddfd59c-tfw7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.24:15021/healthz/ready\": dial tcp 10.132.1.24:15021: connect: connection refused" start-of-body= Oct 30 12:55:44.919224 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:44.918730 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-eab799-openshift-default-574ddfd59c-tfw7h" podUID="61f8676a-8552-4066-a292-26bbd8b64d6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.24:15021/healthz/ready\": dial tcp 10.132.1.24:15021: connect: connection refused" Oct 30 12:55:45.072938 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:45.072895 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-abbf0116-zplfz"] Oct 30 12:55:45.153608 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:45.153569 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-abbf0116-zplfz"] Oct 30 12:55:45.608631 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:45.608591 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="774163b0-110d-4239-80f9-941f9edd4351" path="/var/lib/kubelet/pods/774163b0-110d-4239-80f9-941f9edd4351/volumes" Oct 30 12:55:45.923385 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:45.923307 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-eab799-openshift-default-574ddfd59c-tfw7h" Oct 30 12:55:45.923893 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:45.923818 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-eab799-openshift-default-574ddfd59c-tfw7h" Oct 30 12:55:45.924487 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:55:45.924464 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-eab799-openshift-default-574ddfd59c-tfw7h" Oct 30 12:55:49.685590 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:55:49.685538 4908 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vn6vv" containerName="extractor" Oct 30 12:56:00.246851 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:00.246300 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9wwfc"] Oct 30 12:56:03.550968 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:03.550915 4908 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47fd99/distinct-2-w2-47fd99-kserve-77cf84b469-zxs8s" podUID="baf44d3d-42e6-43f7-a0ad-f0f9932320c1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:56:03.596367 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:03.596314 4908 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6a7148/premium-w3-6a7148-kserve-67b976c888-tgd6c" podUID="13de4441-e7f6-4cbd-97d9-0f079d11cb8e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:56:03.616912 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:03.599017 4908 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f920f7/simulator-w0-f920f7-kserve-599b7ff45f-g79d7" podUID="a85e562f-d655-4c22-b4f4-a299a4e494a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:56:03.735859 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:03.730348 4908 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-512ec5/simulator-w4-512ec5-kserve-7df7674f8-8rqwb" podUID="8a5dfcc9-dec3-4bc6-b0a8-c928350b3596" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:56:03.827019 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:03.826915 4908 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47fd99/premium-w2-47fd99-kserve-5d85f9bb95-hqkmf" podUID="fd2cbeda-dbc4-41f6-a730-e1c6f9343c5e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Oct 30 12:56:03.946966 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:03.946915 4908 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6a7148/distinct-2-w3-6a7148-kserve-67bd5fb976-9nfff" podUID="6046cc02-29fe-4f59-b522-8f57f4557277" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:56:04.154913 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:04.152910 4908 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-512ec5/simulator-w4-512ec5-kserve-7df7674f8-8rqwb" podUID="8a5dfcc9-dec3-4bc6-b0a8-c928350b3596" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:56:04.562033 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:04.561907 4908 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47fd99/distinct-2-w2-47fd99-kserve-77cf84b469-zxs8s" podUID="baf44d3d-42e6-43f7-a0ad-f0f9932320c1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 30 12:56:04.562954 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:04.562929 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-47fd99/distinct-2-w2-47fd99-kserve-77cf84b469-zxs8s" Oct 30 12:56:05.022455 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:05.022418 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-47fd99/distinct-2-w2-47fd99-kserve-77cf84b469-zxs8s" Oct 30 12:56:05.686185 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:05.686136 4908 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f920f7/simulator-w0-f920f7-kserve-599b7ff45f-g79d7" podUID="a85e562f-d655-4c22-b4f4-a299a4e494a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 12:56:05.855335 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:05.855291 4908 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47fd99/premium-w2-47fd99-kserve-5d85f9bb95-hqkmf" podUID="fd2cbeda-dbc4-41f6-a730-e1c6f9343c5e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Oct 30 12:56:05.947515 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:05.947409 4908 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6a7148/distinct-2-w3-6a7148-kserve-67bd5fb976-9nfff" podUID="6046cc02-29fe-4f59-b522-8f57f4557277" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:56:06.034311 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:06.031914 4908 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47fd99/distinct-2-w2-47fd99-kserve-77cf84b469-zxs8s" podUID="baf44d3d-42e6-43f7-a0ad-f0f9932320c1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:56:06.161854 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:06.146918 4908 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-47fd99/distinct-2-w2-47fd99-kserve-77cf84b469-zxs8s" podUID="baf44d3d-42e6-43f7-a0ad-f0f9932320c1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:56:06.161854 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:06.152062 4908 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-512ec5/simulator-w4-512ec5-kserve-7df7674f8-8rqwb" podUID="8a5dfcc9-dec3-4bc6-b0a8-c928350b3596" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:56:06.516853 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:06.513951 4908 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-6a7148/premium-w3-6a7148-kserve-67b976c888-tgd6c" podUID="13de4441-e7f6-4cbd-97d9-0f079d11cb8e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:56:06.597853 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:06.594669 4908 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6a7148/premium-w3-6a7148-kserve-67b976c888-tgd6c" podUID="13de4441-e7f6-4cbd-97d9-0f079d11cb8e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:56:06.714656 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:06.714605 4908 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f920f7/simulator-w0-f920f7-kserve-599b7ff45f-g79d7" podUID="a85e562f-d655-4c22-b4f4-a299a4e494a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:56:06.758143 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:06.758112 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f920f7/simulator-w0-f920f7-kserve-599b7ff45f-g79d7" Oct 30 12:56:07.092883 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:07.069918 4908 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47fd99/distinct-2-w2-47fd99-kserve-77cf84b469-zxs8s" podUID="baf44d3d-42e6-43f7-a0ad-f0f9932320c1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 12:56:07.092883 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:07.070648 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-47fd99/distinct-2-w2-47fd99-kserve-77cf84b469-zxs8s" Oct 30 12:56:07.178708 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:07.178665 4908 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-512ec5/simulator-w4-512ec5-kserve-7df7674f8-8rqwb" podUID="8a5dfcc9-dec3-4bc6-b0a8-c928350b3596" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:56:07.180800 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:07.180776 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-512ec5/simulator-w4-512ec5-kserve-7df7674f8-8rqwb" Oct 30 12:56:07.181031 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:07.181017 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f920f7/simulator-w0-f920f7-kserve-599b7ff45f-g79d7" Oct 30 12:56:07.248359 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:07.248328 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-512ec5/simulator-w4-512ec5-kserve-7df7674f8-8rqwb" Oct 30 12:56:07.350997 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:07.350869 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-47fd99/distinct-2-w2-47fd99-kserve-77cf84b469-zxs8s" Oct 30 12:56:08.565853 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:08.559956 4908 scope.go:122] "RemoveContainer" containerID="c96370d1fb73ea1d884ef0b4b74f994a1a389e6f0f0f6a0c92c8f86eb74b5691" Oct 30 12:56:08.620045 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:08.620014 4908 scope.go:122] "RemoveContainer" containerID="f061546f2cdff92d61853e84d54a5e8d548606aeb2ce06a0498adac38b769692" Oct 30 12:56:14.392525 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:14.392486 4908 generic.go:356] "Generic (PLEG): container finished" podID="f1ca551b-2beb-407b-9e9c-e410199f6f33" containerID="48dc6ab44528d0f5a5162fc8b89d173676851668e94c506a2ab0d5f16d0cb238" exitCode=0 Oct 30 12:56:14.393443 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:14.392565 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-63f4e4-openshift-default-786cd844c9-lxsbx" event={"ID":"f1ca551b-2beb-407b-9e9c-e410199f6f33","Type":"ContainerDied","Data":"48dc6ab44528d0f5a5162fc8b89d173676851668e94c506a2ab0d5f16d0cb238"} Oct 30 12:56:14.393443 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:14.393134 4908 scope.go:122] "RemoveContainer" containerID="48dc6ab44528d0f5a5162fc8b89d173676851668e94c506a2ab0d5f16d0cb238" Oct 30 12:56:14.395132 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:14.395108 4908 generic.go:356] "Generic (PLEG): container finished" podID="0ab626b9-027f-4799-ba80-e47276070fdb" containerID="3c174bcf714e7781bff416a5de8a3c29a0a190b20f792f857b4e573918540ef7" exitCode=0 Oct 30 12:56:14.395247 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:14.395230 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6b67f8-openshift-default-696ff8b6f-6t5q2" event={"ID":"0ab626b9-027f-4799-ba80-e47276070fdb","Type":"ContainerDied","Data":"3c174bcf714e7781bff416a5de8a3c29a0a190b20f792f857b4e573918540ef7"} Oct 30 12:56:14.395437 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:14.395405 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:56:14.395532 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:14.395519 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:56:14.395598 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:14.395566 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:56:14.395660 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:14.395635 4908 scope.go:122] "RemoveContainer" containerID="3c174bcf714e7781bff416a5de8a3c29a0a190b20f792f857b4e573918540ef7" Oct 30 12:56:14.398028 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:14.397993 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:56:14.398113 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:14.398081 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:56:14.398159 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:14.398130 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:56:14.849576 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:14.849540 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e8e9b9f-4wkgh"] Oct 30 12:56:14.852798 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:14.852781 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e8e9b9f-4wkgh" Oct 30 12:56:14.867405 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:14.867383 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 12:56:14.867706 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:14.867675 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 12:56:14.868544 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:14.867954 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 12:56:14.868544 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:14.867958 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s75hq\"" Oct 30 12:56:14.879216 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:14.879189 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e8e9b9f-4wkgh"] Oct 30 12:56:14.937982 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:14.930649 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/085a75a0-26d8-4f54-97aa-cbf5536cfcf6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e8e9b9f-4wkgh\" (UID: \"085a75a0-26d8-4f54-97aa-cbf5536cfcf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e8e9b9f-4wkgh" Oct 30 12:56:14.937982 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:14.930711 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/085a75a0-26d8-4f54-97aa-cbf5536cfcf6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e8e9b9f-4wkgh\" (UID: \"085a75a0-26d8-4f54-97aa-cbf5536cfcf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e8e9b9f-4wkgh" Oct 30 12:56:14.937982 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:14.930900 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lfql\" (UniqueName: \"kubernetes.io/projected/085a75a0-26d8-4f54-97aa-cbf5536cfcf6-kube-api-access-5lfql\") pod \"maas-api-revoke-sub-0e8e9b9f-4wkgh\" (UID: \"085a75a0-26d8-4f54-97aa-cbf5536cfcf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e8e9b9f-4wkgh" Oct 30 12:56:15.032476 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:15.032434 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lfql\" (UniqueName: \"kubernetes.io/projected/085a75a0-26d8-4f54-97aa-cbf5536cfcf6-kube-api-access-5lfql\") pod \"maas-api-revoke-sub-0e8e9b9f-4wkgh\" (UID: \"085a75a0-26d8-4f54-97aa-cbf5536cfcf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e8e9b9f-4wkgh" Oct 30 12:56:15.032677 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:15.032626 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/085a75a0-26d8-4f54-97aa-cbf5536cfcf6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e8e9b9f-4wkgh\" (UID: \"085a75a0-26d8-4f54-97aa-cbf5536cfcf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e8e9b9f-4wkgh" Oct 30 12:56:15.032762 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:15.032677 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/085a75a0-26d8-4f54-97aa-cbf5536cfcf6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e8e9b9f-4wkgh\" (UID: \"085a75a0-26d8-4f54-97aa-cbf5536cfcf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e8e9b9f-4wkgh" Oct 30 12:56:15.033413 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:15.033384 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/085a75a0-26d8-4f54-97aa-cbf5536cfcf6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e8e9b9f-4wkgh\" (UID: \"085a75a0-26d8-4f54-97aa-cbf5536cfcf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e8e9b9f-4wkgh" Oct 30 12:56:15.035102 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:15.035079 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/085a75a0-26d8-4f54-97aa-cbf5536cfcf6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e8e9b9f-4wkgh\" (UID: \"085a75a0-26d8-4f54-97aa-cbf5536cfcf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e8e9b9f-4wkgh" Oct 30 12:56:15.054787 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:15.054756 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lfql\" (UniqueName: \"kubernetes.io/projected/085a75a0-26d8-4f54-97aa-cbf5536cfcf6-kube-api-access-5lfql\") pod \"maas-api-revoke-sub-0e8e9b9f-4wkgh\" (UID: \"085a75a0-26d8-4f54-97aa-cbf5536cfcf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e8e9b9f-4wkgh" Oct 30 12:56:15.168288 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:15.168209 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e8e9b9f-4wkgh" Oct 30 12:56:15.407785 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:15.407742 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6b67f8-openshift-default-696ff8b6f-6t5q2" event={"ID":"0ab626b9-027f-4799-ba80-e47276070fdb","Type":"ContainerStarted","Data":"f5aca2adf11b10aa6418e97b0c907a75af90a650dd5fe67b5c443a53b26f7af3"} Oct 30 12:56:15.411539 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:15.411507 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-63f4e4-openshift-default-786cd844c9-lxsbx" event={"ID":"f1ca551b-2beb-407b-9e9c-e410199f6f33","Type":"ContainerStarted","Data":"b38b1af2792d7db04419d6757727cf8f5967968075a7a33b871fcfbe36c3fa02"} Oct 30 12:56:15.517890 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:15.517859 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-63f4e4-openshift-default-786cd844c9-lxsbx" Oct 30 12:56:15.528046 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:15.522664 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-63f4e4-openshift-default-786cd844c9-lxsbx" Oct 30 12:56:15.532850 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:15.528956 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e8e9b9f-4wkgh"] Oct 30 12:56:15.532850 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:15.529749 4908 patch_prober.go:29] interesting pod/e2e-inf-a-63f4e4-openshift-default-786cd844c9-lxsbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.30:15021/healthz/ready\": dial tcp 10.132.1.30:15021: connect: connection refused" start-of-body= Oct 30 12:56:15.532850 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:15.529801 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-63f4e4-openshift-default-786cd844c9-lxsbx" podUID="f1ca551b-2beb-407b-9e9c-e410199f6f33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.30:15021/healthz/ready\": dial tcp 10.132.1.30:15021: connect: connection refused" Oct 30 12:56:16.302852 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:16.297271 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-6b67f8-openshift-default-696ff8b6f-6t5q2" Oct 30 12:56:16.302852 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:16.297423 4908 patch_prober.go:29] interesting pod/e2e-inf-a-6b67f8-openshift-default-696ff8b6f-6t5q2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused" start-of-body= Oct 30 12:56:16.302852 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:16.297483 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6b67f8-openshift-default-696ff8b6f-6t5q2" podUID="0ab626b9-027f-4799-ba80-e47276070fdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused" Oct 30 12:56:16.440878 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:16.440847 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e8e9b9f-4wkgh" event={"ID":"085a75a0-26d8-4f54-97aa-cbf5536cfcf6","Type":"ContainerStarted","Data":"3296a51a15a59accfb37f0639fb4a0b2097a7a55ec9cd6f757e0d46812afdb57"} Oct 30 12:56:16.441865 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:16.441843 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e8e9b9f-4wkgh" event={"ID":"085a75a0-26d8-4f54-97aa-cbf5536cfcf6","Type":"ContainerStarted","Data":"531cf91593cacae0e39450e0c167fa369525586ce27ef98520a1e1f8475aa521"} Oct 30 12:56:16.499853 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:16.498869 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e8e9b9f-4wkgh" podStartSLOduration=2.498821871 podStartE2EDuration="2.498821871s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:56:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:56:16.483348734 +0000 UTC m=+3732.997777504" watchObservedRunningTime="2026-10-30 12:56:16.498821871 +0000 UTC m=+3733.013250618" Oct 30 12:56:16.521849 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:16.521653 4908 patch_prober.go:29] interesting pod/e2e-inf-a-63f4e4-openshift-default-786cd844c9-lxsbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.30:15021/healthz/ready\": dial tcp 10.132.1.30:15021: connect: connection refused" start-of-body= Oct 30 12:56:16.521849 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:16.521716 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-63f4e4-openshift-default-786cd844c9-lxsbx" podUID="f1ca551b-2beb-407b-9e9c-e410199f6f33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.30:15021/healthz/ready\": dial tcp 10.132.1.30:15021: connect: connection refused" Oct 30 12:56:17.309016 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:17.308971 4908 patch_prober.go:29] interesting pod/e2e-inf-a-6b67f8-openshift-default-696ff8b6f-6t5q2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused" start-of-body= Oct 30 12:56:17.309288 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:17.309255 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6b67f8-openshift-default-696ff8b6f-6t5q2" podUID="0ab626b9-027f-4799-ba80-e47276070fdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused" Oct 30 12:56:17.522902 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:17.499195 4908 generic.go:356] "Generic (PLEG): container finished" podID="085a75a0-26d8-4f54-97aa-cbf5536cfcf6" containerID="3296a51a15a59accfb37f0639fb4a0b2097a7a55ec9cd6f757e0d46812afdb57" exitCode=0 Oct 30 12:56:17.522902 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:17.502586 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e8e9b9f-4wkgh" event={"ID":"085a75a0-26d8-4f54-97aa-cbf5536cfcf6","Type":"ContainerDied","Data":"3296a51a15a59accfb37f0639fb4a0b2097a7a55ec9cd6f757e0d46812afdb57"} Oct 30 12:56:17.522902 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:17.522232 4908 patch_prober.go:29] interesting pod/e2e-inf-a-63f4e4-openshift-default-786cd844c9-lxsbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.30:15021/healthz/ready\": dial tcp 10.132.1.30:15021: connect: connection refused" start-of-body= Oct 30 12:56:17.522902 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:17.522315 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-63f4e4-openshift-default-786cd844c9-lxsbx" podUID="f1ca551b-2beb-407b-9e9c-e410199f6f33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.30:15021/healthz/ready\": dial tcp 10.132.1.30:15021: connect: connection refused" Oct 30 12:56:18.301853 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:18.301208 4908 patch_prober.go:29] interesting pod/e2e-inf-a-6b67f8-openshift-default-696ff8b6f-6t5q2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused" start-of-body= Oct 30 12:56:18.301853 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:18.301270 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6b67f8-openshift-default-696ff8b6f-6t5q2" podUID="0ab626b9-027f-4799-ba80-e47276070fdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused" Oct 30 12:56:18.524611 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:18.523963 4908 patch_prober.go:29] interesting pod/e2e-inf-a-63f4e4-openshift-default-786cd844c9-lxsbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.30:15021/healthz/ready\": dial tcp 10.132.1.30:15021: connect: connection refused" start-of-body= Oct 30 12:56:18.524611 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:18.524043 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-63f4e4-openshift-default-786cd844c9-lxsbx" podUID="f1ca551b-2beb-407b-9e9c-e410199f6f33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.30:15021/healthz/ready\": dial tcp 10.132.1.30:15021: connect: connection refused" Oct 30 12:56:18.931185 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:18.931110 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e8e9b9f-4wkgh" Oct 30 12:56:19.034301 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:19.034241 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/085a75a0-26d8-4f54-97aa-cbf5536cfcf6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/085a75a0-26d8-4f54-97aa-cbf5536cfcf6-maas-api-openshift-ca\") pod \"085a75a0-26d8-4f54-97aa-cbf5536cfcf6\" (UID: \"085a75a0-26d8-4f54-97aa-cbf5536cfcf6\") " Oct 30 12:56:19.034475 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:19.034409 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/085a75a0-26d8-4f54-97aa-cbf5536cfcf6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/085a75a0-26d8-4f54-97aa-cbf5536cfcf6-maas-api-xks-ca\") pod \"085a75a0-26d8-4f54-97aa-cbf5536cfcf6\" (UID: \"085a75a0-26d8-4f54-97aa-cbf5536cfcf6\") " Oct 30 12:56:19.034475 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:19.034465 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/085a75a0-26d8-4f54-97aa-cbf5536cfcf6-kube-api-access-5lfql\" (UniqueName: \"kubernetes.io/projected/085a75a0-26d8-4f54-97aa-cbf5536cfcf6-kube-api-access-5lfql\") pod \"085a75a0-26d8-4f54-97aa-cbf5536cfcf6\" (UID: \"085a75a0-26d8-4f54-97aa-cbf5536cfcf6\") " Oct 30 12:56:19.035639 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:19.035588 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/085a75a0-26d8-4f54-97aa-cbf5536cfcf6-maas-api-openshift-ca" pod "085a75a0-26d8-4f54-97aa-cbf5536cfcf6" (UID: "085a75a0-26d8-4f54-97aa-cbf5536cfcf6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:56:19.037703 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:19.037359 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/085a75a0-26d8-4f54-97aa-cbf5536cfcf6-maas-api-xks-ca" pod "085a75a0-26d8-4f54-97aa-cbf5536cfcf6" (UID: "085a75a0-26d8-4f54-97aa-cbf5536cfcf6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:56:19.037703 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:19.037502 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/085a75a0-26d8-4f54-97aa-cbf5536cfcf6-kube-api-access-5lfql" pod "085a75a0-26d8-4f54-97aa-cbf5536cfcf6" (UID: "085a75a0-26d8-4f54-97aa-cbf5536cfcf6"). InnerVolumeSpecName "kube-api-access-5lfql". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:56:19.135976 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:19.135942 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5lfql\" (UniqueName: \"kubernetes.io/projected/085a75a0-26d8-4f54-97aa-cbf5536cfcf6-kube-api-access-5lfql\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:56:19.135976 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:19.135974 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/085a75a0-26d8-4f54-97aa-cbf5536cfcf6-maas-api-openshift-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:56:19.136195 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:19.135988 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/085a75a0-26d8-4f54-97aa-cbf5536cfcf6-maas-api-xks-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 12:56:19.323856 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:19.321959 4908 patch_prober.go:29] interesting pod/e2e-inf-a-6b67f8-openshift-default-696ff8b6f-6t5q2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused" start-of-body= Oct 30 12:56:19.323856 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:19.322025 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6b67f8-openshift-default-696ff8b6f-6t5q2" podUID="0ab626b9-027f-4799-ba80-e47276070fdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused" Oct 30 12:56:19.530872 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:19.522264 4908 patch_prober.go:29] interesting pod/e2e-inf-a-63f4e4-openshift-default-786cd844c9-lxsbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.30:15021/healthz/ready\": dial tcp 10.132.1.30:15021: connect: connection refused" start-of-body= Oct 30 12:56:19.530872 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:19.522343 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-63f4e4-openshift-default-786cd844c9-lxsbx" podUID="f1ca551b-2beb-407b-9e9c-e410199f6f33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.30:15021/healthz/ready\": dial tcp 10.132.1.30:15021: connect: connection refused" Oct 30 12:56:19.549322 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:19.549183 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e8e9b9f-4wkgh" Oct 30 12:56:19.549636 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:19.549315 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e8e9b9f-4wkgh" event={"ID":"085a75a0-26d8-4f54-97aa-cbf5536cfcf6","Type":"ContainerDied","Data":"531cf91593cacae0e39450e0c167fa369525586ce27ef98520a1e1f8475aa521"} Oct 30 12:56:19.549742 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:19.549672 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="531cf91593cacae0e39450e0c167fa369525586ce27ef98520a1e1f8475aa521" Oct 30 12:56:19.750932 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:56:19.750898 4908 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vn6vv" containerName="extractor" Oct 30 12:56:20.295851 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:20.295271 4908 patch_prober.go:29] interesting pod/e2e-inf-a-6b67f8-openshift-default-696ff8b6f-6t5q2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused" start-of-body= Oct 30 12:56:20.295851 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:20.295350 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6b67f8-openshift-default-696ff8b6f-6t5q2" podUID="0ab626b9-027f-4799-ba80-e47276070fdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused" Oct 30 12:56:20.525852 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:20.521594 4908 patch_prober.go:29] interesting pod/e2e-inf-a-63f4e4-openshift-default-786cd844c9-lxsbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.30:15021/healthz/ready\": dial tcp 10.132.1.30:15021: connect: connection refused" start-of-body= Oct 30 12:56:20.525852 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:20.521686 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-63f4e4-openshift-default-786cd844c9-lxsbx" podUID="f1ca551b-2beb-407b-9e9c-e410199f6f33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.30:15021/healthz/ready\": dial tcp 10.132.1.30:15021: connect: connection refused" Oct 30 12:56:21.321802 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:21.321680 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-6b67f8-openshift-default-696ff8b6f-6t5q2" Oct 30 12:56:21.323261 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:21.323089 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-6b67f8-openshift-default-696ff8b6f-6t5q2" Oct 30 12:56:21.324041 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:21.324018 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-6b67f8-openshift-default-696ff8b6f-6t5q2" Oct 30 12:56:21.538743 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:21.538712 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-63f4e4-openshift-default-786cd844c9-lxsbx" Oct 30 12:56:21.539873 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:21.539812 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-63f4e4-openshift-default-786cd844c9-lxsbx" Oct 30 12:56:34.274170 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:34.274138 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-734ffe-openshift-default-5db45fbc99-5vzmr"] Oct 30 12:56:34.275033 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:34.275010 4908 cpu_manager.go:414] "containerMap: removing container" podUID="085a75a0-26d8-4f54-97aa-cbf5536cfcf6" containerName="revoke-keys" Oct 30 12:56:34.275033 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:34.275032 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="085a75a0-26d8-4f54-97aa-cbf5536cfcf6" containerName="revoke-keys" Oct 30 12:56:34.275246 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:34.275231 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="085a75a0-26d8-4f54-97aa-cbf5536cfcf6" containerName="revoke-keys" Oct 30 12:56:34.278580 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:34.278559 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-734ffe-openshift-default-5db45fbc99-5vzmr" Oct 30 12:56:34.297660 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:34.297635 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-734ffe-openshift-default-dockercfg-qwqdq\"" Oct 30 12:56:34.316116 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:34.316083 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-734ffe-openshift-default-5db45fbc99-5vzmr"] Oct 30 12:56:34.404849 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:34.401857 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e96c960-abe0-4b56-b3f9-ce46f2ce307a-workload-certs\") pod \"e2e-inf-b-734ffe-openshift-default-5db45fbc99-5vzmr\" (UID: \"7e96c960-abe0-4b56-b3f9-ce46f2ce307a\") " pod="openshift-ingress/e2e-inf-b-734ffe-openshift-default-5db45fbc99-5vzmr" Oct 30 12:56:34.404849 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:34.401924 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e96c960-abe0-4b56-b3f9-ce46f2ce307a-credential-socket\") pod \"e2e-inf-b-734ffe-openshift-default-5db45fbc99-5vzmr\" (UID: \"7e96c960-abe0-4b56-b3f9-ce46f2ce307a\") " pod="openshift-ingress/e2e-inf-b-734ffe-openshift-default-5db45fbc99-5vzmr" Oct 30 12:56:34.421853 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:34.414948 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e96c960-abe0-4b56-b3f9-ce46f2ce307a-istio-envoy\") pod \"e2e-inf-b-734ffe-openshift-default-5db45fbc99-5vzmr\" (UID: \"7e96c960-abe0-4b56-b3f9-ce46f2ce307a\") " pod="openshift-ingress/e2e-inf-b-734ffe-openshift-default-5db45fbc99-5vzmr" Oct 30 12:56:34.421853 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:34.415130 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e96c960-abe0-4b56-b3f9-ce46f2ce307a-istiod-ca-cert\") pod \"e2e-inf-b-734ffe-openshift-default-5db45fbc99-5vzmr\" (UID: \"7e96c960-abe0-4b56-b3f9-ce46f2ce307a\") " pod="openshift-ingress/e2e-inf-b-734ffe-openshift-default-5db45fbc99-5vzmr" Oct 30 12:56:34.421853 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:34.415186 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e96c960-abe0-4b56-b3f9-ce46f2ce307a-istio-token\") pod \"e2e-inf-b-734ffe-openshift-default-5db45fbc99-5vzmr\" (UID: \"7e96c960-abe0-4b56-b3f9-ce46f2ce307a\") " pod="openshift-ingress/e2e-inf-b-734ffe-openshift-default-5db45fbc99-5vzmr" Oct 30 12:56:34.421853 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:34.415249 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e96c960-abe0-4b56-b3f9-ce46f2ce307a-istio-data\") pod \"e2e-inf-b-734ffe-openshift-default-5db45fbc99-5vzmr\" (UID: \"7e96c960-abe0-4b56-b3f9-ce46f2ce307a\") " pod="openshift-ingress/e2e-inf-b-734ffe-openshift-default-5db45fbc99-5vzmr" Oct 30 12:56:34.421853 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:34.415305 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9pzp\" (UniqueName: \"kubernetes.io/projected/7e96c960-abe0-4b56-b3f9-ce46f2ce307a-kube-api-access-q9pzp\") pod \"e2e-inf-b-734ffe-openshift-default-5db45fbc99-5vzmr\" (UID: \"7e96c960-abe0-4b56-b3f9-ce46f2ce307a\") " pod="openshift-ingress/e2e-inf-b-734ffe-openshift-default-5db45fbc99-5vzmr" Oct 30 12:56:34.421853 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:34.415391 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e96c960-abe0-4b56-b3f9-ce46f2ce307a-workload-socket\") pod \"e2e-inf-b-734ffe-openshift-default-5db45fbc99-5vzmr\" (UID: \"7e96c960-abe0-4b56-b3f9-ce46f2ce307a\") " pod="openshift-ingress/e2e-inf-b-734ffe-openshift-default-5db45fbc99-5vzmr" Oct 30 12:56:34.421853 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:34.415454 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e96c960-abe0-4b56-b3f9-ce46f2ce307a-istio-podinfo\") pod \"e2e-inf-b-734ffe-openshift-default-5db45fbc99-5vzmr\" (UID: \"7e96c960-abe0-4b56-b3f9-ce46f2ce307a\") " pod="openshift-ingress/e2e-inf-b-734ffe-openshift-default-5db45fbc99-5vzmr" Oct 30 12:56:34.520851 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:34.519847 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9pzp\" (UniqueName: \"kubernetes.io/projected/7e96c960-abe0-4b56-b3f9-ce46f2ce307a-kube-api-access-q9pzp\") pod \"e2e-inf-b-734ffe-openshift-default-5db45fbc99-5vzmr\" (UID: \"7e96c960-abe0-4b56-b3f9-ce46f2ce307a\") " pod="openshift-ingress/e2e-inf-b-734ffe-openshift-default-5db45fbc99-5vzmr" Oct 30 12:56:34.520851 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:34.519947 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e96c960-abe0-4b56-b3f9-ce46f2ce307a-workload-socket\") pod \"e2e-inf-b-734ffe-openshift-default-5db45fbc99-5vzmr\" (UID: \"7e96c960-abe0-4b56-b3f9-ce46f2ce307a\") " pod="openshift-ingress/e2e-inf-b-734ffe-openshift-default-5db45fbc99-5vzmr" Oct 30 12:56:34.520851 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:34.520002 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e96c960-abe0-4b56-b3f9-ce46f2ce307a-istio-podinfo\") pod \"e2e-inf-b-734ffe-openshift-default-5db45fbc99-5vzmr\" (UID: \"7e96c960-abe0-4b56-b3f9-ce46f2ce307a\") " pod="openshift-ingress/e2e-inf-b-734ffe-openshift-default-5db45fbc99-5vzmr" Oct 30 12:56:34.520851 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:34.520129 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e96c960-abe0-4b56-b3f9-ce46f2ce307a-workload-certs\") pod \"e2e-inf-b-734ffe-openshift-default-5db45fbc99-5vzmr\" (UID: \"7e96c960-abe0-4b56-b3f9-ce46f2ce307a\") " pod="openshift-ingress/e2e-inf-b-734ffe-openshift-default-5db45fbc99-5vzmr" Oct 30 12:56:34.520851 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:34.520175 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e96c960-abe0-4b56-b3f9-ce46f2ce307a-credential-socket\") pod \"e2e-inf-b-734ffe-openshift-default-5db45fbc99-5vzmr\" (UID: \"7e96c960-abe0-4b56-b3f9-ce46f2ce307a\") " pod="openshift-ingress/e2e-inf-b-734ffe-openshift-default-5db45fbc99-5vzmr" Oct 30 12:56:34.520851 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:34.520255 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e96c960-abe0-4b56-b3f9-ce46f2ce307a-istio-envoy\") pod \"e2e-inf-b-734ffe-openshift-default-5db45fbc99-5vzmr\" (UID: \"7e96c960-abe0-4b56-b3f9-ce46f2ce307a\") " pod="openshift-ingress/e2e-inf-b-734ffe-openshift-default-5db45fbc99-5vzmr" Oct 30 12:56:34.520851 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:34.520357 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e96c960-abe0-4b56-b3f9-ce46f2ce307a-istiod-ca-cert\") pod \"e2e-inf-b-734ffe-openshift-default-5db45fbc99-5vzmr\" (UID: \"7e96c960-abe0-4b56-b3f9-ce46f2ce307a\") " pod="openshift-ingress/e2e-inf-b-734ffe-openshift-default-5db45fbc99-5vzmr" Oct 30 12:56:34.520851 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:34.520406 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e96c960-abe0-4b56-b3f9-ce46f2ce307a-istio-token\") pod \"e2e-inf-b-734ffe-openshift-default-5db45fbc99-5vzmr\" (UID: \"7e96c960-abe0-4b56-b3f9-ce46f2ce307a\") " pod="openshift-ingress/e2e-inf-b-734ffe-openshift-default-5db45fbc99-5vzmr" Oct 30 12:56:34.520851 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:34.520460 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e96c960-abe0-4b56-b3f9-ce46f2ce307a-istio-data\") pod \"e2e-inf-b-734ffe-openshift-default-5db45fbc99-5vzmr\" (UID: \"7e96c960-abe0-4b56-b3f9-ce46f2ce307a\") " pod="openshift-ingress/e2e-inf-b-734ffe-openshift-default-5db45fbc99-5vzmr" Oct 30 12:56:34.521795 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:34.521006 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e96c960-abe0-4b56-b3f9-ce46f2ce307a-istio-data\") pod \"e2e-inf-b-734ffe-openshift-default-5db45fbc99-5vzmr\" (UID: \"7e96c960-abe0-4b56-b3f9-ce46f2ce307a\") " pod="openshift-ingress/e2e-inf-b-734ffe-openshift-default-5db45fbc99-5vzmr" Oct 30 12:56:34.521795 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:34.521596 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e96c960-abe0-4b56-b3f9-ce46f2ce307a-workload-socket\") pod \"e2e-inf-b-734ffe-openshift-default-5db45fbc99-5vzmr\" (UID: \"7e96c960-abe0-4b56-b3f9-ce46f2ce307a\") " pod="openshift-ingress/e2e-inf-b-734ffe-openshift-default-5db45fbc99-5vzmr" Oct 30 12:56:34.523192 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:34.523164 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e96c960-abe0-4b56-b3f9-ce46f2ce307a-workload-certs\") pod \"e2e-inf-b-734ffe-openshift-default-5db45fbc99-5vzmr\" (UID: \"7e96c960-abe0-4b56-b3f9-ce46f2ce307a\") " pod="openshift-ingress/e2e-inf-b-734ffe-openshift-default-5db45fbc99-5vzmr" Oct 30 12:56:34.524059 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:34.524033 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e96c960-abe0-4b56-b3f9-ce46f2ce307a-istiod-ca-cert\") pod \"e2e-inf-b-734ffe-openshift-default-5db45fbc99-5vzmr\" (UID: \"7e96c960-abe0-4b56-b3f9-ce46f2ce307a\") " pod="openshift-ingress/e2e-inf-b-734ffe-openshift-default-5db45fbc99-5vzmr" Oct 30 12:56:34.535194 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:34.524513 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e96c960-abe0-4b56-b3f9-ce46f2ce307a-credential-socket\") pod \"e2e-inf-b-734ffe-openshift-default-5db45fbc99-5vzmr\" (UID: \"7e96c960-abe0-4b56-b3f9-ce46f2ce307a\") " pod="openshift-ingress/e2e-inf-b-734ffe-openshift-default-5db45fbc99-5vzmr" Oct 30 12:56:34.535393 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:34.535070 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e96c960-abe0-4b56-b3f9-ce46f2ce307a-istio-envoy\") pod \"e2e-inf-b-734ffe-openshift-default-5db45fbc99-5vzmr\" (UID: \"7e96c960-abe0-4b56-b3f9-ce46f2ce307a\") " pod="openshift-ingress/e2e-inf-b-734ffe-openshift-default-5db45fbc99-5vzmr" Oct 30 12:56:34.549525 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:34.549490 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e96c960-abe0-4b56-b3f9-ce46f2ce307a-istio-podinfo\") pod \"e2e-inf-b-734ffe-openshift-default-5db45fbc99-5vzmr\" (UID: \"7e96c960-abe0-4b56-b3f9-ce46f2ce307a\") " pod="openshift-ingress/e2e-inf-b-734ffe-openshift-default-5db45fbc99-5vzmr" Oct 30 12:56:34.572851 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:34.566499 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e96c960-abe0-4b56-b3f9-ce46f2ce307a-istio-token\") pod \"e2e-inf-b-734ffe-openshift-default-5db45fbc99-5vzmr\" (UID: \"7e96c960-abe0-4b56-b3f9-ce46f2ce307a\") " pod="openshift-ingress/e2e-inf-b-734ffe-openshift-default-5db45fbc99-5vzmr" Oct 30 12:56:34.573666 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:34.573638 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9pzp\" (UniqueName: \"kubernetes.io/projected/7e96c960-abe0-4b56-b3f9-ce46f2ce307a-kube-api-access-q9pzp\") pod \"e2e-inf-b-734ffe-openshift-default-5db45fbc99-5vzmr\" (UID: \"7e96c960-abe0-4b56-b3f9-ce46f2ce307a\") " pod="openshift-ingress/e2e-inf-b-734ffe-openshift-default-5db45fbc99-5vzmr" Oct 30 12:56:34.604857 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:34.597089 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-734ffe-openshift-default-5db45fbc99-5vzmr" Oct 30 12:56:35.192696 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:56:35.192658 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e96c960_abe0_4b56_b3f9_ce46f2ce307a.slice/crio-2417315f235ea85756bf0cff4590cf2ed76756a72eaeac8e80e3e7c0c2806024 WatchSource:0}: Error finding container 2417315f235ea85756bf0cff4590cf2ed76756a72eaeac8e80e3e7c0c2806024: Status 404 returned error can't find the container with id 2417315f235ea85756bf0cff4590cf2ed76756a72eaeac8e80e3e7c0c2806024 Oct 30 12:56:35.197581 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:35.197537 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:56:35.197711 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:35.197632 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:56:35.197711 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:35.197677 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:56:35.197875 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:35.197741 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-734ffe-openshift-default-5db45fbc99-5vzmr"] Oct 30 12:56:35.852695 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:35.852659 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-734ffe-openshift-default-5db45fbc99-5vzmr" event={"ID":"7e96c960-abe0-4b56-b3f9-ce46f2ce307a","Type":"ContainerStarted","Data":"0f32041b6b6e591be1af63761a9a5f99db3675b682c4ee06b9fa1e362c1dbf5d"} Oct 30 12:56:35.853137 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:35.852716 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-734ffe-openshift-default-5db45fbc99-5vzmr" event={"ID":"7e96c960-abe0-4b56-b3f9-ce46f2ce307a","Type":"ContainerStarted","Data":"2417315f235ea85756bf0cff4590cf2ed76756a72eaeac8e80e3e7c0c2806024"} Oct 30 12:56:35.911886 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:35.910846 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-734ffe-openshift-default-5db45fbc99-5vzmr" podStartSLOduration=1.91080957 podStartE2EDuration="1.91080957s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:56:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:56:35.908350084 +0000 UTC m=+3752.422778843" watchObservedRunningTime="2026-10-30 12:56:35.91080957 +0000 UTC m=+3752.425238312" Oct 30 12:56:36.598500 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:36.598468 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-734ffe-openshift-default-5db45fbc99-5vzmr" Oct 30 12:56:36.604481 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:36.604454 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-734ffe-openshift-default-5db45fbc99-5vzmr" Oct 30 12:56:36.861464 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:36.861376 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-734ffe-openshift-default-5db45fbc99-5vzmr" Oct 30 12:56:36.862439 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:36.862417 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-734ffe-openshift-default-5db45fbc99-5vzmr" Oct 30 12:56:43.356118 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:43.356068 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-befaa6-openshift-default-767bb98587-6bvqz"] Oct 30 12:56:43.363238 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:43.363212 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-befaa6-openshift-default-767bb98587-6bvqz" Oct 30 12:56:43.397673 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:43.397644 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-befaa6-openshift-default-dockercfg-hxrww\"" Oct 30 12:56:43.423955 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:43.423915 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93f3f694-3b06-430c-b4f0-ed5afbea43d1-istio-token\") pod \"e2e-inf-b-befaa6-openshift-default-767bb98587-6bvqz\" (UID: \"93f3f694-3b06-430c-b4f0-ed5afbea43d1\") " pod="openshift-ingress/e2e-inf-b-befaa6-openshift-default-767bb98587-6bvqz" Oct 30 12:56:43.424140 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:43.424048 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qjkl\" (UniqueName: \"kubernetes.io/projected/93f3f694-3b06-430c-b4f0-ed5afbea43d1-kube-api-access-6qjkl\") pod \"e2e-inf-b-befaa6-openshift-default-767bb98587-6bvqz\" (UID: \"93f3f694-3b06-430c-b4f0-ed5afbea43d1\") " pod="openshift-ingress/e2e-inf-b-befaa6-openshift-default-767bb98587-6bvqz" Oct 30 12:56:43.424206 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:43.424140 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93f3f694-3b06-430c-b4f0-ed5afbea43d1-istio-data\") pod \"e2e-inf-b-befaa6-openshift-default-767bb98587-6bvqz\" (UID: \"93f3f694-3b06-430c-b4f0-ed5afbea43d1\") " pod="openshift-ingress/e2e-inf-b-befaa6-openshift-default-767bb98587-6bvqz" Oct 30 12:56:43.424276 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:43.424231 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93f3f694-3b06-430c-b4f0-ed5afbea43d1-workload-socket\") pod \"e2e-inf-b-befaa6-openshift-default-767bb98587-6bvqz\" (UID: \"93f3f694-3b06-430c-b4f0-ed5afbea43d1\") " pod="openshift-ingress/e2e-inf-b-befaa6-openshift-default-767bb98587-6bvqz" Oct 30 12:56:43.424339 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:43.424296 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93f3f694-3b06-430c-b4f0-ed5afbea43d1-credential-socket\") pod \"e2e-inf-b-befaa6-openshift-default-767bb98587-6bvqz\" (UID: \"93f3f694-3b06-430c-b4f0-ed5afbea43d1\") " pod="openshift-ingress/e2e-inf-b-befaa6-openshift-default-767bb98587-6bvqz" Oct 30 12:56:43.424399 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:43.424374 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93f3f694-3b06-430c-b4f0-ed5afbea43d1-istio-podinfo\") pod \"e2e-inf-b-befaa6-openshift-default-767bb98587-6bvqz\" (UID: \"93f3f694-3b06-430c-b4f0-ed5afbea43d1\") " pod="openshift-ingress/e2e-inf-b-befaa6-openshift-default-767bb98587-6bvqz" Oct 30 12:56:43.424459 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:43.424410 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93f3f694-3b06-430c-b4f0-ed5afbea43d1-workload-certs\") pod \"e2e-inf-b-befaa6-openshift-default-767bb98587-6bvqz\" (UID: \"93f3f694-3b06-430c-b4f0-ed5afbea43d1\") " pod="openshift-ingress/e2e-inf-b-befaa6-openshift-default-767bb98587-6bvqz" Oct 30 12:56:43.424518 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:43.424471 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93f3f694-3b06-430c-b4f0-ed5afbea43d1-istio-envoy\") pod \"e2e-inf-b-befaa6-openshift-default-767bb98587-6bvqz\" (UID: \"93f3f694-3b06-430c-b4f0-ed5afbea43d1\") " pod="openshift-ingress/e2e-inf-b-befaa6-openshift-default-767bb98587-6bvqz" Oct 30 12:56:43.424583 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:43.424518 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93f3f694-3b06-430c-b4f0-ed5afbea43d1-istiod-ca-cert\") pod \"e2e-inf-b-befaa6-openshift-default-767bb98587-6bvqz\" (UID: \"93f3f694-3b06-430c-b4f0-ed5afbea43d1\") " pod="openshift-ingress/e2e-inf-b-befaa6-openshift-default-767bb98587-6bvqz" Oct 30 12:56:43.424929 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:43.424906 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-befaa6-openshift-default-767bb98587-6bvqz"] Oct 30 12:56:43.570851 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:43.566803 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93f3f694-3b06-430c-b4f0-ed5afbea43d1-istio-token\") pod \"e2e-inf-b-befaa6-openshift-default-767bb98587-6bvqz\" (UID: \"93f3f694-3b06-430c-b4f0-ed5afbea43d1\") " pod="openshift-ingress/e2e-inf-b-befaa6-openshift-default-767bb98587-6bvqz" Oct 30 12:56:43.578374 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:43.574994 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qjkl\" (UniqueName: \"kubernetes.io/projected/93f3f694-3b06-430c-b4f0-ed5afbea43d1-kube-api-access-6qjkl\") pod \"e2e-inf-b-befaa6-openshift-default-767bb98587-6bvqz\" (UID: \"93f3f694-3b06-430c-b4f0-ed5afbea43d1\") " pod="openshift-ingress/e2e-inf-b-befaa6-openshift-default-767bb98587-6bvqz" Oct 30 12:56:43.578818 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:43.578786 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93f3f694-3b06-430c-b4f0-ed5afbea43d1-istio-data\") pod \"e2e-inf-b-befaa6-openshift-default-767bb98587-6bvqz\" (UID: \"93f3f694-3b06-430c-b4f0-ed5afbea43d1\") " pod="openshift-ingress/e2e-inf-b-befaa6-openshift-default-767bb98587-6bvqz" Oct 30 12:56:43.579084 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:43.579064 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93f3f694-3b06-430c-b4f0-ed5afbea43d1-workload-socket\") pod \"e2e-inf-b-befaa6-openshift-default-767bb98587-6bvqz\" (UID: \"93f3f694-3b06-430c-b4f0-ed5afbea43d1\") " pod="openshift-ingress/e2e-inf-b-befaa6-openshift-default-767bb98587-6bvqz" Oct 30 12:56:43.579243 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:43.579228 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93f3f694-3b06-430c-b4f0-ed5afbea43d1-credential-socket\") pod \"e2e-inf-b-befaa6-openshift-default-767bb98587-6bvqz\" (UID: \"93f3f694-3b06-430c-b4f0-ed5afbea43d1\") " pod="openshift-ingress/e2e-inf-b-befaa6-openshift-default-767bb98587-6bvqz" Oct 30 12:56:43.579424 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:43.579409 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93f3f694-3b06-430c-b4f0-ed5afbea43d1-istio-podinfo\") pod \"e2e-inf-b-befaa6-openshift-default-767bb98587-6bvqz\" (UID: \"93f3f694-3b06-430c-b4f0-ed5afbea43d1\") " pod="openshift-ingress/e2e-inf-b-befaa6-openshift-default-767bb98587-6bvqz" Oct 30 12:56:43.579552 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:43.579536 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93f3f694-3b06-430c-b4f0-ed5afbea43d1-workload-certs\") pod \"e2e-inf-b-befaa6-openshift-default-767bb98587-6bvqz\" (UID: \"93f3f694-3b06-430c-b4f0-ed5afbea43d1\") " pod="openshift-ingress/e2e-inf-b-befaa6-openshift-default-767bb98587-6bvqz" Oct 30 12:56:43.585138 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:43.579700 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93f3f694-3b06-430c-b4f0-ed5afbea43d1-istio-envoy\") pod \"e2e-inf-b-befaa6-openshift-default-767bb98587-6bvqz\" (UID: \"93f3f694-3b06-430c-b4f0-ed5afbea43d1\") " pod="openshift-ingress/e2e-inf-b-befaa6-openshift-default-767bb98587-6bvqz" Oct 30 12:56:43.585351 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:43.585328 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93f3f694-3b06-430c-b4f0-ed5afbea43d1-istiod-ca-cert\") pod \"e2e-inf-b-befaa6-openshift-default-767bb98587-6bvqz\" (UID: \"93f3f694-3b06-430c-b4f0-ed5afbea43d1\") " pod="openshift-ingress/e2e-inf-b-befaa6-openshift-default-767bb98587-6bvqz" Oct 30 12:56:43.602016 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:43.582850 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93f3f694-3b06-430c-b4f0-ed5afbea43d1-istio-data\") pod \"e2e-inf-b-befaa6-openshift-default-767bb98587-6bvqz\" (UID: \"93f3f694-3b06-430c-b4f0-ed5afbea43d1\") " pod="openshift-ingress/e2e-inf-b-befaa6-openshift-default-767bb98587-6bvqz" Oct 30 12:56:43.602226 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:43.583856 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93f3f694-3b06-430c-b4f0-ed5afbea43d1-workload-certs\") pod \"e2e-inf-b-befaa6-openshift-default-767bb98587-6bvqz\" (UID: \"93f3f694-3b06-430c-b4f0-ed5afbea43d1\") " pod="openshift-ingress/e2e-inf-b-befaa6-openshift-default-767bb98587-6bvqz" Oct 30 12:56:43.602332 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:43.599723 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93f3f694-3b06-430c-b4f0-ed5afbea43d1-istiod-ca-cert\") pod \"e2e-inf-b-befaa6-openshift-default-767bb98587-6bvqz\" (UID: \"93f3f694-3b06-430c-b4f0-ed5afbea43d1\") " pod="openshift-ingress/e2e-inf-b-befaa6-openshift-default-767bb98587-6bvqz" Oct 30 12:56:43.602426 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:43.582277 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93f3f694-3b06-430c-b4f0-ed5afbea43d1-workload-socket\") pod \"e2e-inf-b-befaa6-openshift-default-767bb98587-6bvqz\" (UID: \"93f3f694-3b06-430c-b4f0-ed5afbea43d1\") " pod="openshift-ingress/e2e-inf-b-befaa6-openshift-default-767bb98587-6bvqz" Oct 30 12:56:43.602505 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:43.600011 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93f3f694-3b06-430c-b4f0-ed5afbea43d1-credential-socket\") pod \"e2e-inf-b-befaa6-openshift-default-767bb98587-6bvqz\" (UID: \"93f3f694-3b06-430c-b4f0-ed5afbea43d1\") " pod="openshift-ingress/e2e-inf-b-befaa6-openshift-default-767bb98587-6bvqz" Oct 30 12:56:43.635012 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:43.619782 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93f3f694-3b06-430c-b4f0-ed5afbea43d1-istio-podinfo\") pod \"e2e-inf-b-befaa6-openshift-default-767bb98587-6bvqz\" (UID: \"93f3f694-3b06-430c-b4f0-ed5afbea43d1\") " pod="openshift-ingress/e2e-inf-b-befaa6-openshift-default-767bb98587-6bvqz" Oct 30 12:56:43.694699 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:43.689870 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qjkl\" (UniqueName: \"kubernetes.io/projected/93f3f694-3b06-430c-b4f0-ed5afbea43d1-kube-api-access-6qjkl\") pod \"e2e-inf-b-befaa6-openshift-default-767bb98587-6bvqz\" (UID: \"93f3f694-3b06-430c-b4f0-ed5afbea43d1\") " pod="openshift-ingress/e2e-inf-b-befaa6-openshift-default-767bb98587-6bvqz" Oct 30 12:56:43.708521 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:43.706422 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93f3f694-3b06-430c-b4f0-ed5afbea43d1-istio-token\") pod \"e2e-inf-b-befaa6-openshift-default-767bb98587-6bvqz\" (UID: \"93f3f694-3b06-430c-b4f0-ed5afbea43d1\") " pod="openshift-ingress/e2e-inf-b-befaa6-openshift-default-767bb98587-6bvqz" Oct 30 12:56:43.737194 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:43.737124 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93f3f694-3b06-430c-b4f0-ed5afbea43d1-istio-envoy\") pod \"e2e-inf-b-befaa6-openshift-default-767bb98587-6bvqz\" (UID: \"93f3f694-3b06-430c-b4f0-ed5afbea43d1\") " pod="openshift-ingress/e2e-inf-b-befaa6-openshift-default-767bb98587-6bvqz" Oct 30 12:56:44.004842 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:44.004791 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-befaa6-openshift-default-767bb98587-6bvqz" Oct 30 12:56:44.338743 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:44.338712 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-befaa6-openshift-default-767bb98587-6bvqz"] Oct 30 12:56:44.340801 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:56:44.340772 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93f3f694_3b06_430c_b4f0_ed5afbea43d1.slice/crio-16328ff68490e6046260f446763ea1165bdd57c4280c4f808674ce62cebdef44 WatchSource:0}: Error finding container 16328ff68490e6046260f446763ea1165bdd57c4280c4f808674ce62cebdef44: Status 404 returned error can't find the container with id 16328ff68490e6046260f446763ea1165bdd57c4280c4f808674ce62cebdef44 Oct 30 12:56:44.344357 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:44.344324 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:56:44.344440 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:44.344393 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:56:44.344440 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:44.344422 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:56:44.960656 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:44.960618 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-befaa6-openshift-default-767bb98587-6bvqz" event={"ID":"93f3f694-3b06-430c-b4f0-ed5afbea43d1","Type":"ContainerStarted","Data":"fd793e758613ea073329301aa04b1d67dc621e22ffb6618363f63290a4ffbe51"} Oct 30 12:56:44.961087 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:44.960678 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-befaa6-openshift-default-767bb98587-6bvqz" event={"ID":"93f3f694-3b06-430c-b4f0-ed5afbea43d1","Type":"ContainerStarted","Data":"16328ff68490e6046260f446763ea1165bdd57c4280c4f808674ce62cebdef44"} Oct 30 12:56:45.005487 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:45.005450 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-befaa6-openshift-default-767bb98587-6bvqz" Oct 30 12:56:45.012626 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:45.012600 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-befaa6-openshift-default-767bb98587-6bvqz" Oct 30 12:56:45.015411 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:45.015368 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-befaa6-openshift-default-767bb98587-6bvqz" podStartSLOduration=2.015355441 podStartE2EDuration="2.015355441s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:56:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:56:45.014856302 +0000 UTC m=+3761.529285043" watchObservedRunningTime="2026-10-30 12:56:45.015355441 +0000 UTC m=+3761.529784187" Oct 30 12:56:45.970600 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:45.970562 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-befaa6-openshift-default-767bb98587-6bvqz" Oct 30 12:56:45.971933 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:45.971909 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-befaa6-openshift-default-767bb98587-6bvqz" Oct 30 12:56:49.245706 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:49.245667 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9wwfc"] Oct 30 12:56:49.784853 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:56:49.784087 4908 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vn6vv" containerName="extractor" Oct 30 12:56:53.197853 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:53.196901 4908 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-d9b94 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 12:56:53.197853 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:53.196981 4908 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-d9b94" podUID="a5b1685b-e185-4b12-9c20-3c87e9837fd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 12:56:53.686854 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:53.635413 4908 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6a7148/premium-w3-6a7148-kserve-67b976c888-tgd6c" podUID="13de4441-e7f6-4cbd-97d9-0f079d11cb8e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:56:53.855007 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:53.854960 4908 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47fd99/premium-w2-47fd99-kserve-5d85f9bb95-hqkmf" podUID="fd2cbeda-dbc4-41f6-a730-e1c6f9343c5e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 12:56:53.958913 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:56:53.958032 4908 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6a7148/distinct-2-w3-6a7148-kserve-67bd5fb976-9nfff" podUID="6046cc02-29fe-4f59-b522-8f57f4557277" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:57:02.012904 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:57:02.012866 4908 generic.go:356] "Generic (PLEG): container finished" podID="b1b34ded-198e-48a2-bd18-303865ac83f3" containerID="bc985fb1bf4ea590b514fce128c27da55759701718af051ba7efcee15a3e13a8" exitCode=0 Oct 30 12:57:02.013359 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:57:02.012937 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e9f184-openshift-default-55dd875b9f-jjt74" event={"ID":"b1b34ded-198e-48a2-bd18-303865ac83f3","Type":"ContainerDied","Data":"bc985fb1bf4ea590b514fce128c27da55759701718af051ba7efcee15a3e13a8"} Oct 30 12:57:02.013359 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:57:02.013333 4908 scope.go:122] "RemoveContainer" containerID="bc985fb1bf4ea590b514fce128c27da55759701718af051ba7efcee15a3e13a8" Oct 30 12:57:02.015321 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:57:02.015288 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:57:02.015446 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:57:02.015377 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:57:02.015446 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:57:02.015428 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:57:03.023106 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:57:03.023069 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e9f184-openshift-default-55dd875b9f-jjt74" event={"ID":"b1b34ded-198e-48a2-bd18-303865ac83f3","Type":"ContainerStarted","Data":"2b7f3531e930a52fbe1b088154c9e9fb14cb455b18cd81f2b6e4838fc2a06d84"} Oct 30 12:57:03.139532 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:57:03.139498 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-e9f184-openshift-default-55dd875b9f-jjt74" Oct 30 12:57:03.140535 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:57:03.140509 4908 patch_prober.go:29] interesting pod/e2e-inf-b-e9f184-openshift-default-55dd875b9f-jjt74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.34:15021/healthz/ready\": dial tcp 10.132.1.34:15021: connect: connection refused" start-of-body= Oct 30 12:57:03.140675 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:57:03.140564 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e9f184-openshift-default-55dd875b9f-jjt74" podUID="b1b34ded-198e-48a2-bd18-303865ac83f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.34:15021/healthz/ready\": dial tcp 10.132.1.34:15021: connect: connection refused" Oct 30 12:57:03.142651 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:57:03.142630 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-e9f184-openshift-default-55dd875b9f-jjt74" Oct 30 12:57:04.139011 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:57:04.138978 4908 patch_prober.go:29] interesting pod/e2e-inf-b-e9f184-openshift-default-55dd875b9f-jjt74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.34:15021/healthz/ready\": dial tcp 10.132.1.34:15021: connect: connection refused" start-of-body= Oct 30 12:57:04.139432 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:57:04.139031 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e9f184-openshift-default-55dd875b9f-jjt74" podUID="b1b34ded-198e-48a2-bd18-303865ac83f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.34:15021/healthz/ready\": dial tcp 10.132.1.34:15021: connect: connection refused" Oct 30 12:57:05.138944 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:57:05.138908 4908 patch_prober.go:29] interesting pod/e2e-inf-b-e9f184-openshift-default-55dd875b9f-jjt74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.34:15021/healthz/ready\": dial tcp 10.132.1.34:15021: connect: connection refused" start-of-body= Oct 30 12:57:05.139160 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:57:05.138957 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e9f184-openshift-default-55dd875b9f-jjt74" podUID="b1b34ded-198e-48a2-bd18-303865ac83f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.34:15021/healthz/ready\": dial tcp 10.132.1.34:15021: connect: connection refused" Oct 30 12:57:06.139083 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:57:06.139047 4908 patch_prober.go:29] interesting pod/e2e-inf-b-e9f184-openshift-default-55dd875b9f-jjt74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.34:15021/healthz/ready\": dial tcp 10.132.1.34:15021: connect: connection refused" start-of-body= Oct 30 12:57:06.139403 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:57:06.139107 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e9f184-openshift-default-55dd875b9f-jjt74" podUID="b1b34ded-198e-48a2-bd18-303865ac83f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.34:15021/healthz/ready\": dial tcp 10.132.1.34:15021: connect: connection refused" Oct 30 12:57:07.139038 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:57:07.139002 4908 patch_prober.go:29] interesting pod/e2e-inf-b-e9f184-openshift-default-55dd875b9f-jjt74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.34:15021/healthz/ready\": dial tcp 10.132.1.34:15021: connect: connection refused" start-of-body= Oct 30 12:57:07.139237 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:57:07.139063 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e9f184-openshift-default-55dd875b9f-jjt74" podUID="b1b34ded-198e-48a2-bd18-303865ac83f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.34:15021/healthz/ready\": dial tcp 10.132.1.34:15021: connect: connection refused" Oct 30 12:57:08.139759 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:57:08.139722 4908 patch_prober.go:29] interesting pod/e2e-inf-b-e9f184-openshift-default-55dd875b9f-jjt74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.34:15021/healthz/ready\": dial tcp 10.132.1.34:15021: connect: connection refused" start-of-body= Oct 30 12:57:08.140205 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:57:08.139782 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e9f184-openshift-default-55dd875b9f-jjt74" podUID="b1b34ded-198e-48a2-bd18-303865ac83f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.34:15021/healthz/ready\": dial tcp 10.132.1.34:15021: connect: connection refused" Oct 30 12:57:08.704253 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:57:08.704222 4908 scope.go:122] "RemoveContainer" containerID="263fd0abedbbb4197fcbed3dfd84e76599bc871ca89707c5314f141a76e8026d" Oct 30 12:57:08.714940 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:57:08.714909 4908 scope.go:122] "RemoveContainer" containerID="801e058264f289e9f6c7299977f678cfc70c752c923dc64f27926ca96d385a01" Oct 30 12:57:08.724728 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:57:08.724706 4908 scope.go:122] "RemoveContainer" containerID="3306aa9d5fe035136e3d7e5ddc32c19e086eac9b8a14bd628f4996699619cc0f" Oct 30 12:57:09.144196 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:57:09.144169 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-e9f184-openshift-default-55dd875b9f-jjt74" Oct 30 12:57:09.145066 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:57:09.145044 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-e9f184-openshift-default-55dd875b9f-jjt74" Oct 30 12:57:19.684997 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:57:19.684953 4908 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vn6vv" containerName="extractor" Oct 30 12:57:25.954525 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:57:25.954485 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-9c3f12-openshift-default-78b944cc9c-fdpqh"] Oct 30 12:57:25.958786 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:57:25.958759 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9c3f12-openshift-default-78b944cc9c-fdpqh" Oct 30 12:57:26.005296 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:57:26.002992 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-9c3f12-openshift-default-dockercfg-xnt9w\"" Oct 30 12:57:26.007229 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:57:26.007195 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2029481-e8ad-40ef-97b7-81140abedb5c-istio-podinfo\") pod \"e2e-inf-b-9c3f12-openshift-default-78b944cc9c-fdpqh\" (UID: \"a2029481-e8ad-40ef-97b7-81140abedb5c\") " pod="openshift-ingress/e2e-inf-b-9c3f12-openshift-default-78b944cc9c-fdpqh" Oct 30 12:57:26.007554 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:57:26.007517 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2029481-e8ad-40ef-97b7-81140abedb5c-istio-data\") pod \"e2e-inf-b-9c3f12-openshift-default-78b944cc9c-fdpqh\" (UID: \"a2029481-e8ad-40ef-97b7-81140abedb5c\") " pod="openshift-ingress/e2e-inf-b-9c3f12-openshift-default-78b944cc9c-fdpqh" Oct 30 12:57:26.007725 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:57:26.007704 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2029481-e8ad-40ef-97b7-81140abedb5c-istio-token\") pod \"e2e-inf-b-9c3f12-openshift-default-78b944cc9c-fdpqh\" (UID: \"a2029481-e8ad-40ef-97b7-81140abedb5c\") " pod="openshift-ingress/e2e-inf-b-9c3f12-openshift-default-78b944cc9c-fdpqh" Oct 30 12:57:26.007870 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:57:26.007853 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2029481-e8ad-40ef-97b7-81140abedb5c-workload-certs\") pod \"e2e-inf-b-9c3f12-openshift-default-78b944cc9c-fdpqh\" (UID: \"a2029481-e8ad-40ef-97b7-81140abedb5c\") " pod="openshift-ingress/e2e-inf-b-9c3f12-openshift-default-78b944cc9c-fdpqh" Oct 30 12:57:26.008014 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:57:26.007995 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2029481-e8ad-40ef-97b7-81140abedb5c-istiod-ca-cert\") pod \"e2e-inf-b-9c3f12-openshift-default-78b944cc9c-fdpqh\" (UID: \"a2029481-e8ad-40ef-97b7-81140abedb5c\") " pod="openshift-ingress/e2e-inf-b-9c3f12-openshift-default-78b944cc9c-fdpqh" Oct 30 12:57:26.008158 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:57:26.008140 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2029481-e8ad-40ef-97b7-81140abedb5c-credential-socket\") pod \"e2e-inf-b-9c3f12-openshift-default-78b944cc9c-fdpqh\" (UID: \"a2029481-e8ad-40ef-97b7-81140abedb5c\") " pod="openshift-ingress/e2e-inf-b-9c3f12-openshift-default-78b944cc9c-fdpqh" Oct 30 12:57:26.008331 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:57:26.008314 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2029481-e8ad-40ef-97b7-81140abedb5c-istio-envoy\") pod \"e2e-inf-b-9c3f12-openshift-default-78b944cc9c-fdpqh\" (UID: \"a2029481-e8ad-40ef-97b7-81140abedb5c\") " pod="openshift-ingress/e2e-inf-b-9c3f12-openshift-default-78b944cc9c-fdpqh" Oct 30 12:57:26.008465 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:57:26.008449 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9fd2\" (UniqueName: \"kubernetes.io/projected/a2029481-e8ad-40ef-97b7-81140abedb5c-kube-api-access-f9fd2\") pod \"e2e-inf-b-9c3f12-openshift-default-78b944cc9c-fdpqh\" (UID: \"a2029481-e8ad-40ef-97b7-81140abedb5c\") " pod="openshift-ingress/e2e-inf-b-9c3f12-openshift-default-78b944cc9c-fdpqh" Oct 30 12:57:26.008638 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:57:26.008621 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2029481-e8ad-40ef-97b7-81140abedb5c-workload-socket\") pod \"e2e-inf-b-9c3f12-openshift-default-78b944cc9c-fdpqh\" (UID: \"a2029481-e8ad-40ef-97b7-81140abedb5c\") " pod="openshift-ingress/e2e-inf-b-9c3f12-openshift-default-78b944cc9c-fdpqh" Oct 30 12:57:26.016339 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:57:26.014728 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-9c3f12-openshift-default-78b944cc9c-fdpqh"] Oct 30 12:57:26.153853 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:57:26.127959 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2029481-e8ad-40ef-97b7-81140abedb5c-istio-data\") pod \"e2e-inf-b-9c3f12-openshift-default-78b944cc9c-fdpqh\" (UID: \"a2029481-e8ad-40ef-97b7-81140abedb5c\") " pod="openshift-ingress/e2e-inf-b-9c3f12-openshift-default-78b944cc9c-fdpqh" Oct 30 12:57:26.225905 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:57:26.225440 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2029481-e8ad-40ef-97b7-81140abedb5c-istio-token\") pod \"e2e-inf-b-9c3f12-openshift-default-78b944cc9c-fdpqh\" (UID: \"a2029481-e8ad-40ef-97b7-81140abedb5c\") " pod="openshift-ingress/e2e-inf-b-9c3f12-openshift-default-78b944cc9c-fdpqh" Oct 30 12:57:26.225905 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:57:26.225493 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2029481-e8ad-40ef-97b7-81140abedb5c-workload-certs\") pod \"e2e-inf-b-9c3f12-openshift-default-78b944cc9c-fdpqh\" (UID: \"a2029481-e8ad-40ef-97b7-81140abedb5c\") " pod="openshift-ingress/e2e-inf-b-9c3f12-openshift-default-78b944cc9c-fdpqh" Oct 30 12:57:26.225905 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:57:26.225539 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2029481-e8ad-40ef-97b7-81140abedb5c-istiod-ca-cert\") pod \"e2e-inf-b-9c3f12-openshift-default-78b944cc9c-fdpqh\" (UID: \"a2029481-e8ad-40ef-97b7-81140abedb5c\") " pod="openshift-ingress/e2e-inf-b-9c3f12-openshift-default-78b944cc9c-fdpqh" Oct 30 12:57:26.225905 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:57:26.225569 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2029481-e8ad-40ef-97b7-81140abedb5c-credential-socket\") pod \"e2e-inf-b-9c3f12-openshift-default-78b944cc9c-fdpqh\" (UID: \"a2029481-e8ad-40ef-97b7-81140abedb5c\") " pod="openshift-ingress/e2e-inf-b-9c3f12-openshift-default-78b944cc9c-fdpqh" Oct 30 12:57:26.225905 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:57:26.225659 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2029481-e8ad-40ef-97b7-81140abedb5c-istio-envoy\") pod \"e2e-inf-b-9c3f12-openshift-default-78b944cc9c-fdpqh\" (UID: \"a2029481-e8ad-40ef-97b7-81140abedb5c\") " pod="openshift-ingress/e2e-inf-b-9c3f12-openshift-default-78b944cc9c-fdpqh" Oct 30 12:57:26.225905 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:57:26.225694 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9fd2\" (UniqueName: \"kubernetes.io/projected/a2029481-e8ad-40ef-97b7-81140abedb5c-kube-api-access-f9fd2\") pod \"e2e-inf-b-9c3f12-openshift-default-78b944cc9c-fdpqh\" (UID: \"a2029481-e8ad-40ef-97b7-81140abedb5c\") " pod="openshift-ingress/e2e-inf-b-9c3f12-openshift-default-78b944cc9c-fdpqh" Oct 30 12:57:26.225905 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:57:26.225799 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2029481-e8ad-40ef-97b7-81140abedb5c-workload-socket\") pod \"e2e-inf-b-9c3f12-openshift-default-78b944cc9c-fdpqh\" (UID: \"a2029481-e8ad-40ef-97b7-81140abedb5c\") " pod="openshift-ingress/e2e-inf-b-9c3f12-openshift-default-78b944cc9c-fdpqh" Oct 30 12:57:26.244814 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:57:26.244774 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2029481-e8ad-40ef-97b7-81140abedb5c-istiod-ca-cert\") pod \"e2e-inf-b-9c3f12-openshift-default-78b944cc9c-fdpqh\" (UID: \"a2029481-e8ad-40ef-97b7-81140abedb5c\") " pod="openshift-ingress/e2e-inf-b-9c3f12-openshift-default-78b944cc9c-fdpqh" Oct 30 12:57:26.245063 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:57:26.165237 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2029481-e8ad-40ef-97b7-81140abedb5c-istio-data\") pod \"e2e-inf-b-9c3f12-openshift-default-78b944cc9c-fdpqh\" (UID: \"a2029481-e8ad-40ef-97b7-81140abedb5c\") " pod="openshift-ingress/e2e-inf-b-9c3f12-openshift-default-78b944cc9c-fdpqh" Oct 30 12:57:26.245739 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:57:26.245711 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2029481-e8ad-40ef-97b7-81140abedb5c-workload-certs\") pod \"e2e-inf-b-9c3f12-openshift-default-78b944cc9c-fdpqh\" (UID: \"a2029481-e8ad-40ef-97b7-81140abedb5c\") " pod="openshift-ingress/e2e-inf-b-9c3f12-openshift-default-78b944cc9c-fdpqh" Oct 30 12:57:26.246986 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:57:26.246962 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2029481-e8ad-40ef-97b7-81140abedb5c-credential-socket\") pod \"e2e-inf-b-9c3f12-openshift-default-78b944cc9c-fdpqh\" (UID: \"a2029481-e8ad-40ef-97b7-81140abedb5c\") " pod="openshift-ingress/e2e-inf-b-9c3f12-openshift-default-78b944cc9c-fdpqh" Oct 30 12:57:26.269312 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:57:26.269274 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2029481-e8ad-40ef-97b7-81140abedb5c-workload-socket\") pod \"e2e-inf-b-9c3f12-openshift-default-78b944cc9c-fdpqh\" (UID: \"a2029481-e8ad-40ef-97b7-81140abedb5c\") " pod="openshift-ingress/e2e-inf-b-9c3f12-openshift-default-78b944cc9c-fdpqh" Oct 30 12:57:26.282850 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:57:26.277190 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2029481-e8ad-40ef-97b7-81140abedb5c-istio-podinfo\") pod \"e2e-inf-b-9c3f12-openshift-default-78b944cc9c-fdpqh\" (UID: \"a2029481-e8ad-40ef-97b7-81140abedb5c\") " pod="openshift-ingress/e2e-inf-b-9c3f12-openshift-default-78b944cc9c-fdpqh" Oct 30 12:57:26.339850 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:57:26.318501 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2029481-e8ad-40ef-97b7-81140abedb5c-istio-token\") pod \"e2e-inf-b-9c3f12-openshift-default-78b944cc9c-fdpqh\" (UID: \"a2029481-e8ad-40ef-97b7-81140abedb5c\") " pod="openshift-ingress/e2e-inf-b-9c3f12-openshift-default-78b944cc9c-fdpqh" Oct 30 12:57:26.339850 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:57:26.319768 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9fd2\" (UniqueName: \"kubernetes.io/projected/a2029481-e8ad-40ef-97b7-81140abedb5c-kube-api-access-f9fd2\") pod \"e2e-inf-b-9c3f12-openshift-default-78b944cc9c-fdpqh\" (UID: \"a2029481-e8ad-40ef-97b7-81140abedb5c\") " pod="openshift-ingress/e2e-inf-b-9c3f12-openshift-default-78b944cc9c-fdpqh" Oct 30 12:57:26.342218 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:57:26.342160 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2029481-e8ad-40ef-97b7-81140abedb5c-istio-envoy\") pod \"e2e-inf-b-9c3f12-openshift-default-78b944cc9c-fdpqh\" (UID: \"a2029481-e8ad-40ef-97b7-81140abedb5c\") " pod="openshift-ingress/e2e-inf-b-9c3f12-openshift-default-78b944cc9c-fdpqh" Oct 30 12:57:26.342405 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:57:26.342249 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2029481-e8ad-40ef-97b7-81140abedb5c-istio-podinfo\") pod \"e2e-inf-b-9c3f12-openshift-default-78b944cc9c-fdpqh\" (UID: \"a2029481-e8ad-40ef-97b7-81140abedb5c\") " pod="openshift-ingress/e2e-inf-b-9c3f12-openshift-default-78b944cc9c-fdpqh" Oct 30 12:57:26.628585 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:57:26.628547 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9c3f12-openshift-default-78b944cc9c-fdpqh" Oct 30 12:57:27.203064 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:57:27.202475 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-9c3f12-openshift-default-78b944cc9c-fdpqh"] Oct 30 12:57:27.204455 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:57:27.204398 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2029481_e8ad_40ef_97b7_81140abedb5c.slice/crio-61b3ec88d47aefd6dfbc2beded8887a5c742721ec7696e56beee771ddd47e2f8 WatchSource:0}: Error finding container 61b3ec88d47aefd6dfbc2beded8887a5c742721ec7696e56beee771ddd47e2f8: Status 404 returned error can't find the container with id 61b3ec88d47aefd6dfbc2beded8887a5c742721ec7696e56beee771ddd47e2f8 Oct 30 12:57:27.208675 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:57:27.208627 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:57:27.208821 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:57:27.208715 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:57:27.208821 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:57:27.208759 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:57:27.321647 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:57:27.321604 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9c3f12-openshift-default-78b944cc9c-fdpqh" event={"ID":"a2029481-e8ad-40ef-97b7-81140abedb5c","Type":"ContainerStarted","Data":"61b3ec88d47aefd6dfbc2beded8887a5c742721ec7696e56beee771ddd47e2f8"} Oct 30 12:57:28.236218 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:57:28.236174 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-eab799/test-model-eab799-kserve-7b456659fc-bgzth"] Oct 30 12:57:28.240239 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:57:28.240215 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-eab799/test-model-eab799-kserve-7b456659fc-bgzth" Oct 30 12:57:28.281810 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:57:28.281073 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-eab799\"/\"openshift-service-ca.crt\"" Oct 30 12:57:28.288756 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:57:28.288715 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-eab799/test-model-eab799-kserve-7b456659fc-bgzth"] Oct 30 12:57:28.294297 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:57:28.294270 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-eab799\"/\"kube-root-ca.crt\"" Oct 30 12:57:28.294697 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:57:28.294672 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-eab799\"/\"default-dockercfg-2wts7\"" Oct 30 12:57:28.294865 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:57:28.294797 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-eab799\"/\"test-model-eab799-kserve-self-signed-certs\"" Oct 30 12:57:28.331570 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:57:28.331533 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad52e800-af7e-46e0-b834-0f86ffd55b96-tls-certs\") pod \"test-model-eab799-kserve-7b456659fc-bgzth\" (UID: \"ad52e800-af7e-46e0-b834-0f86ffd55b96\") " pod="ai-tenant-e2e-inf-a-eab799/test-model-eab799-kserve-7b456659fc-bgzth" Oct 30 12:57:28.331865 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:57:28.331841 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7npr4\" (UniqueName: \"kubernetes.io/projected/ad52e800-af7e-46e0-b834-0f86ffd55b96-kube-api-access-7npr4\") pod \"test-model-eab799-kserve-7b456659fc-bgzth\" (UID: \"ad52e800-af7e-46e0-b834-0f86ffd55b96\") " pod="ai-tenant-e2e-inf-a-eab799/test-model-eab799-kserve-7b456659fc-bgzth" Oct 30 12:57:28.332115 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:57:28.332098 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad52e800-af7e-46e0-b834-0f86ffd55b96-tmp-dir\") pod \"test-model-eab799-kserve-7b456659fc-bgzth\" (UID: \"ad52e800-af7e-46e0-b834-0f86ffd55b96\") " pod="ai-tenant-e2e-inf-a-eab799/test-model-eab799-kserve-7b456659fc-bgzth" Oct 30 12:57:28.332340 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:57:28.332324 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad52e800-af7e-46e0-b834-0f86ffd55b96-dshm\") pod \"test-model-eab799-kserve-7b456659fc-bgzth\" (UID: \"ad52e800-af7e-46e0-b834-0f86ffd55b96\") " pod="ai-tenant-e2e-inf-a-eab799/test-model-eab799-kserve-7b456659fc-bgzth" Oct 30 12:57:28.332553 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:57:28.332539 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad52e800-af7e-46e0-b834-0f86ffd55b96-home\") pod \"test-model-eab799-kserve-7b456659fc-bgzth\" (UID: \"ad52e800-af7e-46e0-b834-0f86ffd55b96\") " pod="ai-tenant-e2e-inf-a-eab799/test-model-eab799-kserve-7b456659fc-bgzth" Oct 30 12:57:28.332721 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:57:28.332652 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad52e800-af7e-46e0-b834-0f86ffd55b96-model-cache\") pod \"test-model-eab799-kserve-7b456659fc-bgzth\" (UID: \"ad52e800-af7e-46e0-b834-0f86ffd55b96\") " pod="ai-tenant-e2e-inf-a-eab799/test-model-eab799-kserve-7b456659fc-bgzth" Oct 30 12:57:28.336470 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:57:28.336441 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9c3f12-openshift-default-78b944cc9c-fdpqh" event={"ID":"a2029481-e8ad-40ef-97b7-81140abedb5c","Type":"ContainerStarted","Data":"77cff5c69f3faa3be287996ceaa831fc7ebd6797f0efb89476d52d6b56ce2413"} Oct 30 12:57:28.382814 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:57:28.382755 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-9c3f12-openshift-default-78b944cc9c-fdpqh" podStartSLOduration=3.382742653 podStartE2EDuration="3.382742653s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:57:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:57:28.382577157 +0000 UTC m=+3804.897005905" watchObservedRunningTime="2026-10-30 12:57:28.382742653 +0000 UTC m=+3804.897171398" Oct 30 12:57:28.433869 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:57:28.433791 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7npr4\" (UniqueName: \"kubernetes.io/projected/ad52e800-af7e-46e0-b834-0f86ffd55b96-kube-api-access-7npr4\") pod \"test-model-eab799-kserve-7b456659fc-bgzth\" (UID: \"ad52e800-af7e-46e0-b834-0f86ffd55b96\") " pod="ai-tenant-e2e-inf-a-eab799/test-model-eab799-kserve-7b456659fc-bgzth" Oct 30 12:57:28.434065 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:57:28.433962 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad52e800-af7e-46e0-b834-0f86ffd55b96-tmp-dir\") pod \"test-model-eab799-kserve-7b456659fc-bgzth\" (UID: \"ad52e800-af7e-46e0-b834-0f86ffd55b96\") " pod="ai-tenant-e2e-inf-a-eab799/test-model-eab799-kserve-7b456659fc-bgzth" Oct 30 12:57:28.434144 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:57:28.434131 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad52e800-af7e-46e0-b834-0f86ffd55b96-dshm\") pod \"test-model-eab799-kserve-7b456659fc-bgzth\" (UID: \"ad52e800-af7e-46e0-b834-0f86ffd55b96\") " pod="ai-tenant-e2e-inf-a-eab799/test-model-eab799-kserve-7b456659fc-bgzth" Oct 30 12:57:28.434261 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:57:28.434245 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad52e800-af7e-46e0-b834-0f86ffd55b96-home\") pod \"test-model-eab799-kserve-7b456659fc-bgzth\" (UID: \"ad52e800-af7e-46e0-b834-0f86ffd55b96\") " pod="ai-tenant-e2e-inf-a-eab799/test-model-eab799-kserve-7b456659fc-bgzth" Oct 30 12:57:28.434332 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:57:28.434280 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad52e800-af7e-46e0-b834-0f86ffd55b96-model-cache\") pod \"test-model-eab799-kserve-7b456659fc-bgzth\" (UID: \"ad52e800-af7e-46e0-b834-0f86ffd55b96\") " pod="ai-tenant-e2e-inf-a-eab799/test-model-eab799-kserve-7b456659fc-bgzth" Oct 30 12:57:28.434393 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:57:28.434362 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad52e800-af7e-46e0-b834-0f86ffd55b96-tls-certs\") pod \"test-model-eab799-kserve-7b456659fc-bgzth\" (UID: \"ad52e800-af7e-46e0-b834-0f86ffd55b96\") " pod="ai-tenant-e2e-inf-a-eab799/test-model-eab799-kserve-7b456659fc-bgzth" Oct 30 12:57:28.435260 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:57:28.435228 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad52e800-af7e-46e0-b834-0f86ffd55b96-tmp-dir\") pod \"test-model-eab799-kserve-7b456659fc-bgzth\" (UID: \"ad52e800-af7e-46e0-b834-0f86ffd55b96\") " pod="ai-tenant-e2e-inf-a-eab799/test-model-eab799-kserve-7b456659fc-bgzth" Oct 30 12:57:28.436861 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:57:28.436793 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad52e800-af7e-46e0-b834-0f86ffd55b96-model-cache\") pod \"test-model-eab799-kserve-7b456659fc-bgzth\" (UID: \"ad52e800-af7e-46e0-b834-0f86ffd55b96\") " pod="ai-tenant-e2e-inf-a-eab799/test-model-eab799-kserve-7b456659fc-bgzth" Oct 30 12:57:28.437210 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:57:28.437186 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad52e800-af7e-46e0-b834-0f86ffd55b96-home\") pod \"test-model-eab799-kserve-7b456659fc-bgzth\" (UID: \"ad52e800-af7e-46e0-b834-0f86ffd55b96\") " pod="ai-tenant-e2e-inf-a-eab799/test-model-eab799-kserve-7b456659fc-bgzth" Oct 30 12:57:28.438321 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:57:28.438292 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad52e800-af7e-46e0-b834-0f86ffd55b96-tls-certs\") pod \"test-model-eab799-kserve-7b456659fc-bgzth\" (UID: \"ad52e800-af7e-46e0-b834-0f86ffd55b96\") " pod="ai-tenant-e2e-inf-a-eab799/test-model-eab799-kserve-7b456659fc-bgzth" Oct 30 12:57:28.441040 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:57:28.441019 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad52e800-af7e-46e0-b834-0f86ffd55b96-dshm\") pod \"test-model-eab799-kserve-7b456659fc-bgzth\" (UID: \"ad52e800-af7e-46e0-b834-0f86ffd55b96\") " pod="ai-tenant-e2e-inf-a-eab799/test-model-eab799-kserve-7b456659fc-bgzth" Oct 30 12:57:28.456394 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:57:28.456366 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7npr4\" (UniqueName: \"kubernetes.io/projected/ad52e800-af7e-46e0-b834-0f86ffd55b96-kube-api-access-7npr4\") pod \"test-model-eab799-kserve-7b456659fc-bgzth\" (UID: \"ad52e800-af7e-46e0-b834-0f86ffd55b96\") " pod="ai-tenant-e2e-inf-a-eab799/test-model-eab799-kserve-7b456659fc-bgzth" Oct 30 12:57:28.597908 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:57:28.593140 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-eab799/test-model-eab799-kserve-7b456659fc-bgzth" Oct 30 12:57:28.644506 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:57:28.644473 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-9c3f12-openshift-default-78b944cc9c-fdpqh" Oct 30 12:57:28.808221 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:57:28.808178 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-9c3f12-openshift-default-78b944cc9c-fdpqh" Oct 30 12:57:29.084415 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:57:29.084385 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-eab799/test-model-eab799-kserve-7b456659fc-bgzth"] Oct 30 12:57:29.087538 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:57:29.087503 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podad52e800_af7e_46e0_b834_0f86ffd55b96.slice/crio-9d204b4f2dff3b913ff6383f5c4255a4926aca0ada052f291c6528ddac6ce9fe WatchSource:0}: Error finding container 9d204b4f2dff3b913ff6383f5c4255a4926aca0ada052f291c6528ddac6ce9fe: Status 404 returned error can't find the container with id 9d204b4f2dff3b913ff6383f5c4255a4926aca0ada052f291c6528ddac6ce9fe Oct 30 12:57:29.346759 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:57:29.346716 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-eab799/test-model-eab799-kserve-7b456659fc-bgzth" event={"ID":"ad52e800-af7e-46e0-b834-0f86ffd55b96","Type":"ContainerStarted","Data":"601c535f3ca2f8e911423cae0a4f0ea209d5ccd8da9d10b9edc81d543ce5edeb"} Oct 30 12:57:29.347343 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:57:29.346768 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-eab799/test-model-eab799-kserve-7b456659fc-bgzth" event={"ID":"ad52e800-af7e-46e0-b834-0f86ffd55b96","Type":"ContainerStarted","Data":"9d204b4f2dff3b913ff6383f5c4255a4926aca0ada052f291c6528ddac6ce9fe"} Oct 30 12:57:29.347343 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:57:29.347079 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-9c3f12-openshift-default-78b944cc9c-fdpqh" Oct 30 12:57:29.348070 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:57:29.348048 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-9c3f12-openshift-default-78b944cc9c-fdpqh" Oct 30 12:57:29.392259 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:57:29.390876 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-eab799/test-model-eab799-kserve-7b456659fc-bgzth" podStartSLOduration=1.390821474 podStartE2EDuration="1.390821474s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:57:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:57:29.387893213 +0000 UTC m=+3805.902321984" watchObservedRunningTime="2026-10-30 12:57:29.390821474 +0000 UTC m=+3805.905250223" Oct 30 12:57:29.642087 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:57:29.641907 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-eab799/test-model-eab799-kserve-7b456659fc-bgzth" Oct 30 12:57:31.411225 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:57:31.409261 4908 patch_prober.go:29] interesting pod/e2e-inf-a-1333c8-openshift-default-847ccc6485-tp95k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.33:15021/healthz/ready\": read tcp 10.132.0.2:54044->10.132.1.33:15021: read: connection reset by peer" start-of-body= Oct 30 12:57:31.411225 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:57:31.409370 4908 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-1333c8-openshift-default-847ccc6485-tp95k" podUID="75b90e54-7d8b-44cf-9e8d-06fe342acc3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.33:15021/healthz/ready\": read tcp 10.132.0.2:54044->10.132.1.33:15021: read: connection reset by peer" Oct 30 12:57:32.393702 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:57:32.393669 4908 generic.go:356] "Generic (PLEG): container finished" podID="75b90e54-7d8b-44cf-9e8d-06fe342acc3b" containerID="aeb670d117bd2951334cae801bb431a0a884adf89d173a4d11efdaa3c2610696" exitCode=0 Oct 30 12:57:32.393928 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:57:32.393738 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1333c8-openshift-default-847ccc6485-tp95k" event={"ID":"75b90e54-7d8b-44cf-9e8d-06fe342acc3b","Type":"ContainerDied","Data":"aeb670d117bd2951334cae801bb431a0a884adf89d173a4d11efdaa3c2610696"} Oct 30 12:57:32.394300 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:57:32.394278 4908 scope.go:122] "RemoveContainer" containerID="aeb670d117bd2951334cae801bb431a0a884adf89d173a4d11efdaa3c2610696" Oct 30 12:57:32.396516 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:57:32.396486 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:57:32.396598 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:57:32.396559 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:57:32.396598 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:57:32.396591 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:57:33.544779 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:57:33.544160 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1333c8-openshift-default-847ccc6485-tp95k" event={"ID":"75b90e54-7d8b-44cf-9e8d-06fe342acc3b","Type":"ContainerStarted","Data":"dc3a66352a2581e2b79c62f2c986f51e7f7feeccee51d3d0b74097990e1bc134"} Oct 30 12:57:33.910749 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:57:33.910658 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-1333c8-openshift-default-847ccc6485-tp95k" Oct 30 12:57:33.910953 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:57:33.910889 4908 patch_prober.go:29] interesting pod/e2e-inf-a-1333c8-openshift-default-847ccc6485-tp95k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.33:15021/healthz/ready\": dial tcp 10.132.1.33:15021: connect: connection refused" start-of-body= Oct 30 12:57:33.911041 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:57:33.910961 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1333c8-openshift-default-847ccc6485-tp95k" podUID="75b90e54-7d8b-44cf-9e8d-06fe342acc3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.33:15021/healthz/ready\": dial tcp 10.132.1.33:15021: connect: connection refused" Oct 30 12:57:34.918860 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:57:34.918699 4908 patch_prober.go:29] interesting pod/e2e-inf-a-1333c8-openshift-default-847ccc6485-tp95k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.33:15021/healthz/ready\": dial tcp 10.132.1.33:15021: connect: connection refused" start-of-body= Oct 30 12:57:34.918860 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:57:34.918761 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1333c8-openshift-default-847ccc6485-tp95k" podUID="75b90e54-7d8b-44cf-9e8d-06fe342acc3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.33:15021/healthz/ready\": dial tcp 10.132.1.33:15021: connect: connection refused" Oct 30 12:57:35.911543 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:57:35.911499 4908 patch_prober.go:29] interesting pod/e2e-inf-a-1333c8-openshift-default-847ccc6485-tp95k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.33:15021/healthz/ready\": dial tcp 10.132.1.33:15021: connect: connection refused" start-of-body= Oct 30 12:57:35.911772 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:57:35.911564 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1333c8-openshift-default-847ccc6485-tp95k" podUID="75b90e54-7d8b-44cf-9e8d-06fe342acc3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.33:15021/healthz/ready\": dial tcp 10.132.1.33:15021: connect: connection refused" Oct 30 12:57:36.910949 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:57:36.910901 4908 patch_prober.go:29] interesting pod/e2e-inf-a-1333c8-openshift-default-847ccc6485-tp95k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.33:15021/healthz/ready\": dial tcp 10.132.1.33:15021: connect: connection refused" start-of-body= Oct 30 12:57:36.911354 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:57:36.910984 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1333c8-openshift-default-847ccc6485-tp95k" podUID="75b90e54-7d8b-44cf-9e8d-06fe342acc3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.33:15021/healthz/ready\": dial tcp 10.132.1.33:15021: connect: connection refused" Oct 30 12:57:37.910959 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:57:37.910920 4908 patch_prober.go:29] interesting pod/e2e-inf-a-1333c8-openshift-default-847ccc6485-tp95k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.33:15021/healthz/ready\": dial tcp 10.132.1.33:15021: connect: connection refused" start-of-body= Oct 30 12:57:37.911431 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:57:37.910986 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1333c8-openshift-default-847ccc6485-tp95k" podUID="75b90e54-7d8b-44cf-9e8d-06fe342acc3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.33:15021/healthz/ready\": dial tcp 10.132.1.33:15021: connect: connection refused" Oct 30 12:57:38.595314 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:57:38.595224 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-eab799/test-model-eab799-kserve-7b456659fc-bgzth" Oct 30 12:57:38.613658 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:57:38.613335 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-eab799/test-model-eab799-kserve-7b456659fc-bgzth" Oct 30 12:57:38.719794 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:57:38.719766 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-eab799/test-model-eab799-kserve-7b456659fc-bgzth" Oct 30 12:57:38.910770 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:57:38.910691 4908 patch_prober.go:29] interesting pod/e2e-inf-a-1333c8-openshift-default-847ccc6485-tp95k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.33:15021/healthz/ready\": dial tcp 10.132.1.33:15021: connect: connection refused" start-of-body= Oct 30 12:57:38.911060 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:57:38.911031 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1333c8-openshift-default-847ccc6485-tp95k" podUID="75b90e54-7d8b-44cf-9e8d-06fe342acc3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.33:15021/healthz/ready\": dial tcp 10.132.1.33:15021: connect: connection refused" Oct 30 12:57:39.917488 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:57:39.917453 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-1333c8-openshift-default-847ccc6485-tp95k" Oct 30 12:57:39.918052 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:57:39.917943 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-1333c8-openshift-default-847ccc6485-tp95k" Oct 30 12:57:39.918766 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:57:39.918743 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-1333c8-openshift-default-847ccc6485-tp95k" Oct 30 12:57:49.686430 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:57:49.686335 4908 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vn6vv" containerName="extractor" Oct 30 12:57:51.626525 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:57:51.626487 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-e9f184/test-model-e9f184-kserve-86dbfc9485-g9lm6"] Oct 30 12:57:51.632946 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:57:51.632920 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-e9f184/test-model-e9f184-kserve-86dbfc9485-g9lm6" Oct 30 12:57:51.663916 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:57:51.663881 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-e9f184\"/\"openshift-service-ca.crt\"" Oct 30 12:57:51.664094 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:57:51.663979 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-e9f184\"/\"kube-root-ca.crt\"" Oct 30 12:57:51.670929 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:57:51.670892 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-e9f184/test-model-e9f184-kserve-86dbfc9485-g9lm6"] Oct 30 12:57:51.699347 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:57:51.699312 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8396834c-234e-431a-8e62-97c193ce6b66-model-cache\") pod \"test-model-e9f184-kserve-86dbfc9485-g9lm6\" (UID: \"8396834c-234e-431a-8e62-97c193ce6b66\") " pod="ai-tenant-e2e-inf-b-e9f184/test-model-e9f184-kserve-86dbfc9485-g9lm6" Oct 30 12:57:51.699535 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:57:51.699367 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8396834c-234e-431a-8e62-97c193ce6b66-dshm\") pod \"test-model-e9f184-kserve-86dbfc9485-g9lm6\" (UID: \"8396834c-234e-431a-8e62-97c193ce6b66\") " pod="ai-tenant-e2e-inf-b-e9f184/test-model-e9f184-kserve-86dbfc9485-g9lm6" Oct 30 12:57:51.699593 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:57:51.699545 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8396834c-234e-431a-8e62-97c193ce6b66-home\") pod \"test-model-e9f184-kserve-86dbfc9485-g9lm6\" (UID: \"8396834c-234e-431a-8e62-97c193ce6b66\") " pod="ai-tenant-e2e-inf-b-e9f184/test-model-e9f184-kserve-86dbfc9485-g9lm6" Oct 30 12:57:51.699638 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:57:51.699590 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8396834c-234e-431a-8e62-97c193ce6b66-tls-certs\") pod \"test-model-e9f184-kserve-86dbfc9485-g9lm6\" (UID: \"8396834c-234e-431a-8e62-97c193ce6b66\") " pod="ai-tenant-e2e-inf-b-e9f184/test-model-e9f184-kserve-86dbfc9485-g9lm6" Oct 30 12:57:51.699776 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:57:51.699748 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8396834c-234e-431a-8e62-97c193ce6b66-tmp-dir\") pod \"test-model-e9f184-kserve-86dbfc9485-g9lm6\" (UID: \"8396834c-234e-431a-8e62-97c193ce6b66\") " pod="ai-tenant-e2e-inf-b-e9f184/test-model-e9f184-kserve-86dbfc9485-g9lm6" Oct 30 12:57:51.700168 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:57:51.700114 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kzl8\" (UniqueName: \"kubernetes.io/projected/8396834c-234e-431a-8e62-97c193ce6b66-kube-api-access-4kzl8\") pod \"test-model-e9f184-kserve-86dbfc9485-g9lm6\" (UID: \"8396834c-234e-431a-8e62-97c193ce6b66\") " pod="ai-tenant-e2e-inf-b-e9f184/test-model-e9f184-kserve-86dbfc9485-g9lm6" Oct 30 12:57:51.709544 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:57:51.709506 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-e9f184\"/\"default-dockercfg-lvppr\"" Oct 30 12:57:51.709544 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:57:51.709513 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-e9f184\"/\"test-model-e9f184-kserve-self-signed-certs\"" Oct 30 12:57:51.801845 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:57:51.801792 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8396834c-234e-431a-8e62-97c193ce6b66-home\") pod \"test-model-e9f184-kserve-86dbfc9485-g9lm6\" (UID: \"8396834c-234e-431a-8e62-97c193ce6b66\") " pod="ai-tenant-e2e-inf-b-e9f184/test-model-e9f184-kserve-86dbfc9485-g9lm6" Oct 30 12:57:51.802060 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:57:51.801869 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8396834c-234e-431a-8e62-97c193ce6b66-tls-certs\") pod \"test-model-e9f184-kserve-86dbfc9485-g9lm6\" (UID: \"8396834c-234e-431a-8e62-97c193ce6b66\") " pod="ai-tenant-e2e-inf-b-e9f184/test-model-e9f184-kserve-86dbfc9485-g9lm6" Oct 30 12:57:51.802060 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:57:51.801952 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8396834c-234e-431a-8e62-97c193ce6b66-tmp-dir\") pod \"test-model-e9f184-kserve-86dbfc9485-g9lm6\" (UID: \"8396834c-234e-431a-8e62-97c193ce6b66\") " pod="ai-tenant-e2e-inf-b-e9f184/test-model-e9f184-kserve-86dbfc9485-g9lm6" Oct 30 12:57:51.802206 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:57:51.802186 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kzl8\" (UniqueName: \"kubernetes.io/projected/8396834c-234e-431a-8e62-97c193ce6b66-kube-api-access-4kzl8\") pod \"test-model-e9f184-kserve-86dbfc9485-g9lm6\" (UID: \"8396834c-234e-431a-8e62-97c193ce6b66\") " pod="ai-tenant-e2e-inf-b-e9f184/test-model-e9f184-kserve-86dbfc9485-g9lm6" Oct 30 12:57:51.802300 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:57:51.802262 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8396834c-234e-431a-8e62-97c193ce6b66-model-cache\") pod \"test-model-e9f184-kserve-86dbfc9485-g9lm6\" (UID: \"8396834c-234e-431a-8e62-97c193ce6b66\") " pod="ai-tenant-e2e-inf-b-e9f184/test-model-e9f184-kserve-86dbfc9485-g9lm6" Oct 30 12:57:51.802453 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:57:51.802318 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8396834c-234e-431a-8e62-97c193ce6b66-dshm\") pod \"test-model-e9f184-kserve-86dbfc9485-g9lm6\" (UID: \"8396834c-234e-431a-8e62-97c193ce6b66\") " pod="ai-tenant-e2e-inf-b-e9f184/test-model-e9f184-kserve-86dbfc9485-g9lm6" Oct 30 12:57:51.802453 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:57:51.802326 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8396834c-234e-431a-8e62-97c193ce6b66-home\") pod \"test-model-e9f184-kserve-86dbfc9485-g9lm6\" (UID: \"8396834c-234e-431a-8e62-97c193ce6b66\") " pod="ai-tenant-e2e-inf-b-e9f184/test-model-e9f184-kserve-86dbfc9485-g9lm6" Oct 30 12:57:51.802648 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:57:51.802621 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8396834c-234e-431a-8e62-97c193ce6b66-tmp-dir\") pod \"test-model-e9f184-kserve-86dbfc9485-g9lm6\" (UID: \"8396834c-234e-431a-8e62-97c193ce6b66\") " pod="ai-tenant-e2e-inf-b-e9f184/test-model-e9f184-kserve-86dbfc9485-g9lm6" Oct 30 12:57:51.802725 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:57:51.802692 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8396834c-234e-431a-8e62-97c193ce6b66-model-cache\") pod \"test-model-e9f184-kserve-86dbfc9485-g9lm6\" (UID: \"8396834c-234e-431a-8e62-97c193ce6b66\") " pod="ai-tenant-e2e-inf-b-e9f184/test-model-e9f184-kserve-86dbfc9485-g9lm6" Oct 30 12:57:51.805452 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:57:51.805419 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8396834c-234e-431a-8e62-97c193ce6b66-dshm\") pod \"test-model-e9f184-kserve-86dbfc9485-g9lm6\" (UID: \"8396834c-234e-431a-8e62-97c193ce6b66\") " pod="ai-tenant-e2e-inf-b-e9f184/test-model-e9f184-kserve-86dbfc9485-g9lm6" Oct 30 12:57:51.806036 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:57:51.806014 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8396834c-234e-431a-8e62-97c193ce6b66-tls-certs\") pod \"test-model-e9f184-kserve-86dbfc9485-g9lm6\" (UID: \"8396834c-234e-431a-8e62-97c193ce6b66\") " pod="ai-tenant-e2e-inf-b-e9f184/test-model-e9f184-kserve-86dbfc9485-g9lm6" Oct 30 12:57:51.824416 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:57:51.824388 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kzl8\" (UniqueName: \"kubernetes.io/projected/8396834c-234e-431a-8e62-97c193ce6b66-kube-api-access-4kzl8\") pod \"test-model-e9f184-kserve-86dbfc9485-g9lm6\" (UID: \"8396834c-234e-431a-8e62-97c193ce6b66\") " pod="ai-tenant-e2e-inf-b-e9f184/test-model-e9f184-kserve-86dbfc9485-g9lm6" Oct 30 12:57:52.011853 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:57:51.997886 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-e9f184/test-model-e9f184-kserve-86dbfc9485-g9lm6" Oct 30 12:57:52.481114 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:57:52.481065 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-e9f184/test-model-e9f184-kserve-86dbfc9485-g9lm6"] Oct 30 12:57:52.485113 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:57:52.485077 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8396834c_234e_431a_8e62_97c193ce6b66.slice/crio-67b9efcbbc9fbcf79346699f505696761695d0d889822cf379e2d1fe36ac62b5 WatchSource:0}: Error finding container 67b9efcbbc9fbcf79346699f505696761695d0d889822cf379e2d1fe36ac62b5: Status 404 returned error can't find the container with id 67b9efcbbc9fbcf79346699f505696761695d0d889822cf379e2d1fe36ac62b5 Oct 30 12:57:52.851184 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:57:52.851101 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-e9f184/test-model-e9f184-kserve-86dbfc9485-g9lm6" event={"ID":"8396834c-234e-431a-8e62-97c193ce6b66","Type":"ContainerStarted","Data":"b68646389cae46ea141fd7361c2285df0eaa0dfd5d29ca1a62fea426df161299"} Oct 30 12:57:52.851184 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:57:52.851142 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-e9f184/test-model-e9f184-kserve-86dbfc9485-g9lm6" event={"ID":"8396834c-234e-431a-8e62-97c193ce6b66","Type":"ContainerStarted","Data":"67b9efcbbc9fbcf79346699f505696761695d0d889822cf379e2d1fe36ac62b5"} Oct 30 12:57:52.894593 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:57:52.894544 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-e9f184/test-model-e9f184-kserve-86dbfc9485-g9lm6" podStartSLOduration=1.894528909 podStartE2EDuration="1.894528909s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:57:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:57:52.889994361 +0000 UTC m=+3829.404423109" watchObservedRunningTime="2026-10-30 12:57:52.894528909 +0000 UTC m=+3829.408957676" Oct 30 12:57:53.020855 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:57:53.013152 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-e9f184/test-model-e9f184-kserve-86dbfc9485-g9lm6" Oct 30 12:57:59.800857 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:57:59.800064 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9wwfc"] Oct 30 12:58:02.013424 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:58:02.013387 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-e9f184/test-model-e9f184-kserve-86dbfc9485-g9lm6" Oct 30 12:58:02.027976 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:58:02.027947 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-e9f184/test-model-e9f184-kserve-86dbfc9485-g9lm6" Oct 30 12:58:03.025119 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:58:03.025079 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-e9f184/test-model-e9f184-kserve-86dbfc9485-g9lm6" Oct 30 12:58:12.181312 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:58:12.181278 4908 generic.go:356] "Generic (PLEG): container finished" podID="7e96c960-abe0-4b56-b3f9-ce46f2ce307a" containerID="0f32041b6b6e591be1af63761a9a5f99db3675b682c4ee06b9fa1e362c1dbf5d" exitCode=0 Oct 30 12:58:12.182167 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:58:12.181361 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-734ffe-openshift-default-5db45fbc99-5vzmr" event={"ID":"7e96c960-abe0-4b56-b3f9-ce46f2ce307a","Type":"ContainerDied","Data":"0f32041b6b6e591be1af63761a9a5f99db3675b682c4ee06b9fa1e362c1dbf5d"} Oct 30 12:58:12.182167 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:58:12.181974 4908 scope.go:122] "RemoveContainer" containerID="0f32041b6b6e591be1af63761a9a5f99db3675b682c4ee06b9fa1e362c1dbf5d" Oct 30 12:58:12.184819 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:58:12.184774 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:58:12.184950 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:58:12.184888 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:58:12.184950 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:58:12.184938 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:58:13.194324 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:58:13.194282 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-734ffe-openshift-default-5db45fbc99-5vzmr" event={"ID":"7e96c960-abe0-4b56-b3f9-ce46f2ce307a","Type":"ContainerStarted","Data":"e44f7f042c249b44a8aeea37b5f6c38de531e2e11fbaac541006930b57b32133"} Oct 30 12:58:13.599214 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:58:13.599171 4908 patch_prober.go:29] interesting pod/e2e-inf-b-734ffe-openshift-default-5db45fbc99-5vzmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.36:15021/healthz/ready\": dial tcp 10.132.1.36:15021: connect: connection refused" start-of-body= Oct 30 12:58:13.599433 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:58:13.599244 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-734ffe-openshift-default-5db45fbc99-5vzmr" podUID="7e96c960-abe0-4b56-b3f9-ce46f2ce307a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.36:15021/healthz/ready\": dial tcp 10.132.1.36:15021: connect: connection refused" Oct 30 12:58:13.608454 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:58:13.608424 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-734ffe-openshift-default-5db45fbc99-5vzmr" Oct 30 12:58:14.599726 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:58:14.599248 4908 patch_prober.go:29] interesting pod/e2e-inf-b-734ffe-openshift-default-5db45fbc99-5vzmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.36:15021/healthz/ready\": dial tcp 10.132.1.36:15021: connect: connection refused" start-of-body= Oct 30 12:58:14.599726 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:58:14.599314 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-734ffe-openshift-default-5db45fbc99-5vzmr" podUID="7e96c960-abe0-4b56-b3f9-ce46f2ce307a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.36:15021/healthz/ready\": dial tcp 10.132.1.36:15021: connect: connection refused" Oct 30 12:58:15.601752 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:58:15.601701 4908 patch_prober.go:29] interesting pod/e2e-inf-b-734ffe-openshift-default-5db45fbc99-5vzmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.36:15021/healthz/ready\": dial tcp 10.132.1.36:15021: connect: connection refused" start-of-body= Oct 30 12:58:15.602345 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:58:15.602321 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-734ffe-openshift-default-5db45fbc99-5vzmr" podUID="7e96c960-abe0-4b56-b3f9-ce46f2ce307a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.36:15021/healthz/ready\": dial tcp 10.132.1.36:15021: connect: connection refused" Oct 30 12:58:16.598890 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:58:16.598843 4908 patch_prober.go:29] interesting pod/e2e-inf-b-734ffe-openshift-default-5db45fbc99-5vzmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.36:15021/healthz/ready\": dial tcp 10.132.1.36:15021: connect: connection refused" start-of-body= Oct 30 12:58:16.599114 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:58:16.598909 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-734ffe-openshift-default-5db45fbc99-5vzmr" podUID="7e96c960-abe0-4b56-b3f9-ce46f2ce307a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.36:15021/healthz/ready\": dial tcp 10.132.1.36:15021: connect: connection refused" Oct 30 12:58:17.599477 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:58:17.599421 4908 patch_prober.go:29] interesting pod/e2e-inf-b-734ffe-openshift-default-5db45fbc99-5vzmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.36:15021/healthz/ready\": dial tcp 10.132.1.36:15021: connect: connection refused" start-of-body= Oct 30 12:58:17.600010 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:58:17.599496 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-734ffe-openshift-default-5db45fbc99-5vzmr" podUID="7e96c960-abe0-4b56-b3f9-ce46f2ce307a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.36:15021/healthz/ready\": dial tcp 10.132.1.36:15021: connect: connection refused" Oct 30 12:58:18.599791 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:58:18.599752 4908 patch_prober.go:29] interesting pod/e2e-inf-b-734ffe-openshift-default-5db45fbc99-5vzmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.36:15021/healthz/ready\": dial tcp 10.132.1.36:15021: connect: connection refused" start-of-body= Oct 30 12:58:18.600348 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:58:18.599844 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-734ffe-openshift-default-5db45fbc99-5vzmr" podUID="7e96c960-abe0-4b56-b3f9-ce46f2ce307a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.36:15021/healthz/ready\": dial tcp 10.132.1.36:15021: connect: connection refused" Oct 30 12:58:19.609513 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:58:19.609477 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-734ffe-openshift-default-5db45fbc99-5vzmr" Oct 30 12:58:19.610065 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:58:19.609866 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-734ffe-openshift-default-5db45fbc99-5vzmr" Oct 30 12:58:19.610806 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:58:19.610784 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-734ffe-openshift-default-5db45fbc99-5vzmr" Oct 30 12:58:19.687042 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:58:19.687005 4908 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vn6vv" containerName="extractor" Oct 30 12:58:37.066964 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:58:37.066919 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0cc48ca4-f8pxw"] Oct 30 12:58:37.080386 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:58:37.080353 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0cc48ca4-f8pxw"] Oct 30 12:58:37.609239 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:58:37.609202 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ccb8306-200d-492a-86d6-c381f6e7b34d" path="/var/lib/kubelet/pods/7ccb8306-200d-492a-86d6-c381f6e7b34d/volumes" Oct 30 12:58:37.688968 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:58:37.688933 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-63f4e4/test-model-63f4e4-kserve-57bdcf4b79-cgctl"] Oct 30 12:58:37.695057 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:58:37.695024 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-63f4e4/test-model-63f4e4-kserve-57bdcf4b79-cgctl" Oct 30 12:58:37.730752 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:58:37.730718 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-63f4e4\"/\"test-model-63f4e4-kserve-self-signed-certs\"" Oct 30 12:58:37.731013 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:58:37.730875 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-63f4e4\"/\"default-dockercfg-jf7rb\"" Oct 30 12:58:37.731918 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:58:37.731887 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-63f4e4/test-model-63f4e4-kserve-57bdcf4b79-cgctl"] Oct 30 12:58:37.760180 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:58:37.760134 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3a9d487-206d-4027-b563-81d2db998422-dshm\") pod \"test-model-63f4e4-kserve-57bdcf4b79-cgctl\" (UID: \"b3a9d487-206d-4027-b563-81d2db998422\") " pod="ai-tenant-e2e-inf-a-63f4e4/test-model-63f4e4-kserve-57bdcf4b79-cgctl" Oct 30 12:58:37.760525 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:58:37.760473 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3a9d487-206d-4027-b563-81d2db998422-model-cache\") pod \"test-model-63f4e4-kserve-57bdcf4b79-cgctl\" (UID: \"b3a9d487-206d-4027-b563-81d2db998422\") " pod="ai-tenant-e2e-inf-a-63f4e4/test-model-63f4e4-kserve-57bdcf4b79-cgctl" Oct 30 12:58:37.760701 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:58:37.760684 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3a9d487-206d-4027-b563-81d2db998422-home\") pod \"test-model-63f4e4-kserve-57bdcf4b79-cgctl\" (UID: \"b3a9d487-206d-4027-b563-81d2db998422\") " pod="ai-tenant-e2e-inf-a-63f4e4/test-model-63f4e4-kserve-57bdcf4b79-cgctl" Oct 30 12:58:37.760935 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:58:37.760917 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3a9d487-206d-4027-b563-81d2db998422-tmp-dir\") pod \"test-model-63f4e4-kserve-57bdcf4b79-cgctl\" (UID: \"b3a9d487-206d-4027-b563-81d2db998422\") " pod="ai-tenant-e2e-inf-a-63f4e4/test-model-63f4e4-kserve-57bdcf4b79-cgctl" Oct 30 12:58:37.761082 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:58:37.761062 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3a9d487-206d-4027-b563-81d2db998422-tls-certs\") pod \"test-model-63f4e4-kserve-57bdcf4b79-cgctl\" (UID: \"b3a9d487-206d-4027-b563-81d2db998422\") " pod="ai-tenant-e2e-inf-a-63f4e4/test-model-63f4e4-kserve-57bdcf4b79-cgctl" Oct 30 12:58:37.761337 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:58:37.761297 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mf8jw\" (UniqueName: \"kubernetes.io/projected/b3a9d487-206d-4027-b563-81d2db998422-kube-api-access-mf8jw\") pod \"test-model-63f4e4-kserve-57bdcf4b79-cgctl\" (UID: \"b3a9d487-206d-4027-b563-81d2db998422\") " pod="ai-tenant-e2e-inf-a-63f4e4/test-model-63f4e4-kserve-57bdcf4b79-cgctl" Oct 30 12:58:37.795488 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:58:37.795438 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-63f4e4\"/\"openshift-service-ca.crt\"" Oct 30 12:58:37.795687 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:58:37.795523 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-63f4e4\"/\"kube-root-ca.crt\"" Oct 30 12:58:37.863295 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:58:37.863204 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3a9d487-206d-4027-b563-81d2db998422-dshm\") pod \"test-model-63f4e4-kserve-57bdcf4b79-cgctl\" (UID: \"b3a9d487-206d-4027-b563-81d2db998422\") " pod="ai-tenant-e2e-inf-a-63f4e4/test-model-63f4e4-kserve-57bdcf4b79-cgctl" Oct 30 12:58:37.863295 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:58:37.863266 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3a9d487-206d-4027-b563-81d2db998422-model-cache\") pod \"test-model-63f4e4-kserve-57bdcf4b79-cgctl\" (UID: \"b3a9d487-206d-4027-b563-81d2db998422\") " pod="ai-tenant-e2e-inf-a-63f4e4/test-model-63f4e4-kserve-57bdcf4b79-cgctl" Oct 30 12:58:37.863560 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:58:37.863302 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3a9d487-206d-4027-b563-81d2db998422-home\") pod \"test-model-63f4e4-kserve-57bdcf4b79-cgctl\" (UID: \"b3a9d487-206d-4027-b563-81d2db998422\") " pod="ai-tenant-e2e-inf-a-63f4e4/test-model-63f4e4-kserve-57bdcf4b79-cgctl" Oct 30 12:58:37.863560 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:58:37.863385 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3a9d487-206d-4027-b563-81d2db998422-tmp-dir\") pod \"test-model-63f4e4-kserve-57bdcf4b79-cgctl\" (UID: \"b3a9d487-206d-4027-b563-81d2db998422\") " pod="ai-tenant-e2e-inf-a-63f4e4/test-model-63f4e4-kserve-57bdcf4b79-cgctl" Oct 30 12:58:37.863560 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:58:37.863409 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3a9d487-206d-4027-b563-81d2db998422-tls-certs\") pod \"test-model-63f4e4-kserve-57bdcf4b79-cgctl\" (UID: \"b3a9d487-206d-4027-b563-81d2db998422\") " pod="ai-tenant-e2e-inf-a-63f4e4/test-model-63f4e4-kserve-57bdcf4b79-cgctl" Oct 30 12:58:37.863560 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:58:37.863500 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mf8jw\" (UniqueName: \"kubernetes.io/projected/b3a9d487-206d-4027-b563-81d2db998422-kube-api-access-mf8jw\") pod \"test-model-63f4e4-kserve-57bdcf4b79-cgctl\" (UID: \"b3a9d487-206d-4027-b563-81d2db998422\") " pod="ai-tenant-e2e-inf-a-63f4e4/test-model-63f4e4-kserve-57bdcf4b79-cgctl" Oct 30 12:58:37.865302 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:58:37.865246 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3a9d487-206d-4027-b563-81d2db998422-home\") pod \"test-model-63f4e4-kserve-57bdcf4b79-cgctl\" (UID: \"b3a9d487-206d-4027-b563-81d2db998422\") " pod="ai-tenant-e2e-inf-a-63f4e4/test-model-63f4e4-kserve-57bdcf4b79-cgctl" Oct 30 12:58:37.865695 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:58:37.865668 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3a9d487-206d-4027-b563-81d2db998422-tmp-dir\") pod \"test-model-63f4e4-kserve-57bdcf4b79-cgctl\" (UID: \"b3a9d487-206d-4027-b563-81d2db998422\") " pod="ai-tenant-e2e-inf-a-63f4e4/test-model-63f4e4-kserve-57bdcf4b79-cgctl" Oct 30 12:58:37.866489 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:58:37.866437 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3a9d487-206d-4027-b563-81d2db998422-model-cache\") pod \"test-model-63f4e4-kserve-57bdcf4b79-cgctl\" (UID: \"b3a9d487-206d-4027-b563-81d2db998422\") " pod="ai-tenant-e2e-inf-a-63f4e4/test-model-63f4e4-kserve-57bdcf4b79-cgctl" Oct 30 12:58:37.868668 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:58:37.868624 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3a9d487-206d-4027-b563-81d2db998422-dshm\") pod \"test-model-63f4e4-kserve-57bdcf4b79-cgctl\" (UID: \"b3a9d487-206d-4027-b563-81d2db998422\") " pod="ai-tenant-e2e-inf-a-63f4e4/test-model-63f4e4-kserve-57bdcf4b79-cgctl" Oct 30 12:58:37.869786 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:58:37.869742 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3a9d487-206d-4027-b563-81d2db998422-tls-certs\") pod \"test-model-63f4e4-kserve-57bdcf4b79-cgctl\" (UID: \"b3a9d487-206d-4027-b563-81d2db998422\") " pod="ai-tenant-e2e-inf-a-63f4e4/test-model-63f4e4-kserve-57bdcf4b79-cgctl" Oct 30 12:58:37.911327 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:58:37.911286 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mf8jw\" (UniqueName: \"kubernetes.io/projected/b3a9d487-206d-4027-b563-81d2db998422-kube-api-access-mf8jw\") pod \"test-model-63f4e4-kserve-57bdcf4b79-cgctl\" (UID: \"b3a9d487-206d-4027-b563-81d2db998422\") " pod="ai-tenant-e2e-inf-a-63f4e4/test-model-63f4e4-kserve-57bdcf4b79-cgctl" Oct 30 12:58:38.035739 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:58:38.035104 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-63f4e4/test-model-63f4e4-kserve-57bdcf4b79-cgctl" Oct 30 12:58:38.546941 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:58:38.546905 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb3a9d487_206d_4027_b563_81d2db998422.slice/crio-b483af6ce64f9a474b463379c4797455faf2aa0d31c2df42369c82e167b176ff WatchSource:0}: Error finding container b483af6ce64f9a474b463379c4797455faf2aa0d31c2df42369c82e167b176ff: Status 404 returned error can't find the container with id b483af6ce64f9a474b463379c4797455faf2aa0d31c2df42369c82e167b176ff Oct 30 12:58:38.547671 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:58:38.547642 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-63f4e4/test-model-63f4e4-kserve-57bdcf4b79-cgctl"] Oct 30 12:58:38.680899 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:58:38.680770 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-63f4e4/test-model-63f4e4-kserve-57bdcf4b79-cgctl" event={"ID":"b3a9d487-206d-4027-b563-81d2db998422","Type":"ContainerStarted","Data":"a8acc0048246a2aff4ebe1c30827792d5fb04c84f22516061f37988fc01081e5"} Oct 30 12:58:38.680899 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:58:38.680852 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-63f4e4/test-model-63f4e4-kserve-57bdcf4b79-cgctl" event={"ID":"b3a9d487-206d-4027-b563-81d2db998422","Type":"ContainerStarted","Data":"b483af6ce64f9a474b463379c4797455faf2aa0d31c2df42369c82e167b176ff"} Oct 30 12:58:38.683295 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:58:38.683228 4908 generic.go:356] "Generic (PLEG): container finished" podID="93f3f694-3b06-430c-b4f0-ed5afbea43d1" containerID="fd793e758613ea073329301aa04b1d67dc621e22ffb6618363f63290a4ffbe51" exitCode=0 Oct 30 12:58:38.683460 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:58:38.683302 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-befaa6-openshift-default-767bb98587-6bvqz" event={"ID":"93f3f694-3b06-430c-b4f0-ed5afbea43d1","Type":"ContainerDied","Data":"fd793e758613ea073329301aa04b1d67dc621e22ffb6618363f63290a4ffbe51"} Oct 30 12:58:38.683788 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:58:38.683772 4908 scope.go:122] "RemoveContainer" containerID="fd793e758613ea073329301aa04b1d67dc621e22ffb6618363f63290a4ffbe51" Oct 30 12:58:38.686174 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:58:38.686144 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:58:38.686302 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:58:38.686219 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:58:38.686302 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:58:38.686256 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:58:38.728166 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:58:38.728113 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-63f4e4/test-model-63f4e4-kserve-57bdcf4b79-cgctl" podStartSLOduration=1.728094373 podStartE2EDuration="1.728094373s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:58:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:58:38.72313377 +0000 UTC m=+3875.237562519" watchObservedRunningTime="2026-10-30 12:58:38.728094373 +0000 UTC m=+3875.242523119" Oct 30 12:58:39.038093 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:58:39.036217 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-63f4e4/test-model-63f4e4-kserve-57bdcf4b79-cgctl" Oct 30 12:58:39.699483 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:58:39.699439 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-befaa6-openshift-default-767bb98587-6bvqz" event={"ID":"93f3f694-3b06-430c-b4f0-ed5afbea43d1","Type":"ContainerStarted","Data":"8894ef6754187ee1bbbcb40065dbe64af9a2a5d915d0602b37488466a274eda0"} Oct 30 12:58:40.005323 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:58:40.005284 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-befaa6-openshift-default-767bb98587-6bvqz" Oct 30 12:58:40.005506 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:58:40.005472 4908 patch_prober.go:29] interesting pod/e2e-inf-b-befaa6-openshift-default-767bb98587-6bvqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.37:15021/healthz/ready\": dial tcp 10.132.1.37:15021: connect: connection refused" start-of-body= Oct 30 12:58:40.005586 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:58:40.005530 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-befaa6-openshift-default-767bb98587-6bvqz" podUID="93f3f694-3b06-430c-b4f0-ed5afbea43d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.37:15021/healthz/ready\": dial tcp 10.132.1.37:15021: connect: connection refused" Oct 30 12:58:41.005602 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:58:41.005558 4908 patch_prober.go:29] interesting pod/e2e-inf-b-befaa6-openshift-default-767bb98587-6bvqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.37:15021/healthz/ready\": dial tcp 10.132.1.37:15021: connect: connection refused" start-of-body= Oct 30 12:58:41.006102 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:58:41.005624 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-befaa6-openshift-default-767bb98587-6bvqz" podUID="93f3f694-3b06-430c-b4f0-ed5afbea43d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.37:15021/healthz/ready\": dial tcp 10.132.1.37:15021: connect: connection refused" Oct 30 12:58:42.006179 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:58:42.006143 4908 patch_prober.go:29] interesting pod/e2e-inf-b-befaa6-openshift-default-767bb98587-6bvqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.37:15021/healthz/ready\": dial tcp 10.132.1.37:15021: connect: connection refused" start-of-body= Oct 30 12:58:42.006596 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:58:42.006200 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-befaa6-openshift-default-767bb98587-6bvqz" podUID="93f3f694-3b06-430c-b4f0-ed5afbea43d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.37:15021/healthz/ready\": dial tcp 10.132.1.37:15021: connect: connection refused" Oct 30 12:58:43.005898 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:58:43.005867 4908 patch_prober.go:29] interesting pod/e2e-inf-b-befaa6-openshift-default-767bb98587-6bvqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.37:15021/healthz/ready\": dial tcp 10.132.1.37:15021: connect: connection refused" start-of-body= Oct 30 12:58:43.006143 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:58:43.006122 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-befaa6-openshift-default-767bb98587-6bvqz" podUID="93f3f694-3b06-430c-b4f0-ed5afbea43d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.37:15021/healthz/ready\": dial tcp 10.132.1.37:15021: connect: connection refused" Oct 30 12:58:44.005536 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:58:44.005499 4908 patch_prober.go:29] interesting pod/e2e-inf-b-befaa6-openshift-default-767bb98587-6bvqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.37:15021/healthz/ready\": dial tcp 10.132.1.37:15021: connect: connection refused" start-of-body= Oct 30 12:58:44.005994 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:58:44.005556 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-befaa6-openshift-default-767bb98587-6bvqz" podUID="93f3f694-3b06-430c-b4f0-ed5afbea43d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.37:15021/healthz/ready\": dial tcp 10.132.1.37:15021: connect: connection refused" Oct 30 12:58:45.006229 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:58:45.006180 4908 patch_prober.go:29] interesting pod/e2e-inf-b-befaa6-openshift-default-767bb98587-6bvqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.37:15021/healthz/ready\": dial tcp 10.132.1.37:15021: connect: connection refused" start-of-body= Oct 30 12:58:45.006746 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:58:45.006255 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-befaa6-openshift-default-767bb98587-6bvqz" podUID="93f3f694-3b06-430c-b4f0-ed5afbea43d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.37:15021/healthz/ready\": dial tcp 10.132.1.37:15021: connect: connection refused" Oct 30 12:58:45.970632 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:58:45.970580 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-befaa6-openshift-default-767bb98587-6bvqz" Oct 30 12:58:46.011590 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:58:46.011556 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-befaa6-openshift-default-767bb98587-6bvqz" Oct 30 12:58:46.776635 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:58:46.774746 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-befaa6-openshift-default-767bb98587-6bvqz" Oct 30 12:58:48.036567 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:58:48.036529 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-63f4e4/test-model-63f4e4-kserve-57bdcf4b79-cgctl" Oct 30 12:58:48.049284 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:58:48.049254 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-63f4e4/test-model-63f4e4-kserve-57bdcf4b79-cgctl" Oct 30 12:58:48.808996 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:58:48.808962 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-63f4e4/test-model-63f4e4-kserve-57bdcf4b79-cgctl" Oct 30 12:58:49.686155 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:58:49.686114 4908 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vn6vv" containerName="extractor" Oct 30 12:58:50.039648 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:58:50.039609 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-94eab2bb-hzvrh"] Oct 30 12:58:50.050235 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:58:50.050209 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-94eab2bb-hzvrh"] Oct 30 12:58:51.609153 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:58:51.609118 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5d09275-8c9d-4118-ac23-c7452ecee0c6" path="/var/lib/kubelet/pods/b5d09275-8c9d-4118-ac23-c7452ecee0c6/volumes" Oct 30 12:58:58.672862 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:58:58.670355 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9wwfc"] Oct 30 12:59:00.186856 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:59:00.186345 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5424a42b-knz9w"] Oct 30 12:59:00.341889 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:59:00.305793 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5424a42b-knz9w"] Oct 30 12:59:01.615660 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:59:01.615618 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5abc3d06-43b1-421b-95d2-be659f9668c4" path="/var/lib/kubelet/pods/5abc3d06-43b1-421b-95d2-be659f9668c4/volumes" Oct 30 12:59:03.438176 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:59:03.438136 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-734ffe/test-model-734ffe-kserve-7c69bdf48f-46tft"] Oct 30 12:59:03.443924 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:59:03.443891 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-734ffe/test-model-734ffe-kserve-7c69bdf48f-46tft" Oct 30 12:59:03.479367 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:59:03.479336 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-734ffe\"/\"kube-root-ca.crt\"" Oct 30 12:59:03.480497 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:59:03.480464 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-734ffe\"/\"openshift-service-ca.crt\"" Oct 30 12:59:03.480837 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:59:03.480796 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-734ffe\"/\"test-model-734ffe-kserve-self-signed-certs\"" Oct 30 12:59:03.481279 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:59:03.481263 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-734ffe\"/\"default-dockercfg-xm7lw\"" Oct 30 12:59:03.505188 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:59:03.505160 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-734ffe/test-model-734ffe-kserve-7c69bdf48f-46tft"] Oct 30 12:59:03.517671 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:59:03.517639 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56318ef3-7e27-4b11-9344-46778652e73a-tmp-dir\") pod \"test-model-734ffe-kserve-7c69bdf48f-46tft\" (UID: \"56318ef3-7e27-4b11-9344-46778652e73a\") " pod="ai-tenant-e2e-inf-b-734ffe/test-model-734ffe-kserve-7c69bdf48f-46tft" Oct 30 12:59:03.517912 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:59:03.517878 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56318ef3-7e27-4b11-9344-46778652e73a-home\") pod \"test-model-734ffe-kserve-7c69bdf48f-46tft\" (UID: \"56318ef3-7e27-4b11-9344-46778652e73a\") " pod="ai-tenant-e2e-inf-b-734ffe/test-model-734ffe-kserve-7c69bdf48f-46tft" Oct 30 12:59:03.518304 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:59:03.518280 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6g5nf\" (UniqueName: \"kubernetes.io/projected/56318ef3-7e27-4b11-9344-46778652e73a-kube-api-access-6g5nf\") pod \"test-model-734ffe-kserve-7c69bdf48f-46tft\" (UID: \"56318ef3-7e27-4b11-9344-46778652e73a\") " pod="ai-tenant-e2e-inf-b-734ffe/test-model-734ffe-kserve-7c69bdf48f-46tft" Oct 30 12:59:03.518584 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:59:03.518565 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56318ef3-7e27-4b11-9344-46778652e73a-tls-certs\") pod \"test-model-734ffe-kserve-7c69bdf48f-46tft\" (UID: \"56318ef3-7e27-4b11-9344-46778652e73a\") " pod="ai-tenant-e2e-inf-b-734ffe/test-model-734ffe-kserve-7c69bdf48f-46tft" Oct 30 12:59:03.518737 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:59:03.518721 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56318ef3-7e27-4b11-9344-46778652e73a-dshm\") pod \"test-model-734ffe-kserve-7c69bdf48f-46tft\" (UID: \"56318ef3-7e27-4b11-9344-46778652e73a\") " pod="ai-tenant-e2e-inf-b-734ffe/test-model-734ffe-kserve-7c69bdf48f-46tft" Oct 30 12:59:03.518923 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:59:03.518903 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56318ef3-7e27-4b11-9344-46778652e73a-model-cache\") pod \"test-model-734ffe-kserve-7c69bdf48f-46tft\" (UID: \"56318ef3-7e27-4b11-9344-46778652e73a\") " pod="ai-tenant-e2e-inf-b-734ffe/test-model-734ffe-kserve-7c69bdf48f-46tft" Oct 30 12:59:03.620670 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:59:03.620627 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56318ef3-7e27-4b11-9344-46778652e73a-tmp-dir\") pod \"test-model-734ffe-kserve-7c69bdf48f-46tft\" (UID: \"56318ef3-7e27-4b11-9344-46778652e73a\") " pod="ai-tenant-e2e-inf-b-734ffe/test-model-734ffe-kserve-7c69bdf48f-46tft" Oct 30 12:59:03.621519 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:59:03.621492 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56318ef3-7e27-4b11-9344-46778652e73a-home\") pod \"test-model-734ffe-kserve-7c69bdf48f-46tft\" (UID: \"56318ef3-7e27-4b11-9344-46778652e73a\") " pod="ai-tenant-e2e-inf-b-734ffe/test-model-734ffe-kserve-7c69bdf48f-46tft" Oct 30 12:59:03.621887 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:59:03.621866 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6g5nf\" (UniqueName: \"kubernetes.io/projected/56318ef3-7e27-4b11-9344-46778652e73a-kube-api-access-6g5nf\") pod \"test-model-734ffe-kserve-7c69bdf48f-46tft\" (UID: \"56318ef3-7e27-4b11-9344-46778652e73a\") " pod="ai-tenant-e2e-inf-b-734ffe/test-model-734ffe-kserve-7c69bdf48f-46tft" Oct 30 12:59:03.622488 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:59:03.622428 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56318ef3-7e27-4b11-9344-46778652e73a-tls-certs\") pod \"test-model-734ffe-kserve-7c69bdf48f-46tft\" (UID: \"56318ef3-7e27-4b11-9344-46778652e73a\") " pod="ai-tenant-e2e-inf-b-734ffe/test-model-734ffe-kserve-7c69bdf48f-46tft" Oct 30 12:59:03.622661 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:59:03.622643 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56318ef3-7e27-4b11-9344-46778652e73a-dshm\") pod \"test-model-734ffe-kserve-7c69bdf48f-46tft\" (UID: \"56318ef3-7e27-4b11-9344-46778652e73a\") " pod="ai-tenant-e2e-inf-b-734ffe/test-model-734ffe-kserve-7c69bdf48f-46tft" Oct 30 12:59:03.622822 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:59:03.622807 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56318ef3-7e27-4b11-9344-46778652e73a-model-cache\") pod \"test-model-734ffe-kserve-7c69bdf48f-46tft\" (UID: \"56318ef3-7e27-4b11-9344-46778652e73a\") " pod="ai-tenant-e2e-inf-b-734ffe/test-model-734ffe-kserve-7c69bdf48f-46tft" Oct 30 12:59:03.623624 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:59:03.621391 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56318ef3-7e27-4b11-9344-46778652e73a-tmp-dir\") pod \"test-model-734ffe-kserve-7c69bdf48f-46tft\" (UID: \"56318ef3-7e27-4b11-9344-46778652e73a\") " pod="ai-tenant-e2e-inf-b-734ffe/test-model-734ffe-kserve-7c69bdf48f-46tft" Oct 30 12:59:03.623823 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:59:03.622142 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56318ef3-7e27-4b11-9344-46778652e73a-home\") pod \"test-model-734ffe-kserve-7c69bdf48f-46tft\" (UID: \"56318ef3-7e27-4b11-9344-46778652e73a\") " pod="ai-tenant-e2e-inf-b-734ffe/test-model-734ffe-kserve-7c69bdf48f-46tft" Oct 30 12:59:03.624161 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:59:03.623147 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56318ef3-7e27-4b11-9344-46778652e73a-model-cache\") pod \"test-model-734ffe-kserve-7c69bdf48f-46tft\" (UID: \"56318ef3-7e27-4b11-9344-46778652e73a\") " pod="ai-tenant-e2e-inf-b-734ffe/test-model-734ffe-kserve-7c69bdf48f-46tft" Oct 30 12:59:03.626293 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:59:03.626270 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56318ef3-7e27-4b11-9344-46778652e73a-dshm\") pod \"test-model-734ffe-kserve-7c69bdf48f-46tft\" (UID: \"56318ef3-7e27-4b11-9344-46778652e73a\") " pod="ai-tenant-e2e-inf-b-734ffe/test-model-734ffe-kserve-7c69bdf48f-46tft" Oct 30 12:59:03.628501 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:59:03.628475 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56318ef3-7e27-4b11-9344-46778652e73a-tls-certs\") pod \"test-model-734ffe-kserve-7c69bdf48f-46tft\" (UID: \"56318ef3-7e27-4b11-9344-46778652e73a\") " pod="ai-tenant-e2e-inf-b-734ffe/test-model-734ffe-kserve-7c69bdf48f-46tft" Oct 30 12:59:03.644661 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:59:03.644613 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6g5nf\" (UniqueName: \"kubernetes.io/projected/56318ef3-7e27-4b11-9344-46778652e73a-kube-api-access-6g5nf\") pod \"test-model-734ffe-kserve-7c69bdf48f-46tft\" (UID: \"56318ef3-7e27-4b11-9344-46778652e73a\") " pod="ai-tenant-e2e-inf-b-734ffe/test-model-734ffe-kserve-7c69bdf48f-46tft" Oct 30 12:59:03.799628 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:59:03.799597 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-734ffe\"/\"default-dockercfg-xm7lw\"" Oct 30 12:59:03.815339 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:59:03.815055 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-734ffe/test-model-734ffe-kserve-7c69bdf48f-46tft" Oct 30 12:59:04.427910 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:59:04.424622 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-734ffe/test-model-734ffe-kserve-7c69bdf48f-46tft"] Oct 30 12:59:05.326104 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:59:05.326044 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-734ffe/test-model-734ffe-kserve-7c69bdf48f-46tft" event={"ID":"56318ef3-7e27-4b11-9344-46778652e73a","Type":"ContainerStarted","Data":"8789e596d04f46b5ff698a60c60b6545de34538a8f7a2a76047f67b430ed3f4e"} Oct 30 12:59:05.326693 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:59:05.326662 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-734ffe/test-model-734ffe-kserve-7c69bdf48f-46tft" event={"ID":"56318ef3-7e27-4b11-9344-46778652e73a","Type":"ContainerStarted","Data":"3035f38cbe79aac798f375efaba687b92371c72ccca2b69ead71499ba621935c"} Oct 30 12:59:05.379852 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:59:05.378949 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-734ffe/test-model-734ffe-kserve-7c69bdf48f-46tft" podStartSLOduration=2.378929968 podStartE2EDuration="2.378929968s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:59:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:59:05.376291481 +0000 UTC m=+3901.890720229" watchObservedRunningTime="2026-10-30 12:59:05.378929968 +0000 UTC m=+3901.893358716" Oct 30 12:59:05.815710 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:59:05.815672 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-734ffe/test-model-734ffe-kserve-7c69bdf48f-46tft" Oct 30 12:59:06.458761 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:59:06.458731 4908 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-plczt_a70ee11c-c20e-4a4f-8476-ac5eb02816d0/console-operator/3.log" Oct 30 12:59:06.493604 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:59:06.493570 4908 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b8bxz_644e19b3-10ad-4145-9ed3-595d19ed9b07/ovn-acl-logging/0.log" Oct 30 12:59:07.249012 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:59:07.248979 4908 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-plczt_a70ee11c-c20e-4a4f-8476-ac5eb02816d0/console-operator/3.log" Oct 30 12:59:07.255420 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:59:07.255395 4908 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b8bxz_644e19b3-10ad-4145-9ed3-595d19ed9b07/ovn-acl-logging/0.log" Oct 30 12:59:07.532265 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:59:07.532172 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-6b67f8/test-model-6b67f8-kserve-6cdd49c85c-2ccp8"] Oct 30 12:59:07.536353 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:59:07.536328 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-6b67f8/test-model-6b67f8-kserve-6cdd49c85c-2ccp8" Oct 30 12:59:07.564089 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:59:07.564052 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-6b67f8/test-model-6b67f8-kserve-6cdd49c85c-2ccp8"] Oct 30 12:59:07.568523 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:59:07.568108 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-6b67f8\"/\"openshift-service-ca.crt\"" Oct 30 12:59:07.568713 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:59:07.568313 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-6b67f8\"/\"kube-root-ca.crt\"" Oct 30 12:59:07.568815 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:59:07.568414 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-6b67f8\"/\"default-dockercfg-97lgw\"" Oct 30 12:59:07.568942 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:59:07.568455 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-6b67f8\"/\"test-model-6b67f8-kserve-self-signed-certs\"" Oct 30 12:59:07.658391 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:59:07.658352 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd0cc8f8-87d0-49fe-a25f-51653f60dbdb-tmp-dir\") pod \"test-model-6b67f8-kserve-6cdd49c85c-2ccp8\" (UID: \"cd0cc8f8-87d0-49fe-a25f-51653f60dbdb\") " pod="ai-tenant-e2e-inf-a-6b67f8/test-model-6b67f8-kserve-6cdd49c85c-2ccp8" Oct 30 12:59:07.658626 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:59:07.658535 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd0cc8f8-87d0-49fe-a25f-51653f60dbdb-model-cache\") pod \"test-model-6b67f8-kserve-6cdd49c85c-2ccp8\" (UID: \"cd0cc8f8-87d0-49fe-a25f-51653f60dbdb\") " pod="ai-tenant-e2e-inf-a-6b67f8/test-model-6b67f8-kserve-6cdd49c85c-2ccp8" Oct 30 12:59:07.658753 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:59:07.658715 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd0cc8f8-87d0-49fe-a25f-51653f60dbdb-tls-certs\") pod \"test-model-6b67f8-kserve-6cdd49c85c-2ccp8\" (UID: \"cd0cc8f8-87d0-49fe-a25f-51653f60dbdb\") " pod="ai-tenant-e2e-inf-a-6b67f8/test-model-6b67f8-kserve-6cdd49c85c-2ccp8" Oct 30 12:59:07.659009 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:59:07.658988 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgnw8\" (UniqueName: \"kubernetes.io/projected/cd0cc8f8-87d0-49fe-a25f-51653f60dbdb-kube-api-access-pgnw8\") pod \"test-model-6b67f8-kserve-6cdd49c85c-2ccp8\" (UID: \"cd0cc8f8-87d0-49fe-a25f-51653f60dbdb\") " pod="ai-tenant-e2e-inf-a-6b67f8/test-model-6b67f8-kserve-6cdd49c85c-2ccp8" Oct 30 12:59:07.659261 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:59:07.659244 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd0cc8f8-87d0-49fe-a25f-51653f60dbdb-home\") pod \"test-model-6b67f8-kserve-6cdd49c85c-2ccp8\" (UID: \"cd0cc8f8-87d0-49fe-a25f-51653f60dbdb\") " pod="ai-tenant-e2e-inf-a-6b67f8/test-model-6b67f8-kserve-6cdd49c85c-2ccp8" Oct 30 12:59:07.659444 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:59:07.659430 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd0cc8f8-87d0-49fe-a25f-51653f60dbdb-dshm\") pod \"test-model-6b67f8-kserve-6cdd49c85c-2ccp8\" (UID: \"cd0cc8f8-87d0-49fe-a25f-51653f60dbdb\") " pod="ai-tenant-e2e-inf-a-6b67f8/test-model-6b67f8-kserve-6cdd49c85c-2ccp8" Oct 30 12:59:07.762022 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:59:07.761916 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgnw8\" (UniqueName: \"kubernetes.io/projected/cd0cc8f8-87d0-49fe-a25f-51653f60dbdb-kube-api-access-pgnw8\") pod \"test-model-6b67f8-kserve-6cdd49c85c-2ccp8\" (UID: \"cd0cc8f8-87d0-49fe-a25f-51653f60dbdb\") " pod="ai-tenant-e2e-inf-a-6b67f8/test-model-6b67f8-kserve-6cdd49c85c-2ccp8" Oct 30 12:59:07.762372 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:59:07.762345 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd0cc8f8-87d0-49fe-a25f-51653f60dbdb-home\") pod \"test-model-6b67f8-kserve-6cdd49c85c-2ccp8\" (UID: \"cd0cc8f8-87d0-49fe-a25f-51653f60dbdb\") " pod="ai-tenant-e2e-inf-a-6b67f8/test-model-6b67f8-kserve-6cdd49c85c-2ccp8" Oct 30 12:59:07.762534 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:59:07.762514 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd0cc8f8-87d0-49fe-a25f-51653f60dbdb-dshm\") pod \"test-model-6b67f8-kserve-6cdd49c85c-2ccp8\" (UID: \"cd0cc8f8-87d0-49fe-a25f-51653f60dbdb\") " pod="ai-tenant-e2e-inf-a-6b67f8/test-model-6b67f8-kserve-6cdd49c85c-2ccp8" Oct 30 12:59:07.762783 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:59:07.762761 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd0cc8f8-87d0-49fe-a25f-51653f60dbdb-tmp-dir\") pod \"test-model-6b67f8-kserve-6cdd49c85c-2ccp8\" (UID: \"cd0cc8f8-87d0-49fe-a25f-51653f60dbdb\") " pod="ai-tenant-e2e-inf-a-6b67f8/test-model-6b67f8-kserve-6cdd49c85c-2ccp8" Oct 30 12:59:07.763046 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:59:07.763028 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd0cc8f8-87d0-49fe-a25f-51653f60dbdb-model-cache\") pod \"test-model-6b67f8-kserve-6cdd49c85c-2ccp8\" (UID: \"cd0cc8f8-87d0-49fe-a25f-51653f60dbdb\") " pod="ai-tenant-e2e-inf-a-6b67f8/test-model-6b67f8-kserve-6cdd49c85c-2ccp8" Oct 30 12:59:07.763286 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:59:07.763262 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd0cc8f8-87d0-49fe-a25f-51653f60dbdb-tls-certs\") pod \"test-model-6b67f8-kserve-6cdd49c85c-2ccp8\" (UID: \"cd0cc8f8-87d0-49fe-a25f-51653f60dbdb\") " pod="ai-tenant-e2e-inf-a-6b67f8/test-model-6b67f8-kserve-6cdd49c85c-2ccp8" Oct 30 12:59:07.764514 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:59:07.764479 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd0cc8f8-87d0-49fe-a25f-51653f60dbdb-tmp-dir\") pod \"test-model-6b67f8-kserve-6cdd49c85c-2ccp8\" (UID: \"cd0cc8f8-87d0-49fe-a25f-51653f60dbdb\") " pod="ai-tenant-e2e-inf-a-6b67f8/test-model-6b67f8-kserve-6cdd49c85c-2ccp8" Oct 30 12:59:07.764734 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:59:07.764706 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd0cc8f8-87d0-49fe-a25f-51653f60dbdb-home\") pod \"test-model-6b67f8-kserve-6cdd49c85c-2ccp8\" (UID: \"cd0cc8f8-87d0-49fe-a25f-51653f60dbdb\") " pod="ai-tenant-e2e-inf-a-6b67f8/test-model-6b67f8-kserve-6cdd49c85c-2ccp8" Oct 30 12:59:07.764819 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:59:07.764737 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd0cc8f8-87d0-49fe-a25f-51653f60dbdb-model-cache\") pod \"test-model-6b67f8-kserve-6cdd49c85c-2ccp8\" (UID: \"cd0cc8f8-87d0-49fe-a25f-51653f60dbdb\") " pod="ai-tenant-e2e-inf-a-6b67f8/test-model-6b67f8-kserve-6cdd49c85c-2ccp8" Oct 30 12:59:07.767125 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:59:07.767099 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd0cc8f8-87d0-49fe-a25f-51653f60dbdb-dshm\") pod \"test-model-6b67f8-kserve-6cdd49c85c-2ccp8\" (UID: \"cd0cc8f8-87d0-49fe-a25f-51653f60dbdb\") " pod="ai-tenant-e2e-inf-a-6b67f8/test-model-6b67f8-kserve-6cdd49c85c-2ccp8" Oct 30 12:59:07.813065 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:59:07.797153 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd0cc8f8-87d0-49fe-a25f-51653f60dbdb-tls-certs\") pod \"test-model-6b67f8-kserve-6cdd49c85c-2ccp8\" (UID: \"cd0cc8f8-87d0-49fe-a25f-51653f60dbdb\") " pod="ai-tenant-e2e-inf-a-6b67f8/test-model-6b67f8-kserve-6cdd49c85c-2ccp8" Oct 30 12:59:07.829858 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:59:07.815928 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgnw8\" (UniqueName: \"kubernetes.io/projected/cd0cc8f8-87d0-49fe-a25f-51653f60dbdb-kube-api-access-pgnw8\") pod \"test-model-6b67f8-kserve-6cdd49c85c-2ccp8\" (UID: \"cd0cc8f8-87d0-49fe-a25f-51653f60dbdb\") " pod="ai-tenant-e2e-inf-a-6b67f8/test-model-6b67f8-kserve-6cdd49c85c-2ccp8" Oct 30 12:59:07.873856 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:59:07.870179 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-6b67f8/test-model-6b67f8-kserve-6cdd49c85c-2ccp8" Oct 30 12:59:08.479685 ip-10-0-141-251 kubenswrapper[4908]: W1030 12:59:08.479643 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcd0cc8f8_87d0_49fe_a25f_51653f60dbdb.slice/crio-9ad127f88812530bce29cf6d51f1236ca7db28e94a652f3d41bae4d9ea228fd4 WatchSource:0}: Error finding container 9ad127f88812530bce29cf6d51f1236ca7db28e94a652f3d41bae4d9ea228fd4: Status 404 returned error can't find the container with id 9ad127f88812530bce29cf6d51f1236ca7db28e94a652f3d41bae4d9ea228fd4 Oct 30 12:59:08.480336 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:59:08.480305 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-6b67f8/test-model-6b67f8-kserve-6cdd49c85c-2ccp8"] Oct 30 12:59:09.411345 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:59:09.411308 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-6b67f8/test-model-6b67f8-kserve-6cdd49c85c-2ccp8" event={"ID":"cd0cc8f8-87d0-49fe-a25f-51653f60dbdb","Type":"ContainerStarted","Data":"f83f152572f8b2f95387e5d9c89701a204a807d4c9f18dd0b46140acae559e29"} Oct 30 12:59:09.411757 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:59:09.411354 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-6b67f8/test-model-6b67f8-kserve-6cdd49c85c-2ccp8" event={"ID":"cd0cc8f8-87d0-49fe-a25f-51653f60dbdb","Type":"ContainerStarted","Data":"9ad127f88812530bce29cf6d51f1236ca7db28e94a652f3d41bae4d9ea228fd4"} Oct 30 12:59:09.455279 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:59:09.455215 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-6b67f8/test-model-6b67f8-kserve-6cdd49c85c-2ccp8" podStartSLOduration=2.455200504 podStartE2EDuration="2.455200504s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:59:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:59:09.451095026 +0000 UTC m=+3905.965523784" watchObservedRunningTime="2026-10-30 12:59:09.455200504 +0000 UTC m=+3905.969629249" Oct 30 12:59:09.871024 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:59:09.870978 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-6b67f8/test-model-6b67f8-kserve-6cdd49c85c-2ccp8" Oct 30 12:59:11.429928 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:59:11.429892 4908 generic.go:356] "Generic (PLEG): container finished" podID="a2029481-e8ad-40ef-97b7-81140abedb5c" containerID="77cff5c69f3faa3be287996ceaa831fc7ebd6797f0efb89476d52d6b56ce2413" exitCode=0 Oct 30 12:59:11.430454 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:59:11.429962 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9c3f12-openshift-default-78b944cc9c-fdpqh" event={"ID":"a2029481-e8ad-40ef-97b7-81140abedb5c","Type":"ContainerDied","Data":"77cff5c69f3faa3be287996ceaa831fc7ebd6797f0efb89476d52d6b56ce2413"} Oct 30 12:59:11.430713 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:59:11.430693 4908 scope.go:122] "RemoveContainer" containerID="77cff5c69f3faa3be287996ceaa831fc7ebd6797f0efb89476d52d6b56ce2413" Oct 30 12:59:11.433039 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:59:11.433007 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:59:11.433133 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:59:11.433081 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:59:11.433133 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:59:11.433112 4908 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:59:12.441451 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:59:12.441404 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9c3f12-openshift-default-78b944cc9c-fdpqh" event={"ID":"a2029481-e8ad-40ef-97b7-81140abedb5c","Type":"ContainerStarted","Data":"8c2d59e692b8a4c74147a8d0aae249a6b34db995d531a52e86abb25c5ce863e1"} Oct 30 12:59:12.634080 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:59:12.634042 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-9c3f12-openshift-default-78b944cc9c-fdpqh" Oct 30 12:59:12.635497 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:59:12.635464 4908 patch_prober.go:29] interesting pod/e2e-inf-b-9c3f12-openshift-default-78b944cc9c-fdpqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.38:15021/healthz/ready\": dial tcp 10.132.1.38:15021: connect: connection refused" start-of-body= Oct 30 12:59:12.635626 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:59:12.635526 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9c3f12-openshift-default-78b944cc9c-fdpqh" podUID="a2029481-e8ad-40ef-97b7-81140abedb5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.38:15021/healthz/ready\": dial tcp 10.132.1.38:15021: connect: connection refused" Oct 30 12:59:13.050779 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:59:13.050737 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0294fd6-d9hh2"] Oct 30 12:59:13.082258 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:59:13.082218 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0294fd6-d9hh2"] Oct 30 12:59:13.609495 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:59:13.609460 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4cb971e-145e-403f-8118-9d2e0cde2396" path="/var/lib/kubelet/pods/f4cb971e-145e-403f-8118-9d2e0cde2396/volumes" Oct 30 12:59:13.634106 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:59:13.634075 4908 patch_prober.go:29] interesting pod/e2e-inf-b-9c3f12-openshift-default-78b944cc9c-fdpqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.38:15021/healthz/ready\": dial tcp 10.132.1.38:15021: connect: connection refused" start-of-body= Oct 30 12:59:13.634267 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:59:13.634131 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9c3f12-openshift-default-78b944cc9c-fdpqh" podUID="a2029481-e8ad-40ef-97b7-81140abedb5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.38:15021/healthz/ready\": dial tcp 10.132.1.38:15021: connect: connection refused" Oct 30 12:59:13.816378 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:59:13.816336 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-734ffe/test-model-734ffe-kserve-7c69bdf48f-46tft" Oct 30 12:59:13.830027 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:59:13.830003 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-734ffe/test-model-734ffe-kserve-7c69bdf48f-46tft" Oct 30 12:59:14.347451 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:59:14.347408 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-9c3f12-openshift-default-78b944cc9c-fdpqh" Oct 30 12:59:14.521499 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:59:14.521469 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-734ffe/test-model-734ffe-kserve-7c69bdf48f-46tft" Oct 30 12:59:14.634316 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:59:14.634239 4908 patch_prober.go:29] interesting pod/e2e-inf-b-9c3f12-openshift-default-78b944cc9c-fdpqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.38:15021/healthz/ready\": dial tcp 10.132.1.38:15021: connect: connection refused" start-of-body= Oct 30 12:59:14.634316 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:59:14.634292 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9c3f12-openshift-default-78b944cc9c-fdpqh" podUID="a2029481-e8ad-40ef-97b7-81140abedb5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.38:15021/healthz/ready\": dial tcp 10.132.1.38:15021: connect: connection refused" Oct 30 12:59:15.634269 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:59:15.634239 4908 patch_prober.go:29] interesting pod/e2e-inf-b-9c3f12-openshift-default-78b944cc9c-fdpqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.38:15021/healthz/ready\": dial tcp 10.132.1.38:15021: connect: connection refused" start-of-body= Oct 30 12:59:15.634429 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:59:15.634287 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9c3f12-openshift-default-78b944cc9c-fdpqh" podUID="a2029481-e8ad-40ef-97b7-81140abedb5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.38:15021/healthz/ready\": dial tcp 10.132.1.38:15021: connect: connection refused" Oct 30 12:59:16.634082 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:59:16.634038 4908 patch_prober.go:29] interesting pod/e2e-inf-b-9c3f12-openshift-default-78b944cc9c-fdpqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.38:15021/healthz/ready\": dial tcp 10.132.1.38:15021: connect: connection refused" start-of-body= Oct 30 12:59:16.634287 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:59:16.634102 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9c3f12-openshift-default-78b944cc9c-fdpqh" podUID="a2029481-e8ad-40ef-97b7-81140abedb5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.38:15021/healthz/ready\": dial tcp 10.132.1.38:15021: connect: connection refused" Oct 30 12:59:17.634574 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:59:17.634538 4908 patch_prober.go:29] interesting pod/e2e-inf-b-9c3f12-openshift-default-78b944cc9c-fdpqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.38:15021/healthz/ready\": dial tcp 10.132.1.38:15021: connect: connection refused" start-of-body= Oct 30 12:59:17.634975 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:59:17.634590 4908 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9c3f12-openshift-default-78b944cc9c-fdpqh" podUID="a2029481-e8ad-40ef-97b7-81140abedb5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.38:15021/healthz/ready\": dial tcp 10.132.1.38:15021: connect: connection refused" Oct 30 12:59:17.871685 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:59:17.871645 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-6b67f8/test-model-6b67f8-kserve-6cdd49c85c-2ccp8" Oct 30 12:59:17.890379 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:59:17.890270 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-6b67f8/test-model-6b67f8-kserve-6cdd49c85c-2ccp8" Oct 30 12:59:18.577872 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:59:18.577841 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-6b67f8/test-model-6b67f8-kserve-6cdd49c85c-2ccp8" Oct 30 12:59:18.643879 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:59:18.643207 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-9c3f12-openshift-default-78b944cc9c-fdpqh" Oct 30 12:59:18.645654 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:59:18.645617 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-9c3f12-openshift-default-78b944cc9c-fdpqh" Oct 30 12:59:19.695805 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:59:19.695763 4908 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vn6vv" containerName="extractor" Oct 30 12:59:23.041061 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:59:23.041027 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51f1d960-jtrp5"] Oct 30 12:59:23.051531 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:59:23.051500 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51f1d960-jtrp5"] Oct 30 12:59:23.610921 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:59:23.610880 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="abf51df1-f815-498c-964a-18b7db09014b" path="/var/lib/kubelet/pods/abf51df1-f815-498c-964a-18b7db09014b/volumes" Oct 30 12:59:24.034228 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:59:24.034190 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5da5dee2-x8p4m"] Oct 30 12:59:24.039726 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:59:24.039697 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5da5dee2-x8p4m"] Oct 30 12:59:25.610547 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:59:25.610513 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e518fd27-b28e-47da-9365-52cf1c7c5e05" path="/var/lib/kubelet/pods/e518fd27-b28e-47da-9365-52cf1c7c5e05/volumes" Oct 30 12:59:33.049061 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:59:33.049024 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-908f43fe-75ft7"] Oct 30 12:59:33.059885 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:59:33.059853 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-908f43fe-75ft7"] Oct 30 12:59:33.657278 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:59:33.657233 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4664497f-4066-4a36-874f-ad6bd275944e" path="/var/lib/kubelet/pods/4664497f-4066-4a36-874f-ad6bd275944e/volumes" Oct 30 12:59:44.190949 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:59:44.190460 4908 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f920f7/simulator-w0-f920f7-kserve-599b7ff45f-g79d7" podUID="a85e562f-d655-4c22-b4f4-a299a4e494a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Oct 30 12:59:44.264853 ip-10-0-141-251 kubenswrapper[4908]: I1030 12:59:44.264043 4908 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-512ec5/simulator-w4-512ec5-kserve-7df7674f8-8rqwb" podUID="8a5dfcc9-dec3-4bc6-b0a8-c928350b3596" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:59:49.686003 ip-10-0-141-251 kubenswrapper[4908]: E1030 12:59:49.685958 4908 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vn6vv" containerName="extractor" Oct 30 13:00:00.153040 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.152991 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-12fd33-29889420-cfqsp"] Oct 30 13:00:00.157591 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.157563 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fddaa3-29889420-vzns9"] Oct 30 13:00:00.157732 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.157683 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-12fd33-29889420-cfqsp" Oct 30 13:00:00.161989 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.161960 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-dd0628-29889420-jh88n"] Oct 30 13:00:00.162173 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.162152 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fddaa3-29889420-vzns9" Oct 30 13:00:00.166810 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.166736 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-dd0628-29889420-jh88n" Oct 30 13:00:00.167584 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.167270 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-10f364-29889420-fzd6k"] Oct 30 13:00:00.172339 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.172318 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-10f364-29889420-fzd6k" Oct 30 13:00:00.173233 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.173209 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-73bafc-29889420-4lh9z"] Oct 30 13:00:00.177630 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.177604 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-73bafc-29889420-4lh9z" Oct 30 13:00:00.177815 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.177793 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-dbe123-29889420-bcnzv"] Oct 30 13:00:00.181212 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.181187 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s75hq\"" Oct 30 13:00:00.183736 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.183630 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-dbe123-29889420-bcnzv" Oct 30 13:00:00.184239 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.184189 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 13:00:00.184384 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.184331 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 13:00:00.193591 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.188978 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ecb3aa-29889420-72h7x"] Oct 30 13:00:00.196189 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.196165 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ecb3aa-29889420-72h7x" Oct 30 13:00:00.215877 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.202577 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0bda67-29889420-4z7jt"] Oct 30 13:00:00.234424 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.233442 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-12fd33-29889420-cfqsp"] Oct 30 13:00:00.234424 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.233489 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2675c0-29889420-hqgp5"] Oct 30 13:00:00.234424 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.233653 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0bda67-29889420-4z7jt" Oct 30 13:00:00.238995 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.238940 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fddaa3-29889420-vzns9"] Oct 30 13:00:00.239164 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.239077 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2675c0-29889420-hqgp5" Oct 30 13:00:00.241624 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.241599 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2545c7-29889420-xrknl"] Oct 30 13:00:00.248368 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.247066 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2545c7-29889420-xrknl" Oct 30 13:00:00.249624 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.248996 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6h7x\" (UniqueName: \"kubernetes.io/projected/f2695f98-7dab-4cf3-aaa3-d1717ecc2d30-kube-api-access-p6h7x\") pod \"maas-api-key-cleanup-e2e-extmod-b-12fd33-29889420-cfqsp\" (UID: \"f2695f98-7dab-4cf3-aaa3-d1717ecc2d30\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-12fd33-29889420-cfqsp" Oct 30 13:00:00.249624 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.249129 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/347ca826-af8d-496d-bd0b-dca4071b3cd8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-dd0628-29889420-jh88n\" (UID: \"347ca826-af8d-496d-bd0b-dca4071b3cd8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-dd0628-29889420-jh88n" Oct 30 13:00:00.249624 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.249166 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d506c31-90a7-493f-8b73-649d1ffcd0eb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-73bafc-29889420-4lh9z\" (UID: \"0d506c31-90a7-493f-8b73-649d1ffcd0eb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-73bafc-29889420-4lh9z" Oct 30 13:00:00.249624 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.249193 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tz2tr\" (UniqueName: \"kubernetes.io/projected/cb87d307-def9-4d16-8e7a-54b531f0d2c5-kube-api-access-tz2tr\") pod \"maas-api-key-cleanup-e2e-extmod-b-ecb3aa-29889420-72h7x\" (UID: \"cb87d307-def9-4d16-8e7a-54b531f0d2c5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ecb3aa-29889420-72h7x" Oct 30 13:00:00.249624 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.249241 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qqw7\" (UniqueName: \"kubernetes.io/projected/8165abf7-fbfb-4de2-a94c-5023a834835d-kube-api-access-6qqw7\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-dbe123-29889420-bcnzv\" (UID: \"8165abf7-fbfb-4de2-a94c-5023a834835d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-dbe123-29889420-bcnzv" Oct 30 13:00:00.249624 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.249290 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzfvz\" (UniqueName: \"kubernetes.io/projected/2674b41c-6ab9-4222-aa87-4fc2f5d9f468-kube-api-access-bzfvz\") pod \"maas-api-key-cleanup-e2e-extmod-b-fddaa3-29889420-vzns9\" (UID: \"2674b41c-6ab9-4222-aa87-4fc2f5d9f468\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fddaa3-29889420-vzns9" Oct 30 13:00:00.249624 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.249333 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2695f98-7dab-4cf3-aaa3-d1717ecc2d30-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-12fd33-29889420-cfqsp\" (UID: \"f2695f98-7dab-4cf3-aaa3-d1717ecc2d30\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-12fd33-29889420-cfqsp" Oct 30 13:00:00.249624 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.249362 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8165abf7-fbfb-4de2-a94c-5023a834835d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-dbe123-29889420-bcnzv\" (UID: \"8165abf7-fbfb-4de2-a94c-5023a834835d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-dbe123-29889420-bcnzv" Oct 30 13:00:00.249624 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.249428 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/76cf17b7-c8b9-48d8-934c-f38aa9d1f008-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-10f364-29889420-fzd6k\" (UID: \"76cf17b7-c8b9-48d8-934c-f38aa9d1f008\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-10f364-29889420-fzd6k" Oct 30 13:00:00.249624 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.249502 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2674b41c-6ab9-4222-aa87-4fc2f5d9f468-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-fddaa3-29889420-vzns9\" (UID: \"2674b41c-6ab9-4222-aa87-4fc2f5d9f468\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fddaa3-29889420-vzns9" Oct 30 13:00:00.249624 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.249544 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb87d307-def9-4d16-8e7a-54b531f0d2c5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-ecb3aa-29889420-72h7x\" (UID: \"cb87d307-def9-4d16-8e7a-54b531f0d2c5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ecb3aa-29889420-72h7x" Oct 30 13:00:00.249624 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.249610 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txtcl\" (UniqueName: \"kubernetes.io/projected/0d506c31-90a7-493f-8b73-649d1ffcd0eb-kube-api-access-txtcl\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-73bafc-29889420-4lh9z\" (UID: \"0d506c31-90a7-493f-8b73-649d1ffcd0eb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-73bafc-29889420-4lh9z" Oct 30 13:00:00.250439 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.249734 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxfbg\" (UniqueName: \"kubernetes.io/projected/76cf17b7-c8b9-48d8-934c-f38aa9d1f008-kube-api-access-wxfbg\") pod \"maas-api-key-cleanup-e2e-extmod-a-10f364-29889420-fzd6k\" (UID: \"76cf17b7-c8b9-48d8-934c-f38aa9d1f008\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-10f364-29889420-fzd6k" Oct 30 13:00:00.250439 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.249765 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kq4t\" (UniqueName: \"kubernetes.io/projected/347ca826-af8d-496d-bd0b-dca4071b3cd8-kube-api-access-2kq4t\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-dd0628-29889420-jh88n\" (UID: \"347ca826-af8d-496d-bd0b-dca4071b3cd8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-dd0628-29889420-jh88n" Oct 30 13:00:00.255655 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.253769 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-dd0628-29889420-jh88n"] Oct 30 13:00:00.269641 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.269129 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-73bafc-29889420-4lh9z"] Oct 30 13:00:00.275907 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.275773 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-dbe123-29889420-bcnzv"] Oct 30 13:00:00.315329 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.315284 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a786af-29889420-mrgm2"] Oct 30 13:00:00.327380 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.327326 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a786af-29889420-mrgm2" Oct 30 13:00:00.354568 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.353279 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kq4t\" (UniqueName: \"kubernetes.io/projected/347ca826-af8d-496d-bd0b-dca4071b3cd8-kube-api-access-2kq4t\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-dd0628-29889420-jh88n\" (UID: \"347ca826-af8d-496d-bd0b-dca4071b3cd8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-dd0628-29889420-jh88n" Oct 30 13:00:00.354883 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.354854 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6h7x\" (UniqueName: \"kubernetes.io/projected/f2695f98-7dab-4cf3-aaa3-d1717ecc2d30-kube-api-access-p6h7x\") pod \"maas-api-key-cleanup-e2e-extmod-b-12fd33-29889420-cfqsp\" (UID: \"f2695f98-7dab-4cf3-aaa3-d1717ecc2d30\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-12fd33-29889420-cfqsp" Oct 30 13:00:00.355040 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.355019 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b46f76c5-4139-4fbd-af18-5f35fcb1cb05-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-0bda67-29889420-4z7jt\" (UID: \"b46f76c5-4139-4fbd-af18-5f35fcb1cb05\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0bda67-29889420-4z7jt" Oct 30 13:00:00.355200 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.355179 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrmmn\" (UniqueName: \"kubernetes.io/projected/b46f76c5-4139-4fbd-af18-5f35fcb1cb05-kube-api-access-vrmmn\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-0bda67-29889420-4z7jt\" (UID: \"b46f76c5-4139-4fbd-af18-5f35fcb1cb05\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0bda67-29889420-4z7jt" Oct 30 13:00:00.355355 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.355339 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rwhl\" (UniqueName: \"kubernetes.io/projected/bf7295f0-efb3-487a-926c-c4f19fddde26-kube-api-access-8rwhl\") pod \"maas-api-key-cleanup-e2e-extmod-a-2545c7-29889420-xrknl\" (UID: \"bf7295f0-efb3-487a-926c-c4f19fddde26\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2545c7-29889420-xrknl" Oct 30 13:00:00.355511 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.355496 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/347ca826-af8d-496d-bd0b-dca4071b3cd8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-dd0628-29889420-jh88n\" (UID: \"347ca826-af8d-496d-bd0b-dca4071b3cd8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-dd0628-29889420-jh88n" Oct 30 13:00:00.355665 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.355618 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d506c31-90a7-493f-8b73-649d1ffcd0eb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-73bafc-29889420-4lh9z\" (UID: \"0d506c31-90a7-493f-8b73-649d1ffcd0eb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-73bafc-29889420-4lh9z" Oct 30 13:00:00.357131 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.357108 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tz2tr\" (UniqueName: \"kubernetes.io/projected/cb87d307-def9-4d16-8e7a-54b531f0d2c5-kube-api-access-tz2tr\") pod \"maas-api-key-cleanup-e2e-extmod-b-ecb3aa-29889420-72h7x\" (UID: \"cb87d307-def9-4d16-8e7a-54b531f0d2c5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ecb3aa-29889420-72h7x" Oct 30 13:00:00.357346 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.357327 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qqw7\" (UniqueName: \"kubernetes.io/projected/8165abf7-fbfb-4de2-a94c-5023a834835d-kube-api-access-6qqw7\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-dbe123-29889420-bcnzv\" (UID: \"8165abf7-fbfb-4de2-a94c-5023a834835d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-dbe123-29889420-bcnzv" Oct 30 13:00:00.357475 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.357461 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzfvz\" (UniqueName: \"kubernetes.io/projected/2674b41c-6ab9-4222-aa87-4fc2f5d9f468-kube-api-access-bzfvz\") pod \"maas-api-key-cleanup-e2e-extmod-b-fddaa3-29889420-vzns9\" (UID: \"2674b41c-6ab9-4222-aa87-4fc2f5d9f468\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fddaa3-29889420-vzns9" Oct 30 13:00:00.357598 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.357584 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2695f98-7dab-4cf3-aaa3-d1717ecc2d30-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-12fd33-29889420-cfqsp\" (UID: \"f2695f98-7dab-4cf3-aaa3-d1717ecc2d30\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-12fd33-29889420-cfqsp" Oct 30 13:00:00.357703 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.357690 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8165abf7-fbfb-4de2-a94c-5023a834835d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-dbe123-29889420-bcnzv\" (UID: \"8165abf7-fbfb-4de2-a94c-5023a834835d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-dbe123-29889420-bcnzv" Oct 30 13:00:00.357800 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.357786 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1bb2e64a-3fd0-41bb-aafa-8f234fb2ffa2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-a786af-29889420-mrgm2\" (UID: \"1bb2e64a-3fd0-41bb-aafa-8f234fb2ffa2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a786af-29889420-mrgm2" Oct 30 13:00:00.357973 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.357958 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/76cf17b7-c8b9-48d8-934c-f38aa9d1f008-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-10f364-29889420-fzd6k\" (UID: \"76cf17b7-c8b9-48d8-934c-f38aa9d1f008\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-10f364-29889420-fzd6k" Oct 30 13:00:00.358137 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.358121 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6s4rp\" (UniqueName: \"kubernetes.io/projected/1bb2e64a-3fd0-41bb-aafa-8f234fb2ffa2-kube-api-access-6s4rp\") pod \"maas-api-key-cleanup-e2e-extmod-a-a786af-29889420-mrgm2\" (UID: \"1bb2e64a-3fd0-41bb-aafa-8f234fb2ffa2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a786af-29889420-mrgm2" Oct 30 13:00:00.358247 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.358234 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2674b41c-6ab9-4222-aa87-4fc2f5d9f468-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-fddaa3-29889420-vzns9\" (UID: \"2674b41c-6ab9-4222-aa87-4fc2f5d9f468\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fddaa3-29889420-vzns9" Oct 30 13:00:00.358362 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.358350 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb87d307-def9-4d16-8e7a-54b531f0d2c5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-ecb3aa-29889420-72h7x\" (UID: \"cb87d307-def9-4d16-8e7a-54b531f0d2c5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ecb3aa-29889420-72h7x" Oct 30 13:00:00.358505 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.358492 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txtcl\" (UniqueName: \"kubernetes.io/projected/0d506c31-90a7-493f-8b73-649d1ffcd0eb-kube-api-access-txtcl\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-73bafc-29889420-4lh9z\" (UID: \"0d506c31-90a7-493f-8b73-649d1ffcd0eb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-73bafc-29889420-4lh9z" Oct 30 13:00:00.358613 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.358599 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf7295f0-efb3-487a-926c-c4f19fddde26-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-2545c7-29889420-xrknl\" (UID: \"bf7295f0-efb3-487a-926c-c4f19fddde26\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2545c7-29889420-xrknl" Oct 30 13:00:00.358785 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.358766 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8v6z7\" (UniqueName: \"kubernetes.io/projected/7f4c2c03-089f-4701-87d3-4e0496b08750-kube-api-access-8v6z7\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-2675c0-29889420-hqgp5\" (UID: \"7f4c2c03-089f-4701-87d3-4e0496b08750\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2675c0-29889420-hqgp5" Oct 30 13:00:00.358994 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.358976 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f4c2c03-089f-4701-87d3-4e0496b08750-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-2675c0-29889420-hqgp5\" (UID: \"7f4c2c03-089f-4701-87d3-4e0496b08750\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2675c0-29889420-hqgp5" Oct 30 13:00:00.359230 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.359211 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxfbg\" (UniqueName: \"kubernetes.io/projected/76cf17b7-c8b9-48d8-934c-f38aa9d1f008-kube-api-access-wxfbg\") pod \"maas-api-key-cleanup-e2e-extmod-a-10f364-29889420-fzd6k\" (UID: \"76cf17b7-c8b9-48d8-934c-f38aa9d1f008\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-10f364-29889420-fzd6k" Oct 30 13:00:00.361475 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.361415 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2674b41c-6ab9-4222-aa87-4fc2f5d9f468-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-fddaa3-29889420-vzns9\" (UID: \"2674b41c-6ab9-4222-aa87-4fc2f5d9f468\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fddaa3-29889420-vzns9" Oct 30 13:00:00.363284 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.363255 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb87d307-def9-4d16-8e7a-54b531f0d2c5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-ecb3aa-29889420-72h7x\" (UID: \"cb87d307-def9-4d16-8e7a-54b531f0d2c5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ecb3aa-29889420-72h7x" Oct 30 13:00:00.363693 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.363671 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/76cf17b7-c8b9-48d8-934c-f38aa9d1f008-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-10f364-29889420-fzd6k\" (UID: \"76cf17b7-c8b9-48d8-934c-f38aa9d1f008\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-10f364-29889420-fzd6k" Oct 30 13:00:00.364551 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.364531 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/347ca826-af8d-496d-bd0b-dca4071b3cd8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-dd0628-29889420-jh88n\" (UID: \"347ca826-af8d-496d-bd0b-dca4071b3cd8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-dd0628-29889420-jh88n" Oct 30 13:00:00.384296 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.366279 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2695f98-7dab-4cf3-aaa3-d1717ecc2d30-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-12fd33-29889420-cfqsp\" (UID: \"f2695f98-7dab-4cf3-aaa3-d1717ecc2d30\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-12fd33-29889420-cfqsp" Oct 30 13:00:00.384296 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.366306 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-42220c-29889420-q4fqq"] Oct 30 13:00:00.384600 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.368228 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d506c31-90a7-493f-8b73-649d1ffcd0eb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-73bafc-29889420-4lh9z\" (UID: \"0d506c31-90a7-493f-8b73-649d1ffcd0eb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-73bafc-29889420-4lh9z" Oct 30 13:00:00.384788 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.375137 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8165abf7-fbfb-4de2-a94c-5023a834835d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-dbe123-29889420-bcnzv\" (UID: \"8165abf7-fbfb-4de2-a94c-5023a834835d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-dbe123-29889420-bcnzv" Oct 30 13:00:00.393705 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.393682 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f7464f-29889420-z9mg6"] Oct 30 13:00:00.394928 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.394786 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-42220c-29889420-q4fqq" Oct 30 13:00:00.403272 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.403211 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f7464f-29889420-z9mg6" Oct 30 13:00:00.404333 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.404306 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kq4t\" (UniqueName: \"kubernetes.io/projected/347ca826-af8d-496d-bd0b-dca4071b3cd8-kube-api-access-2kq4t\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-dd0628-29889420-jh88n\" (UID: \"347ca826-af8d-496d-bd0b-dca4071b3cd8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-dd0628-29889420-jh88n" Oct 30 13:00:00.404877 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.404858 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-txtcl\" (UniqueName: \"kubernetes.io/projected/0d506c31-90a7-493f-8b73-649d1ffcd0eb-kube-api-access-txtcl\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-73bafc-29889420-4lh9z\" (UID: \"0d506c31-90a7-493f-8b73-649d1ffcd0eb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-73bafc-29889420-4lh9z" Oct 30 13:00:00.405842 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.405811 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxfbg\" (UniqueName: \"kubernetes.io/projected/76cf17b7-c8b9-48d8-934c-f38aa9d1f008-kube-api-access-wxfbg\") pod \"maas-api-key-cleanup-e2e-extmod-a-10f364-29889420-fzd6k\" (UID: \"76cf17b7-c8b9-48d8-934c-f38aa9d1f008\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-10f364-29889420-fzd6k" Oct 30 13:00:00.406929 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.406564 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzfvz\" (UniqueName: \"kubernetes.io/projected/2674b41c-6ab9-4222-aa87-4fc2f5d9f468-kube-api-access-bzfvz\") pod \"maas-api-key-cleanup-e2e-extmod-b-fddaa3-29889420-vzns9\" (UID: \"2674b41c-6ab9-4222-aa87-4fc2f5d9f468\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fddaa3-29889420-vzns9" Oct 30 13:00:00.407496 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.407057 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tz2tr\" (UniqueName: \"kubernetes.io/projected/cb87d307-def9-4d16-8e7a-54b531f0d2c5-kube-api-access-tz2tr\") pod \"maas-api-key-cleanup-e2e-extmod-b-ecb3aa-29889420-72h7x\" (UID: \"cb87d307-def9-4d16-8e7a-54b531f0d2c5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ecb3aa-29889420-72h7x" Oct 30 13:00:00.407496 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.407108 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2675c0-29889420-hqgp5"] Oct 30 13:00:00.408514 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.408454 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6h7x\" (UniqueName: \"kubernetes.io/projected/f2695f98-7dab-4cf3-aaa3-d1717ecc2d30-kube-api-access-p6h7x\") pod \"maas-api-key-cleanup-e2e-extmod-b-12fd33-29889420-cfqsp\" (UID: \"f2695f98-7dab-4cf3-aaa3-d1717ecc2d30\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-12fd33-29889420-cfqsp" Oct 30 13:00:00.411388 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.411334 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qqw7\" (UniqueName: \"kubernetes.io/projected/8165abf7-fbfb-4de2-a94c-5023a834835d-kube-api-access-6qqw7\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-dbe123-29889420-bcnzv\" (UID: \"8165abf7-fbfb-4de2-a94c-5023a834835d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-dbe123-29889420-bcnzv" Oct 30 13:00:00.414897 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.413699 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-327e31-29889420-2vbrc"] Oct 30 13:00:00.418701 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.418583 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-327e31-29889420-2vbrc" Oct 30 13:00:00.421398 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.421349 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0bda67-29889420-4z7jt"] Oct 30 13:00:00.429918 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.429810 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2545c7-29889420-xrknl"] Oct 30 13:00:00.434870 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.434781 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ecb3aa-29889420-72h7x"] Oct 30 13:00:00.442136 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.442109 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-9d45fb-29889420-5wlx7"] Oct 30 13:00:00.447628 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.447604 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-9d45fb-29889420-5wlx7" Oct 30 13:00:00.448250 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.448226 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a786af-29889420-mrgm2"] Oct 30 13:00:00.453395 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.453367 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-8d2170-29889420-67bzd"] Oct 30 13:00:00.463285 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.458859 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f7464f-29889420-z9mg6"] Oct 30 13:00:00.463285 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.458918 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-8d2170-29889420-67bzd" Oct 30 13:00:00.465231 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.465091 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9rmm\" (UniqueName: \"kubernetes.io/projected/94f72c00-8f0f-4bcf-9218-cd64b7773d1f-kube-api-access-w9rmm\") pod \"maas-api-key-cleanup-e2e-extmod-a-f7464f-29889420-z9mg6\" (UID: \"94f72c00-8f0f-4bcf-9218-cd64b7773d1f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f7464f-29889420-z9mg6" Oct 30 13:00:00.465231 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.465172 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6s4rp\" (UniqueName: \"kubernetes.io/projected/1bb2e64a-3fd0-41bb-aafa-8f234fb2ffa2-kube-api-access-6s4rp\") pod \"maas-api-key-cleanup-e2e-extmod-a-a786af-29889420-mrgm2\" (UID: \"1bb2e64a-3fd0-41bb-aafa-8f234fb2ffa2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a786af-29889420-mrgm2" Oct 30 13:00:00.465420 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.465265 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-10f364-29889420-fzd6k"] Oct 30 13:00:00.465587 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.465561 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94f72c00-8f0f-4bcf-9218-cd64b7773d1f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-f7464f-29889420-z9mg6\" (UID: \"94f72c00-8f0f-4bcf-9218-cd64b7773d1f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f7464f-29889420-z9mg6" Oct 30 13:00:00.465671 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.465647 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf7295f0-efb3-487a-926c-c4f19fddde26-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-2545c7-29889420-xrknl\" (UID: \"bf7295f0-efb3-487a-926c-c4f19fddde26\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2545c7-29889420-xrknl" Oct 30 13:00:00.465740 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.465714 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8v6z7\" (UniqueName: \"kubernetes.io/projected/7f4c2c03-089f-4701-87d3-4e0496b08750-kube-api-access-8v6z7\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-2675c0-29889420-hqgp5\" (UID: \"7f4c2c03-089f-4701-87d3-4e0496b08750\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2675c0-29889420-hqgp5" Oct 30 13:00:00.465809 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.465752 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlbjg\" (UniqueName: \"kubernetes.io/projected/53e599d0-de7d-4c2f-86f9-97cc39405bc0-kube-api-access-nlbjg\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-327e31-29889420-2vbrc\" (UID: \"53e599d0-de7d-4c2f-86f9-97cc39405bc0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-327e31-29889420-2vbrc" Oct 30 13:00:00.465896 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.465849 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f4c2c03-089f-4701-87d3-4e0496b08750-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-2675c0-29889420-hqgp5\" (UID: \"7f4c2c03-089f-4701-87d3-4e0496b08750\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2675c0-29889420-hqgp5" Oct 30 13:00:00.465970 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.465894 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8vk9\" (UniqueName: \"kubernetes.io/projected/ced43030-57a2-418d-bf86-f9c5a3b8cd95-kube-api-access-s8vk9\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-42220c-29889420-q4fqq\" (UID: \"ced43030-57a2-418d-bf86-f9c5a3b8cd95\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-42220c-29889420-q4fqq" Oct 30 13:00:00.466030 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.466006 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b46f76c5-4139-4fbd-af18-5f35fcb1cb05-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-0bda67-29889420-4z7jt\" (UID: \"b46f76c5-4139-4fbd-af18-5f35fcb1cb05\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0bda67-29889420-4z7jt" Oct 30 13:00:00.466093 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.466058 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrmmn\" (UniqueName: \"kubernetes.io/projected/b46f76c5-4139-4fbd-af18-5f35fcb1cb05-kube-api-access-vrmmn\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-0bda67-29889420-4z7jt\" (UID: \"b46f76c5-4139-4fbd-af18-5f35fcb1cb05\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0bda67-29889420-4z7jt" Oct 30 13:00:00.466152 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.466105 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rwhl\" (UniqueName: \"kubernetes.io/projected/bf7295f0-efb3-487a-926c-c4f19fddde26-kube-api-access-8rwhl\") pod \"maas-api-key-cleanup-e2e-extmod-a-2545c7-29889420-xrknl\" (UID: \"bf7295f0-efb3-487a-926c-c4f19fddde26\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2545c7-29889420-xrknl" Oct 30 13:00:00.466233 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.466209 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53e599d0-de7d-4c2f-86f9-97cc39405bc0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-327e31-29889420-2vbrc\" (UID: \"53e599d0-de7d-4c2f-86f9-97cc39405bc0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-327e31-29889420-2vbrc" Oct 30 13:00:00.466384 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.466332 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1bb2e64a-3fd0-41bb-aafa-8f234fb2ffa2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-a786af-29889420-mrgm2\" (UID: \"1bb2e64a-3fd0-41bb-aafa-8f234fb2ffa2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a786af-29889420-mrgm2" Oct 30 13:00:00.466384 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.466368 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ced43030-57a2-418d-bf86-f9c5a3b8cd95-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-42220c-29889420-q4fqq\" (UID: \"ced43030-57a2-418d-bf86-f9c5a3b8cd95\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-42220c-29889420-q4fqq" Oct 30 13:00:00.471867 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.466681 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf7295f0-efb3-487a-926c-c4f19fddde26-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-2545c7-29889420-xrknl\" (UID: \"bf7295f0-efb3-487a-926c-c4f19fddde26\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2545c7-29889420-xrknl" Oct 30 13:00:00.471867 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.467589 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1bb2e64a-3fd0-41bb-aafa-8f234fb2ffa2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-a786af-29889420-mrgm2\" (UID: \"1bb2e64a-3fd0-41bb-aafa-8f234fb2ffa2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a786af-29889420-mrgm2" Oct 30 13:00:00.471867 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.468253 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f4c2c03-089f-4701-87d3-4e0496b08750-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-2675c0-29889420-hqgp5\" (UID: \"7f4c2c03-089f-4701-87d3-4e0496b08750\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2675c0-29889420-hqgp5" Oct 30 13:00:00.471867 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.468932 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b46f76c5-4139-4fbd-af18-5f35fcb1cb05-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-0bda67-29889420-4z7jt\" (UID: \"b46f76c5-4139-4fbd-af18-5f35fcb1cb05\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0bda67-29889420-4z7jt" Oct 30 13:00:00.471867 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.471602 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-327e31-29889420-2vbrc"] Oct 30 13:00:00.477841 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.477798 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6ad1da-29889420-qhlvt"] Oct 30 13:00:00.478805 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.478783 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-12fd33-29889420-cfqsp" Oct 30 13:00:00.498942 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.498912 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8v6z7\" (UniqueName: \"kubernetes.io/projected/7f4c2c03-089f-4701-87d3-4e0496b08750-kube-api-access-8v6z7\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-2675c0-29889420-hqgp5\" (UID: \"7f4c2c03-089f-4701-87d3-4e0496b08750\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2675c0-29889420-hqgp5" Oct 30 13:00:00.504081 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.504056 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-8d2170-29889420-67bzd"] Oct 30 13:00:00.506809 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.504322 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6ad1da-29889420-qhlvt" Oct 30 13:00:00.506999 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.506984 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-42220c-29889420-q4fqq"] Oct 30 13:00:00.511360 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.511335 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rwhl\" (UniqueName: \"kubernetes.io/projected/bf7295f0-efb3-487a-926c-c4f19fddde26-kube-api-access-8rwhl\") pod \"maas-api-key-cleanup-e2e-extmod-a-2545c7-29889420-xrknl\" (UID: \"bf7295f0-efb3-487a-926c-c4f19fddde26\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2545c7-29889420-xrknl" Oct 30 13:00:00.518576 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.518526 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrmmn\" (UniqueName: \"kubernetes.io/projected/b46f76c5-4139-4fbd-af18-5f35fcb1cb05-kube-api-access-vrmmn\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-0bda67-29889420-4z7jt\" (UID: \"b46f76c5-4139-4fbd-af18-5f35fcb1cb05\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0bda67-29889420-4z7jt" Oct 30 13:00:00.518576 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.518526 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6s4rp\" (UniqueName: \"kubernetes.io/projected/1bb2e64a-3fd0-41bb-aafa-8f234fb2ffa2-kube-api-access-6s4rp\") pod \"maas-api-key-cleanup-e2e-extmod-a-a786af-29889420-mrgm2\" (UID: \"1bb2e64a-3fd0-41bb-aafa-8f234fb2ffa2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a786af-29889420-mrgm2" Oct 30 13:00:00.521852 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.521813 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-812990-29889420-ddwkl"] Oct 30 13:00:00.535987 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.535957 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fddaa3-29889420-vzns9" Oct 30 13:00:00.536383 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.536364 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889420-n4zdq"] Oct 30 13:00:00.536537 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.536521 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-812990-29889420-ddwkl" Oct 30 13:00:00.544034 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.544011 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-9d45fb-29889420-5wlx7"] Oct 30 13:00:00.544193 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.544179 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6ad1da-29889420-qhlvt"] Oct 30 13:00:00.544316 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.544109 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889420-n4zdq" Oct 30 13:00:00.546775 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.546758 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-812990-29889420-ddwkl"] Oct 30 13:00:00.548300 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.548281 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-dd0628-29889420-jh88n" Oct 30 13:00:00.550272 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.550206 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e9f184-29889420-sv9rp"] Oct 30 13:00:00.556361 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.556336 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1333c8-29889420-9zjhg"] Oct 30 13:00:00.558991 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.558967 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e9f184-29889420-sv9rp" Oct 30 13:00:00.564061 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.564038 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1333c8-29889420-9zjhg" Oct 30 13:00:00.568518 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.568490 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1daed017-3390-4a05-9457-8d6c97919043-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-6ad1da-29889420-qhlvt\" (UID: \"1daed017-3390-4a05-9457-8d6c97919043\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6ad1da-29889420-qhlvt" Oct 30 13:00:00.568888 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.568811 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbqrv\" (UniqueName: \"kubernetes.io/projected/1baa9cd5-0305-4f77-b7cf-f4751053ace8-kube-api-access-zbqrv\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-812990-29889420-ddwkl\" (UID: \"1baa9cd5-0305-4f77-b7cf-f4751053ace8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-812990-29889420-ddwkl" Oct 30 13:00:00.569063 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.569043 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5af9446-e0d9-4edc-b01f-d049c4b2d3e7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-8d2170-29889420-67bzd\" (UID: \"f5af9446-e0d9-4edc-b01f-d049c4b2d3e7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-8d2170-29889420-67bzd" Oct 30 13:00:00.569197 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.569178 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53e599d0-de7d-4c2f-86f9-97cc39405bc0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-327e31-29889420-2vbrc\" (UID: \"53e599d0-de7d-4c2f-86f9-97cc39405bc0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-327e31-29889420-2vbrc" Oct 30 13:00:00.569375 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.569359 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ced43030-57a2-418d-bf86-f9c5a3b8cd95-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-42220c-29889420-q4fqq\" (UID: \"ced43030-57a2-418d-bf86-f9c5a3b8cd95\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-42220c-29889420-q4fqq" Oct 30 13:00:00.569532 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.569518 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9rmm\" (UniqueName: \"kubernetes.io/projected/94f72c00-8f0f-4bcf-9218-cd64b7773d1f-kube-api-access-w9rmm\") pod \"maas-api-key-cleanup-e2e-extmod-a-f7464f-29889420-z9mg6\" (UID: \"94f72c00-8f0f-4bcf-9218-cd64b7773d1f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f7464f-29889420-z9mg6" Oct 30 13:00:00.569731 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.569711 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94f72c00-8f0f-4bcf-9218-cd64b7773d1f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-f7464f-29889420-z9mg6\" (UID: \"94f72c00-8f0f-4bcf-9218-cd64b7773d1f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f7464f-29889420-z9mg6" Oct 30 13:00:00.569870 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.569853 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9jgd\" (UniqueName: \"kubernetes.io/projected/f5af9446-e0d9-4edc-b01f-d049c4b2d3e7-kube-api-access-z9jgd\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-8d2170-29889420-67bzd\" (UID: \"f5af9446-e0d9-4edc-b01f-d049c4b2d3e7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-8d2170-29889420-67bzd" Oct 30 13:00:00.574206 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.572752 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqvth\" (UniqueName: \"kubernetes.io/projected/a130ae9f-0e32-4b92-b6d3-597640f7e802-kube-api-access-gqvth\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-9d45fb-29889420-5wlx7\" (UID: \"a130ae9f-0e32-4b92-b6d3-597640f7e802\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-9d45fb-29889420-5wlx7" Oct 30 13:00:00.574206 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.572844 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlbjg\" (UniqueName: \"kubernetes.io/projected/53e599d0-de7d-4c2f-86f9-97cc39405bc0-kube-api-access-nlbjg\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-327e31-29889420-2vbrc\" (UID: \"53e599d0-de7d-4c2f-86f9-97cc39405bc0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-327e31-29889420-2vbrc" Oct 30 13:00:00.574206 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.572879 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1baa9cd5-0305-4f77-b7cf-f4751053ace8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-812990-29889420-ddwkl\" (UID: \"1baa9cd5-0305-4f77-b7cf-f4751053ace8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-812990-29889420-ddwkl" Oct 30 13:00:00.574206 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.571074 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53e599d0-de7d-4c2f-86f9-97cc39405bc0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-327e31-29889420-2vbrc\" (UID: \"53e599d0-de7d-4c2f-86f9-97cc39405bc0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-327e31-29889420-2vbrc" Oct 30 13:00:00.574206 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.572986 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8vk9\" (UniqueName: \"kubernetes.io/projected/ced43030-57a2-418d-bf86-f9c5a3b8cd95-kube-api-access-s8vk9\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-42220c-29889420-q4fqq\" (UID: \"ced43030-57a2-418d-bf86-f9c5a3b8cd95\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-42220c-29889420-q4fqq" Oct 30 13:00:00.574206 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.573029 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnfs8\" (UniqueName: \"kubernetes.io/projected/1daed017-3390-4a05-9457-8d6c97919043-kube-api-access-xnfs8\") pod \"maas-api-key-cleanup-e2e-extmod-a-6ad1da-29889420-qhlvt\" (UID: \"1daed017-3390-4a05-9457-8d6c97919043\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6ad1da-29889420-qhlvt" Oct 30 13:00:00.574206 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.573060 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a130ae9f-0e32-4b92-b6d3-597640f7e802-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-9d45fb-29889420-5wlx7\" (UID: \"a130ae9f-0e32-4b92-b6d3-597640f7e802\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-9d45fb-29889420-5wlx7" Oct 30 13:00:00.574206 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.572390 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ced43030-57a2-418d-bf86-f9c5a3b8cd95-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-42220c-29889420-q4fqq\" (UID: \"ced43030-57a2-418d-bf86-f9c5a3b8cd95\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-42220c-29889420-q4fqq" Oct 30 13:00:00.574206 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.573621 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94f72c00-8f0f-4bcf-9218-cd64b7773d1f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-f7464f-29889420-z9mg6\" (UID: \"94f72c00-8f0f-4bcf-9218-cd64b7773d1f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f7464f-29889420-z9mg6" Oct 30 13:00:00.590686 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.590661 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889420-n4zdq"] Oct 30 13:00:00.592402 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.592380 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-10f364-29889420-fzd6k" Oct 30 13:00:00.613462 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.607345 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e9f184-29889420-sv9rp"] Oct 30 13:00:00.613462 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.609562 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9rmm\" (UniqueName: \"kubernetes.io/projected/94f72c00-8f0f-4bcf-9218-cd64b7773d1f-kube-api-access-w9rmm\") pod \"maas-api-key-cleanup-e2e-extmod-a-f7464f-29889420-z9mg6\" (UID: \"94f72c00-8f0f-4bcf-9218-cd64b7773d1f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f7464f-29889420-z9mg6" Oct 30 13:00:00.621027 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.616153 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-73bafc-29889420-4lh9z" Oct 30 13:00:00.621027 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.616956 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6b67f8-29889420-pt2lw"] Oct 30 13:00:00.621027 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.619000 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlbjg\" (UniqueName: \"kubernetes.io/projected/53e599d0-de7d-4c2f-86f9-97cc39405bc0-kube-api-access-nlbjg\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-327e31-29889420-2vbrc\" (UID: \"53e599d0-de7d-4c2f-86f9-97cc39405bc0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-327e31-29889420-2vbrc" Oct 30 13:00:00.621027 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.619708 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8vk9\" (UniqueName: \"kubernetes.io/projected/ced43030-57a2-418d-bf86-f9c5a3b8cd95-kube-api-access-s8vk9\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-42220c-29889420-q4fqq\" (UID: \"ced43030-57a2-418d-bf86-f9c5a3b8cd95\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-42220c-29889420-q4fqq" Oct 30 13:00:00.625206 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.625113 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6b67f8-29889420-pt2lw" Oct 30 13:00:00.625414 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.625386 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1333c8-29889420-9zjhg"] Oct 30 13:00:00.635844 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.634995 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-dbe123-29889420-bcnzv" Oct 30 13:00:00.677967 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.658424 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6b67f8-29889420-pt2lw"] Oct 30 13:00:00.680106 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.680072 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9jgd\" (UniqueName: \"kubernetes.io/projected/f5af9446-e0d9-4edc-b01f-d049c4b2d3e7-kube-api-access-z9jgd\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-8d2170-29889420-67bzd\" (UID: \"f5af9446-e0d9-4edc-b01f-d049c4b2d3e7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-8d2170-29889420-67bzd" Oct 30 13:00:00.680351 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.680329 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqvth\" (UniqueName: \"kubernetes.io/projected/a130ae9f-0e32-4b92-b6d3-597640f7e802-kube-api-access-gqvth\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-9d45fb-29889420-5wlx7\" (UID: \"a130ae9f-0e32-4b92-b6d3-597640f7e802\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-9d45fb-29889420-5wlx7" Oct 30 13:00:00.680487 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.680469 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/71805e6f-86c5-4659-aa70-849ba38f288c-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889420-n4zdq\" (UID: \"71805e6f-86c5-4659-aa70-849ba38f288c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889420-n4zdq" Oct 30 13:00:00.680693 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.680675 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1baa9cd5-0305-4f77-b7cf-f4751053ace8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-812990-29889420-ddwkl\" (UID: \"1baa9cd5-0305-4f77-b7cf-f4751053ace8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-812990-29889420-ddwkl" Oct 30 13:00:00.680890 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.680870 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94bed6ea-d66f-4f63-8859-a5a8d1969a01-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-1333c8-29889420-9zjhg\" (UID: \"94bed6ea-d66f-4f63-8859-a5a8d1969a01\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1333c8-29889420-9zjhg" Oct 30 13:00:00.681040 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.681024 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shzqt\" (UniqueName: \"kubernetes.io/projected/29e5414c-9b69-4713-8b27-80af3af5b4c7-kube-api-access-shzqt\") pod \"maas-api-key-cleanup-e2e-inf-b-e9f184-29889420-sv9rp\" (UID: \"29e5414c-9b69-4713-8b27-80af3af5b4c7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e9f184-29889420-sv9rp" Oct 30 13:00:00.681186 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.681169 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnfs8\" (UniqueName: \"kubernetes.io/projected/1daed017-3390-4a05-9457-8d6c97919043-kube-api-access-xnfs8\") pod \"maas-api-key-cleanup-e2e-extmod-a-6ad1da-29889420-qhlvt\" (UID: \"1daed017-3390-4a05-9457-8d6c97919043\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6ad1da-29889420-qhlvt" Oct 30 13:00:00.681314 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.681298 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a130ae9f-0e32-4b92-b6d3-597640f7e802-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-9d45fb-29889420-5wlx7\" (UID: \"a130ae9f-0e32-4b92-b6d3-597640f7e802\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-9d45fb-29889420-5wlx7" Oct 30 13:00:00.681535 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.681512 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1daed017-3390-4a05-9457-8d6c97919043-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-6ad1da-29889420-qhlvt\" (UID: \"1daed017-3390-4a05-9457-8d6c97919043\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6ad1da-29889420-qhlvt" Oct 30 13:00:00.681756 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.681739 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbqrv\" (UniqueName: \"kubernetes.io/projected/1baa9cd5-0305-4f77-b7cf-f4751053ace8-kube-api-access-zbqrv\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-812990-29889420-ddwkl\" (UID: \"1baa9cd5-0305-4f77-b7cf-f4751053ace8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-812990-29889420-ddwkl" Oct 30 13:00:00.681904 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.681888 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fw5vn\" (UniqueName: \"kubernetes.io/projected/71805e6f-86c5-4659-aa70-849ba38f288c-kube-api-access-fw5vn\") pod \"maas-api-key-cleanup-29889420-n4zdq\" (UID: \"71805e6f-86c5-4659-aa70-849ba38f288c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889420-n4zdq" Oct 30 13:00:00.682050 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.682035 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5af9446-e0d9-4edc-b01f-d049c4b2d3e7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-8d2170-29889420-67bzd\" (UID: \"f5af9446-e0d9-4edc-b01f-d049c4b2d3e7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-8d2170-29889420-67bzd" Oct 30 13:00:00.682193 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.682178 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7bd05c5-f726-4ad3-88bf-b3a351a409bb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-6b67f8-29889420-pt2lw\" (UID: \"a7bd05c5-f726-4ad3-88bf-b3a351a409bb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6b67f8-29889420-pt2lw" Oct 30 13:00:00.683249 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.683224 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/29e5414c-9b69-4713-8b27-80af3af5b4c7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-e9f184-29889420-sv9rp\" (UID: \"29e5414c-9b69-4713-8b27-80af3af5b4c7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e9f184-29889420-sv9rp" Oct 30 13:00:00.683591 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.683561 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krtdq\" (UniqueName: \"kubernetes.io/projected/94bed6ea-d66f-4f63-8859-a5a8d1969a01-kube-api-access-krtdq\") pod \"maas-api-key-cleanup-e2e-inf-a-1333c8-29889420-9zjhg\" (UID: \"94bed6ea-d66f-4f63-8859-a5a8d1969a01\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1333c8-29889420-9zjhg" Oct 30 13:00:00.683768 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.683749 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhgfv\" (UniqueName: \"kubernetes.io/projected/a7bd05c5-f726-4ad3-88bf-b3a351a409bb-kube-api-access-jhgfv\") pod \"maas-api-key-cleanup-e2e-inf-a-6b67f8-29889420-pt2lw\" (UID: \"a7bd05c5-f726-4ad3-88bf-b3a351a409bb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6b67f8-29889420-pt2lw" Oct 30 13:00:00.686151 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.686123 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1baa9cd5-0305-4f77-b7cf-f4751053ace8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-812990-29889420-ddwkl\" (UID: \"1baa9cd5-0305-4f77-b7cf-f4751053ace8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-812990-29889420-ddwkl" Oct 30 13:00:00.686285 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.686196 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5af9446-e0d9-4edc-b01f-d049c4b2d3e7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-8d2170-29889420-67bzd\" (UID: \"f5af9446-e0d9-4edc-b01f-d049c4b2d3e7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-8d2170-29889420-67bzd" Oct 30 13:00:00.688051 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.688024 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ecb3aa-29889420-72h7x" Oct 30 13:00:00.688156 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.688139 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1daed017-3390-4a05-9457-8d6c97919043-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-6ad1da-29889420-qhlvt\" (UID: \"1daed017-3390-4a05-9457-8d6c97919043\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6ad1da-29889420-qhlvt" Oct 30 13:00:00.688613 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.688586 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a130ae9f-0e32-4b92-b6d3-597640f7e802-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-9d45fb-29889420-5wlx7\" (UID: \"a130ae9f-0e32-4b92-b6d3-597640f7e802\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-9d45fb-29889420-5wlx7" Oct 30 13:00:00.700363 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.700341 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2545c7-29889420-xrknl" Oct 30 13:00:00.700773 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.700746 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0bda67-29889420-4z7jt" Oct 30 13:00:00.700909 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.700896 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2675c0-29889420-hqgp5" Oct 30 13:00:00.715737 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.715684 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9jgd\" (UniqueName: \"kubernetes.io/projected/f5af9446-e0d9-4edc-b01f-d049c4b2d3e7-kube-api-access-z9jgd\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-8d2170-29889420-67bzd\" (UID: \"f5af9446-e0d9-4edc-b01f-d049c4b2d3e7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-8d2170-29889420-67bzd" Oct 30 13:00:00.716171 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.716127 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbqrv\" (UniqueName: \"kubernetes.io/projected/1baa9cd5-0305-4f77-b7cf-f4751053ace8-kube-api-access-zbqrv\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-812990-29889420-ddwkl\" (UID: \"1baa9cd5-0305-4f77-b7cf-f4751053ace8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-812990-29889420-ddwkl" Oct 30 13:00:00.716393 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.716347 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnfs8\" (UniqueName: \"kubernetes.io/projected/1daed017-3390-4a05-9457-8d6c97919043-kube-api-access-xnfs8\") pod \"maas-api-key-cleanup-e2e-extmod-a-6ad1da-29889420-qhlvt\" (UID: \"1daed017-3390-4a05-9457-8d6c97919043\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6ad1da-29889420-qhlvt" Oct 30 13:00:00.717265 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.717239 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqvth\" (UniqueName: \"kubernetes.io/projected/a130ae9f-0e32-4b92-b6d3-597640f7e802-kube-api-access-gqvth\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-9d45fb-29889420-5wlx7\" (UID: \"a130ae9f-0e32-4b92-b6d3-597640f7e802\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-9d45fb-29889420-5wlx7" Oct 30 13:00:00.750140 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.749809 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6ad1da-29889420-qhlvt" Oct 30 13:00:00.767873 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.755814 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-812990-29889420-ddwkl" Oct 30 13:00:00.778581 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.778553 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a786af-29889420-mrgm2" Oct 30 13:00:00.785689 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.785519 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fw5vn\" (UniqueName: \"kubernetes.io/projected/71805e6f-86c5-4659-aa70-849ba38f288c-kube-api-access-fw5vn\") pod \"maas-api-key-cleanup-29889420-n4zdq\" (UID: \"71805e6f-86c5-4659-aa70-849ba38f288c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889420-n4zdq" Oct 30 13:00:00.785689 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.785589 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7bd05c5-f726-4ad3-88bf-b3a351a409bb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-6b67f8-29889420-pt2lw\" (UID: \"a7bd05c5-f726-4ad3-88bf-b3a351a409bb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6b67f8-29889420-pt2lw" Oct 30 13:00:00.785689 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.785620 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/29e5414c-9b69-4713-8b27-80af3af5b4c7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-e9f184-29889420-sv9rp\" (UID: \"29e5414c-9b69-4713-8b27-80af3af5b4c7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e9f184-29889420-sv9rp" Oct 30 13:00:00.785930 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.785766 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krtdq\" (UniqueName: \"kubernetes.io/projected/94bed6ea-d66f-4f63-8859-a5a8d1969a01-kube-api-access-krtdq\") pod \"maas-api-key-cleanup-e2e-inf-a-1333c8-29889420-9zjhg\" (UID: \"94bed6ea-d66f-4f63-8859-a5a8d1969a01\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1333c8-29889420-9zjhg" Oct 30 13:00:00.785930 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.785864 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhgfv\" (UniqueName: \"kubernetes.io/projected/a7bd05c5-f726-4ad3-88bf-b3a351a409bb-kube-api-access-jhgfv\") pod \"maas-api-key-cleanup-e2e-inf-a-6b67f8-29889420-pt2lw\" (UID: \"a7bd05c5-f726-4ad3-88bf-b3a351a409bb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6b67f8-29889420-pt2lw" Oct 30 13:00:00.786051 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.786000 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/71805e6f-86c5-4659-aa70-849ba38f288c-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889420-n4zdq\" (UID: \"71805e6f-86c5-4659-aa70-849ba38f288c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889420-n4zdq" Oct 30 13:00:00.786884 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.786141 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94bed6ea-d66f-4f63-8859-a5a8d1969a01-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-1333c8-29889420-9zjhg\" (UID: \"94bed6ea-d66f-4f63-8859-a5a8d1969a01\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1333c8-29889420-9zjhg" Oct 30 13:00:00.786884 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.786195 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shzqt\" (UniqueName: \"kubernetes.io/projected/29e5414c-9b69-4713-8b27-80af3af5b4c7-kube-api-access-shzqt\") pod \"maas-api-key-cleanup-e2e-inf-b-e9f184-29889420-sv9rp\" (UID: \"29e5414c-9b69-4713-8b27-80af3af5b4c7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e9f184-29889420-sv9rp" Oct 30 13:00:00.789526 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.787733 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/71805e6f-86c5-4659-aa70-849ba38f288c-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889420-n4zdq\" (UID: \"71805e6f-86c5-4659-aa70-849ba38f288c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889420-n4zdq" Oct 30 13:00:00.789526 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.787801 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7bd05c5-f726-4ad3-88bf-b3a351a409bb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-6b67f8-29889420-pt2lw\" (UID: \"a7bd05c5-f726-4ad3-88bf-b3a351a409bb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6b67f8-29889420-pt2lw" Oct 30 13:00:00.789526 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.788632 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94bed6ea-d66f-4f63-8859-a5a8d1969a01-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-1333c8-29889420-9zjhg\" (UID: \"94bed6ea-d66f-4f63-8859-a5a8d1969a01\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1333c8-29889420-9zjhg" Oct 30 13:00:00.789526 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.789400 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/29e5414c-9b69-4713-8b27-80af3af5b4c7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-e9f184-29889420-sv9rp\" (UID: \"29e5414c-9b69-4713-8b27-80af3af5b4c7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e9f184-29889420-sv9rp" Oct 30 13:00:00.798290 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.796356 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-42220c-29889420-q4fqq" Oct 30 13:00:00.822917 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.820132 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhgfv\" (UniqueName: \"kubernetes.io/projected/a7bd05c5-f726-4ad3-88bf-b3a351a409bb-kube-api-access-jhgfv\") pod \"maas-api-key-cleanup-e2e-inf-a-6b67f8-29889420-pt2lw\" (UID: \"a7bd05c5-f726-4ad3-88bf-b3a351a409bb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6b67f8-29889420-pt2lw" Oct 30 13:00:00.828700 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.827259 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fw5vn\" (UniqueName: \"kubernetes.io/projected/71805e6f-86c5-4659-aa70-849ba38f288c-kube-api-access-fw5vn\") pod \"maas-api-key-cleanup-29889420-n4zdq\" (UID: \"71805e6f-86c5-4659-aa70-849ba38f288c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889420-n4zdq" Oct 30 13:00:00.828700 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.828538 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-shzqt\" (UniqueName: \"kubernetes.io/projected/29e5414c-9b69-4713-8b27-80af3af5b4c7-kube-api-access-shzqt\") pod \"maas-api-key-cleanup-e2e-inf-b-e9f184-29889420-sv9rp\" (UID: \"29e5414c-9b69-4713-8b27-80af3af5b4c7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e9f184-29889420-sv9rp" Oct 30 13:00:00.834505 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.834477 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-krtdq\" (UniqueName: \"kubernetes.io/projected/94bed6ea-d66f-4f63-8859-a5a8d1969a01-kube-api-access-krtdq\") pod \"maas-api-key-cleanup-e2e-inf-a-1333c8-29889420-9zjhg\" (UID: \"94bed6ea-d66f-4f63-8859-a5a8d1969a01\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1333c8-29889420-9zjhg" Oct 30 13:00:00.841771 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.841736 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1333c8-29889420-9zjhg" Oct 30 13:00:00.851776 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.851751 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6b67f8-29889420-pt2lw" Oct 30 13:00:00.931940 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.919230 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f7464f-29889420-z9mg6" Oct 30 13:00:00.931940 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.922182 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-327e31-29889420-2vbrc" Oct 30 13:00:00.953630 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.940478 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-9d45fb-29889420-5wlx7" Oct 30 13:00:00.957649 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:00.957556 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-8d2170-29889420-67bzd" Oct 30 13:00:01.119593 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:01.119513 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889420-n4zdq" Oct 30 13:00:01.124621 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:01.124596 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e9f184-29889420-sv9rp" Oct 30 13:00:01.303110 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:01.303049 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-12fd33-29889420-cfqsp"] Oct 30 13:00:01.535764 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:01.535711 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-12fd33-29889420-cfqsp" event={"ID":"f2695f98-7dab-4cf3-aaa3-d1717ecc2d30","Type":"ContainerStarted","Data":"0b5aeb3bbdac21c1a9ff7080ac0756bbea9207ff367102227c9e4805748767e1"} Oct 30 13:00:01.797114 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:01.797083 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fddaa3-29889420-vzns9"] Oct 30 13:00:01.831323 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:01.831294 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-dd0628-29889420-jh88n"] Oct 30 13:00:01.846799 ip-10-0-141-251 kubenswrapper[4908]: W1030 13:00:01.846765 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod347ca826_af8d_496d_bd0b_dca4071b3cd8.slice/crio-309a0efd465068d7d45cc7167f1e9b3a8d0ffb8c76c145eb2e0142384800cf0f WatchSource:0}: Error finding container 309a0efd465068d7d45cc7167f1e9b3a8d0ffb8c76c145eb2e0142384800cf0f: Status 404 returned error can't find the container with id 309a0efd465068d7d45cc7167f1e9b3a8d0ffb8c76c145eb2e0142384800cf0f Oct 30 13:00:01.940185 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:01.940152 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2545c7-29889420-xrknl"] Oct 30 13:00:01.940733 ip-10-0-141-251 kubenswrapper[4908]: W1030 13:00:01.940706 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf7295f0_efb3_487a_926c_c4f19fddde26.slice/crio-5dc636445502e263bc215f39f516c71d35510dd0f2cc65055d6bcf01ad449db7 WatchSource:0}: Error finding container 5dc636445502e263bc215f39f516c71d35510dd0f2cc65055d6bcf01ad449db7: Status 404 returned error can't find the container with id 5dc636445502e263bc215f39f516c71d35510dd0f2cc65055d6bcf01ad449db7 Oct 30 13:00:01.965271 ip-10-0-141-251 kubenswrapper[4908]: W1030 13:00:01.965228 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1baa9cd5_0305_4f77_b7cf_f4751053ace8.slice/crio-2259400db520020aceeca60e8e16c05abc23c24dc587a8fc0e56d3edb117c8a3 WatchSource:0}: Error finding container 2259400db520020aceeca60e8e16c05abc23c24dc587a8fc0e56d3edb117c8a3: Status 404 returned error can't find the container with id 2259400db520020aceeca60e8e16c05abc23c24dc587a8fc0e56d3edb117c8a3 Oct 30 13:00:01.968739 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:01.968710 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-812990-29889420-ddwkl"] Oct 30 13:00:01.974924 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:01.974895 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a786af-29889420-mrgm2"] Oct 30 13:00:02.038808 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:02.037480 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-dbe123-29889420-bcnzv"] Oct 30 13:00:02.038808 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:02.037604 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6b67f8-29889420-pt2lw"] Oct 30 13:00:02.044530 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:02.044456 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-42220c-29889420-q4fqq"] Oct 30 13:00:02.055953 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:02.055923 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0bda67-29889420-4z7jt"] Oct 30 13:00:02.080930 ip-10-0-141-251 kubenswrapper[4908]: W1030 13:00:02.080895 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bb2e64a_3fd0_41bb_aafa_8f234fb2ffa2.slice/crio-d4bfd60afba0eeaddbf8a5e8037417b67960769e1be5d55f338d4f3511834004 WatchSource:0}: Error finding container d4bfd60afba0eeaddbf8a5e8037417b67960769e1be5d55f338d4f3511834004: Status 404 returned error can't find the container with id d4bfd60afba0eeaddbf8a5e8037417b67960769e1be5d55f338d4f3511834004 Oct 30 13:00:02.112691 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:02.112662 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2675c0-29889420-hqgp5"] Oct 30 13:00:02.129457 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:02.117239 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-73bafc-29889420-4lh9z"] Oct 30 13:00:02.149254 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:02.143636 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-10f364-29889420-fzd6k"] Oct 30 13:00:02.154319 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:02.154289 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ecb3aa-29889420-72h7x"] Oct 30 13:00:02.165607 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:02.165577 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-9d45fb-29889420-5wlx7"] Oct 30 13:00:02.172126 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:02.172089 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-8d2170-29889420-67bzd"] Oct 30 13:00:02.207128 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:02.207093 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1333c8-29889420-9zjhg"] Oct 30 13:00:02.220068 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:02.220044 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f7464f-29889420-z9mg6"] Oct 30 13:00:02.232113 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:02.232086 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6ad1da-29889420-qhlvt"] Oct 30 13:00:02.288978 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:02.288939 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e9f184-29889420-sv9rp"] Oct 30 13:00:02.309433 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:02.303113 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889420-n4zdq"] Oct 30 13:00:02.313049 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:02.313003 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-327e31-29889420-2vbrc"] Oct 30 13:00:02.313363 ip-10-0-141-251 kubenswrapper[4908]: W1030 13:00:02.313338 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94bed6ea_d66f_4f63_8859_a5a8d1969a01.slice/crio-9e432b4bdccd9305c88f99cc2a0d907fd7611f0539c79c827ee9310cda914fa2 WatchSource:0}: Error finding container 9e432b4bdccd9305c88f99cc2a0d907fd7611f0539c79c827ee9310cda914fa2: Status 404 returned error can't find the container with id 9e432b4bdccd9305c88f99cc2a0d907fd7611f0539c79c827ee9310cda914fa2 Oct 30 13:00:02.316426 ip-10-0-141-251 kubenswrapper[4908]: W1030 13:00:02.315287 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94f72c00_8f0f_4bcf_9218_cd64b7773d1f.slice/crio-2b260926cc3df5a112845dc372bbef135eb1285b7c6743a00e55e92eaefcc7f1 WatchSource:0}: Error finding container 2b260926cc3df5a112845dc372bbef135eb1285b7c6743a00e55e92eaefcc7f1: Status 404 returned error can't find the container with id 2b260926cc3df5a112845dc372bbef135eb1285b7c6743a00e55e92eaefcc7f1 Oct 30 13:00:02.317305 ip-10-0-141-251 kubenswrapper[4908]: W1030 13:00:02.317288 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1daed017_3390_4a05_9457_8d6c97919043.slice/crio-b65e4e06839268c5d9394433ba58fc5b5aada07507422b202276d358bf995d85 WatchSource:0}: Error finding container b65e4e06839268c5d9394433ba58fc5b5aada07507422b202276d358bf995d85: Status 404 returned error can't find the container with id b65e4e06839268c5d9394433ba58fc5b5aada07507422b202276d358bf995d85 Oct 30 13:00:02.328016 ip-10-0-141-251 kubenswrapper[4908]: W1030 13:00:02.327987 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71805e6f_86c5_4659_aa70_849ba38f288c.slice/crio-b5faf1282d0746dd4abc1df79d6a637d7577f53091b08933b3abe0bb2dfbb42e WatchSource:0}: Error finding container b5faf1282d0746dd4abc1df79d6a637d7577f53091b08933b3abe0bb2dfbb42e: Status 404 returned error can't find the container with id b5faf1282d0746dd4abc1df79d6a637d7577f53091b08933b3abe0bb2dfbb42e Oct 30 13:00:02.329502 ip-10-0-141-251 kubenswrapper[4908]: W1030 13:00:02.329480 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53e599d0_de7d_4c2f_86f9_97cc39405bc0.slice/crio-45d8d9e47c598e40b01ee4c2cdb2f276b87cc730a1277d99f75955b81f102044 WatchSource:0}: Error finding container 45d8d9e47c598e40b01ee4c2cdb2f276b87cc730a1277d99f75955b81f102044: Status 404 returned error can't find the container with id 45d8d9e47c598e40b01ee4c2cdb2f276b87cc730a1277d99f75955b81f102044 Oct 30 13:00:02.546525 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:02.546487 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-812990-29889420-ddwkl" event={"ID":"1baa9cd5-0305-4f77-b7cf-f4751053ace8","Type":"ContainerStarted","Data":"2259400db520020aceeca60e8e16c05abc23c24dc587a8fc0e56d3edb117c8a3"} Oct 30 13:00:02.547929 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:02.547899 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a786af-29889420-mrgm2" event={"ID":"1bb2e64a-3fd0-41bb-aafa-8f234fb2ffa2","Type":"ContainerStarted","Data":"d4bfd60afba0eeaddbf8a5e8037417b67960769e1be5d55f338d4f3511834004"} Oct 30 13:00:02.549376 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:02.549349 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-73bafc-29889420-4lh9z" event={"ID":"0d506c31-90a7-493f-8b73-649d1ffcd0eb","Type":"ContainerStarted","Data":"a9c9d092be559fa34863e8f550db72c10feb14609962d2237923b7386b914f89"} Oct 30 13:00:02.550607 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:02.550581 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f7464f-29889420-z9mg6" event={"ID":"94f72c00-8f0f-4bcf-9218-cd64b7773d1f","Type":"ContainerStarted","Data":"2b260926cc3df5a112845dc372bbef135eb1285b7c6743a00e55e92eaefcc7f1"} Oct 30 13:00:02.551777 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:02.551756 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0bda67-29889420-4z7jt" event={"ID":"b46f76c5-4139-4fbd-af18-5f35fcb1cb05","Type":"ContainerStarted","Data":"02852f50bfad7fef40b45541888fefbdf644770d1e0e25d3aec116ae4c7ccc4c"} Oct 30 13:00:02.553377 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:02.553351 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fddaa3-29889420-vzns9" event={"ID":"2674b41c-6ab9-4222-aa87-4fc2f5d9f468","Type":"ContainerStarted","Data":"638fc8b4170a52d61066b2036007068185945e90565ee614ca88dda2b57ac377"} Oct 30 13:00:02.553480 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:02.553392 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fddaa3-29889420-vzns9" event={"ID":"2674b41c-6ab9-4222-aa87-4fc2f5d9f468","Type":"ContainerStarted","Data":"aa8e2f32d8efd0c45e445cc661f8f8d2632731114baf424804542db284a4d13d"} Oct 30 13:00:02.555124 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:02.555103 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2675c0-29889420-hqgp5" event={"ID":"7f4c2c03-089f-4701-87d3-4e0496b08750","Type":"ContainerStarted","Data":"c52d54912c2428f6b3d8013053604e55b5febfaf57164b299ab190aa06c15134"} Oct 30 13:00:02.556601 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:02.556574 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-9d45fb-29889420-5wlx7" event={"ID":"a130ae9f-0e32-4b92-b6d3-597640f7e802","Type":"ContainerStarted","Data":"d96461af740976a95c0c0d28890dfcce381e6945b2f8d12ab755f6418020cb2f"} Oct 30 13:00:02.559292 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:02.559220 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889420-n4zdq" event={"ID":"71805e6f-86c5-4659-aa70-849ba38f288c","Type":"ContainerStarted","Data":"b5faf1282d0746dd4abc1df79d6a637d7577f53091b08933b3abe0bb2dfbb42e"} Oct 30 13:00:02.561695 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:02.561675 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-10f364-29889420-fzd6k" event={"ID":"76cf17b7-c8b9-48d8-934c-f38aa9d1f008","Type":"ContainerStarted","Data":"e85e8bc46c3d1d00cd575e8a7f54b04aeeb98287a941a11d8f74ab359ea42240"} Oct 30 13:00:02.569205 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:02.569158 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-42220c-29889420-q4fqq" event={"ID":"ced43030-57a2-418d-bf86-f9c5a3b8cd95","Type":"ContainerStarted","Data":"55f13598a2701932512ed3ec59b2667608bcfd0fd79e9ebf0c9371e9fbe1dc37"} Oct 30 13:00:02.585163 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:02.585126 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6b67f8-29889420-pt2lw" event={"ID":"a7bd05c5-f726-4ad3-88bf-b3a351a409bb","Type":"ContainerStarted","Data":"1e89b15fefb97c237a3e16fed867f31b2e9b8db91fcca87f84632c13fa1156d6"} Oct 30 13:00:02.586597 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:02.586568 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2545c7-29889420-xrknl" event={"ID":"bf7295f0-efb3-487a-926c-c4f19fddde26","Type":"ContainerStarted","Data":"5dc636445502e263bc215f39f516c71d35510dd0f2cc65055d6bcf01ad449db7"} Oct 30 13:00:02.588254 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:02.588231 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-dbe123-29889420-bcnzv" event={"ID":"8165abf7-fbfb-4de2-a94c-5023a834835d","Type":"ContainerStarted","Data":"e6d83f69d1d25872b48f126575ad3ffd30082fbc764d7a8a6d1b8c81610631d0"} Oct 30 13:00:02.589798 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:02.589775 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ecb3aa-29889420-72h7x" event={"ID":"cb87d307-def9-4d16-8e7a-54b531f0d2c5","Type":"ContainerStarted","Data":"be7278e70d878bf7d90abec9a1503f4420017e0e04f2a359e204e4daa14fa393"} Oct 30 13:00:02.590864 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:02.590841 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e9f184-29889420-sv9rp" event={"ID":"29e5414c-9b69-4713-8b27-80af3af5b4c7","Type":"ContainerStarted","Data":"94f766340546261b7d2640463d0155ed5928d6c426ef481b1d1cc2ac73f45dfc"} Oct 30 13:00:02.597591 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:02.597559 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6ad1da-29889420-qhlvt" event={"ID":"1daed017-3390-4a05-9457-8d6c97919043","Type":"ContainerStarted","Data":"b65e4e06839268c5d9394433ba58fc5b5aada07507422b202276d358bf995d85"} Oct 30 13:00:02.606427 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:02.606368 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fddaa3-29889420-vzns9" podStartSLOduration=2.606349745 podStartE2EDuration="2.606349745s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:00:02.591679052 +0000 UTC m=+3959.106107811" watchObservedRunningTime="2026-10-30 13:00:02.606349745 +0000 UTC m=+3959.120778492" Oct 30 13:00:02.611851 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:02.610750 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1333c8-29889420-9zjhg" event={"ID":"94bed6ea-d66f-4f63-8859-a5a8d1969a01","Type":"ContainerStarted","Data":"9e432b4bdccd9305c88f99cc2a0d907fd7611f0539c79c827ee9310cda914fa2"} Oct 30 13:00:02.617356 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:02.617327 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-8d2170-29889420-67bzd" event={"ID":"f5af9446-e0d9-4edc-b01f-d049c4b2d3e7","Type":"ContainerStarted","Data":"7d21043dd6b80eea3b2dc088f18a053e5c3fd41919d51f2ffc512b58cc72a8b3"} Oct 30 13:00:02.621229 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:02.621200 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-12fd33-29889420-cfqsp" event={"ID":"f2695f98-7dab-4cf3-aaa3-d1717ecc2d30","Type":"ContainerStarted","Data":"b5531959958a19ed3a61b27af974543f7bb5c72cafe2917402b2bace5c1ab943"} Oct 30 13:00:02.625324 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:02.625271 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-327e31-29889420-2vbrc" event={"ID":"53e599d0-de7d-4c2f-86f9-97cc39405bc0","Type":"ContainerStarted","Data":"45d8d9e47c598e40b01ee4c2cdb2f276b87cc730a1277d99f75955b81f102044"} Oct 30 13:00:02.626708 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:02.626661 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-dd0628-29889420-jh88n" event={"ID":"347ca826-af8d-496d-bd0b-dca4071b3cd8","Type":"ContainerStarted","Data":"309a0efd465068d7d45cc7167f1e9b3a8d0ffb8c76c145eb2e0142384800cf0f"} Oct 30 13:00:03.637135 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:03.637094 4908 generic.go:356] "Generic (PLEG): container finished" podID="2674b41c-6ab9-4222-aa87-4fc2f5d9f468" containerID="638fc8b4170a52d61066b2036007068185945e90565ee614ca88dda2b57ac377" exitCode=60 Oct 30 13:00:03.637645 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:03.637181 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fddaa3-29889420-vzns9" event={"ID":"2674b41c-6ab9-4222-aa87-4fc2f5d9f468","Type":"ContainerDied","Data":"638fc8b4170a52d61066b2036007068185945e90565ee614ca88dda2b57ac377"} Oct 30 13:00:03.637645 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:03.637528 4908 scope.go:122] "RemoveContainer" containerID="638fc8b4170a52d61066b2036007068185945e90565ee614ca88dda2b57ac377" Oct 30 13:00:03.639225 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:03.639196 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2675c0-29889420-hqgp5" event={"ID":"7f4c2c03-089f-4701-87d3-4e0496b08750","Type":"ContainerStarted","Data":"f7dc138f00c048e8584eecb0ff3248ef7ed1ea2795242980d6c399c866fe4811"} Oct 30 13:00:03.642058 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:03.641999 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-9d45fb-29889420-5wlx7" event={"ID":"a130ae9f-0e32-4b92-b6d3-597640f7e802","Type":"ContainerStarted","Data":"a429cdd857d6c0bfe41e4b7f9d2f1f82482af156f0914252e692e1c2168fbbc3"} Oct 30 13:00:03.644897 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:03.644255 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889420-n4zdq" event={"ID":"71805e6f-86c5-4659-aa70-849ba38f288c","Type":"ContainerStarted","Data":"2453eca363563f4152cf90c08ddd077539d9b3b8c9cb5809161cac63ad11a011"} Oct 30 13:00:03.646915 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:03.646886 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-10f364-29889420-fzd6k" event={"ID":"76cf17b7-c8b9-48d8-934c-f38aa9d1f008","Type":"ContainerStarted","Data":"abca2067b9d040e3fd8d0b2c403baf3f57b119cb66994d0197d551b967a9ca67"} Oct 30 13:00:03.649349 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:03.649317 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6b67f8-29889420-pt2lw" event={"ID":"a7bd05c5-f726-4ad3-88bf-b3a351a409bb","Type":"ContainerStarted","Data":"6802d3602f1edaa0752d289899d4de07b7ba19337bfb1afb4615a724c46b9d70"} Oct 30 13:00:03.652008 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:03.651957 4908 generic.go:356] "Generic (PLEG): container finished" podID="bf7295f0-efb3-487a-926c-c4f19fddde26" containerID="8ecb244dfe34af3d365f79e3c10aca566e25f2866f2338c1afaac578233491fd" exitCode=60 Oct 30 13:00:03.652123 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:03.652030 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2545c7-29889420-xrknl" event={"ID":"bf7295f0-efb3-487a-926c-c4f19fddde26","Type":"ContainerDied","Data":"8ecb244dfe34af3d365f79e3c10aca566e25f2866f2338c1afaac578233491fd"} Oct 30 13:00:03.652959 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:03.652906 4908 scope.go:122] "RemoveContainer" containerID="8ecb244dfe34af3d365f79e3c10aca566e25f2866f2338c1afaac578233491fd" Oct 30 13:00:03.655060 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:03.655024 4908 generic.go:356] "Generic (PLEG): container finished" podID="8165abf7-fbfb-4de2-a94c-5023a834835d" containerID="63a800dcd0d1919fc3ec148e82d565d775b77dc1fa26cbb968308bebd1fe39d5" exitCode=60 Oct 30 13:00:03.655165 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:03.655084 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-dbe123-29889420-bcnzv" event={"ID":"8165abf7-fbfb-4de2-a94c-5023a834835d","Type":"ContainerDied","Data":"63a800dcd0d1919fc3ec148e82d565d775b77dc1fa26cbb968308bebd1fe39d5"} Oct 30 13:00:03.655494 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:03.655294 4908 scope.go:122] "RemoveContainer" containerID="63a800dcd0d1919fc3ec148e82d565d775b77dc1fa26cbb968308bebd1fe39d5" Oct 30 13:00:03.664642 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:03.659805 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ecb3aa-29889420-72h7x" event={"ID":"cb87d307-def9-4d16-8e7a-54b531f0d2c5","Type":"ContainerStarted","Data":"fbd79d47f8654aad5bf35f568a012acfe9923462523af28cdbb1a730f35d3592"} Oct 30 13:00:03.674495 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:03.669765 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e9f184-29889420-sv9rp" event={"ID":"29e5414c-9b69-4713-8b27-80af3af5b4c7","Type":"ContainerStarted","Data":"d00dbb68e8a49c4b6f3530f36896b81b12d3888d0c2eefc2f202d0cb8eeb9e35"} Oct 30 13:00:03.677851 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:03.677203 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6ad1da-29889420-qhlvt" event={"ID":"1daed017-3390-4a05-9457-8d6c97919043","Type":"ContainerStarted","Data":"6e89ee35e7c462c448676e2fa4dc2e1e11040ab5807efbd094389f0273b5de31"} Oct 30 13:00:03.686650 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:03.682236 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-12fd33-29889420-cfqsp" podStartSLOduration=3.682216576 podStartE2EDuration="3.682216576s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:00:02.667001605 +0000 UTC m=+3959.181430353" watchObservedRunningTime="2026-10-30 13:00:03.682216576 +0000 UTC m=+3960.196645331" Oct 30 13:00:03.711697 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:03.708291 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1333c8-29889420-9zjhg" event={"ID":"94bed6ea-d66f-4f63-8859-a5a8d1969a01","Type":"ContainerStarted","Data":"c402cb9fcea37536e64e7fabb391504349c3ddcfb6ac71d9c4f4e27730681fc0"} Oct 30 13:00:03.718482 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:03.718130 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-8d2170-29889420-67bzd" event={"ID":"f5af9446-e0d9-4edc-b01f-d049c4b2d3e7","Type":"ContainerStarted","Data":"22449e40d4cff41f0e2c7e5051fcdd239608573532a09c47da81361e17c52f2d"} Oct 30 13:00:03.746327 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:03.735268 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ecb3aa-29889420-72h7x" podStartSLOduration=3.7352499359999998 podStartE2EDuration="3.735249936s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:00:03.734165063 +0000 UTC m=+3960.248593824" watchObservedRunningTime="2026-10-30 13:00:03.735249936 +0000 UTC m=+3960.249678680" Oct 30 13:00:03.758546 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:03.756508 4908 generic.go:356] "Generic (PLEG): container finished" podID="f2695f98-7dab-4cf3-aaa3-d1717ecc2d30" containerID="b5531959958a19ed3a61b27af974543f7bb5c72cafe2917402b2bace5c1ab943" exitCode=60 Oct 30 13:00:03.758546 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:03.756599 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-12fd33-29889420-cfqsp" event={"ID":"f2695f98-7dab-4cf3-aaa3-d1717ecc2d30","Type":"ContainerDied","Data":"b5531959958a19ed3a61b27af974543f7bb5c72cafe2917402b2bace5c1ab943"} Oct 30 13:00:03.784158 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:03.759046 4908 scope.go:122] "RemoveContainer" containerID="b5531959958a19ed3a61b27af974543f7bb5c72cafe2917402b2bace5c1ab943" Oct 30 13:00:03.784158 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:03.778274 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e9f184-29889420-sv9rp" podStartSLOduration=3.7782542120000002 podStartE2EDuration="3.778254212s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:00:03.774282689 +0000 UTC m=+3960.288711436" watchObservedRunningTime="2026-10-30 13:00:03.778254212 +0000 UTC m=+3960.292682963" Oct 30 13:00:03.799373 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:03.791793 4908 generic.go:356] "Generic (PLEG): container finished" podID="347ca826-af8d-496d-bd0b-dca4071b3cd8" containerID="d6bcaeceda4ae67b3907d15663f840db8de8e13454149782953ccfc2fe1c0a46" exitCode=60 Oct 30 13:00:03.799373 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:03.792030 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-dd0628-29889420-jh88n" event={"ID":"347ca826-af8d-496d-bd0b-dca4071b3cd8","Type":"ContainerDied","Data":"d6bcaeceda4ae67b3907d15663f840db8de8e13454149782953ccfc2fe1c0a46"} Oct 30 13:00:03.799373 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:03.792538 4908 scope.go:122] "RemoveContainer" containerID="d6bcaeceda4ae67b3907d15663f840db8de8e13454149782953ccfc2fe1c0a46" Oct 30 13:00:03.799373 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:03.796587 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-812990-29889420-ddwkl" event={"ID":"1baa9cd5-0305-4f77-b7cf-f4751053ace8","Type":"ContainerStarted","Data":"2f2a41ee12eb4b73c2721aa6ee7180fceba973d35ae43775c561e83ea5562f87"} Oct 30 13:00:03.799373 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:03.797364 4908 scope.go:122] "RemoveContainer" containerID="2f2a41ee12eb4b73c2721aa6ee7180fceba973d35ae43775c561e83ea5562f87" Oct 30 13:00:03.803884 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:03.801722 4908 generic.go:356] "Generic (PLEG): container finished" podID="1bb2e64a-3fd0-41bb-aafa-8f234fb2ffa2" containerID="2ba70496bb0e3d8a76a2ca8de05fc18e522893d93c65be886f601c5525ee36f5" exitCode=60 Oct 30 13:00:03.803884 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:03.801800 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a786af-29889420-mrgm2" event={"ID":"1bb2e64a-3fd0-41bb-aafa-8f234fb2ffa2","Type":"ContainerDied","Data":"2ba70496bb0e3d8a76a2ca8de05fc18e522893d93c65be886f601c5525ee36f5"} Oct 30 13:00:03.803884 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:03.802259 4908 scope.go:122] "RemoveContainer" containerID="2ba70496bb0e3d8a76a2ca8de05fc18e522893d93c65be886f601c5525ee36f5" Oct 30 13:00:03.813629 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:03.810613 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-73bafc-29889420-4lh9z" event={"ID":"0d506c31-90a7-493f-8b73-649d1ffcd0eb","Type":"ContainerStarted","Data":"8cd9370751836cce67f9ce51599ada523d5f075add6c38b57d53e941fb5d3daa"} Oct 30 13:00:03.820741 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:03.819644 4908 generic.go:356] "Generic (PLEG): container finished" podID="b46f76c5-4139-4fbd-af18-5f35fcb1cb05" containerID="d533cc5219e4cfe0083f969dec08c23e0db6b5b68e638933a2b37a6f42e962bc" exitCode=60 Oct 30 13:00:03.820741 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:03.819692 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0bda67-29889420-4z7jt" event={"ID":"b46f76c5-4139-4fbd-af18-5f35fcb1cb05","Type":"ContainerDied","Data":"d533cc5219e4cfe0083f969dec08c23e0db6b5b68e638933a2b37a6f42e962bc"} Oct 30 13:00:03.820741 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:03.820216 4908 scope.go:122] "RemoveContainer" containerID="d533cc5219e4cfe0083f969dec08c23e0db6b5b68e638933a2b37a6f42e962bc" Oct 30 13:00:03.878906 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:03.870320 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-10f364-29889420-fzd6k" podStartSLOduration=3.870298978 podStartE2EDuration="3.870298978s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:00:03.854949994 +0000 UTC m=+3960.369378753" watchObservedRunningTime="2026-10-30 13:00:03.870298978 +0000 UTC m=+3960.384727724" Oct 30 13:00:03.904979 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:03.904918 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6b67f8-29889420-pt2lw" podStartSLOduration=3.904899488 podStartE2EDuration="3.904899488s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:00:03.899735512 +0000 UTC m=+3960.414164257" watchObservedRunningTime="2026-10-30 13:00:03.904899488 +0000 UTC m=+3960.419328235" Oct 30 13:00:03.981499 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:03.981447 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-9d45fb-29889420-5wlx7" podStartSLOduration=3.981430624 podStartE2EDuration="3.981430624s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:00:03.977471288 +0000 UTC m=+3960.491900034" watchObservedRunningTime="2026-10-30 13:00:03.981430624 +0000 UTC m=+3960.495859371" Oct 30 13:00:04.019787 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:04.019671 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889420-n4zdq" podStartSLOduration=4.019656127 podStartE2EDuration="4.019656127s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:00:04.015019727 +0000 UTC m=+3960.529448508" watchObservedRunningTime="2026-10-30 13:00:04.019656127 +0000 UTC m=+3960.534084873" Oct 30 13:00:04.057018 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:04.056962 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2675c0-29889420-hqgp5" podStartSLOduration=4.056947777 podStartE2EDuration="4.056947777s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:00:04.052191477 +0000 UTC m=+3960.566620225" watchObservedRunningTime="2026-10-30 13:00:04.056947777 +0000 UTC m=+3960.571376523" Oct 30 13:00:04.128573 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:04.128530 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1333c8-29889420-9zjhg" podStartSLOduration=4.128517795 podStartE2EDuration="4.128517795s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:00:04.12816258 +0000 UTC m=+3960.642591326" watchObservedRunningTime="2026-10-30 13:00:04.128517795 +0000 UTC m=+3960.642946540" Oct 30 13:00:04.210353 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:04.210289 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-73bafc-29889420-4lh9z" podStartSLOduration=4.21026988 podStartE2EDuration="4.21026988s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:00:04.207787675 +0000 UTC m=+3960.722216449" watchObservedRunningTime="2026-10-30 13:00:04.21026988 +0000 UTC m=+3960.724698626" Oct 30 13:00:04.311445 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:04.311041 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-8d2170-29889420-67bzd" podStartSLOduration=4.311019257 podStartE2EDuration="4.311019257s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:00:04.300240572 +0000 UTC m=+3960.814669319" watchObservedRunningTime="2026-10-30 13:00:04.311019257 +0000 UTC m=+3960.825448004" Oct 30 13:00:04.330467 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:04.330398 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-9c3f12/test-model-9c3f12-kserve-cc6c75df5-ldnbh"] Oct 30 13:00:04.338039 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:04.338010 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-9c3f12/test-model-9c3f12-kserve-cc6c75df5-ldnbh"] Oct 30 13:00:04.338185 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:04.338127 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-9c3f12/test-model-9c3f12-kserve-cc6c75df5-ldnbh" Oct 30 13:00:04.354344 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:04.354318 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-9c3f12\"/\"default-dockercfg-nhskc\"" Oct 30 13:00:04.358746 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:04.358708 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-9c3f12\"/\"openshift-service-ca.crt\"" Oct 30 13:00:04.366114 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:04.358884 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-9c3f12\"/\"test-model-9c3f12-kserve-self-signed-certs\"" Oct 30 13:00:04.366114 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:04.358969 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-9c3f12\"/\"kube-root-ca.crt\"" Oct 30 13:00:04.434870 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:04.434807 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6ad1da-29889420-qhlvt" podStartSLOduration=4.434792023 podStartE2EDuration="4.434792023s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:00:04.428986736 +0000 UTC m=+3960.943415482" watchObservedRunningTime="2026-10-30 13:00:04.434792023 +0000 UTC m=+3960.949220829" Oct 30 13:00:04.470663 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:04.470630 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b21ead65-1a34-4368-a2ee-b47795620cd3-dshm\") pod \"test-model-9c3f12-kserve-cc6c75df5-ldnbh\" (UID: \"b21ead65-1a34-4368-a2ee-b47795620cd3\") " pod="ai-tenant-e2e-inf-b-9c3f12/test-model-9c3f12-kserve-cc6c75df5-ldnbh" Oct 30 13:00:04.470951 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:04.470930 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b21ead65-1a34-4368-a2ee-b47795620cd3-home\") pod \"test-model-9c3f12-kserve-cc6c75df5-ldnbh\" (UID: \"b21ead65-1a34-4368-a2ee-b47795620cd3\") " pod="ai-tenant-e2e-inf-b-9c3f12/test-model-9c3f12-kserve-cc6c75df5-ldnbh" Oct 30 13:00:04.471061 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:04.470970 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gh9cp\" (UniqueName: \"kubernetes.io/projected/b21ead65-1a34-4368-a2ee-b47795620cd3-kube-api-access-gh9cp\") pod \"test-model-9c3f12-kserve-cc6c75df5-ldnbh\" (UID: \"b21ead65-1a34-4368-a2ee-b47795620cd3\") " pod="ai-tenant-e2e-inf-b-9c3f12/test-model-9c3f12-kserve-cc6c75df5-ldnbh" Oct 30 13:00:04.471061 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:04.471004 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b21ead65-1a34-4368-a2ee-b47795620cd3-model-cache\") pod \"test-model-9c3f12-kserve-cc6c75df5-ldnbh\" (UID: \"b21ead65-1a34-4368-a2ee-b47795620cd3\") " pod="ai-tenant-e2e-inf-b-9c3f12/test-model-9c3f12-kserve-cc6c75df5-ldnbh" Oct 30 13:00:04.471061 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:04.471034 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b21ead65-1a34-4368-a2ee-b47795620cd3-tls-certs\") pod \"test-model-9c3f12-kserve-cc6c75df5-ldnbh\" (UID: \"b21ead65-1a34-4368-a2ee-b47795620cd3\") " pod="ai-tenant-e2e-inf-b-9c3f12/test-model-9c3f12-kserve-cc6c75df5-ldnbh" Oct 30 13:00:04.471241 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:04.471068 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b21ead65-1a34-4368-a2ee-b47795620cd3-tmp-dir\") pod \"test-model-9c3f12-kserve-cc6c75df5-ldnbh\" (UID: \"b21ead65-1a34-4368-a2ee-b47795620cd3\") " pod="ai-tenant-e2e-inf-b-9c3f12/test-model-9c3f12-kserve-cc6c75df5-ldnbh" Oct 30 13:00:04.572437 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:04.572400 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b21ead65-1a34-4368-a2ee-b47795620cd3-dshm\") pod \"test-model-9c3f12-kserve-cc6c75df5-ldnbh\" (UID: \"b21ead65-1a34-4368-a2ee-b47795620cd3\") " pod="ai-tenant-e2e-inf-b-9c3f12/test-model-9c3f12-kserve-cc6c75df5-ldnbh" Oct 30 13:00:04.572668 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:04.572651 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b21ead65-1a34-4368-a2ee-b47795620cd3-home\") pod \"test-model-9c3f12-kserve-cc6c75df5-ldnbh\" (UID: \"b21ead65-1a34-4368-a2ee-b47795620cd3\") " pod="ai-tenant-e2e-inf-b-9c3f12/test-model-9c3f12-kserve-cc6c75df5-ldnbh" Oct 30 13:00:04.572747 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:04.572690 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gh9cp\" (UniqueName: \"kubernetes.io/projected/b21ead65-1a34-4368-a2ee-b47795620cd3-kube-api-access-gh9cp\") pod \"test-model-9c3f12-kserve-cc6c75df5-ldnbh\" (UID: \"b21ead65-1a34-4368-a2ee-b47795620cd3\") " pod="ai-tenant-e2e-inf-b-9c3f12/test-model-9c3f12-kserve-cc6c75df5-ldnbh" Oct 30 13:00:04.572747 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:04.572721 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b21ead65-1a34-4368-a2ee-b47795620cd3-model-cache\") pod \"test-model-9c3f12-kserve-cc6c75df5-ldnbh\" (UID: \"b21ead65-1a34-4368-a2ee-b47795620cd3\") " pod="ai-tenant-e2e-inf-b-9c3f12/test-model-9c3f12-kserve-cc6c75df5-ldnbh" Oct 30 13:00:04.572889 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:04.572752 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b21ead65-1a34-4368-a2ee-b47795620cd3-tls-certs\") pod \"test-model-9c3f12-kserve-cc6c75df5-ldnbh\" (UID: \"b21ead65-1a34-4368-a2ee-b47795620cd3\") " pod="ai-tenant-e2e-inf-b-9c3f12/test-model-9c3f12-kserve-cc6c75df5-ldnbh" Oct 30 13:00:04.572889 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:04.572792 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b21ead65-1a34-4368-a2ee-b47795620cd3-tmp-dir\") pod \"test-model-9c3f12-kserve-cc6c75df5-ldnbh\" (UID: \"b21ead65-1a34-4368-a2ee-b47795620cd3\") " pod="ai-tenant-e2e-inf-b-9c3f12/test-model-9c3f12-kserve-cc6c75df5-ldnbh" Oct 30 13:00:04.573481 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:04.573459 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b21ead65-1a34-4368-a2ee-b47795620cd3-tmp-dir\") pod \"test-model-9c3f12-kserve-cc6c75df5-ldnbh\" (UID: \"b21ead65-1a34-4368-a2ee-b47795620cd3\") " pod="ai-tenant-e2e-inf-b-9c3f12/test-model-9c3f12-kserve-cc6c75df5-ldnbh" Oct 30 13:00:04.574239 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:04.574055 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b21ead65-1a34-4368-a2ee-b47795620cd3-home\") pod \"test-model-9c3f12-kserve-cc6c75df5-ldnbh\" (UID: \"b21ead65-1a34-4368-a2ee-b47795620cd3\") " pod="ai-tenant-e2e-inf-b-9c3f12/test-model-9c3f12-kserve-cc6c75df5-ldnbh" Oct 30 13:00:04.574597 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:04.574564 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b21ead65-1a34-4368-a2ee-b47795620cd3-model-cache\") pod \"test-model-9c3f12-kserve-cc6c75df5-ldnbh\" (UID: \"b21ead65-1a34-4368-a2ee-b47795620cd3\") " pod="ai-tenant-e2e-inf-b-9c3f12/test-model-9c3f12-kserve-cc6c75df5-ldnbh" Oct 30 13:00:04.576392 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:04.576339 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b21ead65-1a34-4368-a2ee-b47795620cd3-dshm\") pod \"test-model-9c3f12-kserve-cc6c75df5-ldnbh\" (UID: \"b21ead65-1a34-4368-a2ee-b47795620cd3\") " pod="ai-tenant-e2e-inf-b-9c3f12/test-model-9c3f12-kserve-cc6c75df5-ldnbh" Oct 30 13:00:04.578475 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:04.578447 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b21ead65-1a34-4368-a2ee-b47795620cd3-tls-certs\") pod \"test-model-9c3f12-kserve-cc6c75df5-ldnbh\" (UID: \"b21ead65-1a34-4368-a2ee-b47795620cd3\") " pod="ai-tenant-e2e-inf-b-9c3f12/test-model-9c3f12-kserve-cc6c75df5-ldnbh" Oct 30 13:00:04.599020 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:04.597909 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gh9cp\" (UniqueName: \"kubernetes.io/projected/b21ead65-1a34-4368-a2ee-b47795620cd3-kube-api-access-gh9cp\") pod \"test-model-9c3f12-kserve-cc6c75df5-ldnbh\" (UID: \"b21ead65-1a34-4368-a2ee-b47795620cd3\") " pod="ai-tenant-e2e-inf-b-9c3f12/test-model-9c3f12-kserve-cc6c75df5-ldnbh" Oct 30 13:00:04.930896 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:04.915789 4908 generic.go:356] "Generic (PLEG): container finished" podID="1baa9cd5-0305-4f77-b7cf-f4751053ace8" containerID="2f2a41ee12eb4b73c2721aa6ee7180fceba973d35ae43775c561e83ea5562f87" exitCode=60 Oct 30 13:00:04.930896 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:04.915910 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-812990-29889420-ddwkl" event={"ID":"1baa9cd5-0305-4f77-b7cf-f4751053ace8","Type":"ContainerDied","Data":"2f2a41ee12eb4b73c2721aa6ee7180fceba973d35ae43775c561e83ea5562f87"} Oct 30 13:00:04.930896 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:04.915962 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-812990-29889420-ddwkl" event={"ID":"1baa9cd5-0305-4f77-b7cf-f4751053ace8","Type":"ContainerStarted","Data":"45287a04f0139cd48af47d49ad0ea72f6da58375218e51ee6d9f8ca3a953bf85"} Oct 30 13:00:04.930896 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:04.927698 4908 generic.go:356] "Generic (PLEG): container finished" podID="0d506c31-90a7-493f-8b73-649d1ffcd0eb" containerID="8cd9370751836cce67f9ce51599ada523d5f075add6c38b57d53e941fb5d3daa" exitCode=60 Oct 30 13:00:04.930896 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:04.927797 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-73bafc-29889420-4lh9z" event={"ID":"0d506c31-90a7-493f-8b73-649d1ffcd0eb","Type":"ContainerDied","Data":"8cd9370751836cce67f9ce51599ada523d5f075add6c38b57d53e941fb5d3daa"} Oct 30 13:00:04.930896 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:04.928284 4908 scope.go:122] "RemoveContainer" containerID="8cd9370751836cce67f9ce51599ada523d5f075add6c38b57d53e941fb5d3daa" Oct 30 13:00:04.944036 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:04.943616 4908 generic.go:356] "Generic (PLEG): container finished" podID="94f72c00-8f0f-4bcf-9218-cd64b7773d1f" containerID="72994127e7d0f026d2c1c233b772e5cc1922058d4cb5391400706b7695b19f21" exitCode=60 Oct 30 13:00:04.944036 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:04.943712 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f7464f-29889420-z9mg6" event={"ID":"94f72c00-8f0f-4bcf-9218-cd64b7773d1f","Type":"ContainerDied","Data":"72994127e7d0f026d2c1c233b772e5cc1922058d4cb5391400706b7695b19f21"} Oct 30 13:00:04.947473 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:04.944253 4908 scope.go:122] "RemoveContainer" containerID="72994127e7d0f026d2c1c233b772e5cc1922058d4cb5391400706b7695b19f21" Oct 30 13:00:04.957448 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:04.957418 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-9c3f12/test-model-9c3f12-kserve-cc6c75df5-ldnbh" Oct 30 13:00:04.960322 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:04.960290 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fddaa3-29889420-vzns9" event={"ID":"2674b41c-6ab9-4222-aa87-4fc2f5d9f468","Type":"ContainerStarted","Data":"a1e193ed4b30c47e5eb85bda06f874bc2f83a6ef1e393fbedba7f42f690ca371"} Oct 30 13:00:04.964663 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:04.964636 4908 generic.go:356] "Generic (PLEG): container finished" podID="7f4c2c03-089f-4701-87d3-4e0496b08750" containerID="f7dc138f00c048e8584eecb0ff3248ef7ed1ea2795242980d6c399c866fe4811" exitCode=60 Oct 30 13:00:04.964817 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:04.964778 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2675c0-29889420-hqgp5" event={"ID":"7f4c2c03-089f-4701-87d3-4e0496b08750","Type":"ContainerDied","Data":"f7dc138f00c048e8584eecb0ff3248ef7ed1ea2795242980d6c399c866fe4811"} Oct 30 13:00:04.965236 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:04.965216 4908 scope.go:122] "RemoveContainer" containerID="f7dc138f00c048e8584eecb0ff3248ef7ed1ea2795242980d6c399c866fe4811" Oct 30 13:00:04.977554 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:04.976255 4908 generic.go:356] "Generic (PLEG): container finished" podID="a130ae9f-0e32-4b92-b6d3-597640f7e802" containerID="a429cdd857d6c0bfe41e4b7f9d2f1f82482af156f0914252e692e1c2168fbbc3" exitCode=60 Oct 30 13:00:04.977554 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:04.976380 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-9d45fb-29889420-5wlx7" event={"ID":"a130ae9f-0e32-4b92-b6d3-597640f7e802","Type":"ContainerDied","Data":"a429cdd857d6c0bfe41e4b7f9d2f1f82482af156f0914252e692e1c2168fbbc3"} Oct 30 13:00:04.981269 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:04.980332 4908 generic.go:356] "Generic (PLEG): container finished" podID="71805e6f-86c5-4659-aa70-849ba38f288c" containerID="2453eca363563f4152cf90c08ddd077539d9b3b8c9cb5809161cac63ad11a011" exitCode=60 Oct 30 13:00:04.981269 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:04.980411 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889420-n4zdq" event={"ID":"71805e6f-86c5-4659-aa70-849ba38f288c","Type":"ContainerDied","Data":"2453eca363563f4152cf90c08ddd077539d9b3b8c9cb5809161cac63ad11a011"} Oct 30 13:00:04.986843 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:04.982957 4908 generic.go:356] "Generic (PLEG): container finished" podID="76cf17b7-c8b9-48d8-934c-f38aa9d1f008" containerID="abca2067b9d040e3fd8d0b2c403baf3f57b119cb66994d0197d551b967a9ca67" exitCode=60 Oct 30 13:00:04.986843 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:04.983048 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-10f364-29889420-fzd6k" event={"ID":"76cf17b7-c8b9-48d8-934c-f38aa9d1f008","Type":"ContainerDied","Data":"abca2067b9d040e3fd8d0b2c403baf3f57b119cb66994d0197d551b967a9ca67"} Oct 30 13:00:04.986843 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:04.985193 4908 generic.go:356] "Generic (PLEG): container finished" podID="ced43030-57a2-418d-bf86-f9c5a3b8cd95" containerID="b1739406b94f2e5bbe4dd2e3fabdb518c3a8880d2184ed1cd94cfff0dd7e86e1" exitCode=60 Oct 30 13:00:04.986843 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:04.985282 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-42220c-29889420-q4fqq" event={"ID":"ced43030-57a2-418d-bf86-f9c5a3b8cd95","Type":"ContainerDied","Data":"b1739406b94f2e5bbe4dd2e3fabdb518c3a8880d2184ed1cd94cfff0dd7e86e1"} Oct 30 13:00:04.990451 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:04.988468 4908 scope.go:122] "RemoveContainer" containerID="a429cdd857d6c0bfe41e4b7f9d2f1f82482af156f0914252e692e1c2168fbbc3" Oct 30 13:00:04.990451 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:04.989131 4908 scope.go:122] "RemoveContainer" containerID="2453eca363563f4152cf90c08ddd077539d9b3b8c9cb5809161cac63ad11a011" Oct 30 13:00:04.990451 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:04.989579 4908 scope.go:122] "RemoveContainer" containerID="abca2067b9d040e3fd8d0b2c403baf3f57b119cb66994d0197d551b967a9ca67" Oct 30 13:00:04.990451 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:04.989991 4908 scope.go:122] "RemoveContainer" containerID="b1739406b94f2e5bbe4dd2e3fabdb518c3a8880d2184ed1cd94cfff0dd7e86e1" Oct 30 13:00:04.992345 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:04.991502 4908 generic.go:356] "Generic (PLEG): container finished" podID="a7bd05c5-f726-4ad3-88bf-b3a351a409bb" containerID="6802d3602f1edaa0752d289899d4de07b7ba19337bfb1afb4615a724c46b9d70" exitCode=60 Oct 30 13:00:04.992345 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:04.991612 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6b67f8-29889420-pt2lw" event={"ID":"a7bd05c5-f726-4ad3-88bf-b3a351a409bb","Type":"ContainerDied","Data":"6802d3602f1edaa0752d289899d4de07b7ba19337bfb1afb4615a724c46b9d70"} Oct 30 13:00:04.992345 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:04.991866 4908 scope.go:122] "RemoveContainer" containerID="6802d3602f1edaa0752d289899d4de07b7ba19337bfb1afb4615a724c46b9d70" Oct 30 13:00:04.996871 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:04.995766 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-812990-29889420-ddwkl" podStartSLOduration=4.995749275 podStartE2EDuration="4.995749275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:00:04.963096417 +0000 UTC m=+3961.477525164" watchObservedRunningTime="2026-10-30 13:00:04.995749275 +0000 UTC m=+3961.510178024" Oct 30 13:00:04.996871 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:04.996737 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-dbe123-29889420-bcnzv" event={"ID":"8165abf7-fbfb-4de2-a94c-5023a834835d","Type":"ContainerStarted","Data":"d79974ed63f49d74bbd0e00ed623030be41359dccc8277ca8703302f35142ca5"} Oct 30 13:00:04.999558 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:04.999534 4908 generic.go:356] "Generic (PLEG): container finished" podID="cb87d307-def9-4d16-8e7a-54b531f0d2c5" containerID="fbd79d47f8654aad5bf35f568a012acfe9923462523af28cdbb1a730f35d3592" exitCode=60 Oct 30 13:00:04.999685 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:04.999623 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ecb3aa-29889420-72h7x" event={"ID":"cb87d307-def9-4d16-8e7a-54b531f0d2c5","Type":"ContainerDied","Data":"fbd79d47f8654aad5bf35f568a012acfe9923462523af28cdbb1a730f35d3592"} Oct 30 13:00:04.999999 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:04.999979 4908 scope.go:122] "RemoveContainer" containerID="fbd79d47f8654aad5bf35f568a012acfe9923462523af28cdbb1a730f35d3592" Oct 30 13:00:05.007338 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:05.007313 4908 generic.go:356] "Generic (PLEG): container finished" podID="29e5414c-9b69-4713-8b27-80af3af5b4c7" containerID="d00dbb68e8a49c4b6f3530f36896b81b12d3888d0c2eefc2f202d0cb8eeb9e35" exitCode=60 Oct 30 13:00:05.007403 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:05.007391 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e9f184-29889420-sv9rp" event={"ID":"29e5414c-9b69-4713-8b27-80af3af5b4c7","Type":"ContainerDied","Data":"d00dbb68e8a49c4b6f3530f36896b81b12d3888d0c2eefc2f202d0cb8eeb9e35"} Oct 30 13:00:05.007685 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:05.007666 4908 scope.go:122] "RemoveContainer" containerID="d00dbb68e8a49c4b6f3530f36896b81b12d3888d0c2eefc2f202d0cb8eeb9e35" Oct 30 13:00:05.009807 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:05.009780 4908 generic.go:356] "Generic (PLEG): container finished" podID="1daed017-3390-4a05-9457-8d6c97919043" containerID="6e89ee35e7c462c448676e2fa4dc2e1e11040ab5807efbd094389f0273b5de31" exitCode=60 Oct 30 13:00:05.009970 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:05.009952 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6ad1da-29889420-qhlvt" event={"ID":"1daed017-3390-4a05-9457-8d6c97919043","Type":"ContainerDied","Data":"6e89ee35e7c462c448676e2fa4dc2e1e11040ab5807efbd094389f0273b5de31"} Oct 30 13:00:05.010301 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:05.010284 4908 scope.go:122] "RemoveContainer" containerID="6e89ee35e7c462c448676e2fa4dc2e1e11040ab5807efbd094389f0273b5de31" Oct 30 13:00:05.016794 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:05.016770 4908 generic.go:356] "Generic (PLEG): container finished" podID="94bed6ea-d66f-4f63-8859-a5a8d1969a01" containerID="c402cb9fcea37536e64e7fabb391504349c3ddcfb6ac71d9c4f4e27730681fc0" exitCode=60 Oct 30 13:00:05.016906 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:05.016889 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1333c8-29889420-9zjhg" event={"ID":"94bed6ea-d66f-4f63-8859-a5a8d1969a01","Type":"ContainerDied","Data":"c402cb9fcea37536e64e7fabb391504349c3ddcfb6ac71d9c4f4e27730681fc0"} Oct 30 13:00:05.017334 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:05.017312 4908 scope.go:122] "RemoveContainer" containerID="c402cb9fcea37536e64e7fabb391504349c3ddcfb6ac71d9c4f4e27730681fc0" Oct 30 13:00:05.021864 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:05.021256 4908 generic.go:356] "Generic (PLEG): container finished" podID="f5af9446-e0d9-4edc-b01f-d049c4b2d3e7" containerID="22449e40d4cff41f0e2c7e5051fcdd239608573532a09c47da81361e17c52f2d" exitCode=60 Oct 30 13:00:05.021864 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:05.021329 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-8d2170-29889420-67bzd" event={"ID":"f5af9446-e0d9-4edc-b01f-d049c4b2d3e7","Type":"ContainerDied","Data":"22449e40d4cff41f0e2c7e5051fcdd239608573532a09c47da81361e17c52f2d"} Oct 30 13:00:05.022456 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:05.022149 4908 scope.go:122] "RemoveContainer" containerID="22449e40d4cff41f0e2c7e5051fcdd239608573532a09c47da81361e17c52f2d" Oct 30 13:00:05.024945 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:05.024897 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-12fd33-29889420-cfqsp" event={"ID":"f2695f98-7dab-4cf3-aaa3-d1717ecc2d30","Type":"ContainerStarted","Data":"690ec2b414b11ab39bef37bd18421e14c295a32577da7d213917ab9cb7ae45b5"} Oct 30 13:00:05.031595 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:05.030700 4908 generic.go:356] "Generic (PLEG): container finished" podID="53e599d0-de7d-4c2f-86f9-97cc39405bc0" containerID="035e9724d59bf4b08b5bc36a702a866689e87b8ca5da243df5b1a9e6bac5081a" exitCode=60 Oct 30 13:00:05.031595 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:05.030797 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-327e31-29889420-2vbrc" event={"ID":"53e599d0-de7d-4c2f-86f9-97cc39405bc0","Type":"ContainerDied","Data":"035e9724d59bf4b08b5bc36a702a866689e87b8ca5da243df5b1a9e6bac5081a"} Oct 30 13:00:05.031595 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:05.031177 4908 scope.go:122] "RemoveContainer" containerID="035e9724d59bf4b08b5bc36a702a866689e87b8ca5da243df5b1a9e6bac5081a" Oct 30 13:00:05.034114 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:05.033941 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-dd0628-29889420-jh88n" event={"ID":"347ca826-af8d-496d-bd0b-dca4071b3cd8","Type":"ContainerStarted","Data":"96b0ea8ecfe4f3439a866d2f770a7b57d4ad8b8f0b714aebead350c5999327b0"} Oct 30 13:00:05.438702 ip-10-0-141-251 kubenswrapper[4908]: W1030 13:00:05.438666 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb21ead65_1a34_4368_a2ee_b47795620cd3.slice/crio-b1b27702d3ad7ad000f8f3a463a1c24b20fdc04acfcc86d4d4c927ffe8dd53b6 WatchSource:0}: Error finding container b1b27702d3ad7ad000f8f3a463a1c24b20fdc04acfcc86d4d4c927ffe8dd53b6: Status 404 returned error can't find the container with id b1b27702d3ad7ad000f8f3a463a1c24b20fdc04acfcc86d4d4c927ffe8dd53b6 Oct 30 13:00:05.443493 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:05.443450 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-9c3f12/test-model-9c3f12-kserve-cc6c75df5-ldnbh"] Oct 30 13:00:05.473869 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:05.473712 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-dd0628-29889420-jh88n" podStartSLOduration=5.473694787 podStartE2EDuration="5.473694787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:00:05.47132774 +0000 UTC m=+3961.985756498" watchObservedRunningTime="2026-10-30 13:00:05.473694787 +0000 UTC m=+3961.988123537" Oct 30 13:00:05.762626 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:05.762103 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-dbe123-29889420-bcnzv" podStartSLOduration=5.7620877 podStartE2EDuration="5.7620877s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:00:05.759100468 +0000 UTC m=+3962.273529216" watchObservedRunningTime="2026-10-30 13:00:05.7620877 +0000 UTC m=+3962.276516446" Oct 30 13:00:06.047200 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:06.047156 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889420-n4zdq" event={"ID":"71805e6f-86c5-4659-aa70-849ba38f288c","Type":"ContainerStarted","Data":"be667f188369ff762f1c63b473896b06f415609784a3d0bc7c9dffb5f9fff403"} Oct 30 13:00:06.049371 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:06.049339 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-10f364-29889420-fzd6k" event={"ID":"76cf17b7-c8b9-48d8-934c-f38aa9d1f008","Type":"ContainerStarted","Data":"e03b25b8e2910b878c1499548f3ab6f0b62e4548f50e313da7ee0484c111f575"} Oct 30 13:00:06.051404 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:06.051377 4908 generic.go:356] "Generic (PLEG): container finished" podID="bf7295f0-efb3-487a-926c-c4f19fddde26" containerID="455c3c383fe1a940a1d2b692834c5f2398232ceafa93c0f8d8ad91295ff035d1" exitCode=60 Oct 30 13:00:06.051526 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:06.051460 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2545c7-29889420-xrknl" event={"ID":"bf7295f0-efb3-487a-926c-c4f19fddde26","Type":"ContainerDied","Data":"455c3c383fe1a940a1d2b692834c5f2398232ceafa93c0f8d8ad91295ff035d1"} Oct 30 13:00:06.051594 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:06.051532 4908 scope.go:122] "RemoveContainer" containerID="8ecb244dfe34af3d365f79e3c10aca566e25f2866f2338c1afaac578233491fd" Oct 30 13:00:06.051725 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:06.051707 4908 scope.go:122] "RemoveContainer" containerID="455c3c383fe1a940a1d2b692834c5f2398232ceafa93c0f8d8ad91295ff035d1" Oct 30 13:00:06.052164 ip-10-0-141-251 kubenswrapper[4908]: E1030 13:00:06.052136 4908 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-2545c7-29889420-xrknl_odh-ai-gateway-infra(bf7295f0-efb3-487a-926c-c4f19fddde26)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2545c7-29889420-xrknl" podUID="bf7295f0-efb3-487a-926c-c4f19fddde26" Oct 30 13:00:06.053625 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:06.053602 4908 generic.go:356] "Generic (PLEG): container finished" podID="8165abf7-fbfb-4de2-a94c-5023a834835d" containerID="d79974ed63f49d74bbd0e00ed623030be41359dccc8277ca8703302f35142ca5" exitCode=60 Oct 30 13:00:06.053727 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:06.053635 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-dbe123-29889420-bcnzv" event={"ID":"8165abf7-fbfb-4de2-a94c-5023a834835d","Type":"ContainerDied","Data":"d79974ed63f49d74bbd0e00ed623030be41359dccc8277ca8703302f35142ca5"} Oct 30 13:00:06.053954 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:06.053935 4908 scope.go:122] "RemoveContainer" containerID="d79974ed63f49d74bbd0e00ed623030be41359dccc8277ca8703302f35142ca5" Oct 30 13:00:06.054246 ip-10-0-141-251 kubenswrapper[4908]: E1030 13:00:06.054223 4908 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-dbe123-29889420-bcnzv_odh-ai-gateway-infra(8165abf7-fbfb-4de2-a94c-5023a834835d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-dbe123-29889420-bcnzv" podUID="8165abf7-fbfb-4de2-a94c-5023a834835d" Oct 30 13:00:06.055745 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:06.055720 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ecb3aa-29889420-72h7x" event={"ID":"cb87d307-def9-4d16-8e7a-54b531f0d2c5","Type":"ContainerStarted","Data":"9e90edddf33c1b728e9595a6acdcdee922d7af72d81fe611f108bf0947790bc7"} Oct 30 13:00:06.057960 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:06.057936 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e9f184-29889420-sv9rp" event={"ID":"29e5414c-9b69-4713-8b27-80af3af5b4c7","Type":"ContainerStarted","Data":"9f340a37a3e8d33f47f2c6b0d7bc2357f171968f4c8f8f230eed09a502acbd7e"} Oct 30 13:00:06.066259 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:06.060361 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6ad1da-29889420-qhlvt" event={"ID":"1daed017-3390-4a05-9457-8d6c97919043","Type":"ContainerStarted","Data":"81f9b9853015d1c8504e652f5e16222bead314f4f516d286f8af942da7d2e044"} Oct 30 13:00:06.089125 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:06.089086 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-9c3f12/test-model-9c3f12-kserve-cc6c75df5-ldnbh" event={"ID":"b21ead65-1a34-4368-a2ee-b47795620cd3","Type":"ContainerStarted","Data":"e4cb88e766fbc3c9abe05612e236a691207719e4222ed4d75d4e35f1eb832736"} Oct 30 13:00:06.089312 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:06.089141 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-9c3f12/test-model-9c3f12-kserve-cc6c75df5-ldnbh" event={"ID":"b21ead65-1a34-4368-a2ee-b47795620cd3","Type":"ContainerStarted","Data":"b1b27702d3ad7ad000f8f3a463a1c24b20fdc04acfcc86d4d4c927ffe8dd53b6"} Oct 30 13:00:06.090332 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:06.090308 4908 scope.go:122] "RemoveContainer" containerID="63a800dcd0d1919fc3ec148e82d565d775b77dc1fa26cbb968308bebd1fe39d5" Oct 30 13:00:06.095382 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:06.095352 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1333c8-29889420-9zjhg" event={"ID":"94bed6ea-d66f-4f63-8859-a5a8d1969a01","Type":"ContainerStarted","Data":"84574467f1500cea18a9ddcfc697eace7267e137c6bd81b2b06762279c05543f"} Oct 30 13:00:06.123267 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:06.123188 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-8d2170-29889420-67bzd" event={"ID":"f5af9446-e0d9-4edc-b01f-d049c4b2d3e7","Type":"ContainerStarted","Data":"1159c140b26e0a148bbfa6c979acfbad6eac5a31cb67f6f9a4b71037310633a5"} Oct 30 13:00:06.127916 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:06.127781 4908 generic.go:356] "Generic (PLEG): container finished" podID="f2695f98-7dab-4cf3-aaa3-d1717ecc2d30" containerID="690ec2b414b11ab39bef37bd18421e14c295a32577da7d213917ab9cb7ae45b5" exitCode=60 Oct 30 13:00:06.127916 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:06.127859 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-12fd33-29889420-cfqsp" event={"ID":"f2695f98-7dab-4cf3-aaa3-d1717ecc2d30","Type":"ContainerDied","Data":"690ec2b414b11ab39bef37bd18421e14c295a32577da7d213917ab9cb7ae45b5"} Oct 30 13:00:06.128123 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:06.127922 4908 scope.go:122] "RemoveContainer" containerID="b5531959958a19ed3a61b27af974543f7bb5c72cafe2917402b2bace5c1ab943" Oct 30 13:00:06.128632 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:06.128228 4908 scope.go:122] "RemoveContainer" containerID="690ec2b414b11ab39bef37bd18421e14c295a32577da7d213917ab9cb7ae45b5" Oct 30 13:00:06.128740 ip-10-0-141-251 kubenswrapper[4908]: E1030 13:00:06.128661 4908 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-12fd33-29889420-cfqsp_odh-ai-gateway-infra(f2695f98-7dab-4cf3-aaa3-d1717ecc2d30)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-12fd33-29889420-cfqsp" podUID="f2695f98-7dab-4cf3-aaa3-d1717ecc2d30" Oct 30 13:00:06.131213 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:06.130404 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-327e31-29889420-2vbrc" event={"ID":"53e599d0-de7d-4c2f-86f9-97cc39405bc0","Type":"ContainerStarted","Data":"53c7a0cb1a17dbfeff3c3e09682d3f006a52ad471b22c7089d0965bfa316bacb"} Oct 30 13:00:06.133533 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:06.132662 4908 generic.go:356] "Generic (PLEG): container finished" podID="347ca826-af8d-496d-bd0b-dca4071b3cd8" containerID="96b0ea8ecfe4f3439a866d2f770a7b57d4ad8b8f0b714aebead350c5999327b0" exitCode=60 Oct 30 13:00:06.133533 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:06.132958 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-dd0628-29889420-jh88n" event={"ID":"347ca826-af8d-496d-bd0b-dca4071b3cd8","Type":"ContainerDied","Data":"96b0ea8ecfe4f3439a866d2f770a7b57d4ad8b8f0b714aebead350c5999327b0"} Oct 30 13:00:06.133533 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:06.133134 4908 scope.go:122] "RemoveContainer" containerID="96b0ea8ecfe4f3439a866d2f770a7b57d4ad8b8f0b714aebead350c5999327b0" Oct 30 13:00:06.133533 ip-10-0-141-251 kubenswrapper[4908]: E1030 13:00:06.133484 4908 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-dd0628-29889420-jh88n_odh-ai-gateway-infra(347ca826-af8d-496d-bd0b-dca4071b3cd8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-dd0628-29889420-jh88n" podUID="347ca826-af8d-496d-bd0b-dca4071b3cd8" Oct 30 13:00:06.138498 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:06.137710 4908 generic.go:356] "Generic (PLEG): container finished" podID="1baa9cd5-0305-4f77-b7cf-f4751053ace8" containerID="45287a04f0139cd48af47d49ad0ea72f6da58375218e51ee6d9f8ca3a953bf85" exitCode=60 Oct 30 13:00:06.138498 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:06.137795 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-812990-29889420-ddwkl" event={"ID":"1baa9cd5-0305-4f77-b7cf-f4751053ace8","Type":"ContainerDied","Data":"45287a04f0139cd48af47d49ad0ea72f6da58375218e51ee6d9f8ca3a953bf85"} Oct 30 13:00:06.138498 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:06.138044 4908 scope.go:122] "RemoveContainer" containerID="45287a04f0139cd48af47d49ad0ea72f6da58375218e51ee6d9f8ca3a953bf85" Oct 30 13:00:06.138498 ip-10-0-141-251 kubenswrapper[4908]: E1030 13:00:06.138418 4908 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-812990-29889420-ddwkl_odh-ai-gateway-infra(1baa9cd5-0305-4f77-b7cf-f4751053ace8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-812990-29889420-ddwkl" podUID="1baa9cd5-0305-4f77-b7cf-f4751053ace8" Oct 30 13:00:06.141575 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:06.141553 4908 generic.go:356] "Generic (PLEG): container finished" podID="1bb2e64a-3fd0-41bb-aafa-8f234fb2ffa2" containerID="57b813b5ea4b03fd240c4cce6470b80c2af653f1942979d749d2dc031006c211" exitCode=60 Oct 30 13:00:06.141944 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:06.141729 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a786af-29889420-mrgm2" event={"ID":"1bb2e64a-3fd0-41bb-aafa-8f234fb2ffa2","Type":"ContainerDied","Data":"57b813b5ea4b03fd240c4cce6470b80c2af653f1942979d749d2dc031006c211"} Oct 30 13:00:06.142452 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:06.142227 4908 scope.go:122] "RemoveContainer" containerID="57b813b5ea4b03fd240c4cce6470b80c2af653f1942979d749d2dc031006c211" Oct 30 13:00:06.142913 ip-10-0-141-251 kubenswrapper[4908]: E1030 13:00:06.142881 4908 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-a786af-29889420-mrgm2_odh-ai-gateway-infra(1bb2e64a-3fd0-41bb-aafa-8f234fb2ffa2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a786af-29889420-mrgm2" podUID="1bb2e64a-3fd0-41bb-aafa-8f234fb2ffa2" Oct 30 13:00:06.145440 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:06.145416 4908 generic.go:356] "Generic (PLEG): container finished" podID="b46f76c5-4139-4fbd-af18-5f35fcb1cb05" containerID="3fc05ea0c4ac70dad85480064954e620d2dd91e97bc6ea2605e4a3b5f8cb8eb5" exitCode=60 Oct 30 13:00:06.145544 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:06.145499 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0bda67-29889420-4z7jt" event={"ID":"b46f76c5-4139-4fbd-af18-5f35fcb1cb05","Type":"ContainerDied","Data":"3fc05ea0c4ac70dad85480064954e620d2dd91e97bc6ea2605e4a3b5f8cb8eb5"} Oct 30 13:00:06.145781 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:06.145765 4908 scope.go:122] "RemoveContainer" containerID="3fc05ea0c4ac70dad85480064954e620d2dd91e97bc6ea2605e4a3b5f8cb8eb5" Oct 30 13:00:06.146059 ip-10-0-141-251 kubenswrapper[4908]: E1030 13:00:06.146037 4908 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-0bda67-29889420-4z7jt_odh-ai-gateway-infra(b46f76c5-4139-4fbd-af18-5f35fcb1cb05)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0bda67-29889420-4z7jt" podUID="b46f76c5-4139-4fbd-af18-5f35fcb1cb05" Oct 30 13:00:06.148098 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:06.147898 4908 generic.go:356] "Generic (PLEG): container finished" podID="2674b41c-6ab9-4222-aa87-4fc2f5d9f468" containerID="a1e193ed4b30c47e5eb85bda06f874bc2f83a6ef1e393fbedba7f42f690ca371" exitCode=60 Oct 30 13:00:06.148098 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:06.147984 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fddaa3-29889420-vzns9" event={"ID":"2674b41c-6ab9-4222-aa87-4fc2f5d9f468","Type":"ContainerDied","Data":"a1e193ed4b30c47e5eb85bda06f874bc2f83a6ef1e393fbedba7f42f690ca371"} Oct 30 13:00:06.148098 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:06.148066 4908 scope.go:122] "RemoveContainer" containerID="d6bcaeceda4ae67b3907d15663f840db8de8e13454149782953ccfc2fe1c0a46" Oct 30 13:00:06.148343 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:06.148219 4908 scope.go:122] "RemoveContainer" containerID="a1e193ed4b30c47e5eb85bda06f874bc2f83a6ef1e393fbedba7f42f690ca371" Oct 30 13:00:06.148460 ip-10-0-141-251 kubenswrapper[4908]: E1030 13:00:06.148435 4908 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-fddaa3-29889420-vzns9_odh-ai-gateway-infra(2674b41c-6ab9-4222-aa87-4fc2f5d9f468)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fddaa3-29889420-vzns9" podUID="2674b41c-6ab9-4222-aa87-4fc2f5d9f468" Oct 30 13:00:06.151027 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:06.150998 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2675c0-29889420-hqgp5" event={"ID":"7f4c2c03-089f-4701-87d3-4e0496b08750","Type":"ContainerStarted","Data":"4799cf96c861e895a43d5f99bf0921f0786601ec68f4cda7620b009d9f987c1d"} Oct 30 13:00:06.152612 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:06.152591 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-9d45fb-29889420-5wlx7" event={"ID":"a130ae9f-0e32-4b92-b6d3-597640f7e802","Type":"ContainerStarted","Data":"efa00ec66ca15fd57805025c4f950ddaf483969da95c71ff50ce93f6e478f31d"} Oct 30 13:00:06.185268 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:06.185208 4908 scope.go:122] "RemoveContainer" containerID="2f2a41ee12eb4b73c2721aa6ee7180fceba973d35ae43775c561e83ea5562f87" Oct 30 13:00:06.211079 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:06.211048 4908 scope.go:122] "RemoveContainer" containerID="2ba70496bb0e3d8a76a2ca8de05fc18e522893d93c65be886f601c5525ee36f5" Oct 30 13:00:06.227079 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:06.227048 4908 scope.go:122] "RemoveContainer" containerID="d533cc5219e4cfe0083f969dec08c23e0db6b5b68e638933a2b37a6f42e962bc" Oct 30 13:00:06.243131 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:06.243041 4908 scope.go:122] "RemoveContainer" containerID="638fc8b4170a52d61066b2036007068185945e90565ee614ca88dda2b57ac377" Oct 30 13:00:06.528075 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:06.528006 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-327e31-29889420-2vbrc" podStartSLOduration=6.527986413 podStartE2EDuration="6.527986413s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:00:06.515104621 +0000 UTC m=+3963.029533368" watchObservedRunningTime="2026-10-30 13:00:06.527986413 +0000 UTC m=+3963.042415157" Oct 30 13:00:06.845590 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:06.845527 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-9c3f12/test-model-9c3f12-kserve-cc6c75df5-ldnbh" podStartSLOduration=2.845508389 podStartE2EDuration="2.845508389s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:00:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:00:06.842945426 +0000 UTC m=+3963.357374171" watchObservedRunningTime="2026-10-30 13:00:06.845508389 +0000 UTC m=+3963.359937136" Oct 30 13:00:06.959511 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:06.959467 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-9c3f12/test-model-9c3f12-kserve-cc6c75df5-ldnbh" Oct 30 13:00:07.167185 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:07.167133 4908 generic.go:356] "Generic (PLEG): container finished" podID="f5af9446-e0d9-4edc-b01f-d049c4b2d3e7" containerID="1159c140b26e0a148bbfa6c979acfbad6eac5a31cb67f6f9a4b71037310633a5" exitCode=60 Oct 30 13:00:07.167696 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:07.167223 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-8d2170-29889420-67bzd" event={"ID":"f5af9446-e0d9-4edc-b01f-d049c4b2d3e7","Type":"ContainerDied","Data":"1159c140b26e0a148bbfa6c979acfbad6eac5a31cb67f6f9a4b71037310633a5"} Oct 30 13:00:07.167696 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:07.167331 4908 scope.go:122] "RemoveContainer" containerID="22449e40d4cff41f0e2c7e5051fcdd239608573532a09c47da81361e17c52f2d" Oct 30 13:00:07.167696 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:07.167604 4908 scope.go:122] "RemoveContainer" containerID="1159c140b26e0a148bbfa6c979acfbad6eac5a31cb67f6f9a4b71037310633a5" Oct 30 13:00:07.168123 ip-10-0-141-251 kubenswrapper[4908]: E1030 13:00:07.168092 4908 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-8d2170-29889420-67bzd_odh-ai-gateway-infra(f5af9446-e0d9-4edc-b01f-d049c4b2d3e7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-8d2170-29889420-67bzd" podUID="f5af9446-e0d9-4edc-b01f-d049c4b2d3e7" Oct 30 13:00:07.171566 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:07.171543 4908 scope.go:122] "RemoveContainer" containerID="690ec2b414b11ab39bef37bd18421e14c295a32577da7d213917ab9cb7ae45b5" Oct 30 13:00:07.172019 ip-10-0-141-251 kubenswrapper[4908]: E1030 13:00:07.171992 4908 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-12fd33-29889420-cfqsp_odh-ai-gateway-infra(f2695f98-7dab-4cf3-aaa3-d1717ecc2d30)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-12fd33-29889420-cfqsp" podUID="f2695f98-7dab-4cf3-aaa3-d1717ecc2d30" Oct 30 13:00:07.173709 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:07.173619 4908 generic.go:356] "Generic (PLEG): container finished" podID="53e599d0-de7d-4c2f-86f9-97cc39405bc0" containerID="53c7a0cb1a17dbfeff3c3e09682d3f006a52ad471b22c7089d0965bfa316bacb" exitCode=60 Oct 30 13:00:07.173709 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:07.173655 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-327e31-29889420-2vbrc" event={"ID":"53e599d0-de7d-4c2f-86f9-97cc39405bc0","Type":"ContainerDied","Data":"53c7a0cb1a17dbfeff3c3e09682d3f006a52ad471b22c7089d0965bfa316bacb"} Oct 30 13:00:07.174072 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:07.174051 4908 scope.go:122] "RemoveContainer" containerID="53c7a0cb1a17dbfeff3c3e09682d3f006a52ad471b22c7089d0965bfa316bacb" Oct 30 13:00:07.174607 ip-10-0-141-251 kubenswrapper[4908]: E1030 13:00:07.174565 4908 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-327e31-29889420-2vbrc_odh-ai-gateway-infra(53e599d0-de7d-4c2f-86f9-97cc39405bc0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-327e31-29889420-2vbrc" podUID="53e599d0-de7d-4c2f-86f9-97cc39405bc0" Oct 30 13:00:07.176128 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:07.176108 4908 scope.go:122] "RemoveContainer" containerID="96b0ea8ecfe4f3439a866d2f770a7b57d4ad8b8f0b714aebead350c5999327b0" Oct 30 13:00:07.176444 ip-10-0-141-251 kubenswrapper[4908]: E1030 13:00:07.176418 4908 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-dd0628-29889420-jh88n_odh-ai-gateway-infra(347ca826-af8d-496d-bd0b-dca4071b3cd8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-dd0628-29889420-jh88n" podUID="347ca826-af8d-496d-bd0b-dca4071b3cd8" Oct 30 13:00:07.179526 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:07.178625 4908 scope.go:122] "RemoveContainer" containerID="45287a04f0139cd48af47d49ad0ea72f6da58375218e51ee6d9f8ca3a953bf85" Oct 30 13:00:07.179526 ip-10-0-141-251 kubenswrapper[4908]: E1030 13:00:07.178867 4908 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-812990-29889420-ddwkl_odh-ai-gateway-infra(1baa9cd5-0305-4f77-b7cf-f4751053ace8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-812990-29889420-ddwkl" podUID="1baa9cd5-0305-4f77-b7cf-f4751053ace8" Oct 30 13:00:07.183148 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:07.183124 4908 scope.go:122] "RemoveContainer" containerID="57b813b5ea4b03fd240c4cce6470b80c2af653f1942979d749d2dc031006c211" Oct 30 13:00:07.183461 ip-10-0-141-251 kubenswrapper[4908]: E1030 13:00:07.183433 4908 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-a786af-29889420-mrgm2_odh-ai-gateway-infra(1bb2e64a-3fd0-41bb-aafa-8f234fb2ffa2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a786af-29889420-mrgm2" podUID="1bb2e64a-3fd0-41bb-aafa-8f234fb2ffa2" Oct 30 13:00:07.186105 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:07.186080 4908 generic.go:356] "Generic (PLEG): container finished" podID="0d506c31-90a7-493f-8b73-649d1ffcd0eb" containerID="684e2f653cdd517f57cf566510baf88f8ccf5c83fbe6ee79be6565cde142ffbf" exitCode=60 Oct 30 13:00:07.186226 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:07.186177 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-73bafc-29889420-4lh9z" event={"ID":"0d506c31-90a7-493f-8b73-649d1ffcd0eb","Type":"ContainerDied","Data":"684e2f653cdd517f57cf566510baf88f8ccf5c83fbe6ee79be6565cde142ffbf"} Oct 30 13:00:07.186873 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:07.186809 4908 scope.go:122] "RemoveContainer" containerID="684e2f653cdd517f57cf566510baf88f8ccf5c83fbe6ee79be6565cde142ffbf" Oct 30 13:00:07.187161 ip-10-0-141-251 kubenswrapper[4908]: E1030 13:00:07.187139 4908 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-73bafc-29889420-4lh9z_odh-ai-gateway-infra(0d506c31-90a7-493f-8b73-649d1ffcd0eb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-73bafc-29889420-4lh9z" podUID="0d506c31-90a7-493f-8b73-649d1ffcd0eb" Oct 30 13:00:07.189328 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:07.189303 4908 generic.go:356] "Generic (PLEG): container finished" podID="94f72c00-8f0f-4bcf-9218-cd64b7773d1f" containerID="624e7a813c919a43fbebe646cc29d37292bc07604dd331dd58451a3b9c7b305e" exitCode=60 Oct 30 13:00:07.190002 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:07.189509 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f7464f-29889420-z9mg6" event={"ID":"94f72c00-8f0f-4bcf-9218-cd64b7773d1f","Type":"ContainerDied","Data":"624e7a813c919a43fbebe646cc29d37292bc07604dd331dd58451a3b9c7b305e"} Oct 30 13:00:07.190185 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:07.189795 4908 scope.go:122] "RemoveContainer" containerID="624e7a813c919a43fbebe646cc29d37292bc07604dd331dd58451a3b9c7b305e" Oct 30 13:00:07.193350 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:07.193329 4908 scope.go:122] "RemoveContainer" containerID="3fc05ea0c4ac70dad85480064954e620d2dd91e97bc6ea2605e4a3b5f8cb8eb5" Oct 30 13:00:07.193620 ip-10-0-141-251 kubenswrapper[4908]: E1030 13:00:07.193590 4908 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-0bda67-29889420-4z7jt_odh-ai-gateway-infra(b46f76c5-4139-4fbd-af18-5f35fcb1cb05)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0bda67-29889420-4z7jt" podUID="b46f76c5-4139-4fbd-af18-5f35fcb1cb05" Oct 30 13:00:07.195655 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:07.195633 4908 scope.go:122] "RemoveContainer" containerID="a1e193ed4b30c47e5eb85bda06f874bc2f83a6ef1e393fbedba7f42f690ca371" Oct 30 13:00:07.195951 ip-10-0-141-251 kubenswrapper[4908]: E1030 13:00:07.195917 4908 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-fddaa3-29889420-vzns9_odh-ai-gateway-infra(2674b41c-6ab9-4222-aa87-4fc2f5d9f468)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fddaa3-29889420-vzns9" podUID="2674b41c-6ab9-4222-aa87-4fc2f5d9f468" Oct 30 13:00:07.198538 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:07.198516 4908 generic.go:356] "Generic (PLEG): container finished" podID="7f4c2c03-089f-4701-87d3-4e0496b08750" containerID="4799cf96c861e895a43d5f99bf0921f0786601ec68f4cda7620b009d9f987c1d" exitCode=60 Oct 30 13:00:07.198682 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:07.198595 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2675c0-29889420-hqgp5" event={"ID":"7f4c2c03-089f-4701-87d3-4e0496b08750","Type":"ContainerDied","Data":"4799cf96c861e895a43d5f99bf0921f0786601ec68f4cda7620b009d9f987c1d"} Oct 30 13:00:07.198942 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:07.198923 4908 scope.go:122] "RemoveContainer" containerID="4799cf96c861e895a43d5f99bf0921f0786601ec68f4cda7620b009d9f987c1d" Oct 30 13:00:07.199222 ip-10-0-141-251 kubenswrapper[4908]: E1030 13:00:07.199177 4908 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-2675c0-29889420-hqgp5_odh-ai-gateway-infra(7f4c2c03-089f-4701-87d3-4e0496b08750)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2675c0-29889420-hqgp5" podUID="7f4c2c03-089f-4701-87d3-4e0496b08750" Oct 30 13:00:07.201209 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:07.201189 4908 generic.go:356] "Generic (PLEG): container finished" podID="a130ae9f-0e32-4b92-b6d3-597640f7e802" containerID="efa00ec66ca15fd57805025c4f950ddaf483969da95c71ff50ce93f6e478f31d" exitCode=60 Oct 30 13:00:07.201408 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:07.201265 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-9d45fb-29889420-5wlx7" event={"ID":"a130ae9f-0e32-4b92-b6d3-597640f7e802","Type":"ContainerDied","Data":"efa00ec66ca15fd57805025c4f950ddaf483969da95c71ff50ce93f6e478f31d"} Oct 30 13:00:07.201596 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:07.201578 4908 scope.go:122] "RemoveContainer" containerID="efa00ec66ca15fd57805025c4f950ddaf483969da95c71ff50ce93f6e478f31d" Oct 30 13:00:07.202389 ip-10-0-141-251 kubenswrapper[4908]: E1030 13:00:07.202364 4908 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-9d45fb-29889420-5wlx7_odh-ai-gateway-infra(a130ae9f-0e32-4b92-b6d3-597640f7e802)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-9d45fb-29889420-5wlx7" podUID="a130ae9f-0e32-4b92-b6d3-597640f7e802" Oct 30 13:00:07.203593 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:07.203568 4908 generic.go:356] "Generic (PLEG): container finished" podID="71805e6f-86c5-4659-aa70-849ba38f288c" containerID="be667f188369ff762f1c63b473896b06f415609784a3d0bc7c9dffb5f9fff403" exitCode=60 Oct 30 13:00:07.203722 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:07.203646 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889420-n4zdq" event={"ID":"71805e6f-86c5-4659-aa70-849ba38f288c","Type":"ContainerDied","Data":"be667f188369ff762f1c63b473896b06f415609784a3d0bc7c9dffb5f9fff403"} Oct 30 13:00:07.203995 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:07.203976 4908 scope.go:122] "RemoveContainer" containerID="be667f188369ff762f1c63b473896b06f415609784a3d0bc7c9dffb5f9fff403" Oct 30 13:00:07.204242 ip-10-0-141-251 kubenswrapper[4908]: E1030 13:00:07.204218 4908 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889420-n4zdq_odh-ai-gateway-infra(71805e6f-86c5-4659-aa70-849ba38f288c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889420-n4zdq" podUID="71805e6f-86c5-4659-aa70-849ba38f288c" Oct 30 13:00:07.206128 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:07.206105 4908 generic.go:356] "Generic (PLEG): container finished" podID="76cf17b7-c8b9-48d8-934c-f38aa9d1f008" containerID="e03b25b8e2910b878c1499548f3ab6f0b62e4548f50e313da7ee0484c111f575" exitCode=60 Oct 30 13:00:07.206329 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:07.206310 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-10f364-29889420-fzd6k" event={"ID":"76cf17b7-c8b9-48d8-934c-f38aa9d1f008","Type":"ContainerDied","Data":"e03b25b8e2910b878c1499548f3ab6f0b62e4548f50e313da7ee0484c111f575"} Oct 30 13:00:07.207072 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:07.206495 4908 scope.go:122] "RemoveContainer" containerID="e03b25b8e2910b878c1499548f3ab6f0b62e4548f50e313da7ee0484c111f575" Oct 30 13:00:07.207072 ip-10-0-141-251 kubenswrapper[4908]: E1030 13:00:07.206817 4908 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-10f364-29889420-fzd6k_odh-ai-gateway-infra(76cf17b7-c8b9-48d8-934c-f38aa9d1f008)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-10f364-29889420-fzd6k" podUID="76cf17b7-c8b9-48d8-934c-f38aa9d1f008" Oct 30 13:00:07.209282 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:07.209258 4908 generic.go:356] "Generic (PLEG): container finished" podID="ced43030-57a2-418d-bf86-f9c5a3b8cd95" containerID="b150b641deb5a9f13128f282c2081a39ebb087d1b77f9b5b70ca47012c325a3b" exitCode=60 Oct 30 13:00:07.209621 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:07.209601 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-42220c-29889420-q4fqq" event={"ID":"ced43030-57a2-418d-bf86-f9c5a3b8cd95","Type":"ContainerDied","Data":"b150b641deb5a9f13128f282c2081a39ebb087d1b77f9b5b70ca47012c325a3b"} Oct 30 13:00:07.210158 ip-10-0-141-251 kubenswrapper[4908]: E1030 13:00:07.210131 4908 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-f7464f-29889420-z9mg6_odh-ai-gateway-infra(94f72c00-8f0f-4bcf-9218-cd64b7773d1f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f7464f-29889420-z9mg6" podUID="94f72c00-8f0f-4bcf-9218-cd64b7773d1f" Oct 30 13:00:07.210316 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:07.210297 4908 scope.go:122] "RemoveContainer" containerID="b150b641deb5a9f13128f282c2081a39ebb087d1b77f9b5b70ca47012c325a3b" Oct 30 13:00:07.210638 ip-10-0-141-251 kubenswrapper[4908]: E1030 13:00:07.210615 4908 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-42220c-29889420-q4fqq_odh-ai-gateway-infra(ced43030-57a2-418d-bf86-f9c5a3b8cd95)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-42220c-29889420-q4fqq" podUID="ced43030-57a2-418d-bf86-f9c5a3b8cd95" Oct 30 13:00:07.212193 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:07.212169 4908 generic.go:356] "Generic (PLEG): container finished" podID="a7bd05c5-f726-4ad3-88bf-b3a351a409bb" containerID="2fa4249a504b685d16cd39267ecb54320052aa377205c69d5fb179ebd7807720" exitCode=60 Oct 30 13:00:07.212312 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:07.212250 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6b67f8-29889420-pt2lw" event={"ID":"a7bd05c5-f726-4ad3-88bf-b3a351a409bb","Type":"ContainerDied","Data":"2fa4249a504b685d16cd39267ecb54320052aa377205c69d5fb179ebd7807720"} Oct 30 13:00:07.212507 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:07.212491 4908 scope.go:122] "RemoveContainer" containerID="2fa4249a504b685d16cd39267ecb54320052aa377205c69d5fb179ebd7807720" Oct 30 13:00:07.212772 ip-10-0-141-251 kubenswrapper[4908]: E1030 13:00:07.212752 4908 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-6b67f8-29889420-pt2lw_odh-ai-gateway-infra(a7bd05c5-f726-4ad3-88bf-b3a351a409bb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6b67f8-29889420-pt2lw" podUID="a7bd05c5-f726-4ad3-88bf-b3a351a409bb" Oct 30 13:00:07.214088 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:07.214067 4908 scope.go:122] "RemoveContainer" containerID="455c3c383fe1a940a1d2b692834c5f2398232ceafa93c0f8d8ad91295ff035d1" Oct 30 13:00:07.214299 ip-10-0-141-251 kubenswrapper[4908]: E1030 13:00:07.214280 4908 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-2545c7-29889420-xrknl_odh-ai-gateway-infra(bf7295f0-efb3-487a-926c-c4f19fddde26)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2545c7-29889420-xrknl" podUID="bf7295f0-efb3-487a-926c-c4f19fddde26" Oct 30 13:00:07.216036 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:07.216014 4908 scope.go:122] "RemoveContainer" containerID="d79974ed63f49d74bbd0e00ed623030be41359dccc8277ca8703302f35142ca5" Oct 30 13:00:07.216285 ip-10-0-141-251 kubenswrapper[4908]: E1030 13:00:07.216261 4908 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-dbe123-29889420-bcnzv_odh-ai-gateway-infra(8165abf7-fbfb-4de2-a94c-5023a834835d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-dbe123-29889420-bcnzv" podUID="8165abf7-fbfb-4de2-a94c-5023a834835d" Oct 30 13:00:07.218345 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:07.218323 4908 generic.go:356] "Generic (PLEG): container finished" podID="cb87d307-def9-4d16-8e7a-54b531f0d2c5" containerID="9e90edddf33c1b728e9595a6acdcdee922d7af72d81fe611f108bf0947790bc7" exitCode=60 Oct 30 13:00:07.218466 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:07.218348 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ecb3aa-29889420-72h7x" event={"ID":"cb87d307-def9-4d16-8e7a-54b531f0d2c5","Type":"ContainerDied","Data":"9e90edddf33c1b728e9595a6acdcdee922d7af72d81fe611f108bf0947790bc7"} Oct 30 13:00:07.218613 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:07.218586 4908 scope.go:122] "RemoveContainer" containerID="9e90edddf33c1b728e9595a6acdcdee922d7af72d81fe611f108bf0947790bc7" Oct 30 13:00:07.218895 ip-10-0-141-251 kubenswrapper[4908]: E1030 13:00:07.218871 4908 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-ecb3aa-29889420-72h7x_odh-ai-gateway-infra(cb87d307-def9-4d16-8e7a-54b531f0d2c5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ecb3aa-29889420-72h7x" podUID="cb87d307-def9-4d16-8e7a-54b531f0d2c5" Oct 30 13:00:07.220637 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:07.220616 4908 generic.go:356] "Generic (PLEG): container finished" podID="29e5414c-9b69-4713-8b27-80af3af5b4c7" containerID="9f340a37a3e8d33f47f2c6b0d7bc2357f171968f4c8f8f230eed09a502acbd7e" exitCode=60 Oct 30 13:00:07.220736 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:07.220692 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e9f184-29889420-sv9rp" event={"ID":"29e5414c-9b69-4713-8b27-80af3af5b4c7","Type":"ContainerDied","Data":"9f340a37a3e8d33f47f2c6b0d7bc2357f171968f4c8f8f230eed09a502acbd7e"} Oct 30 13:00:07.220921 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:07.220904 4908 scope.go:122] "RemoveContainer" containerID="9f340a37a3e8d33f47f2c6b0d7bc2357f171968f4c8f8f230eed09a502acbd7e" Oct 30 13:00:07.221136 ip-10-0-141-251 kubenswrapper[4908]: E1030 13:00:07.221115 4908 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-e9f184-29889420-sv9rp_odh-ai-gateway-infra(29e5414c-9b69-4713-8b27-80af3af5b4c7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e9f184-29889420-sv9rp" podUID="29e5414c-9b69-4713-8b27-80af3af5b4c7" Oct 30 13:00:07.222596 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:07.222576 4908 generic.go:356] "Generic (PLEG): container finished" podID="1daed017-3390-4a05-9457-8d6c97919043" containerID="81f9b9853015d1c8504e652f5e16222bead314f4f516d286f8af942da7d2e044" exitCode=60 Oct 30 13:00:07.222708 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:07.222695 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6ad1da-29889420-qhlvt" event={"ID":"1daed017-3390-4a05-9457-8d6c97919043","Type":"ContainerDied","Data":"81f9b9853015d1c8504e652f5e16222bead314f4f516d286f8af942da7d2e044"} Oct 30 13:00:07.223033 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:07.223011 4908 scope.go:122] "RemoveContainer" containerID="81f9b9853015d1c8504e652f5e16222bead314f4f516d286f8af942da7d2e044" Oct 30 13:00:07.223269 ip-10-0-141-251 kubenswrapper[4908]: E1030 13:00:07.223245 4908 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-6ad1da-29889420-qhlvt_odh-ai-gateway-infra(1daed017-3390-4a05-9457-8d6c97919043)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6ad1da-29889420-qhlvt" podUID="1daed017-3390-4a05-9457-8d6c97919043" Oct 30 13:00:07.224811 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:07.224792 4908 generic.go:356] "Generic (PLEG): container finished" podID="94bed6ea-d66f-4f63-8859-a5a8d1969a01" containerID="84574467f1500cea18a9ddcfc697eace7267e137c6bd81b2b06762279c05543f" exitCode=60 Oct 30 13:00:07.224939 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:07.224855 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1333c8-29889420-9zjhg" event={"ID":"94bed6ea-d66f-4f63-8859-a5a8d1969a01","Type":"ContainerDied","Data":"84574467f1500cea18a9ddcfc697eace7267e137c6bd81b2b06762279c05543f"} Oct 30 13:00:07.225118 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:07.225104 4908 scope.go:122] "RemoveContainer" containerID="84574467f1500cea18a9ddcfc697eace7267e137c6bd81b2b06762279c05543f" Oct 30 13:00:07.225379 ip-10-0-141-251 kubenswrapper[4908]: E1030 13:00:07.225356 4908 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-1333c8-29889420-9zjhg_odh-ai-gateway-infra(94bed6ea-d66f-4f63-8859-a5a8d1969a01)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1333c8-29889420-9zjhg" podUID="94bed6ea-d66f-4f63-8859-a5a8d1969a01" Oct 30 13:00:07.319648 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:07.319614 4908 scope.go:122] "RemoveContainer" containerID="035e9724d59bf4b08b5bc36a702a866689e87b8ca5da243df5b1a9e6bac5081a" Oct 30 13:00:07.336058 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:07.336035 4908 scope.go:122] "RemoveContainer" containerID="8cd9370751836cce67f9ce51599ada523d5f075add6c38b57d53e941fb5d3daa" Oct 30 13:00:07.351210 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:07.351179 4908 scope.go:122] "RemoveContainer" containerID="72994127e7d0f026d2c1c233b772e5cc1922058d4cb5391400706b7695b19f21" Oct 30 13:00:07.371560 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:07.371529 4908 scope.go:122] "RemoveContainer" containerID="f7dc138f00c048e8584eecb0ff3248ef7ed1ea2795242980d6c399c866fe4811" Oct 30 13:00:07.391747 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:07.391722 4908 scope.go:122] "RemoveContainer" containerID="a429cdd857d6c0bfe41e4b7f9d2f1f82482af156f0914252e692e1c2168fbbc3" Oct 30 13:00:07.409499 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:07.409476 4908 scope.go:122] "RemoveContainer" containerID="2453eca363563f4152cf90c08ddd077539d9b3b8c9cb5809161cac63ad11a011" Oct 30 13:00:07.424813 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:07.424785 4908 scope.go:122] "RemoveContainer" containerID="abca2067b9d040e3fd8d0b2c403baf3f57b119cb66994d0197d551b967a9ca67" Oct 30 13:00:07.439675 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:07.439650 4908 scope.go:122] "RemoveContainer" containerID="b1739406b94f2e5bbe4dd2e3fabdb518c3a8880d2184ed1cd94cfff0dd7e86e1" Oct 30 13:00:07.453980 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:07.453954 4908 scope.go:122] "RemoveContainer" containerID="6802d3602f1edaa0752d289899d4de07b7ba19337bfb1afb4615a724c46b9d70" Oct 30 13:00:07.470358 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:07.470334 4908 scope.go:122] "RemoveContainer" containerID="fbd79d47f8654aad5bf35f568a012acfe9923462523af28cdbb1a730f35d3592" Oct 30 13:00:07.487498 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:07.487454 4908 scope.go:122] "RemoveContainer" containerID="d00dbb68e8a49c4b6f3530f36896b81b12d3888d0c2eefc2f202d0cb8eeb9e35" Oct 30 13:00:07.506242 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:07.506126 4908 scope.go:122] "RemoveContainer" containerID="6e89ee35e7c462c448676e2fa4dc2e1e11040ab5807efbd094389f0273b5de31" Oct 30 13:00:07.524673 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:07.524649 4908 scope.go:122] "RemoveContainer" containerID="c402cb9fcea37536e64e7fabb391504349c3ddcfb6ac71d9c4f4e27730681fc0" Oct 30 13:00:08.235810 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:08.235765 4908 scope.go:122] "RemoveContainer" containerID="9f340a37a3e8d33f47f2c6b0d7bc2357f171968f4c8f8f230eed09a502acbd7e" Oct 30 13:00:08.236251 ip-10-0-141-251 kubenswrapper[4908]: E1030 13:00:08.236192 4908 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-e9f184-29889420-sv9rp_odh-ai-gateway-infra(29e5414c-9b69-4713-8b27-80af3af5b4c7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e9f184-29889420-sv9rp" podUID="29e5414c-9b69-4713-8b27-80af3af5b4c7" Oct 30 13:00:08.237624 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:08.237601 4908 scope.go:122] "RemoveContainer" containerID="81f9b9853015d1c8504e652f5e16222bead314f4f516d286f8af942da7d2e044" Oct 30 13:00:08.238028 ip-10-0-141-251 kubenswrapper[4908]: E1030 13:00:08.238000 4908 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-6ad1da-29889420-qhlvt_odh-ai-gateway-infra(1daed017-3390-4a05-9457-8d6c97919043)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6ad1da-29889420-qhlvt" podUID="1daed017-3390-4a05-9457-8d6c97919043" Oct 30 13:00:08.239413 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:08.239392 4908 scope.go:122] "RemoveContainer" containerID="84574467f1500cea18a9ddcfc697eace7267e137c6bd81b2b06762279c05543f" Oct 30 13:00:08.239734 ip-10-0-141-251 kubenswrapper[4908]: E1030 13:00:08.239707 4908 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-1333c8-29889420-9zjhg_odh-ai-gateway-infra(94bed6ea-d66f-4f63-8859-a5a8d1969a01)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1333c8-29889420-9zjhg" podUID="94bed6ea-d66f-4f63-8859-a5a8d1969a01" Oct 30 13:00:08.241123 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:08.241096 4908 scope.go:122] "RemoveContainer" containerID="1159c140b26e0a148bbfa6c979acfbad6eac5a31cb67f6f9a4b71037310633a5" Oct 30 13:00:08.241372 ip-10-0-141-251 kubenswrapper[4908]: E1030 13:00:08.241350 4908 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-8d2170-29889420-67bzd_odh-ai-gateway-infra(f5af9446-e0d9-4edc-b01f-d049c4b2d3e7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-8d2170-29889420-67bzd" podUID="f5af9446-e0d9-4edc-b01f-d049c4b2d3e7" Oct 30 13:00:08.242730 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:08.242712 4908 scope.go:122] "RemoveContainer" containerID="53c7a0cb1a17dbfeff3c3e09682d3f006a52ad471b22c7089d0965bfa316bacb" Oct 30 13:00:08.243020 ip-10-0-141-251 kubenswrapper[4908]: E1030 13:00:08.242994 4908 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-327e31-29889420-2vbrc_odh-ai-gateway-infra(53e599d0-de7d-4c2f-86f9-97cc39405bc0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-327e31-29889420-2vbrc" podUID="53e599d0-de7d-4c2f-86f9-97cc39405bc0" Oct 30 13:00:08.244324 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:08.244301 4908 scope.go:122] "RemoveContainer" containerID="684e2f653cdd517f57cf566510baf88f8ccf5c83fbe6ee79be6565cde142ffbf" Oct 30 13:00:08.244577 ip-10-0-141-251 kubenswrapper[4908]: E1030 13:00:08.244558 4908 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-73bafc-29889420-4lh9z_odh-ai-gateway-infra(0d506c31-90a7-493f-8b73-649d1ffcd0eb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-73bafc-29889420-4lh9z" podUID="0d506c31-90a7-493f-8b73-649d1ffcd0eb" Oct 30 13:00:08.245861 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:08.245823 4908 scope.go:122] "RemoveContainer" containerID="624e7a813c919a43fbebe646cc29d37292bc07604dd331dd58451a3b9c7b305e" Oct 30 13:00:08.246115 ip-10-0-141-251 kubenswrapper[4908]: E1030 13:00:08.246090 4908 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-f7464f-29889420-z9mg6_odh-ai-gateway-infra(94f72c00-8f0f-4bcf-9218-cd64b7773d1f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f7464f-29889420-z9mg6" podUID="94f72c00-8f0f-4bcf-9218-cd64b7773d1f" Oct 30 13:00:08.247577 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:08.247558 4908 scope.go:122] "RemoveContainer" containerID="4799cf96c861e895a43d5f99bf0921f0786601ec68f4cda7620b009d9f987c1d" Oct 30 13:00:08.247928 ip-10-0-141-251 kubenswrapper[4908]: E1030 13:00:08.247812 4908 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-2675c0-29889420-hqgp5_odh-ai-gateway-infra(7f4c2c03-089f-4701-87d3-4e0496b08750)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2675c0-29889420-hqgp5" podUID="7f4c2c03-089f-4701-87d3-4e0496b08750" Oct 30 13:00:08.249412 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:08.249386 4908 scope.go:122] "RemoveContainer" containerID="efa00ec66ca15fd57805025c4f950ddaf483969da95c71ff50ce93f6e478f31d" Oct 30 13:00:08.249623 ip-10-0-141-251 kubenswrapper[4908]: E1030 13:00:08.249604 4908 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-9d45fb-29889420-5wlx7_odh-ai-gateway-infra(a130ae9f-0e32-4b92-b6d3-597640f7e802)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-9d45fb-29889420-5wlx7" podUID="a130ae9f-0e32-4b92-b6d3-597640f7e802" Oct 30 13:00:08.251175 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:08.251154 4908 scope.go:122] "RemoveContainer" containerID="be667f188369ff762f1c63b473896b06f415609784a3d0bc7c9dffb5f9fff403" Oct 30 13:00:08.251396 ip-10-0-141-251 kubenswrapper[4908]: E1030 13:00:08.251373 4908 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889420-n4zdq_odh-ai-gateway-infra(71805e6f-86c5-4659-aa70-849ba38f288c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889420-n4zdq" podUID="71805e6f-86c5-4659-aa70-849ba38f288c" Oct 30 13:00:08.252886 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:08.252865 4908 scope.go:122] "RemoveContainer" containerID="e03b25b8e2910b878c1499548f3ab6f0b62e4548f50e313da7ee0484c111f575" Oct 30 13:00:08.253141 ip-10-0-141-251 kubenswrapper[4908]: E1030 13:00:08.253120 4908 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-10f364-29889420-fzd6k_odh-ai-gateway-infra(76cf17b7-c8b9-48d8-934c-f38aa9d1f008)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-10f364-29889420-fzd6k" podUID="76cf17b7-c8b9-48d8-934c-f38aa9d1f008" Oct 30 13:00:08.254524 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:08.254503 4908 scope.go:122] "RemoveContainer" containerID="b150b641deb5a9f13128f282c2081a39ebb087d1b77f9b5b70ca47012c325a3b" Oct 30 13:00:08.254779 ip-10-0-141-251 kubenswrapper[4908]: E1030 13:00:08.254756 4908 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-42220c-29889420-q4fqq_odh-ai-gateway-infra(ced43030-57a2-418d-bf86-f9c5a3b8cd95)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-42220c-29889420-q4fqq" podUID="ced43030-57a2-418d-bf86-f9c5a3b8cd95" Oct 30 13:00:08.256131 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:08.256109 4908 scope.go:122] "RemoveContainer" containerID="2fa4249a504b685d16cd39267ecb54320052aa377205c69d5fb179ebd7807720" Oct 30 13:00:08.256390 ip-10-0-141-251 kubenswrapper[4908]: E1030 13:00:08.256359 4908 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-6b67f8-29889420-pt2lw_odh-ai-gateway-infra(a7bd05c5-f726-4ad3-88bf-b3a351a409bb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6b67f8-29889420-pt2lw" podUID="a7bd05c5-f726-4ad3-88bf-b3a351a409bb" Oct 30 13:00:08.258135 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:08.258118 4908 scope.go:122] "RemoveContainer" containerID="9e90edddf33c1b728e9595a6acdcdee922d7af72d81fe611f108bf0947790bc7" Oct 30 13:00:08.258388 ip-10-0-141-251 kubenswrapper[4908]: E1030 13:00:08.258368 4908 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-ecb3aa-29889420-72h7x_odh-ai-gateway-infra(cb87d307-def9-4d16-8e7a-54b531f0d2c5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ecb3aa-29889420-72h7x" podUID="cb87d307-def9-4d16-8e7a-54b531f0d2c5" Oct 30 13:00:08.798938 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:08.798905 4908 scope.go:122] "RemoveContainer" containerID="d741cbe455d300b615158f42d49ea382c5c1d260941cc2511902141b5d8166d7" Oct 30 13:00:08.812485 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:08.812460 4908 scope.go:122] "RemoveContainer" containerID="57969536ae6c3f0b27edfd023b2089bea00b2166168c54c70c09447e1b12a857" Oct 30 13:00:08.824324 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:08.824299 4908 scope.go:122] "RemoveContainer" containerID="f4d3843e83c76011c2d7b6c8292837396ce1b8e29643f00fc2d16c80afec8d95" Oct 30 13:00:11.723957 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:11.723917 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9wwfc"] Oct 30 13:00:14.958914 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:14.958878 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-9c3f12/test-model-9c3f12-kserve-cc6c75df5-ldnbh" Oct 30 13:00:14.971974 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:14.971945 4908 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-9c3f12/test-model-9c3f12-kserve-cc6c75df5-ldnbh" Oct 30 13:00:15.447036 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:15.447001 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-9c3f12/test-model-9c3f12-kserve-cc6c75df5-ldnbh" Oct 30 13:00:15.601871 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:15.598738 4908 scope.go:122] "RemoveContainer" containerID="d79974ed63f49d74bbd0e00ed623030be41359dccc8277ca8703302f35142ca5" Oct 30 13:00:16.447971 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:16.447871 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-dbe123-29889420-bcnzv" event={"ID":"8165abf7-fbfb-4de2-a94c-5023a834835d","Type":"ContainerStarted","Data":"33518cd1d49bcb2993c7b2da2bec20338c19ad839b636ad184fab4f582eb671a"} Oct 30 13:00:17.459183 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:17.459144 4908 generic.go:356] "Generic (PLEG): container finished" podID="8165abf7-fbfb-4de2-a94c-5023a834835d" containerID="33518cd1d49bcb2993c7b2da2bec20338c19ad839b636ad184fab4f582eb671a" exitCode=60 Oct 30 13:00:17.459715 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:17.459302 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-dbe123-29889420-bcnzv" event={"ID":"8165abf7-fbfb-4de2-a94c-5023a834835d","Type":"ContainerDied","Data":"33518cd1d49bcb2993c7b2da2bec20338c19ad839b636ad184fab4f582eb671a"} Oct 30 13:00:17.459715 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:17.459399 4908 scope.go:122] "RemoveContainer" containerID="d79974ed63f49d74bbd0e00ed623030be41359dccc8277ca8703302f35142ca5" Oct 30 13:00:17.459976 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:17.459933 4908 scope.go:122] "RemoveContainer" containerID="33518cd1d49bcb2993c7b2da2bec20338c19ad839b636ad184fab4f582eb671a" Oct 30 13:00:17.460398 ip-10-0-141-251 kubenswrapper[4908]: E1030 13:00:17.460372 4908 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-dbe123-29889420-bcnzv_odh-ai-gateway-infra(8165abf7-fbfb-4de2-a94c-5023a834835d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-dbe123-29889420-bcnzv" podUID="8165abf7-fbfb-4de2-a94c-5023a834835d" Oct 30 13:00:17.500207 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:17.500165 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-dbe123-29889420-bcnzv"] Oct 30 13:00:17.634441 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:17.633623 4908 scope.go:122] "RemoveContainer" containerID="b150b641deb5a9f13128f282c2081a39ebb087d1b77f9b5b70ca47012c325a3b" Oct 30 13:00:17.634666 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:17.634594 4908 scope.go:122] "RemoveContainer" containerID="be667f188369ff762f1c63b473896b06f415609784a3d0bc7c9dffb5f9fff403" Oct 30 13:00:17.636324 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:17.635408 4908 scope.go:122] "RemoveContainer" containerID="a1e193ed4b30c47e5eb85bda06f874bc2f83a6ef1e393fbedba7f42f690ca371" Oct 30 13:00:17.653788 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:17.652678 4908 scope.go:122] "RemoveContainer" containerID="45287a04f0139cd48af47d49ad0ea72f6da58375218e51ee6d9f8ca3a953bf85" Oct 30 13:00:18.036045 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:18.036006 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98ccc618-wz48b"] Oct 30 13:00:18.043491 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:18.043438 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98ccc618-wz48b"] Oct 30 13:00:18.472385 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:18.472344 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-812990-29889420-ddwkl" event={"ID":"1baa9cd5-0305-4f77-b7cf-f4751053ace8","Type":"ContainerStarted","Data":"cde8f2e102c47a71e4fc9dbd2dd19a3350d6b3724268567606a5d3b7dc9b98a6"} Oct 30 13:00:18.474589 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:18.474561 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fddaa3-29889420-vzns9" event={"ID":"2674b41c-6ab9-4222-aa87-4fc2f5d9f468","Type":"ContainerStarted","Data":"0c4da7cac071345b7d449ab8386016399592e123be31d1573ea11a77de33e917"} Oct 30 13:00:18.476810 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:18.476788 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889420-n4zdq" event={"ID":"71805e6f-86c5-4659-aa70-849ba38f288c","Type":"ContainerStarted","Data":"5d4074378e178369be11ca5ff16f41f1a207315bc48d918653be127f15cd8561"} Oct 30 13:00:18.478903 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:18.478877 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-42220c-29889420-q4fqq" event={"ID":"ced43030-57a2-418d-bf86-f9c5a3b8cd95","Type":"ContainerStarted","Data":"6e130d3becfc1417a0be6d2f562314cedeb3deff5b2c495d640f66e5da8d6686"} Oct 30 13:00:18.611848 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:18.601752 4908 scope.go:122] "RemoveContainer" containerID="690ec2b414b11ab39bef37bd18421e14c295a32577da7d213917ab9cb7ae45b5" Oct 30 13:00:18.611848 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:18.602162 4908 scope.go:122] "RemoveContainer" containerID="57b813b5ea4b03fd240c4cce6470b80c2af653f1942979d749d2dc031006c211" Oct 30 13:00:18.611848 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:18.603681 4908 scope.go:122] "RemoveContainer" containerID="684e2f653cdd517f57cf566510baf88f8ccf5c83fbe6ee79be6565cde142ffbf" Oct 30 13:00:18.611848 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:18.604133 4908 scope.go:122] "RemoveContainer" containerID="81f9b9853015d1c8504e652f5e16222bead314f4f516d286f8af942da7d2e044" Oct 30 13:00:18.611848 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:18.604448 4908 scope.go:122] "RemoveContainer" containerID="96b0ea8ecfe4f3439a866d2f770a7b57d4ad8b8f0b714aebead350c5999327b0" Oct 30 13:00:18.611848 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:18.606140 4908 scope.go:122] "RemoveContainer" containerID="3fc05ea0c4ac70dad85480064954e620d2dd91e97bc6ea2605e4a3b5f8cb8eb5" Oct 30 13:00:18.701182 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:18.701120 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-42220c-29889420-q4fqq" podStartSLOduration=18.701098055 podStartE2EDuration="18.701098055s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:00:18.650971421 +0000 UTC m=+3975.165400167" watchObservedRunningTime="2026-10-30 13:00:18.701098055 +0000 UTC m=+3975.215526803" Oct 30 13:00:18.752890 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:18.752861 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-dbe123-29889420-bcnzv" Oct 30 13:00:18.858533 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:18.858429 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8165abf7-fbfb-4de2-a94c-5023a834835d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8165abf7-fbfb-4de2-a94c-5023a834835d-maas-api-service-ca\") pod \"8165abf7-fbfb-4de2-a94c-5023a834835d\" (UID: \"8165abf7-fbfb-4de2-a94c-5023a834835d\") " Oct 30 13:00:18.858726 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:18.858705 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8165abf7-fbfb-4de2-a94c-5023a834835d-kube-api-access-6qqw7\" (UniqueName: \"kubernetes.io/projected/8165abf7-fbfb-4de2-a94c-5023a834835d-kube-api-access-6qqw7\") pod \"8165abf7-fbfb-4de2-a94c-5023a834835d\" (UID: \"8165abf7-fbfb-4de2-a94c-5023a834835d\") " Oct 30 13:00:18.864514 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:18.858917 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8165abf7-fbfb-4de2-a94c-5023a834835d-maas-api-service-ca" pod "8165abf7-fbfb-4de2-a94c-5023a834835d" (UID: "8165abf7-fbfb-4de2-a94c-5023a834835d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:00:18.864514 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:18.859428 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8165abf7-fbfb-4de2-a94c-5023a834835d-maas-api-service-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:00:19.062446 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:19.061851 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8165abf7-fbfb-4de2-a94c-5023a834835d-kube-api-access-6qqw7" pod "8165abf7-fbfb-4de2-a94c-5023a834835d" (UID: "8165abf7-fbfb-4de2-a94c-5023a834835d"). InnerVolumeSpecName "kube-api-access-6qqw7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:00:19.062541 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:19.062470 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8165abf7-fbfb-4de2-a94c-5023a834835d-kube-api-access-6qqw7\" (UniqueName: \"kubernetes.io/projected/8165abf7-fbfb-4de2-a94c-5023a834835d-kube-api-access-6qqw7\") pod \"8165abf7-fbfb-4de2-a94c-5023a834835d\" (UID: \"8165abf7-fbfb-4de2-a94c-5023a834835d\") " Oct 30 13:00:19.062907 ip-10-0-141-251 kubenswrapper[4908]: W1030 13:00:19.062620 4908 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/8165abf7-fbfb-4de2-a94c-5023a834835d/volumes/kubernetes.io~projected/kube-api-access-6qqw7 Oct 30 13:00:19.062907 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:19.062675 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8165abf7-fbfb-4de2-a94c-5023a834835d-kube-api-access-6qqw7" pod "8165abf7-fbfb-4de2-a94c-5023a834835d" (UID: "8165abf7-fbfb-4de2-a94c-5023a834835d"). InnerVolumeSpecName "kube-api-access-6qqw7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:00:19.063421 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:19.063395 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6qqw7\" (UniqueName: \"kubernetes.io/projected/8165abf7-fbfb-4de2-a94c-5023a834835d-kube-api-access-6qqw7\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:00:19.493004 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:19.492973 4908 generic.go:356] "Generic (PLEG): container finished" podID="71805e6f-86c5-4659-aa70-849ba38f288c" containerID="5d4074378e178369be11ca5ff16f41f1a207315bc48d918653be127f15cd8561" exitCode=60 Oct 30 13:00:19.493442 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:19.493043 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889420-n4zdq" event={"ID":"71805e6f-86c5-4659-aa70-849ba38f288c","Type":"ContainerDied","Data":"5d4074378e178369be11ca5ff16f41f1a207315bc48d918653be127f15cd8561"} Oct 30 13:00:19.493442 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:19.493105 4908 scope.go:122] "RemoveContainer" containerID="be667f188369ff762f1c63b473896b06f415609784a3d0bc7c9dffb5f9fff403" Oct 30 13:00:19.493566 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:19.493551 4908 scope.go:122] "RemoveContainer" containerID="5d4074378e178369be11ca5ff16f41f1a207315bc48d918653be127f15cd8561" Oct 30 13:00:19.494025 ip-10-0-141-251 kubenswrapper[4908]: E1030 13:00:19.493999 4908 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889420-n4zdq_odh-ai-gateway-infra(71805e6f-86c5-4659-aa70-849ba38f288c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889420-n4zdq" podUID="71805e6f-86c5-4659-aa70-849ba38f288c" Oct 30 13:00:19.496053 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:19.496026 4908 generic.go:356] "Generic (PLEG): container finished" podID="ced43030-57a2-418d-bf86-f9c5a3b8cd95" containerID="6e130d3becfc1417a0be6d2f562314cedeb3deff5b2c495d640f66e5da8d6686" exitCode=60 Oct 30 13:00:19.496166 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:19.496097 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-42220c-29889420-q4fqq" event={"ID":"ced43030-57a2-418d-bf86-f9c5a3b8cd95","Type":"ContainerDied","Data":"6e130d3becfc1417a0be6d2f562314cedeb3deff5b2c495d640f66e5da8d6686"} Oct 30 13:00:19.496499 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:19.496481 4908 scope.go:122] "RemoveContainer" containerID="6e130d3becfc1417a0be6d2f562314cedeb3deff5b2c495d640f66e5da8d6686" Oct 30 13:00:19.496841 ip-10-0-141-251 kubenswrapper[4908]: E1030 13:00:19.496803 4908 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-42220c-29889420-q4fqq_odh-ai-gateway-infra(ced43030-57a2-418d-bf86-f9c5a3b8cd95)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-42220c-29889420-q4fqq" podUID="ced43030-57a2-418d-bf86-f9c5a3b8cd95" Oct 30 13:00:19.497901 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:19.497880 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-dbe123-29889420-bcnzv" Oct 30 13:00:19.498292 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:19.497880 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-dbe123-29889420-bcnzv" event={"ID":"8165abf7-fbfb-4de2-a94c-5023a834835d","Type":"ContainerDied","Data":"e6d83f69d1d25872b48f126575ad3ffd30082fbc764d7a8a6d1b8c81610631d0"} Oct 30 13:00:19.500340 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:19.500316 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6ad1da-29889420-qhlvt" event={"ID":"1daed017-3390-4a05-9457-8d6c97919043","Type":"ContainerStarted","Data":"c1f20626c06387b86180fddac51ce0f421b0137e95519eb0fde6d5cfb437f95a"} Oct 30 13:00:19.503378 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:19.503352 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-12fd33-29889420-cfqsp" event={"ID":"f2695f98-7dab-4cf3-aaa3-d1717ecc2d30","Type":"ContainerStarted","Data":"66820f87e56f0df0eca4c13701679399273dff674e679e8be54b32d46dd5321f"} Oct 30 13:00:19.509372 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:19.508964 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-dd0628-29889420-jh88n" event={"ID":"347ca826-af8d-496d-bd0b-dca4071b3cd8","Type":"ContainerStarted","Data":"e5fa46add936fd1ef6c54f822f9796e6bcf14f3fb460f1d4bbaacf16d58932a9"} Oct 30 13:00:19.512104 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:19.512082 4908 scope.go:122] "RemoveContainer" containerID="b150b641deb5a9f13128f282c2081a39ebb087d1b77f9b5b70ca47012c325a3b" Oct 30 13:00:19.517917 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:19.517894 4908 generic.go:356] "Generic (PLEG): container finished" podID="1baa9cd5-0305-4f77-b7cf-f4751053ace8" containerID="cde8f2e102c47a71e4fc9dbd2dd19a3350d6b3724268567606a5d3b7dc9b98a6" exitCode=60 Oct 30 13:00:19.517993 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:19.517964 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-812990-29889420-ddwkl" event={"ID":"1baa9cd5-0305-4f77-b7cf-f4751053ace8","Type":"ContainerDied","Data":"cde8f2e102c47a71e4fc9dbd2dd19a3350d6b3724268567606a5d3b7dc9b98a6"} Oct 30 13:00:19.518313 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:19.518295 4908 scope.go:122] "RemoveContainer" containerID="cde8f2e102c47a71e4fc9dbd2dd19a3350d6b3724268567606a5d3b7dc9b98a6" Oct 30 13:00:19.518596 ip-10-0-141-251 kubenswrapper[4908]: E1030 13:00:19.518575 4908 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-812990-29889420-ddwkl_odh-ai-gateway-infra(1baa9cd5-0305-4f77-b7cf-f4751053ace8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-812990-29889420-ddwkl" podUID="1baa9cd5-0305-4f77-b7cf-f4751053ace8" Oct 30 13:00:19.520201 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:19.520171 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a786af-29889420-mrgm2" event={"ID":"1bb2e64a-3fd0-41bb-aafa-8f234fb2ffa2","Type":"ContainerStarted","Data":"b5c6c942fa2e19110e664120347f288f16aa29d06f68ce3919d28e045988e510"} Oct 30 13:00:19.522964 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:19.522942 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-73bafc-29889420-4lh9z" event={"ID":"0d506c31-90a7-493f-8b73-649d1ffcd0eb","Type":"ContainerStarted","Data":"5741e34fe4825e6fb116d013822277813d033f4fb261dfb98e0fe9b4502ac9eb"} Oct 30 13:00:19.526342 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:19.526316 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0bda67-29889420-4z7jt" event={"ID":"b46f76c5-4139-4fbd-af18-5f35fcb1cb05","Type":"ContainerStarted","Data":"39efd3265556ea5b7c689af9f63c13a8bbabfe99b01b8b58805ee36681e3028e"} Oct 30 13:00:19.528421 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:19.528400 4908 generic.go:356] "Generic (PLEG): container finished" podID="2674b41c-6ab9-4222-aa87-4fc2f5d9f468" containerID="0c4da7cac071345b7d449ab8386016399592e123be31d1573ea11a77de33e917" exitCode=60 Oct 30 13:00:19.528509 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:19.528468 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fddaa3-29889420-vzns9" event={"ID":"2674b41c-6ab9-4222-aa87-4fc2f5d9f468","Type":"ContainerDied","Data":"0c4da7cac071345b7d449ab8386016399592e123be31d1573ea11a77de33e917"} Oct 30 13:00:19.528757 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:19.528744 4908 scope.go:122] "RemoveContainer" containerID="0c4da7cac071345b7d449ab8386016399592e123be31d1573ea11a77de33e917" Oct 30 13:00:19.529017 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:19.528998 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-812990-29889420-ddwkl"] Oct 30 13:00:19.529060 ip-10-0-141-251 kubenswrapper[4908]: E1030 13:00:19.529003 4908 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-fddaa3-29889420-vzns9_odh-ai-gateway-infra(2674b41c-6ab9-4222-aa87-4fc2f5d9f468)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fddaa3-29889420-vzns9" podUID="2674b41c-6ab9-4222-aa87-4fc2f5d9f468" Oct 30 13:00:19.532761 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:19.532743 4908 scope.go:122] "RemoveContainer" containerID="33518cd1d49bcb2993c7b2da2bec20338c19ad839b636ad184fab4f582eb671a" Oct 30 13:00:19.544025 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:19.544003 4908 scope.go:122] "RemoveContainer" containerID="45287a04f0139cd48af47d49ad0ea72f6da58375218e51ee6d9f8ca3a953bf85" Oct 30 13:00:19.559959 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:19.559877 4908 scope.go:122] "RemoveContainer" containerID="a1e193ed4b30c47e5eb85bda06f874bc2f83a6ef1e393fbedba7f42f690ca371" Oct 30 13:00:19.568081 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:19.568055 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fddaa3-29889420-vzns9"] Oct 30 13:00:19.579689 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:19.579662 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-dbe123-29889420-bcnzv"] Oct 30 13:00:19.594693 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:19.594638 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-dbe123-29889420-bcnzv"] Oct 30 13:00:19.599785 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:19.599197 4908 scope.go:122] "RemoveContainer" containerID="455c3c383fe1a940a1d2b692834c5f2398232ceafa93c0f8d8ad91295ff035d1" Oct 30 13:00:19.600212 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:19.600180 4908 scope.go:122] "RemoveContainer" containerID="9e90edddf33c1b728e9595a6acdcdee922d7af72d81fe611f108bf0947790bc7" Oct 30 13:00:19.601331 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:19.600633 4908 scope.go:122] "RemoveContainer" containerID="1159c140b26e0a148bbfa6c979acfbad6eac5a31cb67f6f9a4b71037310633a5" Oct 30 13:00:19.614653 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:19.614611 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8165abf7-fbfb-4de2-a94c-5023a834835d" path="/var/lib/kubelet/pods/8165abf7-fbfb-4de2-a94c-5023a834835d/volumes" Oct 30 13:00:19.615288 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:19.615270 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9208b4e7-5b24-45ad-a6ba-faea07eae2b6" path="/var/lib/kubelet/pods/9208b4e7-5b24-45ad-a6ba-faea07eae2b6/volumes" Oct 30 13:00:19.656385 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:19.656334 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0bda67-29889420-4z7jt" podStartSLOduration=19.656315041 podStartE2EDuration="19.656315041s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:00:19.651004331 +0000 UTC m=+3976.165433078" watchObservedRunningTime="2026-10-30 13:00:19.656315041 +0000 UTC m=+3976.170743788" Oct 30 13:00:19.664568 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:19.664517 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-42220c-29889420-q4fqq"] Oct 30 13:00:19.707919 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:19.707877 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889420-n4zdq"] Oct 30 13:00:19.955704 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:19.955638 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a786af-29889420-mrgm2" podStartSLOduration=19.955613136 podStartE2EDuration="19.955613136s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:00:19.953767226 +0000 UTC m=+3976.468195973" watchObservedRunningTime="2026-10-30 13:00:19.955613136 +0000 UTC m=+3976.470041887" Oct 30 13:00:19.982001 ip-10-0-141-251 kubenswrapper[4908]: E1030 13:00:19.981963 4908 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vn6vv" containerName="extractor" Oct 30 13:00:20.540606 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:20.540568 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-8d2170-29889420-67bzd" event={"ID":"f5af9446-e0d9-4edc-b01f-d049c4b2d3e7","Type":"ContainerStarted","Data":"b30cb1fd83670b766226afcefa22d0a2562f286688614391ab3d70c6e93c4be3"} Oct 30 13:00:20.542723 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:20.542690 4908 generic.go:356] "Generic (PLEG): container finished" podID="f2695f98-7dab-4cf3-aaa3-d1717ecc2d30" containerID="66820f87e56f0df0eca4c13701679399273dff674e679e8be54b32d46dd5321f" exitCode=60 Oct 30 13:00:20.542890 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:20.542758 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-12fd33-29889420-cfqsp" event={"ID":"f2695f98-7dab-4cf3-aaa3-d1717ecc2d30","Type":"ContainerDied","Data":"66820f87e56f0df0eca4c13701679399273dff674e679e8be54b32d46dd5321f"} Oct 30 13:00:20.542890 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:20.542817 4908 scope.go:122] "RemoveContainer" containerID="690ec2b414b11ab39bef37bd18421e14c295a32577da7d213917ab9cb7ae45b5" Oct 30 13:00:20.543158 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:20.543143 4908 scope.go:122] "RemoveContainer" containerID="66820f87e56f0df0eca4c13701679399273dff674e679e8be54b32d46dd5321f" Oct 30 13:00:20.543444 ip-10-0-141-251 kubenswrapper[4908]: E1030 13:00:20.543410 4908 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-12fd33-29889420-cfqsp_odh-ai-gateway-infra(f2695f98-7dab-4cf3-aaa3-d1717ecc2d30)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-12fd33-29889420-cfqsp" podUID="f2695f98-7dab-4cf3-aaa3-d1717ecc2d30" Oct 30 13:00:20.545129 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:20.545103 4908 generic.go:356] "Generic (PLEG): container finished" podID="347ca826-af8d-496d-bd0b-dca4071b3cd8" containerID="e5fa46add936fd1ef6c54f822f9796e6bcf14f3fb460f1d4bbaacf16d58932a9" exitCode=60 Oct 30 13:00:20.545236 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:20.545129 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-dd0628-29889420-jh88n" event={"ID":"347ca826-af8d-496d-bd0b-dca4071b3cd8","Type":"ContainerDied","Data":"e5fa46add936fd1ef6c54f822f9796e6bcf14f3fb460f1d4bbaacf16d58932a9"} Oct 30 13:00:20.545503 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:20.545481 4908 scope.go:122] "RemoveContainer" containerID="e5fa46add936fd1ef6c54f822f9796e6bcf14f3fb460f1d4bbaacf16d58932a9" Oct 30 13:00:20.545761 ip-10-0-141-251 kubenswrapper[4908]: E1030 13:00:20.545734 4908 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-dd0628-29889420-jh88n_odh-ai-gateway-infra(347ca826-af8d-496d-bd0b-dca4071b3cd8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-dd0628-29889420-jh88n" podUID="347ca826-af8d-496d-bd0b-dca4071b3cd8" Oct 30 13:00:20.549789 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:20.549764 4908 generic.go:356] "Generic (PLEG): container finished" podID="1bb2e64a-3fd0-41bb-aafa-8f234fb2ffa2" containerID="b5c6c942fa2e19110e664120347f288f16aa29d06f68ce3919d28e045988e510" exitCode=60 Oct 30 13:00:20.549921 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:20.549819 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a786af-29889420-mrgm2" event={"ID":"1bb2e64a-3fd0-41bb-aafa-8f234fb2ffa2","Type":"ContainerDied","Data":"b5c6c942fa2e19110e664120347f288f16aa29d06f68ce3919d28e045988e510"} Oct 30 13:00:20.550208 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:20.550191 4908 scope.go:122] "RemoveContainer" containerID="b5c6c942fa2e19110e664120347f288f16aa29d06f68ce3919d28e045988e510" Oct 30 13:00:20.550493 ip-10-0-141-251 kubenswrapper[4908]: E1030 13:00:20.550466 4908 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-a786af-29889420-mrgm2_odh-ai-gateway-infra(1bb2e64a-3fd0-41bb-aafa-8f234fb2ffa2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a786af-29889420-mrgm2" podUID="1bb2e64a-3fd0-41bb-aafa-8f234fb2ffa2" Oct 30 13:00:20.552058 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:20.552039 4908 generic.go:356] "Generic (PLEG): container finished" podID="0d506c31-90a7-493f-8b73-649d1ffcd0eb" containerID="5741e34fe4825e6fb116d013822277813d033f4fb261dfb98e0fe9b4502ac9eb" exitCode=60 Oct 30 13:00:20.552161 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:20.552122 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-73bafc-29889420-4lh9z" event={"ID":"0d506c31-90a7-493f-8b73-649d1ffcd0eb","Type":"ContainerDied","Data":"5741e34fe4825e6fb116d013822277813d033f4fb261dfb98e0fe9b4502ac9eb"} Oct 30 13:00:20.552522 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:20.552503 4908 scope.go:122] "RemoveContainer" containerID="5741e34fe4825e6fb116d013822277813d033f4fb261dfb98e0fe9b4502ac9eb" Oct 30 13:00:20.552858 ip-10-0-141-251 kubenswrapper[4908]: E1030 13:00:20.552808 4908 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-73bafc-29889420-4lh9z_odh-ai-gateway-infra(0d506c31-90a7-493f-8b73-649d1ffcd0eb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-73bafc-29889420-4lh9z" podUID="0d506c31-90a7-493f-8b73-649d1ffcd0eb" Oct 30 13:00:20.554278 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:20.554253 4908 generic.go:356] "Generic (PLEG): container finished" podID="b46f76c5-4139-4fbd-af18-5f35fcb1cb05" containerID="39efd3265556ea5b7c689af9f63c13a8bbabfe99b01b8b58805ee36681e3028e" exitCode=60 Oct 30 13:00:20.554409 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:20.554381 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0bda67-29889420-4z7jt" event={"ID":"b46f76c5-4139-4fbd-af18-5f35fcb1cb05","Type":"ContainerDied","Data":"39efd3265556ea5b7c689af9f63c13a8bbabfe99b01b8b58805ee36681e3028e"} Oct 30 13:00:20.554765 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:20.554743 4908 scope.go:122] "RemoveContainer" containerID="39efd3265556ea5b7c689af9f63c13a8bbabfe99b01b8b58805ee36681e3028e" Oct 30 13:00:20.555055 ip-10-0-141-251 kubenswrapper[4908]: E1030 13:00:20.555033 4908 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-0bda67-29889420-4z7jt_odh-ai-gateway-infra(b46f76c5-4139-4fbd-af18-5f35fcb1cb05)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0bda67-29889420-4z7jt" podUID="b46f76c5-4139-4fbd-af18-5f35fcb1cb05" Oct 30 13:00:20.608900 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:20.606295 4908 scope.go:122] "RemoveContainer" containerID="96b0ea8ecfe4f3439a866d2f770a7b57d4ad8b8f0b714aebead350c5999327b0" Oct 30 13:00:20.617146 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:20.617111 4908 scope.go:122] "RemoveContainer" containerID="53c7a0cb1a17dbfeff3c3e09682d3f006a52ad471b22c7089d0965bfa316bacb" Oct 30 13:00:20.619172 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:20.619152 4908 scope.go:122] "RemoveContainer" containerID="efa00ec66ca15fd57805025c4f950ddaf483969da95c71ff50ce93f6e478f31d" Oct 30 13:00:20.619663 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:20.619645 4908 scope.go:122] "RemoveContainer" containerID="9f340a37a3e8d33f47f2c6b0d7bc2357f171968f4c8f8f230eed09a502acbd7e" Oct 30 13:00:20.620931 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:20.620913 4908 scope.go:122] "RemoveContainer" containerID="84574467f1500cea18a9ddcfc697eace7267e137c6bd81b2b06762279c05543f" Oct 30 13:00:20.624330 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:20.623524 4908 scope.go:122] "RemoveContainer" containerID="e03b25b8e2910b878c1499548f3ab6f0b62e4548f50e313da7ee0484c111f575" Oct 30 13:00:20.625244 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:20.625208 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2545c7-29889420-xrknl" event={"ID":"bf7295f0-efb3-487a-926c-c4f19fddde26","Type":"ContainerStarted","Data":"c40ee8e5c01ee6b6a0c721d8b6e037228302cf1ca0a91b935c2f91a9e00648b9"} Oct 30 13:00:20.627574 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:20.625554 4908 scope.go:122] "RemoveContainer" containerID="4799cf96c861e895a43d5f99bf0921f0786601ec68f4cda7620b009d9f987c1d" Oct 30 13:00:20.651869 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:20.643777 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ecb3aa-29889420-72h7x" event={"ID":"cb87d307-def9-4d16-8e7a-54b531f0d2c5","Type":"ContainerStarted","Data":"b8083906749acf4ae49e94058ac022af403bb244db7bc4c3b3b7f568d6d861da"} Oct 30 13:00:20.656545 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:20.655907 4908 generic.go:356] "Generic (PLEG): container finished" podID="1daed017-3390-4a05-9457-8d6c97919043" containerID="c1f20626c06387b86180fddac51ce0f421b0137e95519eb0fde6d5cfb437f95a" exitCode=60 Oct 30 13:00:20.656545 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:20.656029 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6ad1da-29889420-qhlvt" event={"ID":"1daed017-3390-4a05-9457-8d6c97919043","Type":"ContainerDied","Data":"c1f20626c06387b86180fddac51ce0f421b0137e95519eb0fde6d5cfb437f95a"} Oct 30 13:00:20.672239 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:20.657261 4908 scope.go:122] "RemoveContainer" containerID="c1f20626c06387b86180fddac51ce0f421b0137e95519eb0fde6d5cfb437f95a" Oct 30 13:00:20.672239 ip-10-0-141-251 kubenswrapper[4908]: E1030 13:00:20.657742 4908 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-6ad1da-29889420-qhlvt_odh-ai-gateway-infra(1daed017-3390-4a05-9457-8d6c97919043)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6ad1da-29889420-qhlvt" podUID="1daed017-3390-4a05-9457-8d6c97919043" Oct 30 13:00:20.687590 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:20.687564 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0bda67-29889420-4z7jt"] Oct 30 13:00:20.694898 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:20.694868 4908 scope.go:122] "RemoveContainer" containerID="57b813b5ea4b03fd240c4cce6470b80c2af653f1942979d749d2dc031006c211" Oct 30 13:00:20.721633 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:20.721573 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2545c7-29889420-xrknl" podStartSLOduration=20.721553918 podStartE2EDuration="20.721553918s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:00:20.715132048 +0000 UTC m=+3977.229560795" watchObservedRunningTime="2026-10-30 13:00:20.721553918 +0000 UTC m=+3977.235982667" Oct 30 13:00:20.751315 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:20.751245 4908 scope.go:122] "RemoveContainer" containerID="684e2f653cdd517f57cf566510baf88f8ccf5c83fbe6ee79be6565cde142ffbf" Oct 30 13:00:20.783904 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:20.783866 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-12fd33-29889420-cfqsp"] Oct 30 13:00:20.815883 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:20.815796 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6ad1da-29889420-qhlvt"] Oct 30 13:00:20.950021 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:20.949993 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-dd0628-29889420-jh88n"] Oct 30 13:00:20.967682 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:20.967600 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-73bafc-29889420-4lh9z"] Oct 30 13:00:21.041214 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:21.040451 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a786af-29889420-mrgm2"] Oct 30 13:00:21.050931 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:21.048711 4908 scope.go:122] "RemoveContainer" containerID="3fc05ea0c4ac70dad85480064954e620d2dd91e97bc6ea2605e4a3b5f8cb8eb5" Oct 30 13:00:21.061216 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:21.061129 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-812990-29889420-ddwkl" Oct 30 13:00:21.080385 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:21.080357 4908 scope.go:122] "RemoveContainer" containerID="81f9b9853015d1c8504e652f5e16222bead314f4f516d286f8af942da7d2e044" Oct 30 13:00:21.081245 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:21.081219 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fddaa3-29889420-vzns9" Oct 30 13:00:21.085333 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:21.085313 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889420-n4zdq" Oct 30 13:00:21.088383 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:21.088358 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-42220c-29889420-q4fqq" Oct 30 13:00:21.143951 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:21.143805 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1baa9cd5-0305-4f77-b7cf-f4751053ace8-kube-api-access-zbqrv\" (UniqueName: \"kubernetes.io/projected/1baa9cd5-0305-4f77-b7cf-f4751053ace8-kube-api-access-zbqrv\") pod \"1baa9cd5-0305-4f77-b7cf-f4751053ace8\" (UID: \"1baa9cd5-0305-4f77-b7cf-f4751053ace8\") " Oct 30 13:00:21.143951 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:21.143880 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1baa9cd5-0305-4f77-b7cf-f4751053ace8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1baa9cd5-0305-4f77-b7cf-f4751053ace8-maas-api-service-ca\") pod \"1baa9cd5-0305-4f77-b7cf-f4751053ace8\" (UID: \"1baa9cd5-0305-4f77-b7cf-f4751053ace8\") " Oct 30 13:00:21.144385 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:21.144362 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ced43030-57a2-418d-bf86-f9c5a3b8cd95-kube-api-access-s8vk9\" (UniqueName: \"kubernetes.io/projected/ced43030-57a2-418d-bf86-f9c5a3b8cd95-kube-api-access-s8vk9\") pod \"ced43030-57a2-418d-bf86-f9c5a3b8cd95\" (UID: \"ced43030-57a2-418d-bf86-f9c5a3b8cd95\") " Oct 30 13:00:21.144454 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:21.144429 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ced43030-57a2-418d-bf86-f9c5a3b8cd95-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ced43030-57a2-418d-bf86-f9c5a3b8cd95-maas-api-service-ca\") pod \"ced43030-57a2-418d-bf86-f9c5a3b8cd95\" (UID: \"ced43030-57a2-418d-bf86-f9c5a3b8cd95\") " Oct 30 13:00:21.144508 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:21.144363 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1baa9cd5-0305-4f77-b7cf-f4751053ace8-maas-api-service-ca" pod "1baa9cd5-0305-4f77-b7cf-f4751053ace8" (UID: "1baa9cd5-0305-4f77-b7cf-f4751053ace8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:00:21.144566 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:21.144513 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71805e6f-86c5-4659-aa70-849ba38f288c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/71805e6f-86c5-4659-aa70-849ba38f288c-maas-api-service-ca\") pod \"71805e6f-86c5-4659-aa70-849ba38f288c\" (UID: \"71805e6f-86c5-4659-aa70-849ba38f288c\") " Oct 30 13:00:21.144867 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:21.144845 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2674b41c-6ab9-4222-aa87-4fc2f5d9f468-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2674b41c-6ab9-4222-aa87-4fc2f5d9f468-maas-api-service-ca\") pod \"2674b41c-6ab9-4222-aa87-4fc2f5d9f468\" (UID: \"2674b41c-6ab9-4222-aa87-4fc2f5d9f468\") " Oct 30 13:00:21.144971 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:21.144871 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ced43030-57a2-418d-bf86-f9c5a3b8cd95-maas-api-service-ca" pod "ced43030-57a2-418d-bf86-f9c5a3b8cd95" (UID: "ced43030-57a2-418d-bf86-f9c5a3b8cd95"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:00:21.144971 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:21.144906 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2674b41c-6ab9-4222-aa87-4fc2f5d9f468-kube-api-access-bzfvz\" (UniqueName: \"kubernetes.io/projected/2674b41c-6ab9-4222-aa87-4fc2f5d9f468-kube-api-access-bzfvz\") pod \"2674b41c-6ab9-4222-aa87-4fc2f5d9f468\" (UID: \"2674b41c-6ab9-4222-aa87-4fc2f5d9f468\") " Oct 30 13:00:21.145089 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:21.144983 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71805e6f-86c5-4659-aa70-849ba38f288c-maas-api-service-ca" pod "71805e6f-86c5-4659-aa70-849ba38f288c" (UID: "71805e6f-86c5-4659-aa70-849ba38f288c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:00:21.145089 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:21.145007 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71805e6f-86c5-4659-aa70-849ba38f288c-kube-api-access-fw5vn\" (UniqueName: \"kubernetes.io/projected/71805e6f-86c5-4659-aa70-849ba38f288c-kube-api-access-fw5vn\") pod \"71805e6f-86c5-4659-aa70-849ba38f288c\" (UID: \"71805e6f-86c5-4659-aa70-849ba38f288c\") " Oct 30 13:00:21.145291 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:21.145255 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2674b41c-6ab9-4222-aa87-4fc2f5d9f468-maas-api-service-ca" pod "2674b41c-6ab9-4222-aa87-4fc2f5d9f468" (UID: "2674b41c-6ab9-4222-aa87-4fc2f5d9f468"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:00:21.145719 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:21.145703 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1baa9cd5-0305-4f77-b7cf-f4751053ace8-maas-api-service-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:00:21.145812 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:21.145722 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ced43030-57a2-418d-bf86-f9c5a3b8cd95-maas-api-service-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:00:21.145812 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:21.145732 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/71805e6f-86c5-4659-aa70-849ba38f288c-maas-api-service-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:00:21.145812 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:21.145741 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2674b41c-6ab9-4222-aa87-4fc2f5d9f468-maas-api-service-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:00:21.273917 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:21.268571 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ced43030-57a2-418d-bf86-f9c5a3b8cd95-kube-api-access-s8vk9" pod "ced43030-57a2-418d-bf86-f9c5a3b8cd95" (UID: "ced43030-57a2-418d-bf86-f9c5a3b8cd95"). InnerVolumeSpecName "kube-api-access-s8vk9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:00:21.287181 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:21.287084 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2674b41c-6ab9-4222-aa87-4fc2f5d9f468-kube-api-access-bzfvz" pod "2674b41c-6ab9-4222-aa87-4fc2f5d9f468" (UID: "2674b41c-6ab9-4222-aa87-4fc2f5d9f468"). InnerVolumeSpecName "kube-api-access-bzfvz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:00:21.287331 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:21.287264 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71805e6f-86c5-4659-aa70-849ba38f288c-kube-api-access-fw5vn" pod "71805e6f-86c5-4659-aa70-849ba38f288c" (UID: "71805e6f-86c5-4659-aa70-849ba38f288c"). InnerVolumeSpecName "kube-api-access-fw5vn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:00:21.287398 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:21.287347 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1baa9cd5-0305-4f77-b7cf-f4751053ace8-kube-api-access-zbqrv" pod "1baa9cd5-0305-4f77-b7cf-f4751053ace8" (UID: "1baa9cd5-0305-4f77-b7cf-f4751053ace8"). InnerVolumeSpecName "kube-api-access-zbqrv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:00:21.356891 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:21.351109 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bzfvz\" (UniqueName: \"kubernetes.io/projected/2674b41c-6ab9-4222-aa87-4fc2f5d9f468-kube-api-access-bzfvz\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:00:21.356891 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:21.351154 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fw5vn\" (UniqueName: \"kubernetes.io/projected/71805e6f-86c5-4659-aa70-849ba38f288c-kube-api-access-fw5vn\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:00:21.356891 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:21.351169 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zbqrv\" (UniqueName: \"kubernetes.io/projected/1baa9cd5-0305-4f77-b7cf-f4751053ace8-kube-api-access-zbqrv\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:00:21.356891 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:21.351183 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8vk9\" (UniqueName: \"kubernetes.io/projected/ced43030-57a2-418d-bf86-f9c5a3b8cd95-kube-api-access-s8vk9\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:00:21.613863 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:21.613325 4908 scope.go:122] "RemoveContainer" containerID="2fa4249a504b685d16cd39267ecb54320052aa377205c69d5fb179ebd7807720" Oct 30 13:00:21.617706 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:21.614909 4908 scope.go:122] "RemoveContainer" containerID="624e7a813c919a43fbebe646cc29d37292bc07604dd331dd58451a3b9c7b305e" Oct 30 13:00:21.649412 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:21.649330 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-8d2170-29889420-67bzd"] Oct 30 13:00:21.672385 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:21.672347 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889420-n4zdq" event={"ID":"71805e6f-86c5-4659-aa70-849ba38f288c","Type":"ContainerDied","Data":"b5faf1282d0746dd4abc1df79d6a637d7577f53091b08933b3abe0bb2dfbb42e"} Oct 30 13:00:21.672555 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:21.672364 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889420-n4zdq" Oct 30 13:00:21.672555 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:21.672409 4908 scope.go:122] "RemoveContainer" containerID="5d4074378e178369be11ca5ff16f41f1a207315bc48d918653be127f15cd8561" Oct 30 13:00:21.675306 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:21.675279 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-10f364-29889420-fzd6k" event={"ID":"76cf17b7-c8b9-48d8-934c-f38aa9d1f008","Type":"ContainerStarted","Data":"d288a9e8e015573c24a8a482c0d655f152d339a97df995d68634141ba7ae1b65"} Oct 30 13:00:21.677240 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:21.677221 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-42220c-29889420-q4fqq" Oct 30 13:00:21.677616 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:21.677586 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-42220c-29889420-q4fqq" event={"ID":"ced43030-57a2-418d-bf86-f9c5a3b8cd95","Type":"ContainerDied","Data":"55f13598a2701932512ed3ec59b2667608bcfd0fd79e9ebf0c9371e9fbe1dc37"} Oct 30 13:00:21.679617 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:21.679596 4908 generic.go:356] "Generic (PLEG): container finished" podID="bf7295f0-efb3-487a-926c-c4f19fddde26" containerID="c40ee8e5c01ee6b6a0c721d8b6e037228302cf1ca0a91b935c2f91a9e00648b9" exitCode=60 Oct 30 13:00:21.679724 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:21.679667 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2545c7-29889420-xrknl" event={"ID":"bf7295f0-efb3-487a-926c-c4f19fddde26","Type":"ContainerDied","Data":"c40ee8e5c01ee6b6a0c721d8b6e037228302cf1ca0a91b935c2f91a9e00648b9"} Oct 30 13:00:21.680186 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:21.680166 4908 scope.go:122] "RemoveContainer" containerID="c40ee8e5c01ee6b6a0c721d8b6e037228302cf1ca0a91b935c2f91a9e00648b9" Oct 30 13:00:21.680585 ip-10-0-141-251 kubenswrapper[4908]: E1030 13:00:21.680557 4908 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-2545c7-29889420-xrknl_odh-ai-gateway-infra(bf7295f0-efb3-487a-926c-c4f19fddde26)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2545c7-29889420-xrknl" podUID="bf7295f0-efb3-487a-926c-c4f19fddde26" Oct 30 13:00:21.681959 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:21.681934 4908 generic.go:356] "Generic (PLEG): container finished" podID="cb87d307-def9-4d16-8e7a-54b531f0d2c5" containerID="b8083906749acf4ae49e94058ac022af403bb244db7bc4c3b3b7f568d6d861da" exitCode=60 Oct 30 13:00:21.682143 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:21.682008 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ecb3aa-29889420-72h7x" event={"ID":"cb87d307-def9-4d16-8e7a-54b531f0d2c5","Type":"ContainerDied","Data":"b8083906749acf4ae49e94058ac022af403bb244db7bc4c3b3b7f568d6d861da"} Oct 30 13:00:21.682520 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:21.682497 4908 scope.go:122] "RemoveContainer" containerID="b8083906749acf4ae49e94058ac022af403bb244db7bc4c3b3b7f568d6d861da" Oct 30 13:00:21.682876 ip-10-0-141-251 kubenswrapper[4908]: E1030 13:00:21.682850 4908 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-ecb3aa-29889420-72h7x_odh-ai-gateway-infra(cb87d307-def9-4d16-8e7a-54b531f0d2c5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ecb3aa-29889420-72h7x" podUID="cb87d307-def9-4d16-8e7a-54b531f0d2c5" Oct 30 13:00:21.684141 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:21.684119 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e9f184-29889420-sv9rp" event={"ID":"29e5414c-9b69-4713-8b27-80af3af5b4c7","Type":"ContainerStarted","Data":"22b206c76e5b75ed81bd6bc9ca1947834388bd1d5336634d293b7f8d2b33ce1c"} Oct 30 13:00:21.687516 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:21.687498 4908 scope.go:122] "RemoveContainer" containerID="6e130d3becfc1417a0be6d2f562314cedeb3deff5b2c495d640f66e5da8d6686" Oct 30 13:00:21.688229 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:21.688203 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1333c8-29889420-9zjhg" event={"ID":"94bed6ea-d66f-4f63-8859-a5a8d1969a01","Type":"ContainerStarted","Data":"7a4dcf1e8c2048091150357906372c27e54b35d8e6d386eaf82f15f13f26bb1b"} Oct 30 13:00:21.690698 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:21.690675 4908 generic.go:356] "Generic (PLEG): container finished" podID="f5af9446-e0d9-4edc-b01f-d049c4b2d3e7" containerID="b30cb1fd83670b766226afcefa22d0a2562f286688614391ab3d70c6e93c4be3" exitCode=60 Oct 30 13:00:21.690794 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:21.690728 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-8d2170-29889420-67bzd" event={"ID":"f5af9446-e0d9-4edc-b01f-d049c4b2d3e7","Type":"ContainerDied","Data":"b30cb1fd83670b766226afcefa22d0a2562f286688614391ab3d70c6e93c4be3"} Oct 30 13:00:21.701230 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:21.695444 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-327e31-29889420-2vbrc" event={"ID":"53e599d0-de7d-4c2f-86f9-97cc39405bc0","Type":"ContainerStarted","Data":"d7db4f0218d1ed4f1222db34cc4acac8e04189280707cbb967be00df6551ea92"} Oct 30 13:00:21.701230 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:21.699275 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-812990-29889420-ddwkl" event={"ID":"1baa9cd5-0305-4f77-b7cf-f4751053ace8","Type":"ContainerDied","Data":"2259400db520020aceeca60e8e16c05abc23c24dc587a8fc0e56d3edb117c8a3"} Oct 30 13:00:21.701230 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:21.699372 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-812990-29889420-ddwkl" Oct 30 13:00:21.707055 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:21.707023 4908 scope.go:122] "RemoveContainer" containerID="455c3c383fe1a940a1d2b692834c5f2398232ceafa93c0f8d8ad91295ff035d1" Oct 30 13:00:21.739920 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:21.738353 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fddaa3-29889420-vzns9" event={"ID":"2674b41c-6ab9-4222-aa87-4fc2f5d9f468","Type":"ContainerDied","Data":"aa8e2f32d8efd0c45e445cc661f8f8d2632731114baf424804542db284a4d13d"} Oct 30 13:00:21.739920 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:21.738493 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fddaa3-29889420-vzns9" Oct 30 13:00:21.745424 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:21.744076 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2675c0-29889420-hqgp5" event={"ID":"7f4c2c03-089f-4701-87d3-4e0496b08750","Type":"ContainerStarted","Data":"730cebe4e2e1f45b85943a7edbddd63d2394b6ebc1c0b12f183f685160d286ac"} Oct 30 13:00:21.768797 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:21.768416 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-9d45fb-29889420-5wlx7" event={"ID":"a130ae9f-0e32-4b92-b6d3-597640f7e802","Type":"ContainerStarted","Data":"64db43b65040950f3c4d64c6e760cdfe2a4cecf1965fbb53ed466b70b55ed58a"} Oct 30 13:00:21.784145 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:21.784119 4908 scope.go:122] "RemoveContainer" containerID="9e90edddf33c1b728e9595a6acdcdee922d7af72d81fe611f108bf0947790bc7" Oct 30 13:00:21.790558 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:21.790525 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2545c7-29889420-xrknl"] Oct 30 13:00:21.839907 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:21.839597 4908 scope.go:122] "RemoveContainer" containerID="1159c140b26e0a148bbfa6c979acfbad6eac5a31cb67f6f9a4b71037310633a5" Oct 30 13:00:21.903499 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:21.903460 4908 scope.go:122] "RemoveContainer" containerID="cde8f2e102c47a71e4fc9dbd2dd19a3350d6b3724268567606a5d3b7dc9b98a6" Oct 30 13:00:21.922717 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:21.921757 4908 scope.go:122] "RemoveContainer" containerID="0c4da7cac071345b7d449ab8386016399592e123be31d1573ea11a77de33e917" Oct 30 13:00:22.109911 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:22.109881 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-12fd33-29889420-cfqsp" Oct 30 13:00:22.197663 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:22.197627 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2695f98-7dab-4cf3-aaa3-d1717ecc2d30-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2695f98-7dab-4cf3-aaa3-d1717ecc2d30-maas-api-service-ca\") pod \"f2695f98-7dab-4cf3-aaa3-d1717ecc2d30\" (UID: \"f2695f98-7dab-4cf3-aaa3-d1717ecc2d30\") " Oct 30 13:00:22.197895 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:22.197749 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2695f98-7dab-4cf3-aaa3-d1717ecc2d30-kube-api-access-p6h7x\" (UniqueName: \"kubernetes.io/projected/f2695f98-7dab-4cf3-aaa3-d1717ecc2d30-kube-api-access-p6h7x\") pod \"f2695f98-7dab-4cf3-aaa3-d1717ecc2d30\" (UID: \"f2695f98-7dab-4cf3-aaa3-d1717ecc2d30\") " Oct 30 13:00:22.198079 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:22.198049 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2695f98-7dab-4cf3-aaa3-d1717ecc2d30-maas-api-service-ca" pod "f2695f98-7dab-4cf3-aaa3-d1717ecc2d30" (UID: "f2695f98-7dab-4cf3-aaa3-d1717ecc2d30"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:00:22.198777 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:22.198749 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2695f98-7dab-4cf3-aaa3-d1717ecc2d30-maas-api-service-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:00:22.229703 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:22.229661 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fddaa3-29889420-vzns9"] Oct 30 13:00:22.237533 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:22.237487 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fddaa3-29889420-vzns9"] Oct 30 13:00:22.261053 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:22.259894 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2695f98-7dab-4cf3-aaa3-d1717ecc2d30-kube-api-access-p6h7x" pod "f2695f98-7dab-4cf3-aaa3-d1717ecc2d30" (UID: "f2695f98-7dab-4cf3-aaa3-d1717ecc2d30"). InnerVolumeSpecName "kube-api-access-p6h7x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:00:22.300975 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:22.300937 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6h7x\" (UniqueName: \"kubernetes.io/projected/f2695f98-7dab-4cf3-aaa3-d1717ecc2d30-kube-api-access-p6h7x\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:00:22.438686 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:22.438592 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-42220c-29889420-q4fqq"] Oct 30 13:00:22.446396 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:22.446338 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-42220c-29889420-q4fqq"] Oct 30 13:00:22.480499 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:22.480462 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889420-n4zdq"] Oct 30 13:00:22.489137 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:22.489095 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889420-n4zdq"] Oct 30 13:00:22.510932 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:22.510893 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ecb3aa-29889420-72h7x"] Oct 30 13:00:22.514277 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:22.514249 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-dd0628-29889420-jh88n" Oct 30 13:00:22.522185 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:22.522160 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-8d2170-29889420-67bzd" Oct 30 13:00:22.529539 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:22.529512 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6ad1da-29889420-qhlvt" Oct 30 13:00:22.533562 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:22.533531 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0bda67-29889420-4z7jt" Oct 30 13:00:22.538167 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:22.538138 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a786af-29889420-mrgm2" Oct 30 13:00:22.543120 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:22.543096 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-73bafc-29889420-4lh9z" Oct 30 13:00:22.591817 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:22.591779 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-812990-29889420-ddwkl"] Oct 30 13:00:22.599559 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:22.599528 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-812990-29889420-ddwkl"] Oct 30 13:00:22.605749 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:22.605720 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/347ca826-af8d-496d-bd0b-dca4071b3cd8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/347ca826-af8d-496d-bd0b-dca4071b3cd8-maas-api-service-ca\") pod \"347ca826-af8d-496d-bd0b-dca4071b3cd8\" (UID: \"347ca826-af8d-496d-bd0b-dca4071b3cd8\") " Oct 30 13:00:22.605896 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:22.605806 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d506c31-90a7-493f-8b73-649d1ffcd0eb-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d506c31-90a7-493f-8b73-649d1ffcd0eb-maas-api-service-ca\") pod \"0d506c31-90a7-493f-8b73-649d1ffcd0eb\" (UID: \"0d506c31-90a7-493f-8b73-649d1ffcd0eb\") " Oct 30 13:00:22.605896 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:22.605841 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1bb2e64a-3fd0-41bb-aafa-8f234fb2ffa2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1bb2e64a-3fd0-41bb-aafa-8f234fb2ffa2-maas-api-service-ca\") pod \"1bb2e64a-3fd0-41bb-aafa-8f234fb2ffa2\" (UID: \"1bb2e64a-3fd0-41bb-aafa-8f234fb2ffa2\") " Oct 30 13:00:22.606013 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:22.605946 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/347ca826-af8d-496d-bd0b-dca4071b3cd8-kube-api-access-2kq4t\" (UniqueName: \"kubernetes.io/projected/347ca826-af8d-496d-bd0b-dca4071b3cd8-kube-api-access-2kq4t\") pod \"347ca826-af8d-496d-bd0b-dca4071b3cd8\" (UID: \"347ca826-af8d-496d-bd0b-dca4071b3cd8\") " Oct 30 13:00:22.606068 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:22.606035 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b46f76c5-4139-4fbd-af18-5f35fcb1cb05-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b46f76c5-4139-4fbd-af18-5f35fcb1cb05-maas-api-service-ca\") pod \"b46f76c5-4139-4fbd-af18-5f35fcb1cb05\" (UID: \"b46f76c5-4139-4fbd-af18-5f35fcb1cb05\") " Oct 30 13:00:22.606143 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:22.606122 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d506c31-90a7-493f-8b73-649d1ffcd0eb-kube-api-access-txtcl\" (UniqueName: \"kubernetes.io/projected/0d506c31-90a7-493f-8b73-649d1ffcd0eb-kube-api-access-txtcl\") pod \"0d506c31-90a7-493f-8b73-649d1ffcd0eb\" (UID: \"0d506c31-90a7-493f-8b73-649d1ffcd0eb\") " Oct 30 13:00:22.606201 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:22.606167 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5af9446-e0d9-4edc-b01f-d049c4b2d3e7-kube-api-access-z9jgd\" (UniqueName: \"kubernetes.io/projected/f5af9446-e0d9-4edc-b01f-d049c4b2d3e7-kube-api-access-z9jgd\") pod \"f5af9446-e0d9-4edc-b01f-d049c4b2d3e7\" (UID: \"f5af9446-e0d9-4edc-b01f-d049c4b2d3e7\") " Oct 30 13:00:22.606252 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:22.606204 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/347ca826-af8d-496d-bd0b-dca4071b3cd8-maas-api-service-ca" pod "347ca826-af8d-496d-bd0b-dca4071b3cd8" (UID: "347ca826-af8d-496d-bd0b-dca4071b3cd8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:00:22.606252 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:22.606220 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5af9446-e0d9-4edc-b01f-d049c4b2d3e7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5af9446-e0d9-4edc-b01f-d049c4b2d3e7-maas-api-service-ca\") pod \"f5af9446-e0d9-4edc-b01f-d049c4b2d3e7\" (UID: \"f5af9446-e0d9-4edc-b01f-d049c4b2d3e7\") " Oct 30 13:00:22.606435 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:22.606268 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bb2e64a-3fd0-41bb-aafa-8f234fb2ffa2-maas-api-service-ca" pod "1bb2e64a-3fd0-41bb-aafa-8f234fb2ffa2" (UID: "1bb2e64a-3fd0-41bb-aafa-8f234fb2ffa2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:00:22.606435 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:22.606284 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b46f76c5-4139-4fbd-af18-5f35fcb1cb05-kube-api-access-vrmmn\" (UniqueName: \"kubernetes.io/projected/b46f76c5-4139-4fbd-af18-5f35fcb1cb05-kube-api-access-vrmmn\") pod \"b46f76c5-4139-4fbd-af18-5f35fcb1cb05\" (UID: \"b46f76c5-4139-4fbd-af18-5f35fcb1cb05\") " Oct 30 13:00:22.606435 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:22.606341 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1bb2e64a-3fd0-41bb-aafa-8f234fb2ffa2-kube-api-access-6s4rp\" (UniqueName: \"kubernetes.io/projected/1bb2e64a-3fd0-41bb-aafa-8f234fb2ffa2-kube-api-access-6s4rp\") pod \"1bb2e64a-3fd0-41bb-aafa-8f234fb2ffa2\" (UID: \"1bb2e64a-3fd0-41bb-aafa-8f234fb2ffa2\") " Oct 30 13:00:22.606592 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:22.606509 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1daed017-3390-4a05-9457-8d6c97919043-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1daed017-3390-4a05-9457-8d6c97919043-maas-api-service-ca\") pod \"1daed017-3390-4a05-9457-8d6c97919043\" (UID: \"1daed017-3390-4a05-9457-8d6c97919043\") " Oct 30 13:00:22.606592 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:22.606541 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1daed017-3390-4a05-9457-8d6c97919043-kube-api-access-xnfs8\" (UniqueName: \"kubernetes.io/projected/1daed017-3390-4a05-9457-8d6c97919043-kube-api-access-xnfs8\") pod \"1daed017-3390-4a05-9457-8d6c97919043\" (UID: \"1daed017-3390-4a05-9457-8d6c97919043\") " Oct 30 13:00:22.606713 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:22.606624 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b46f76c5-4139-4fbd-af18-5f35fcb1cb05-maas-api-service-ca" pod "b46f76c5-4139-4fbd-af18-5f35fcb1cb05" (UID: "b46f76c5-4139-4fbd-af18-5f35fcb1cb05"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:00:22.607407 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:22.606948 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5af9446-e0d9-4edc-b01f-d049c4b2d3e7-maas-api-service-ca" pod "f5af9446-e0d9-4edc-b01f-d049c4b2d3e7" (UID: "f5af9446-e0d9-4edc-b01f-d049c4b2d3e7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:00:22.607407 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:22.607287 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d506c31-90a7-493f-8b73-649d1ffcd0eb-maas-api-service-ca" pod "0d506c31-90a7-493f-8b73-649d1ffcd0eb" (UID: "0d506c31-90a7-493f-8b73-649d1ffcd0eb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:00:22.607407 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:22.607372 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1daed017-3390-4a05-9457-8d6c97919043-maas-api-service-ca" pod "1daed017-3390-4a05-9457-8d6c97919043" (UID: "1daed017-3390-4a05-9457-8d6c97919043"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:00:22.607679 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:22.607546 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b46f76c5-4139-4fbd-af18-5f35fcb1cb05-maas-api-service-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:00:22.607679 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:22.607567 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5af9446-e0d9-4edc-b01f-d049c4b2d3e7-maas-api-service-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:00:22.607679 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:22.607582 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1daed017-3390-4a05-9457-8d6c97919043-maas-api-service-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:00:22.607679 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:22.607595 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/347ca826-af8d-496d-bd0b-dca4071b3cd8-maas-api-service-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:00:22.607679 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:22.607609 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d506c31-90a7-493f-8b73-649d1ffcd0eb-maas-api-service-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:00:22.607679 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:22.607622 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1bb2e64a-3fd0-41bb-aafa-8f234fb2ffa2-maas-api-service-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:00:22.609246 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:22.609224 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/347ca826-af8d-496d-bd0b-dca4071b3cd8-kube-api-access-2kq4t" pod "347ca826-af8d-496d-bd0b-dca4071b3cd8" (UID: "347ca826-af8d-496d-bd0b-dca4071b3cd8"). InnerVolumeSpecName "kube-api-access-2kq4t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:00:22.609345 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:22.609301 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bb2e64a-3fd0-41bb-aafa-8f234fb2ffa2-kube-api-access-6s4rp" pod "1bb2e64a-3fd0-41bb-aafa-8f234fb2ffa2" (UID: "1bb2e64a-3fd0-41bb-aafa-8f234fb2ffa2"). InnerVolumeSpecName "kube-api-access-6s4rp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:00:22.609403 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:22.609367 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d506c31-90a7-493f-8b73-649d1ffcd0eb-kube-api-access-txtcl" pod "0d506c31-90a7-493f-8b73-649d1ffcd0eb" (UID: "0d506c31-90a7-493f-8b73-649d1ffcd0eb"). InnerVolumeSpecName "kube-api-access-txtcl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:00:22.609524 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:22.609500 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b46f76c5-4139-4fbd-af18-5f35fcb1cb05-kube-api-access-vrmmn" pod "b46f76c5-4139-4fbd-af18-5f35fcb1cb05" (UID: "b46f76c5-4139-4fbd-af18-5f35fcb1cb05"). InnerVolumeSpecName "kube-api-access-vrmmn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:00:22.609901 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:22.609882 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5af9446-e0d9-4edc-b01f-d049c4b2d3e7-kube-api-access-z9jgd" pod "f5af9446-e0d9-4edc-b01f-d049c4b2d3e7" (UID: "f5af9446-e0d9-4edc-b01f-d049c4b2d3e7"). InnerVolumeSpecName "kube-api-access-z9jgd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:00:22.610699 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:22.610683 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1daed017-3390-4a05-9457-8d6c97919043-kube-api-access-xnfs8" pod "1daed017-3390-4a05-9457-8d6c97919043" (UID: "1daed017-3390-4a05-9457-8d6c97919043"). InnerVolumeSpecName "kube-api-access-xnfs8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:00:22.709790 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:22.709710 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6s4rp\" (UniqueName: \"kubernetes.io/projected/1bb2e64a-3fd0-41bb-aafa-8f234fb2ffa2-kube-api-access-6s4rp\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:00:22.710676 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:22.710649 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xnfs8\" (UniqueName: \"kubernetes.io/projected/1daed017-3390-4a05-9457-8d6c97919043-kube-api-access-xnfs8\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:00:22.711226 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:22.711204 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2kq4t\" (UniqueName: \"kubernetes.io/projected/347ca826-af8d-496d-bd0b-dca4071b3cd8-kube-api-access-2kq4t\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:00:22.711339 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:22.711326 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-txtcl\" (UniqueName: \"kubernetes.io/projected/0d506c31-90a7-493f-8b73-649d1ffcd0eb-kube-api-access-txtcl\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:00:22.711425 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:22.711413 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9jgd\" (UniqueName: \"kubernetes.io/projected/f5af9446-e0d9-4edc-b01f-d049c4b2d3e7-kube-api-access-z9jgd\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:00:22.711556 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:22.711541 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrmmn\" (UniqueName: \"kubernetes.io/projected/b46f76c5-4139-4fbd-af18-5f35fcb1cb05-kube-api-access-vrmmn\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:00:22.802650 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:22.802617 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e9f184-29889420-sv9rp"] Oct 30 13:00:22.817632 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:22.814128 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-8d2170-29889420-67bzd" Oct 30 13:00:22.817632 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:22.814126 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-8d2170-29889420-67bzd" event={"ID":"f5af9446-e0d9-4edc-b01f-d049c4b2d3e7","Type":"ContainerDied","Data":"7d21043dd6b80eea3b2dc088f18a053e5c3fd41919d51f2ffc512b58cc72a8b3"} Oct 30 13:00:22.817632 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:22.815964 4908 scope.go:122] "RemoveContainer" containerID="b30cb1fd83670b766226afcefa22d0a2562f286688614391ab3d70c6e93c4be3" Oct 30 13:00:22.817921 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:22.817648 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-12fd33-29889420-cfqsp" event={"ID":"f2695f98-7dab-4cf3-aaa3-d1717ecc2d30","Type":"ContainerDied","Data":"0b5aeb3bbdac21c1a9ff7080ac0756bbea9207ff367102227c9e4805748767e1"} Oct 30 13:00:22.817921 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:22.817746 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-12fd33-29889420-cfqsp" Oct 30 13:00:22.841043 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:22.834599 4908 generic.go:356] "Generic (PLEG): container finished" podID="53e599d0-de7d-4c2f-86f9-97cc39405bc0" containerID="d7db4f0218d1ed4f1222db34cc4acac8e04189280707cbb967be00df6551ea92" exitCode=60 Oct 30 13:00:22.841043 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:22.834668 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-327e31-29889420-2vbrc" event={"ID":"53e599d0-de7d-4c2f-86f9-97cc39405bc0","Type":"ContainerDied","Data":"d7db4f0218d1ed4f1222db34cc4acac8e04189280707cbb967be00df6551ea92"} Oct 30 13:00:22.841043 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:22.835523 4908 scope.go:122] "RemoveContainer" containerID="d7db4f0218d1ed4f1222db34cc4acac8e04189280707cbb967be00df6551ea92" Oct 30 13:00:22.841043 ip-10-0-141-251 kubenswrapper[4908]: E1030 13:00:22.835866 4908 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-327e31-29889420-2vbrc_odh-ai-gateway-infra(53e599d0-de7d-4c2f-86f9-97cc39405bc0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-327e31-29889420-2vbrc" podUID="53e599d0-de7d-4c2f-86f9-97cc39405bc0" Oct 30 13:00:22.841043 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:22.837641 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-dd0628-29889420-jh88n" event={"ID":"347ca826-af8d-496d-bd0b-dca4071b3cd8","Type":"ContainerDied","Data":"309a0efd465068d7d45cc7167f1e9b3a8d0ffb8c76c145eb2e0142384800cf0f"} Oct 30 13:00:22.841043 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:22.837728 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-dd0628-29889420-jh88n" Oct 30 13:00:22.845643 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:22.845615 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a786af-29889420-mrgm2" event={"ID":"1bb2e64a-3fd0-41bb-aafa-8f234fb2ffa2","Type":"ContainerDied","Data":"d4bfd60afba0eeaddbf8a5e8037417b67960769e1be5d55f338d4f3511834004"} Oct 30 13:00:22.845772 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:22.845700 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a786af-29889420-mrgm2" Oct 30 13:00:22.849554 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:22.849527 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-73bafc-29889420-4lh9z" event={"ID":"0d506c31-90a7-493f-8b73-649d1ffcd0eb","Type":"ContainerDied","Data":"a9c9d092be559fa34863e8f550db72c10feb14609962d2237923b7386b914f89"} Oct 30 13:00:22.849666 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:22.849624 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-73bafc-29889420-4lh9z" Oct 30 13:00:22.854939 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:22.854749 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f7464f-29889420-z9mg6" event={"ID":"94f72c00-8f0f-4bcf-9218-cd64b7773d1f","Type":"ContainerStarted","Data":"040854c5c1cc903e60e075ed19e264b56b65133c897ab8b72ce5277d0b88b69a"} Oct 30 13:00:22.855160 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:22.855037 4908 scope.go:122] "RemoveContainer" containerID="66820f87e56f0df0eca4c13701679399273dff674e679e8be54b32d46dd5321f" Oct 30 13:00:22.883212 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:22.879796 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0bda67-29889420-4z7jt" Oct 30 13:00:22.883212 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:22.880030 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0bda67-29889420-4z7jt" event={"ID":"b46f76c5-4139-4fbd-af18-5f35fcb1cb05","Type":"ContainerDied","Data":"02852f50bfad7fef40b45541888fefbdf644770d1e0e25d3aec116ae4c7ccc4c"} Oct 30 13:00:22.887752 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:22.886546 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-10f364-29889420-fzd6k"] Oct 30 13:00:22.887752 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:22.887074 4908 generic.go:356] "Generic (PLEG): container finished" podID="7f4c2c03-089f-4701-87d3-4e0496b08750" containerID="730cebe4e2e1f45b85943a7edbddd63d2394b6ebc1c0b12f183f685160d286ac" exitCode=60 Oct 30 13:00:22.887752 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:22.887152 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2675c0-29889420-hqgp5" event={"ID":"7f4c2c03-089f-4701-87d3-4e0496b08750","Type":"ContainerDied","Data":"730cebe4e2e1f45b85943a7edbddd63d2394b6ebc1c0b12f183f685160d286ac"} Oct 30 13:00:22.887752 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:22.887617 4908 scope.go:122] "RemoveContainer" containerID="730cebe4e2e1f45b85943a7edbddd63d2394b6ebc1c0b12f183f685160d286ac" Oct 30 13:00:22.888161 ip-10-0-141-251 kubenswrapper[4908]: E1030 13:00:22.888030 4908 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-2675c0-29889420-hqgp5_odh-ai-gateway-infra(7f4c2c03-089f-4701-87d3-4e0496b08750)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2675c0-29889420-hqgp5" podUID="7f4c2c03-089f-4701-87d3-4e0496b08750" Oct 30 13:00:22.893584 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:22.893055 4908 generic.go:356] "Generic (PLEG): container finished" podID="a130ae9f-0e32-4b92-b6d3-597640f7e802" containerID="64db43b65040950f3c4d64c6e760cdfe2a4cecf1965fbb53ed466b70b55ed58a" exitCode=60 Oct 30 13:00:22.893584 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:22.893172 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-9d45fb-29889420-5wlx7" event={"ID":"a130ae9f-0e32-4b92-b6d3-597640f7e802","Type":"ContainerDied","Data":"64db43b65040950f3c4d64c6e760cdfe2a4cecf1965fbb53ed466b70b55ed58a"} Oct 30 13:00:22.897193 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:22.896364 4908 scope.go:122] "RemoveContainer" containerID="64db43b65040950f3c4d64c6e760cdfe2a4cecf1965fbb53ed466b70b55ed58a" Oct 30 13:00:22.897193 ip-10-0-141-251 kubenswrapper[4908]: E1030 13:00:22.896744 4908 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-9d45fb-29889420-5wlx7_odh-ai-gateway-infra(a130ae9f-0e32-4b92-b6d3-597640f7e802)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-9d45fb-29889420-5wlx7" podUID="a130ae9f-0e32-4b92-b6d3-597640f7e802" Oct 30 13:00:22.904106 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:22.904075 4908 generic.go:356] "Generic (PLEG): container finished" podID="76cf17b7-c8b9-48d8-934c-f38aa9d1f008" containerID="d288a9e8e015573c24a8a482c0d655f152d339a97df995d68634141ba7ae1b65" exitCode=60 Oct 30 13:00:22.904185 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:22.904157 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-10f364-29889420-fzd6k" event={"ID":"76cf17b7-c8b9-48d8-934c-f38aa9d1f008","Type":"ContainerDied","Data":"d288a9e8e015573c24a8a482c0d655f152d339a97df995d68634141ba7ae1b65"} Oct 30 13:00:22.914284 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:22.909032 4908 generic.go:356] "Generic (PLEG): container finished" podID="a7bd05c5-f726-4ad3-88bf-b3a351a409bb" containerID="3bcd34c02206ba70d1e06ea09f1e520b64d2a70331e14deb7c8b5ef686e82703" exitCode=60 Oct 30 13:00:22.914284 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:22.909121 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6b67f8-29889420-pt2lw" event={"ID":"a7bd05c5-f726-4ad3-88bf-b3a351a409bb","Type":"ContainerDied","Data":"3bcd34c02206ba70d1e06ea09f1e520b64d2a70331e14deb7c8b5ef686e82703"} Oct 30 13:00:22.914284 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:22.909552 4908 scope.go:122] "RemoveContainer" containerID="3bcd34c02206ba70d1e06ea09f1e520b64d2a70331e14deb7c8b5ef686e82703" Oct 30 13:00:22.914284 ip-10-0-141-251 kubenswrapper[4908]: E1030 13:00:22.909941 4908 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-6b67f8-29889420-pt2lw_odh-ai-gateway-infra(a7bd05c5-f726-4ad3-88bf-b3a351a409bb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6b67f8-29889420-pt2lw" podUID="a7bd05c5-f726-4ad3-88bf-b3a351a409bb" Oct 30 13:00:22.921522 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:22.920495 4908 generic.go:356] "Generic (PLEG): container finished" podID="29e5414c-9b69-4713-8b27-80af3af5b4c7" containerID="22b206c76e5b75ed81bd6bc9ca1947834388bd1d5336634d293b7f8d2b33ce1c" exitCode=60 Oct 30 13:00:22.921522 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:22.920564 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e9f184-29889420-sv9rp" event={"ID":"29e5414c-9b69-4713-8b27-80af3af5b4c7","Type":"ContainerDied","Data":"22b206c76e5b75ed81bd6bc9ca1947834388bd1d5336634d293b7f8d2b33ce1c"} Oct 30 13:00:22.925664 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:22.923682 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6ad1da-29889420-qhlvt" event={"ID":"1daed017-3390-4a05-9457-8d6c97919043","Type":"ContainerDied","Data":"b65e4e06839268c5d9394433ba58fc5b5aada07507422b202276d358bf995d85"} Oct 30 13:00:22.925664 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:22.923772 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6ad1da-29889420-qhlvt" Oct 30 13:00:22.935623 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:22.935601 4908 scope.go:122] "RemoveContainer" containerID="53c7a0cb1a17dbfeff3c3e09682d3f006a52ad471b22c7089d0965bfa316bacb" Oct 30 13:00:22.940886 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:22.940859 4908 generic.go:356] "Generic (PLEG): container finished" podID="94bed6ea-d66f-4f63-8859-a5a8d1969a01" containerID="7a4dcf1e8c2048091150357906372c27e54b35d8e6d386eaf82f15f13f26bb1b" exitCode=60 Oct 30 13:00:22.941265 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:22.941130 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1333c8-29889420-9zjhg" event={"ID":"94bed6ea-d66f-4f63-8859-a5a8d1969a01","Type":"ContainerDied","Data":"7a4dcf1e8c2048091150357906372c27e54b35d8e6d386eaf82f15f13f26bb1b"} Oct 30 13:00:22.982308 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:22.978720 4908 scope.go:122] "RemoveContainer" containerID="7a4dcf1e8c2048091150357906372c27e54b35d8e6d386eaf82f15f13f26bb1b" Oct 30 13:00:22.982308 ip-10-0-141-251 kubenswrapper[4908]: E1030 13:00:22.979474 4908 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-1333c8-29889420-9zjhg_odh-ai-gateway-infra(94bed6ea-d66f-4f63-8859-a5a8d1969a01)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1333c8-29889420-9zjhg" podUID="94bed6ea-d66f-4f63-8859-a5a8d1969a01" Oct 30 13:00:23.026047 ip-10-0-141-251 kubenswrapper[4908]: E1030 13:00:23.025979 4908 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5af9446_e0d9_4edc_b01f_d049c4b2d3e7.slice/crio-7d21043dd6b80eea3b2dc088f18a053e5c3fd41919d51f2ffc512b58cc72a8b3\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bb2e64a_3fd0_41bb_aafa_8f234fb2ffa2.slice/crio-d4bfd60afba0eeaddbf8a5e8037417b67960769e1be5d55f338d4f3511834004\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7bd05c5_f726_4ad3_88bf_b3a351a409bb.slice/crio-conmon-3bcd34c02206ba70d1e06ea09f1e520b64d2a70331e14deb7c8b5ef686e82703.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bb2e64a_3fd0_41bb_aafa_8f234fb2ffa2.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d506c31_90a7_493f_8b73_649d1ffcd0eb.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2695f98_7dab_4cf3_aaa3_d1717ecc2d30.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d506c31_90a7_493f_8b73_649d1ffcd0eb.slice/crio-a9c9d092be559fa34863e8f550db72c10feb14609962d2237923b7386b914f89\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2695f98_7dab_4cf3_aaa3_d1717ecc2d30.slice/crio-0b5aeb3bbdac21c1a9ff7080ac0756bbea9207ff367102227c9e4805748767e1\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod347ca826_af8d_496d_bd0b_dca4071b3cd8.slice/crio-309a0efd465068d7d45cc7167f1e9b3a8d0ffb8c76c145eb2e0142384800cf0f\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb46f76c5_4139_4fbd_af18_5f35fcb1cb05.slice/crio-02852f50bfad7fef40b45541888fefbdf644770d1e0e25d3aec116ae4c7ccc4c\": RecentStats: unable to find data in memory cache]" Oct 30 13:00:23.052868 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:23.052779 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1333c8-29889420-9zjhg"] Oct 30 13:00:23.066590 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:23.066560 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0bda67-29889420-4z7jt"] Oct 30 13:00:23.072290 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:23.072268 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0bda67-29889420-4z7jt"] Oct 30 13:00:23.075318 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:23.075217 4908 scope.go:122] "RemoveContainer" containerID="e5fa46add936fd1ef6c54f822f9796e6bcf14f3fb460f1d4bbaacf16d58932a9" Oct 30 13:00:23.189074 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:23.188821 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-12fd33-29889420-cfqsp"] Oct 30 13:00:23.192881 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:23.192850 4908 scope.go:122] "RemoveContainer" containerID="b5c6c942fa2e19110e664120347f288f16aa29d06f68ce3919d28e045988e510" Oct 30 13:00:23.198173 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:23.198142 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-12fd33-29889420-cfqsp"] Oct 30 13:00:23.208092 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:23.208064 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-327e31-29889420-2vbrc"] Oct 30 13:00:23.212251 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:23.211956 4908 scope.go:122] "RemoveContainer" containerID="5741e34fe4825e6fb116d013822277813d033f4fb261dfb98e0fe9b4502ac9eb" Oct 30 13:00:23.228722 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:23.228386 4908 scope.go:122] "RemoveContainer" containerID="39efd3265556ea5b7c689af9f63c13a8bbabfe99b01b8b58805ee36681e3028e" Oct 30 13:00:23.247515 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:23.247211 4908 scope.go:122] "RemoveContainer" containerID="4799cf96c861e895a43d5f99bf0921f0786601ec68f4cda7620b009d9f987c1d" Oct 30 13:00:23.260069 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:23.260000 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-dd0628-29889420-jh88n"] Oct 30 13:00:23.279324 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:23.279299 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-dd0628-29889420-jh88n"] Oct 30 13:00:23.280362 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:23.280313 4908 scope.go:122] "RemoveContainer" containerID="efa00ec66ca15fd57805025c4f950ddaf483969da95c71ff50ce93f6e478f31d" Oct 30 13:00:23.301812 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:23.301791 4908 scope.go:122] "RemoveContainer" containerID="e03b25b8e2910b878c1499548f3ab6f0b62e4548f50e313da7ee0484c111f575" Oct 30 13:00:23.318399 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:23.318369 4908 scope.go:122] "RemoveContainer" containerID="2fa4249a504b685d16cd39267ecb54320052aa377205c69d5fb179ebd7807720" Oct 30 13:00:23.351254 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:23.351176 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f7464f-29889420-z9mg6" podStartSLOduration=23.35113784 podStartE2EDuration="23.35113784s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:00:23.299789917 +0000 UTC m=+3979.814218664" watchObservedRunningTime="2026-10-30 13:00:23.35113784 +0000 UTC m=+3979.865566590" Oct 30 13:00:23.362022 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:23.358404 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-73bafc-29889420-4lh9z"] Oct 30 13:00:23.376380 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:23.376341 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-73bafc-29889420-4lh9z"] Oct 30 13:00:23.381789 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:23.381757 4908 scope.go:122] "RemoveContainer" containerID="9f340a37a3e8d33f47f2c6b0d7bc2357f171968f4c8f8f230eed09a502acbd7e" Oct 30 13:00:23.417120 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:23.417084 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-9d45fb-29889420-5wlx7"] Oct 30 13:00:23.432681 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:23.432647 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-10f364-29889420-fzd6k" Oct 30 13:00:23.434741 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:23.434714 4908 scope.go:122] "RemoveContainer" containerID="c1f20626c06387b86180fddac51ce0f421b0137e95519eb0fde6d5cfb437f95a" Oct 30 13:00:23.448506 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:23.448410 4908 scope.go:122] "RemoveContainer" containerID="84574467f1500cea18a9ddcfc697eace7267e137c6bd81b2b06762279c05543f" Oct 30 13:00:23.470588 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:23.470564 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ecb3aa-29889420-72h7x" Oct 30 13:00:23.483763 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:23.483736 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e9f184-29889420-sv9rp" Oct 30 13:00:23.486254 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:23.486232 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2545c7-29889420-xrknl" Oct 30 13:00:23.546258 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:23.546178 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a786af-29889420-mrgm2"] Oct 30 13:00:23.553283 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:23.553246 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a786af-29889420-mrgm2"] Oct 30 13:00:23.555872 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:23.555837 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76cf17b7-c8b9-48d8-934c-f38aa9d1f008-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/76cf17b7-c8b9-48d8-934c-f38aa9d1f008-maas-api-service-ca\") pod \"76cf17b7-c8b9-48d8-934c-f38aa9d1f008\" (UID: \"76cf17b7-c8b9-48d8-934c-f38aa9d1f008\") " Oct 30 13:00:23.555992 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:23.555927 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb87d307-def9-4d16-8e7a-54b531f0d2c5-kube-api-access-tz2tr\" (UniqueName: \"kubernetes.io/projected/cb87d307-def9-4d16-8e7a-54b531f0d2c5-kube-api-access-tz2tr\") pod \"cb87d307-def9-4d16-8e7a-54b531f0d2c5\" (UID: \"cb87d307-def9-4d16-8e7a-54b531f0d2c5\") " Oct 30 13:00:23.556088 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:23.556073 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb87d307-def9-4d16-8e7a-54b531f0d2c5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb87d307-def9-4d16-8e7a-54b531f0d2c5-maas-api-service-ca\") pod \"cb87d307-def9-4d16-8e7a-54b531f0d2c5\" (UID: \"cb87d307-def9-4d16-8e7a-54b531f0d2c5\") " Oct 30 13:00:23.556169 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:23.556136 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76cf17b7-c8b9-48d8-934c-f38aa9d1f008-kube-api-access-wxfbg\" (UniqueName: \"kubernetes.io/projected/76cf17b7-c8b9-48d8-934c-f38aa9d1f008-kube-api-access-wxfbg\") pod \"76cf17b7-c8b9-48d8-934c-f38aa9d1f008\" (UID: \"76cf17b7-c8b9-48d8-934c-f38aa9d1f008\") " Oct 30 13:00:23.556234 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:23.556174 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76cf17b7-c8b9-48d8-934c-f38aa9d1f008-maas-api-service-ca" pod "76cf17b7-c8b9-48d8-934c-f38aa9d1f008" (UID: "76cf17b7-c8b9-48d8-934c-f38aa9d1f008"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:00:23.556493 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:23.556467 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb87d307-def9-4d16-8e7a-54b531f0d2c5-maas-api-service-ca" pod "cb87d307-def9-4d16-8e7a-54b531f0d2c5" (UID: "cb87d307-def9-4d16-8e7a-54b531f0d2c5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:00:23.556900 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:23.556878 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb87d307-def9-4d16-8e7a-54b531f0d2c5-maas-api-service-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:00:23.556900 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:23.556898 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/76cf17b7-c8b9-48d8-934c-f38aa9d1f008-maas-api-service-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:00:23.558277 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:23.558258 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb87d307-def9-4d16-8e7a-54b531f0d2c5-kube-api-access-tz2tr" pod "cb87d307-def9-4d16-8e7a-54b531f0d2c5" (UID: "cb87d307-def9-4d16-8e7a-54b531f0d2c5"). InnerVolumeSpecName "kube-api-access-tz2tr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:00:23.558357 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:23.558284 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76cf17b7-c8b9-48d8-934c-f38aa9d1f008-kube-api-access-wxfbg" pod "76cf17b7-c8b9-48d8-934c-f38aa9d1f008" (UID: "76cf17b7-c8b9-48d8-934c-f38aa9d1f008"). InnerVolumeSpecName "kube-api-access-wxfbg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:00:23.571273 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:23.571242 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2675c0-29889420-hqgp5"] Oct 30 13:00:23.586730 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:23.586701 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-8d2170-29889420-67bzd"] Oct 30 13:00:23.596005 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:23.595978 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-8d2170-29889420-67bzd"] Oct 30 13:00:23.608387 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:23.608357 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d506c31-90a7-493f-8b73-649d1ffcd0eb" path="/var/lib/kubelet/pods/0d506c31-90a7-493f-8b73-649d1ffcd0eb/volumes" Oct 30 13:00:23.608958 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:23.608942 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1baa9cd5-0305-4f77-b7cf-f4751053ace8" path="/var/lib/kubelet/pods/1baa9cd5-0305-4f77-b7cf-f4751053ace8/volumes" Oct 30 13:00:23.609413 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:23.609398 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1bb2e64a-3fd0-41bb-aafa-8f234fb2ffa2" path="/var/lib/kubelet/pods/1bb2e64a-3fd0-41bb-aafa-8f234fb2ffa2/volumes" Oct 30 13:00:23.610192 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:23.610175 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2674b41c-6ab9-4222-aa87-4fc2f5d9f468" path="/var/lib/kubelet/pods/2674b41c-6ab9-4222-aa87-4fc2f5d9f468/volumes" Oct 30 13:00:23.610655 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:23.610631 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="347ca826-af8d-496d-bd0b-dca4071b3cd8" path="/var/lib/kubelet/pods/347ca826-af8d-496d-bd0b-dca4071b3cd8/volumes" Oct 30 13:00:23.611153 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:23.611137 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71805e6f-86c5-4659-aa70-849ba38f288c" path="/var/lib/kubelet/pods/71805e6f-86c5-4659-aa70-849ba38f288c/volumes" Oct 30 13:00:23.611862 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:23.611846 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b46f76c5-4139-4fbd-af18-5f35fcb1cb05" path="/var/lib/kubelet/pods/b46f76c5-4139-4fbd-af18-5f35fcb1cb05/volumes" Oct 30 13:00:23.612296 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:23.612278 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ced43030-57a2-418d-bf86-f9c5a3b8cd95" path="/var/lib/kubelet/pods/ced43030-57a2-418d-bf86-f9c5a3b8cd95/volumes" Oct 30 13:00:23.612855 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:23.612813 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2695f98-7dab-4cf3-aaa3-d1717ecc2d30" path="/var/lib/kubelet/pods/f2695f98-7dab-4cf3-aaa3-d1717ecc2d30/volumes" Oct 30 13:00:23.613517 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:23.613501 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5af9446-e0d9-4edc-b01f-d049c4b2d3e7" path="/var/lib/kubelet/pods/f5af9446-e0d9-4edc-b01f-d049c4b2d3e7/volumes" Oct 30 13:00:23.657567 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:23.657534 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf7295f0-efb3-487a-926c-c4f19fddde26-kube-api-access-8rwhl\" (UniqueName: \"kubernetes.io/projected/bf7295f0-efb3-487a-926c-c4f19fddde26-kube-api-access-8rwhl\") pod \"bf7295f0-efb3-487a-926c-c4f19fddde26\" (UID: \"bf7295f0-efb3-487a-926c-c4f19fddde26\") " Oct 30 13:00:23.657715 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:23.657586 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29e5414c-9b69-4713-8b27-80af3af5b4c7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/29e5414c-9b69-4713-8b27-80af3af5b4c7-maas-api-service-ca\") pod \"29e5414c-9b69-4713-8b27-80af3af5b4c7\" (UID: \"29e5414c-9b69-4713-8b27-80af3af5b4c7\") " Oct 30 13:00:23.657781 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:23.657754 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf7295f0-efb3-487a-926c-c4f19fddde26-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf7295f0-efb3-487a-926c-c4f19fddde26-maas-api-service-ca\") pod \"bf7295f0-efb3-487a-926c-c4f19fddde26\" (UID: \"bf7295f0-efb3-487a-926c-c4f19fddde26\") " Oct 30 13:00:23.657979 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:23.657958 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29e5414c-9b69-4713-8b27-80af3af5b4c7-kube-api-access-shzqt\" (UniqueName: \"kubernetes.io/projected/29e5414c-9b69-4713-8b27-80af3af5b4c7-kube-api-access-shzqt\") pod \"29e5414c-9b69-4713-8b27-80af3af5b4c7\" (UID: \"29e5414c-9b69-4713-8b27-80af3af5b4c7\") " Oct 30 13:00:23.658129 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:23.658048 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29e5414c-9b69-4713-8b27-80af3af5b4c7-maas-api-service-ca" pod "29e5414c-9b69-4713-8b27-80af3af5b4c7" (UID: "29e5414c-9b69-4713-8b27-80af3af5b4c7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:00:23.658195 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:23.658141 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf7295f0-efb3-487a-926c-c4f19fddde26-maas-api-service-ca" pod "bf7295f0-efb3-487a-926c-c4f19fddde26" (UID: "bf7295f0-efb3-487a-926c-c4f19fddde26"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:00:23.658661 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:23.658640 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/29e5414c-9b69-4713-8b27-80af3af5b4c7-maas-api-service-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:00:23.658661 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:23.658665 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxfbg\" (UniqueName: \"kubernetes.io/projected/76cf17b7-c8b9-48d8-934c-f38aa9d1f008-kube-api-access-wxfbg\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:00:23.658806 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:23.658681 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf7295f0-efb3-487a-926c-c4f19fddde26-maas-api-service-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:00:23.658806 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:23.658696 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tz2tr\" (UniqueName: \"kubernetes.io/projected/cb87d307-def9-4d16-8e7a-54b531f0d2c5-kube-api-access-tz2tr\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:00:23.659990 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:23.659971 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29e5414c-9b69-4713-8b27-80af3af5b4c7-kube-api-access-shzqt" pod "29e5414c-9b69-4713-8b27-80af3af5b4c7" (UID: "29e5414c-9b69-4713-8b27-80af3af5b4c7"). InnerVolumeSpecName "kube-api-access-shzqt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:00:23.660144 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:23.660121 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf7295f0-efb3-487a-926c-c4f19fddde26-kube-api-access-8rwhl" pod "bf7295f0-efb3-487a-926c-c4f19fddde26" (UID: "bf7295f0-efb3-487a-926c-c4f19fddde26"). InnerVolumeSpecName "kube-api-access-8rwhl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:00:23.760699 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:23.760659 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8rwhl\" (UniqueName: \"kubernetes.io/projected/bf7295f0-efb3-487a-926c-c4f19fddde26-kube-api-access-8rwhl\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:00:23.760699 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:23.760700 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shzqt\" (UniqueName: \"kubernetes.io/projected/29e5414c-9b69-4713-8b27-80af3af5b4c7-kube-api-access-shzqt\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:00:23.798615 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:23.798508 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6ad1da-29889420-qhlvt"] Oct 30 13:00:23.806164 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:23.806127 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6ad1da-29889420-qhlvt"] Oct 30 13:00:23.981565 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:23.981534 4908 generic.go:356] "Generic (PLEG): container finished" podID="94f72c00-8f0f-4bcf-9218-cd64b7773d1f" containerID="040854c5c1cc903e60e075ed19e264b56b65133c897ab8b72ce5277d0b88b69a" exitCode=60 Oct 30 13:00:23.981763 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:23.981740 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f7464f-29889420-z9mg6" event={"ID":"94f72c00-8f0f-4bcf-9218-cd64b7773d1f","Type":"ContainerDied","Data":"040854c5c1cc903e60e075ed19e264b56b65133c897ab8b72ce5277d0b88b69a"} Oct 30 13:00:23.981920 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:23.981875 4908 scope.go:122] "RemoveContainer" containerID="624e7a813c919a43fbebe646cc29d37292bc07604dd331dd58451a3b9c7b305e" Oct 30 13:00:23.982391 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:23.982370 4908 scope.go:122] "RemoveContainer" containerID="040854c5c1cc903e60e075ed19e264b56b65133c897ab8b72ce5277d0b88b69a" Oct 30 13:00:23.982978 ip-10-0-141-251 kubenswrapper[4908]: E1030 13:00:23.982952 4908 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-f7464f-29889420-z9mg6_odh-ai-gateway-infra(94f72c00-8f0f-4bcf-9218-cd64b7773d1f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f7464f-29889420-z9mg6" podUID="94f72c00-8f0f-4bcf-9218-cd64b7773d1f" Oct 30 13:00:23.994880 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:23.993692 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-10f364-29889420-fzd6k" event={"ID":"76cf17b7-c8b9-48d8-934c-f38aa9d1f008","Type":"ContainerDied","Data":"e85e8bc46c3d1d00cd575e8a7f54b04aeeb98287a941a11d8f74ab359ea42240"} Oct 30 13:00:23.994880 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:23.993804 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-10f364-29889420-fzd6k" Oct 30 13:00:24.002849 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:23.999456 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2545c7-29889420-xrknl" event={"ID":"bf7295f0-efb3-487a-926c-c4f19fddde26","Type":"ContainerDied","Data":"5dc636445502e263bc215f39f516c71d35510dd0f2cc65055d6bcf01ad449db7"} Oct 30 13:00:24.002849 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:23.999633 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2545c7-29889420-xrknl" Oct 30 13:00:24.020342 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:24.020302 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ecb3aa-29889420-72h7x" event={"ID":"cb87d307-def9-4d16-8e7a-54b531f0d2c5","Type":"ContainerDied","Data":"be7278e70d878bf7d90abec9a1503f4420017e0e04f2a359e204e4daa14fa393"} Oct 30 13:00:24.028759 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:24.028722 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e9f184-29889420-sv9rp" event={"ID":"29e5414c-9b69-4713-8b27-80af3af5b4c7","Type":"ContainerDied","Data":"94f766340546261b7d2640463d0155ed5928d6c426ef481b1d1cc2ac73f45dfc"} Oct 30 13:00:24.029028 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:24.026963 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e9f184-29889420-sv9rp" Oct 30 13:00:24.030877 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:24.021732 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ecb3aa-29889420-72h7x" Oct 30 13:00:24.059421 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:24.052298 4908 scope.go:122] "RemoveContainer" containerID="d288a9e8e015573c24a8a482c0d655f152d339a97df995d68634141ba7ae1b65" Oct 30 13:00:24.083996 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:24.083612 4908 scope.go:122] "RemoveContainer" containerID="c40ee8e5c01ee6b6a0c721d8b6e037228302cf1ca0a91b935c2f91a9e00648b9" Oct 30 13:00:24.103081 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:24.102240 4908 scope.go:122] "RemoveContainer" containerID="b8083906749acf4ae49e94058ac022af403bb244db7bc4c3b3b7f568d6d861da" Oct 30 13:00:24.121290 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:24.121089 4908 scope.go:122] "RemoveContainer" containerID="22b206c76e5b75ed81bd6bc9ca1947834388bd1d5336634d293b7f8d2b33ce1c" Oct 30 13:00:24.155985 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:24.155511 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-10f364-29889420-fzd6k"] Oct 30 13:00:24.162694 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:24.162655 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-10f364-29889420-fzd6k"] Oct 30 13:00:24.229084 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:24.229062 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-327e31-29889420-2vbrc" Oct 30 13:00:24.280851 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:24.280150 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ecb3aa-29889420-72h7x"] Oct 30 13:00:24.285052 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:24.285019 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ecb3aa-29889420-72h7x"] Oct 30 13:00:24.348655 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:24.347798 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f7464f-29889420-z9mg6"] Oct 30 13:00:24.356275 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:24.356021 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e9f184-29889420-sv9rp"] Oct 30 13:00:24.368985 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:24.368917 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e9f184-29889420-sv9rp"] Oct 30 13:00:24.381883 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:24.375728 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53e599d0-de7d-4c2f-86f9-97cc39405bc0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53e599d0-de7d-4c2f-86f9-97cc39405bc0-maas-api-service-ca\") pod \"53e599d0-de7d-4c2f-86f9-97cc39405bc0\" (UID: \"53e599d0-de7d-4c2f-86f9-97cc39405bc0\") " Oct 30 13:00:24.381883 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:24.376365 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53e599d0-de7d-4c2f-86f9-97cc39405bc0-kube-api-access-nlbjg\" (UniqueName: \"kubernetes.io/projected/53e599d0-de7d-4c2f-86f9-97cc39405bc0-kube-api-access-nlbjg\") pod \"53e599d0-de7d-4c2f-86f9-97cc39405bc0\" (UID: \"53e599d0-de7d-4c2f-86f9-97cc39405bc0\") " Oct 30 13:00:24.381883 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:24.376620 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53e599d0-de7d-4c2f-86f9-97cc39405bc0-maas-api-service-ca" pod "53e599d0-de7d-4c2f-86f9-97cc39405bc0" (UID: "53e599d0-de7d-4c2f-86f9-97cc39405bc0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:00:24.381883 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:24.377327 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53e599d0-de7d-4c2f-86f9-97cc39405bc0-maas-api-service-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:00:24.383532 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:24.383506 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53e599d0-de7d-4c2f-86f9-97cc39405bc0-kube-api-access-nlbjg" pod "53e599d0-de7d-4c2f-86f9-97cc39405bc0" (UID: "53e599d0-de7d-4c2f-86f9-97cc39405bc0"). InnerVolumeSpecName "kube-api-access-nlbjg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:00:24.389767 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:24.389711 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2545c7-29889420-xrknl"] Oct 30 13:00:24.395532 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:24.395506 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2545c7-29889420-xrknl"] Oct 30 13:00:24.406346 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:24.406322 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-9d45fb-29889420-5wlx7" Oct 30 13:00:24.408430 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:24.408408 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1333c8-29889420-9zjhg" Oct 30 13:00:24.412586 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:24.412564 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2675c0-29889420-hqgp5" Oct 30 13:00:24.478467 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:24.478420 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f4c2c03-089f-4701-87d3-4e0496b08750-kube-api-access-8v6z7\" (UniqueName: \"kubernetes.io/projected/7f4c2c03-089f-4701-87d3-4e0496b08750-kube-api-access-8v6z7\") pod \"7f4c2c03-089f-4701-87d3-4e0496b08750\" (UID: \"7f4c2c03-089f-4701-87d3-4e0496b08750\") " Oct 30 13:00:24.478676 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:24.478490 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a130ae9f-0e32-4b92-b6d3-597640f7e802-kube-api-access-gqvth\" (UniqueName: \"kubernetes.io/projected/a130ae9f-0e32-4b92-b6d3-597640f7e802-kube-api-access-gqvth\") pod \"a130ae9f-0e32-4b92-b6d3-597640f7e802\" (UID: \"a130ae9f-0e32-4b92-b6d3-597640f7e802\") " Oct 30 13:00:24.478676 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:24.478555 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94bed6ea-d66f-4f63-8859-a5a8d1969a01-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94bed6ea-d66f-4f63-8859-a5a8d1969a01-maas-api-service-ca\") pod \"94bed6ea-d66f-4f63-8859-a5a8d1969a01\" (UID: \"94bed6ea-d66f-4f63-8859-a5a8d1969a01\") " Oct 30 13:00:24.478676 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:24.478626 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94bed6ea-d66f-4f63-8859-a5a8d1969a01-kube-api-access-krtdq\" (UniqueName: \"kubernetes.io/projected/94bed6ea-d66f-4f63-8859-a5a8d1969a01-kube-api-access-krtdq\") pod \"94bed6ea-d66f-4f63-8859-a5a8d1969a01\" (UID: \"94bed6ea-d66f-4f63-8859-a5a8d1969a01\") " Oct 30 13:00:24.479244 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:24.478953 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a130ae9f-0e32-4b92-b6d3-597640f7e802-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a130ae9f-0e32-4b92-b6d3-597640f7e802-maas-api-service-ca\") pod \"a130ae9f-0e32-4b92-b6d3-597640f7e802\" (UID: \"a130ae9f-0e32-4b92-b6d3-597640f7e802\") " Oct 30 13:00:24.479244 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:24.478998 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f4c2c03-089f-4701-87d3-4e0496b08750-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f4c2c03-089f-4701-87d3-4e0496b08750-maas-api-service-ca\") pod \"7f4c2c03-089f-4701-87d3-4e0496b08750\" (UID: \"7f4c2c03-089f-4701-87d3-4e0496b08750\") " Oct 30 13:00:24.479793 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:24.479764 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nlbjg\" (UniqueName: \"kubernetes.io/projected/53e599d0-de7d-4c2f-86f9-97cc39405bc0-kube-api-access-nlbjg\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:00:24.480153 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:24.480129 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f4c2c03-089f-4701-87d3-4e0496b08750-maas-api-service-ca" pod "7f4c2c03-089f-4701-87d3-4e0496b08750" (UID: "7f4c2c03-089f-4701-87d3-4e0496b08750"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:00:24.480344 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:24.480317 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94bed6ea-d66f-4f63-8859-a5a8d1969a01-maas-api-service-ca" pod "94bed6ea-d66f-4f63-8859-a5a8d1969a01" (UID: "94bed6ea-d66f-4f63-8859-a5a8d1969a01"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:00:24.480538 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:24.480514 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a130ae9f-0e32-4b92-b6d3-597640f7e802-maas-api-service-ca" pod "a130ae9f-0e32-4b92-b6d3-597640f7e802" (UID: "a130ae9f-0e32-4b92-b6d3-597640f7e802"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:00:24.482109 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:24.482079 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f4c2c03-089f-4701-87d3-4e0496b08750-kube-api-access-8v6z7" pod "7f4c2c03-089f-4701-87d3-4e0496b08750" (UID: "7f4c2c03-089f-4701-87d3-4e0496b08750"). InnerVolumeSpecName "kube-api-access-8v6z7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:00:24.482512 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:24.482492 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a130ae9f-0e32-4b92-b6d3-597640f7e802-kube-api-access-gqvth" pod "a130ae9f-0e32-4b92-b6d3-597640f7e802" (UID: "a130ae9f-0e32-4b92-b6d3-597640f7e802"). InnerVolumeSpecName "kube-api-access-gqvth". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:00:24.484113 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:24.484092 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94bed6ea-d66f-4f63-8859-a5a8d1969a01-kube-api-access-krtdq" pod "94bed6ea-d66f-4f63-8859-a5a8d1969a01" (UID: "94bed6ea-d66f-4f63-8859-a5a8d1969a01"). InnerVolumeSpecName "kube-api-access-krtdq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:00:24.580998 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:24.580961 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8v6z7\" (UniqueName: \"kubernetes.io/projected/7f4c2c03-089f-4701-87d3-4e0496b08750-kube-api-access-8v6z7\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:00:24.580998 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:24.580993 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqvth\" (UniqueName: \"kubernetes.io/projected/a130ae9f-0e32-4b92-b6d3-597640f7e802-kube-api-access-gqvth\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:00:24.580998 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:24.581003 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94bed6ea-d66f-4f63-8859-a5a8d1969a01-maas-api-service-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:00:24.581290 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:24.581014 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krtdq\" (UniqueName: \"kubernetes.io/projected/94bed6ea-d66f-4f63-8859-a5a8d1969a01-kube-api-access-krtdq\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:00:24.581290 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:24.581023 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a130ae9f-0e32-4b92-b6d3-597640f7e802-maas-api-service-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:00:24.581290 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:24.581031 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f4c2c03-089f-4701-87d3-4e0496b08750-maas-api-service-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:00:24.677306 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:24.677203 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6b67f8-29889420-pt2lw"] Oct 30 13:00:24.979665 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:24.979628 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6b67f8-29889420-pt2lw" Oct 30 13:00:25.030736 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:25.030696 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7bd05c5-f726-4ad3-88bf-b3a351a409bb-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7bd05c5-f726-4ad3-88bf-b3a351a409bb-maas-api-service-ca\") pod \"a7bd05c5-f726-4ad3-88bf-b3a351a409bb\" (UID: \"a7bd05c5-f726-4ad3-88bf-b3a351a409bb\") " Oct 30 13:00:25.031016 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:25.030992 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7bd05c5-f726-4ad3-88bf-b3a351a409bb-kube-api-access-jhgfv\" (UniqueName: \"kubernetes.io/projected/a7bd05c5-f726-4ad3-88bf-b3a351a409bb-kube-api-access-jhgfv\") pod \"a7bd05c5-f726-4ad3-88bf-b3a351a409bb\" (UID: \"a7bd05c5-f726-4ad3-88bf-b3a351a409bb\") " Oct 30 13:00:25.034284 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:25.034233 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7bd05c5-f726-4ad3-88bf-b3a351a409bb-maas-api-service-ca" pod "a7bd05c5-f726-4ad3-88bf-b3a351a409bb" (UID: "a7bd05c5-f726-4ad3-88bf-b3a351a409bb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:00:25.036032 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:25.036004 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7bd05c5-f726-4ad3-88bf-b3a351a409bb-kube-api-access-jhgfv" pod "a7bd05c5-f726-4ad3-88bf-b3a351a409bb" (UID: "a7bd05c5-f726-4ad3-88bf-b3a351a409bb"). InnerVolumeSpecName "kube-api-access-jhgfv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:00:25.047517 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:25.047485 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6b67f8-29889420-pt2lw" Oct 30 13:00:25.047636 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:25.047487 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6b67f8-29889420-pt2lw" event={"ID":"a7bd05c5-f726-4ad3-88bf-b3a351a409bb","Type":"ContainerDied","Data":"1e89b15fefb97c237a3e16fed867f31b2e9b8db91fcca87f84632c13fa1156d6"} Oct 30 13:00:25.047710 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:25.047664 4908 scope.go:122] "RemoveContainer" containerID="3bcd34c02206ba70d1e06ea09f1e520b64d2a70331e14deb7c8b5ef686e82703" Oct 30 13:00:25.053205 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:25.053182 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1333c8-29889420-9zjhg" Oct 30 13:00:25.053335 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:25.053216 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1333c8-29889420-9zjhg" event={"ID":"94bed6ea-d66f-4f63-8859-a5a8d1969a01","Type":"ContainerDied","Data":"9e432b4bdccd9305c88f99cc2a0d907fd7611f0539c79c827ee9310cda914fa2"} Oct 30 13:00:25.055122 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:25.055094 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-327e31-29889420-2vbrc" Oct 30 13:00:25.055258 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:25.055160 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-327e31-29889420-2vbrc" event={"ID":"53e599d0-de7d-4c2f-86f9-97cc39405bc0","Type":"ContainerDied","Data":"45d8d9e47c598e40b01ee4c2cdb2f276b87cc730a1277d99f75955b81f102044"} Oct 30 13:00:25.062197 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:25.062176 4908 scope.go:122] "RemoveContainer" containerID="7a4dcf1e8c2048091150357906372c27e54b35d8e6d386eaf82f15f13f26bb1b" Oct 30 13:00:25.067010 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:25.066978 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2675c0-29889420-hqgp5" event={"ID":"7f4c2c03-089f-4701-87d3-4e0496b08750","Type":"ContainerDied","Data":"c52d54912c2428f6b3d8013053604e55b5febfaf57164b299ab190aa06c15134"} Oct 30 13:00:25.067569 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:25.067247 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2675c0-29889420-hqgp5" Oct 30 13:00:25.069277 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:25.069250 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-9d45fb-29889420-5wlx7" event={"ID":"a130ae9f-0e32-4b92-b6d3-597640f7e802","Type":"ContainerDied","Data":"d96461af740976a95c0c0d28890dfcce381e6945b2f8d12ab755f6418020cb2f"} Oct 30 13:00:25.069438 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:25.069367 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-9d45fb-29889420-5wlx7" Oct 30 13:00:25.081691 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:25.081667 4908 scope.go:122] "RemoveContainer" containerID="d7db4f0218d1ed4f1222db34cc4acac8e04189280707cbb967be00df6551ea92" Oct 30 13:00:25.104937 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:25.104517 4908 scope.go:122] "RemoveContainer" containerID="730cebe4e2e1f45b85943a7edbddd63d2394b6ebc1c0b12f183f685160d286ac" Oct 30 13:00:25.119591 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:25.119568 4908 scope.go:122] "RemoveContainer" containerID="64db43b65040950f3c4d64c6e760cdfe2a4cecf1965fbb53ed466b70b55ed58a" Oct 30 13:00:25.134414 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:25.134350 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhgfv\" (UniqueName: \"kubernetes.io/projected/a7bd05c5-f726-4ad3-88bf-b3a351a409bb-kube-api-access-jhgfv\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:00:25.134414 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:25.134388 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7bd05c5-f726-4ad3-88bf-b3a351a409bb-maas-api-service-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:00:25.135521 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:25.135503 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1333c8-29889420-9zjhg"] Oct 30 13:00:25.140135 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:25.140110 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1333c8-29889420-9zjhg"] Oct 30 13:00:25.210727 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:25.210629 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-327e31-29889420-2vbrc"] Oct 30 13:00:25.215051 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:25.215016 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-327e31-29889420-2vbrc"] Oct 30 13:00:25.245687 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:25.245653 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6b67f8-29889420-pt2lw"] Oct 30 13:00:25.250514 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:25.250472 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6b67f8-29889420-pt2lw"] Oct 30 13:00:25.253673 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:25.253652 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f7464f-29889420-z9mg6" Oct 30 13:00:25.280585 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:25.280539 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-9d45fb-29889420-5wlx7"] Oct 30 13:00:25.292556 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:25.292515 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-9d45fb-29889420-5wlx7"] Oct 30 13:00:25.316442 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:25.316399 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2675c0-29889420-hqgp5"] Oct 30 13:00:25.324449 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:25.324408 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2675c0-29889420-hqgp5"] Oct 30 13:00:25.338019 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:25.337979 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94f72c00-8f0f-4bcf-9218-cd64b7773d1f-kube-api-access-w9rmm\" (UniqueName: \"kubernetes.io/projected/94f72c00-8f0f-4bcf-9218-cd64b7773d1f-kube-api-access-w9rmm\") pod \"94f72c00-8f0f-4bcf-9218-cd64b7773d1f\" (UID: \"94f72c00-8f0f-4bcf-9218-cd64b7773d1f\") " Oct 30 13:00:25.338186 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:25.338112 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94f72c00-8f0f-4bcf-9218-cd64b7773d1f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94f72c00-8f0f-4bcf-9218-cd64b7773d1f-maas-api-service-ca\") pod \"94f72c00-8f0f-4bcf-9218-cd64b7773d1f\" (UID: \"94f72c00-8f0f-4bcf-9218-cd64b7773d1f\") " Oct 30 13:00:25.338506 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:25.338450 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94f72c00-8f0f-4bcf-9218-cd64b7773d1f-maas-api-service-ca" pod "94f72c00-8f0f-4bcf-9218-cd64b7773d1f" (UID: "94f72c00-8f0f-4bcf-9218-cd64b7773d1f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:00:25.338974 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:25.338951 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94f72c00-8f0f-4bcf-9218-cd64b7773d1f-maas-api-service-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:00:25.340428 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:25.340400 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94f72c00-8f0f-4bcf-9218-cd64b7773d1f-kube-api-access-w9rmm" pod "94f72c00-8f0f-4bcf-9218-cd64b7773d1f" (UID: "94f72c00-8f0f-4bcf-9218-cd64b7773d1f"). InnerVolumeSpecName "kube-api-access-w9rmm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:00:25.440571 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:25.440532 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9rmm\" (UniqueName: \"kubernetes.io/projected/94f72c00-8f0f-4bcf-9218-cd64b7773d1f-kube-api-access-w9rmm\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:00:25.609415 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:25.609376 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1daed017-3390-4a05-9457-8d6c97919043" path="/var/lib/kubelet/pods/1daed017-3390-4a05-9457-8d6c97919043/volumes" Oct 30 13:00:25.609945 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:25.609929 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29e5414c-9b69-4713-8b27-80af3af5b4c7" path="/var/lib/kubelet/pods/29e5414c-9b69-4713-8b27-80af3af5b4c7/volumes" Oct 30 13:00:25.610373 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:25.610359 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53e599d0-de7d-4c2f-86f9-97cc39405bc0" path="/var/lib/kubelet/pods/53e599d0-de7d-4c2f-86f9-97cc39405bc0/volumes" Oct 30 13:00:25.611132 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:25.611115 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76cf17b7-c8b9-48d8-934c-f38aa9d1f008" path="/var/lib/kubelet/pods/76cf17b7-c8b9-48d8-934c-f38aa9d1f008/volumes" Oct 30 13:00:25.611547 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:25.611533 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f4c2c03-089f-4701-87d3-4e0496b08750" path="/var/lib/kubelet/pods/7f4c2c03-089f-4701-87d3-4e0496b08750/volumes" Oct 30 13:00:25.611974 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:25.611959 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94bed6ea-d66f-4f63-8859-a5a8d1969a01" path="/var/lib/kubelet/pods/94bed6ea-d66f-4f63-8859-a5a8d1969a01/volumes" Oct 30 13:00:25.612652 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:25.612636 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a130ae9f-0e32-4b92-b6d3-597640f7e802" path="/var/lib/kubelet/pods/a130ae9f-0e32-4b92-b6d3-597640f7e802/volumes" Oct 30 13:00:25.613066 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:25.613052 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7bd05c5-f726-4ad3-88bf-b3a351a409bb" path="/var/lib/kubelet/pods/a7bd05c5-f726-4ad3-88bf-b3a351a409bb/volumes" Oct 30 13:00:25.613458 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:25.613446 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf7295f0-efb3-487a-926c-c4f19fddde26" path="/var/lib/kubelet/pods/bf7295f0-efb3-487a-926c-c4f19fddde26/volumes" Oct 30 13:00:25.614159 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:25.614134 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb87d307-def9-4d16-8e7a-54b531f0d2c5" path="/var/lib/kubelet/pods/cb87d307-def9-4d16-8e7a-54b531f0d2c5/volumes" Oct 30 13:00:26.082819 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:26.082776 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f7464f-29889420-z9mg6" event={"ID":"94f72c00-8f0f-4bcf-9218-cd64b7773d1f","Type":"ContainerDied","Data":"2b260926cc3df5a112845dc372bbef135eb1285b7c6743a00e55e92eaefcc7f1"} Oct 30 13:00:26.082819 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:26.082786 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f7464f-29889420-z9mg6" Oct 30 13:00:26.083420 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:26.082899 4908 scope.go:122] "RemoveContainer" containerID="040854c5c1cc903e60e075ed19e264b56b65133c897ab8b72ce5277d0b88b69a" Oct 30 13:00:26.133141 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:26.133099 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f7464f-29889420-z9mg6"] Oct 30 13:00:26.144316 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:26.144283 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f7464f-29889420-z9mg6"] Oct 30 13:00:27.609364 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:00:27.609331 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94f72c00-8f0f-4bcf-9218-cd64b7773d1f" path="/var/lib/kubelet/pods/94f72c00-8f0f-4bcf-9218-cd64b7773d1f/volumes" Oct 30 13:00:49.687753 ip-10-0-141-251 kubenswrapper[4908]: E1030 13:00:49.687659 4908 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vn6vv" containerName="extractor" Oct 30 13:01:08.878104 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:08.878062 4908 scope.go:122] "RemoveContainer" containerID="03586bfef5de1e35c8f5ba0bc84d8d26d39255b0ec477ec6bbf1fca09e36c93e" Oct 30 13:01:08.889895 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:08.889868 4908 scope.go:122] "RemoveContainer" containerID="e2b110faabc6ddcec4e788fc67643af0c14e892227da86bb30e0d9905b31f103" Oct 30 13:01:08.900945 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:08.900920 4908 scope.go:122] "RemoveContainer" containerID="aef7cea00feba7d1bf708a6ca9a5ba0b5a15efebe16ef79d4272e56ddb2cd87a" Oct 30 13:01:08.912672 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:08.912643 4908 scope.go:122] "RemoveContainer" containerID="15ca7ceefb401dfd5c367a43bebcd9bb81e686557cef9c5fe612fe52003ae561" Oct 30 13:01:19.033821 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:19.033783 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e8e9b9f-4wkgh"] Oct 30 13:01:19.044299 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:19.044263 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e8e9b9f-4wkgh"] Oct 30 13:01:19.611946 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:19.611908 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="085a75a0-26d8-4f54-97aa-cbf5536cfcf6" path="/var/lib/kubelet/pods/085a75a0-26d8-4f54-97aa-cbf5536cfcf6/volumes" Oct 30 13:01:19.694204 ip-10-0-141-251 kubenswrapper[4908]: E1030 13:01:19.694170 4908 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vn6vv" containerName="extractor" Oct 30 13:01:27.352724 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:27.352682 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9wwfc"] Oct 30 13:01:31.810468 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.810427 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd707416-8fx5n"] Oct 30 13:01:31.811318 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.811292 4908 cpu_manager.go:414] "containerMap: removing container" podUID="1bb2e64a-3fd0-41bb-aafa-8f234fb2ffa2" containerName="cleanup" Oct 30 13:01:31.811318 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.811318 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1bb2e64a-3fd0-41bb-aafa-8f234fb2ffa2" containerName="cleanup" Oct 30 13:01:31.811497 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.811333 4908 cpu_manager.go:414] "containerMap: removing container" podUID="76cf17b7-c8b9-48d8-934c-f38aa9d1f008" containerName="cleanup" Oct 30 13:01:31.811497 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.811340 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76cf17b7-c8b9-48d8-934c-f38aa9d1f008" containerName="cleanup" Oct 30 13:01:31.811497 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.811356 4908 cpu_manager.go:414] "containerMap: removing container" podUID="71805e6f-86c5-4659-aa70-849ba38f288c" containerName="cleanup" Oct 30 13:01:31.811497 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.811364 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71805e6f-86c5-4659-aa70-849ba38f288c" containerName="cleanup" Oct 30 13:01:31.811497 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.811378 4908 cpu_manager.go:414] "containerMap: removing container" podUID="1baa9cd5-0305-4f77-b7cf-f4751053ace8" containerName="cleanup" Oct 30 13:01:31.811497 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.811385 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1baa9cd5-0305-4f77-b7cf-f4751053ace8" containerName="cleanup" Oct 30 13:01:31.811497 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.811399 4908 cpu_manager.go:414] "containerMap: removing container" podUID="7f4c2c03-089f-4701-87d3-4e0496b08750" containerName="cleanup" Oct 30 13:01:31.811497 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.811405 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f4c2c03-089f-4701-87d3-4e0496b08750" containerName="cleanup" Oct 30 13:01:31.811497 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.811414 4908 cpu_manager.go:414] "containerMap: removing container" podUID="a7bd05c5-f726-4ad3-88bf-b3a351a409bb" containerName="cleanup" Oct 30 13:01:31.811497 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.811420 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7bd05c5-f726-4ad3-88bf-b3a351a409bb" containerName="cleanup" Oct 30 13:01:31.811497 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.811429 4908 cpu_manager.go:414] "containerMap: removing container" podUID="0d506c31-90a7-493f-8b73-649d1ffcd0eb" containerName="cleanup" Oct 30 13:01:31.811497 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.811435 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d506c31-90a7-493f-8b73-649d1ffcd0eb" containerName="cleanup" Oct 30 13:01:31.811497 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.811445 4908 cpu_manager.go:414] "containerMap: removing container" podUID="94bed6ea-d66f-4f63-8859-a5a8d1969a01" containerName="cleanup" Oct 30 13:01:31.811497 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.811452 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94bed6ea-d66f-4f63-8859-a5a8d1969a01" containerName="cleanup" Oct 30 13:01:31.811497 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.811464 4908 cpu_manager.go:414] "containerMap: removing container" podUID="a130ae9f-0e32-4b92-b6d3-597640f7e802" containerName="cleanup" Oct 30 13:01:31.811497 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.811471 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a130ae9f-0e32-4b92-b6d3-597640f7e802" containerName="cleanup" Oct 30 13:01:31.811497 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.811481 4908 cpu_manager.go:414] "containerMap: removing container" podUID="1baa9cd5-0305-4f77-b7cf-f4751053ace8" containerName="cleanup" Oct 30 13:01:31.811497 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.811488 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1baa9cd5-0305-4f77-b7cf-f4751053ace8" containerName="cleanup" Oct 30 13:01:31.811497 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.811497 4908 cpu_manager.go:414] "containerMap: removing container" podUID="1daed017-3390-4a05-9457-8d6c97919043" containerName="cleanup" Oct 30 13:01:31.811497 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.811504 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1daed017-3390-4a05-9457-8d6c97919043" containerName="cleanup" Oct 30 13:01:31.812666 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.811513 4908 cpu_manager.go:414] "containerMap: removing container" podUID="347ca826-af8d-496d-bd0b-dca4071b3cd8" containerName="cleanup" Oct 30 13:01:31.812666 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.811520 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="347ca826-af8d-496d-bd0b-dca4071b3cd8" containerName="cleanup" Oct 30 13:01:31.812666 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.811543 4908 cpu_manager.go:414] "containerMap: removing container" podUID="ced43030-57a2-418d-bf86-f9c5a3b8cd95" containerName="cleanup" Oct 30 13:01:31.812666 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.811552 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ced43030-57a2-418d-bf86-f9c5a3b8cd95" containerName="cleanup" Oct 30 13:01:31.812666 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.811561 4908 cpu_manager.go:414] "containerMap: removing container" podUID="53e599d0-de7d-4c2f-86f9-97cc39405bc0" containerName="cleanup" Oct 30 13:01:31.812666 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.811568 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53e599d0-de7d-4c2f-86f9-97cc39405bc0" containerName="cleanup" Oct 30 13:01:31.812666 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.811578 4908 cpu_manager.go:414] "containerMap: removing container" podUID="1bb2e64a-3fd0-41bb-aafa-8f234fb2ffa2" containerName="cleanup" Oct 30 13:01:31.812666 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.811585 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1bb2e64a-3fd0-41bb-aafa-8f234fb2ffa2" containerName="cleanup" Oct 30 13:01:31.812666 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.811601 4908 cpu_manager.go:414] "containerMap: removing container" podUID="347ca826-af8d-496d-bd0b-dca4071b3cd8" containerName="cleanup" Oct 30 13:01:31.812666 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.811608 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="347ca826-af8d-496d-bd0b-dca4071b3cd8" containerName="cleanup" Oct 30 13:01:31.812666 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.811621 4908 cpu_manager.go:414] "containerMap: removing container" podUID="29e5414c-9b69-4713-8b27-80af3af5b4c7" containerName="cleanup" Oct 30 13:01:31.812666 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.811628 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29e5414c-9b69-4713-8b27-80af3af5b4c7" containerName="cleanup" Oct 30 13:01:31.812666 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.811637 4908 cpu_manager.go:414] "containerMap: removing container" podUID="a130ae9f-0e32-4b92-b6d3-597640f7e802" containerName="cleanup" Oct 30 13:01:31.812666 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.811644 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a130ae9f-0e32-4b92-b6d3-597640f7e802" containerName="cleanup" Oct 30 13:01:31.812666 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.811654 4908 cpu_manager.go:414] "containerMap: removing container" podUID="94f72c00-8f0f-4bcf-9218-cd64b7773d1f" containerName="cleanup" Oct 30 13:01:31.812666 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.811661 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94f72c00-8f0f-4bcf-9218-cd64b7773d1f" containerName="cleanup" Oct 30 13:01:31.812666 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.811672 4908 cpu_manager.go:414] "containerMap: removing container" podUID="bf7295f0-efb3-487a-926c-c4f19fddde26" containerName="cleanup" Oct 30 13:01:31.812666 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.811679 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf7295f0-efb3-487a-926c-c4f19fddde26" containerName="cleanup" Oct 30 13:01:31.812666 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.811689 4908 cpu_manager.go:414] "containerMap: removing container" podUID="b46f76c5-4139-4fbd-af18-5f35fcb1cb05" containerName="cleanup" Oct 30 13:01:31.812666 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.811698 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b46f76c5-4139-4fbd-af18-5f35fcb1cb05" containerName="cleanup" Oct 30 13:01:31.812666 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.811711 4908 cpu_manager.go:414] "containerMap: removing container" podUID="b46f76c5-4139-4fbd-af18-5f35fcb1cb05" containerName="cleanup" Oct 30 13:01:31.812666 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.811718 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b46f76c5-4139-4fbd-af18-5f35fcb1cb05" containerName="cleanup" Oct 30 13:01:31.812666 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.811729 4908 cpu_manager.go:414] "containerMap: removing container" podUID="94bed6ea-d66f-4f63-8859-a5a8d1969a01" containerName="cleanup" Oct 30 13:01:31.812666 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.811736 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94bed6ea-d66f-4f63-8859-a5a8d1969a01" containerName="cleanup" Oct 30 13:01:31.812666 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.811746 4908 cpu_manager.go:414] "containerMap: removing container" podUID="bf7295f0-efb3-487a-926c-c4f19fddde26" containerName="cleanup" Oct 30 13:01:31.812666 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.811753 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf7295f0-efb3-487a-926c-c4f19fddde26" containerName="cleanup" Oct 30 13:01:31.812666 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.811769 4908 cpu_manager.go:414] "containerMap: removing container" podUID="f2695f98-7dab-4cf3-aaa3-d1717ecc2d30" containerName="cleanup" Oct 30 13:01:31.812666 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.811777 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2695f98-7dab-4cf3-aaa3-d1717ecc2d30" containerName="cleanup" Oct 30 13:01:31.812666 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.811786 4908 cpu_manager.go:414] "containerMap: removing container" podUID="ced43030-57a2-418d-bf86-f9c5a3b8cd95" containerName="cleanup" Oct 30 13:01:31.812666 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.811794 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ced43030-57a2-418d-bf86-f9c5a3b8cd95" containerName="cleanup" Oct 30 13:01:31.812666 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.811806 4908 cpu_manager.go:414] "containerMap: removing container" podUID="a7bd05c5-f726-4ad3-88bf-b3a351a409bb" containerName="cleanup" Oct 30 13:01:31.812666 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.811814 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7bd05c5-f726-4ad3-88bf-b3a351a409bb" containerName="cleanup" Oct 30 13:01:31.812666 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.811841 4908 cpu_manager.go:414] "containerMap: removing container" podUID="f5af9446-e0d9-4edc-b01f-d049c4b2d3e7" containerName="cleanup" Oct 30 13:01:31.812666 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.811849 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5af9446-e0d9-4edc-b01f-d049c4b2d3e7" containerName="cleanup" Oct 30 13:01:31.812666 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.811859 4908 cpu_manager.go:414] "containerMap: removing container" podUID="a130ae9f-0e32-4b92-b6d3-597640f7e802" containerName="cleanup" Oct 30 13:01:31.812666 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.811866 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a130ae9f-0e32-4b92-b6d3-597640f7e802" containerName="cleanup" Oct 30 13:01:31.812666 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.811878 4908 cpu_manager.go:414] "containerMap: removing container" podUID="b46f76c5-4139-4fbd-af18-5f35fcb1cb05" containerName="cleanup" Oct 30 13:01:31.812666 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.811885 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b46f76c5-4139-4fbd-af18-5f35fcb1cb05" containerName="cleanup" Oct 30 13:01:31.812666 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.811894 4908 cpu_manager.go:414] "containerMap: removing container" podUID="7f4c2c03-089f-4701-87d3-4e0496b08750" containerName="cleanup" Oct 30 13:01:31.812666 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.811902 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f4c2c03-089f-4701-87d3-4e0496b08750" containerName="cleanup" Oct 30 13:01:31.812666 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.811917 4908 cpu_manager.go:414] "containerMap: removing container" podUID="29e5414c-9b69-4713-8b27-80af3af5b4c7" containerName="cleanup" Oct 30 13:01:31.812666 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.811925 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29e5414c-9b69-4713-8b27-80af3af5b4c7" containerName="cleanup" Oct 30 13:01:31.812666 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.811937 4908 cpu_manager.go:414] "containerMap: removing container" podUID="8165abf7-fbfb-4de2-a94c-5023a834835d" containerName="cleanup" Oct 30 13:01:31.812666 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.811945 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8165abf7-fbfb-4de2-a94c-5023a834835d" containerName="cleanup" Oct 30 13:01:31.812666 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.811959 4908 cpu_manager.go:414] "containerMap: removing container" podUID="7f4c2c03-089f-4701-87d3-4e0496b08750" containerName="cleanup" Oct 30 13:01:31.812666 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.811966 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f4c2c03-089f-4701-87d3-4e0496b08750" containerName="cleanup" Oct 30 13:01:31.812666 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.811975 4908 cpu_manager.go:414] "containerMap: removing container" podUID="1daed017-3390-4a05-9457-8d6c97919043" containerName="cleanup" Oct 30 13:01:31.812666 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.811982 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1daed017-3390-4a05-9457-8d6c97919043" containerName="cleanup" Oct 30 13:01:31.812666 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.811996 4908 cpu_manager.go:414] "containerMap: removing container" podUID="347ca826-af8d-496d-bd0b-dca4071b3cd8" containerName="cleanup" Oct 30 13:01:31.812666 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.812003 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="347ca826-af8d-496d-bd0b-dca4071b3cd8" containerName="cleanup" Oct 30 13:01:31.812666 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.812015 4908 cpu_manager.go:414] "containerMap: removing container" podUID="f5af9446-e0d9-4edc-b01f-d049c4b2d3e7" containerName="cleanup" Oct 30 13:01:31.812666 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.812023 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5af9446-e0d9-4edc-b01f-d049c4b2d3e7" containerName="cleanup" Oct 30 13:01:31.812666 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.812037 4908 cpu_manager.go:414] "containerMap: removing container" podUID="a7bd05c5-f726-4ad3-88bf-b3a351a409bb" containerName="cleanup" Oct 30 13:01:31.812666 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.812044 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7bd05c5-f726-4ad3-88bf-b3a351a409bb" containerName="cleanup" Oct 30 13:01:31.812666 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.812055 4908 cpu_manager.go:414] "containerMap: removing container" podUID="0d506c31-90a7-493f-8b73-649d1ffcd0eb" containerName="cleanup" Oct 30 13:01:31.812666 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.812063 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d506c31-90a7-493f-8b73-649d1ffcd0eb" containerName="cleanup" Oct 30 13:01:31.812666 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.812074 4908 cpu_manager.go:414] "containerMap: removing container" podUID="29e5414c-9b69-4713-8b27-80af3af5b4c7" containerName="cleanup" Oct 30 13:01:31.812666 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.812081 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29e5414c-9b69-4713-8b27-80af3af5b4c7" containerName="cleanup" Oct 30 13:01:31.812666 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.812098 4908 cpu_manager.go:414] "containerMap: removing container" podUID="1baa9cd5-0305-4f77-b7cf-f4751053ace8" containerName="cleanup" Oct 30 13:01:31.812666 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.812107 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1baa9cd5-0305-4f77-b7cf-f4751053ace8" containerName="cleanup" Oct 30 13:01:31.812666 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.812122 4908 cpu_manager.go:414] "containerMap: removing container" podUID="f2695f98-7dab-4cf3-aaa3-d1717ecc2d30" containerName="cleanup" Oct 30 13:01:31.812666 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.812129 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2695f98-7dab-4cf3-aaa3-d1717ecc2d30" containerName="cleanup" Oct 30 13:01:31.812666 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.812139 4908 cpu_manager.go:414] "containerMap: removing container" podUID="ced43030-57a2-418d-bf86-f9c5a3b8cd95" containerName="cleanup" Oct 30 13:01:31.812666 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.812146 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ced43030-57a2-418d-bf86-f9c5a3b8cd95" containerName="cleanup" Oct 30 13:01:31.812666 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.812159 4908 cpu_manager.go:414] "containerMap: removing container" podUID="71805e6f-86c5-4659-aa70-849ba38f288c" containerName="cleanup" Oct 30 13:01:31.812666 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.812166 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71805e6f-86c5-4659-aa70-849ba38f288c" containerName="cleanup" Oct 30 13:01:31.812666 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.812176 4908 cpu_manager.go:414] "containerMap: removing container" podUID="cb87d307-def9-4d16-8e7a-54b531f0d2c5" containerName="cleanup" Oct 30 13:01:31.812666 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.812183 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb87d307-def9-4d16-8e7a-54b531f0d2c5" containerName="cleanup" Oct 30 13:01:31.812666 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.812193 4908 cpu_manager.go:414] "containerMap: removing container" podUID="0d506c31-90a7-493f-8b73-649d1ffcd0eb" containerName="cleanup" Oct 30 13:01:31.812666 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.812200 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d506c31-90a7-493f-8b73-649d1ffcd0eb" containerName="cleanup" Oct 30 13:01:31.812666 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.812217 4908 cpu_manager.go:414] "containerMap: removing container" podUID="2674b41c-6ab9-4222-aa87-4fc2f5d9f468" containerName="cleanup" Oct 30 13:01:31.812666 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.812225 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2674b41c-6ab9-4222-aa87-4fc2f5d9f468" containerName="cleanup" Oct 30 13:01:31.812666 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.812239 4908 cpu_manager.go:414] "containerMap: removing container" podUID="8165abf7-fbfb-4de2-a94c-5023a834835d" containerName="cleanup" Oct 30 13:01:31.812666 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.812246 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8165abf7-fbfb-4de2-a94c-5023a834835d" containerName="cleanup" Oct 30 13:01:31.812666 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.812257 4908 cpu_manager.go:414] "containerMap: removing container" podUID="1daed017-3390-4a05-9457-8d6c97919043" containerName="cleanup" Oct 30 13:01:31.812666 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.812264 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1daed017-3390-4a05-9457-8d6c97919043" containerName="cleanup" Oct 30 13:01:31.812666 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.812280 4908 cpu_manager.go:414] "containerMap: removing container" podUID="1bb2e64a-3fd0-41bb-aafa-8f234fb2ffa2" containerName="cleanup" Oct 30 13:01:31.812666 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.812287 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1bb2e64a-3fd0-41bb-aafa-8f234fb2ffa2" containerName="cleanup" Oct 30 13:01:31.812666 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.812297 4908 cpu_manager.go:414] "containerMap: removing container" podUID="76cf17b7-c8b9-48d8-934c-f38aa9d1f008" containerName="cleanup" Oct 30 13:01:31.812666 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.812306 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76cf17b7-c8b9-48d8-934c-f38aa9d1f008" containerName="cleanup" Oct 30 13:01:31.812666 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.812320 4908 cpu_manager.go:414] "containerMap: removing container" podUID="f5af9446-e0d9-4edc-b01f-d049c4b2d3e7" containerName="cleanup" Oct 30 13:01:31.812666 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.812326 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5af9446-e0d9-4edc-b01f-d049c4b2d3e7" containerName="cleanup" Oct 30 13:01:31.812666 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.812339 4908 cpu_manager.go:414] "containerMap: removing container" podUID="2674b41c-6ab9-4222-aa87-4fc2f5d9f468" containerName="cleanup" Oct 30 13:01:31.812666 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.812347 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2674b41c-6ab9-4222-aa87-4fc2f5d9f468" containerName="cleanup" Oct 30 13:01:31.812666 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.812358 4908 cpu_manager.go:414] "containerMap: removing container" podUID="8165abf7-fbfb-4de2-a94c-5023a834835d" containerName="cleanup" Oct 30 13:01:31.812666 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.812366 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8165abf7-fbfb-4de2-a94c-5023a834835d" containerName="cleanup" Oct 30 13:01:31.812666 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.812377 4908 cpu_manager.go:414] "containerMap: removing container" podUID="cb87d307-def9-4d16-8e7a-54b531f0d2c5" containerName="cleanup" Oct 30 13:01:31.812666 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.812383 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb87d307-def9-4d16-8e7a-54b531f0d2c5" containerName="cleanup" Oct 30 13:01:31.812666 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.812393 4908 cpu_manager.go:414] "containerMap: removing container" podUID="2674b41c-6ab9-4222-aa87-4fc2f5d9f468" containerName="cleanup" Oct 30 13:01:31.812666 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.812399 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2674b41c-6ab9-4222-aa87-4fc2f5d9f468" containerName="cleanup" Oct 30 13:01:31.812666 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.812416 4908 cpu_manager.go:414] "containerMap: removing container" podUID="53e599d0-de7d-4c2f-86f9-97cc39405bc0" containerName="cleanup" Oct 30 13:01:31.812666 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.812424 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53e599d0-de7d-4c2f-86f9-97cc39405bc0" containerName="cleanup" Oct 30 13:01:31.812666 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.812436 4908 cpu_manager.go:414] "containerMap: removing container" podUID="94f72c00-8f0f-4bcf-9218-cd64b7773d1f" containerName="cleanup" Oct 30 13:01:31.812666 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.812443 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94f72c00-8f0f-4bcf-9218-cd64b7773d1f" containerName="cleanup" Oct 30 13:01:31.812666 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.812649 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="94bed6ea-d66f-4f63-8859-a5a8d1969a01" containerName="cleanup" Oct 30 13:01:31.812666 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.812666 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="a130ae9f-0e32-4b92-b6d3-597640f7e802" containerName="cleanup" Oct 30 13:01:31.812666 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.812676 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="1baa9cd5-0305-4f77-b7cf-f4751053ace8" containerName="cleanup" Oct 30 13:01:31.812666 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.812686 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="1daed017-3390-4a05-9457-8d6c97919043" containerName="cleanup" Oct 30 13:01:31.812666 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.812695 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="347ca826-af8d-496d-bd0b-dca4071b3cd8" containerName="cleanup" Oct 30 13:01:31.812666 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.812703 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="ced43030-57a2-418d-bf86-f9c5a3b8cd95" containerName="cleanup" Oct 30 13:01:31.812666 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.812711 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="53e599d0-de7d-4c2f-86f9-97cc39405bc0" containerName="cleanup" Oct 30 13:01:31.812666 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.812719 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="1bb2e64a-3fd0-41bb-aafa-8f234fb2ffa2" containerName="cleanup" Oct 30 13:01:31.812666 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.812728 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="347ca826-af8d-496d-bd0b-dca4071b3cd8" containerName="cleanup" Oct 30 13:01:31.817045 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.812738 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="29e5414c-9b69-4713-8b27-80af3af5b4c7" containerName="cleanup" Oct 30 13:01:31.817045 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.812746 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="a130ae9f-0e32-4b92-b6d3-597640f7e802" containerName="cleanup" Oct 30 13:01:31.817045 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.812754 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="94f72c00-8f0f-4bcf-9218-cd64b7773d1f" containerName="cleanup" Oct 30 13:01:31.817045 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.812764 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf7295f0-efb3-487a-926c-c4f19fddde26" containerName="cleanup" Oct 30 13:01:31.817045 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.812774 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="b46f76c5-4139-4fbd-af18-5f35fcb1cb05" containerName="cleanup" Oct 30 13:01:31.817045 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.812784 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="b46f76c5-4139-4fbd-af18-5f35fcb1cb05" containerName="cleanup" Oct 30 13:01:31.817045 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.812794 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="94bed6ea-d66f-4f63-8859-a5a8d1969a01" containerName="cleanup" Oct 30 13:01:31.817045 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.812804 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf7295f0-efb3-487a-926c-c4f19fddde26" containerName="cleanup" Oct 30 13:01:31.817045 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.812813 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2695f98-7dab-4cf3-aaa3-d1717ecc2d30" containerName="cleanup" Oct 30 13:01:31.817045 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.812821 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="ced43030-57a2-418d-bf86-f9c5a3b8cd95" containerName="cleanup" Oct 30 13:01:31.817045 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.812849 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7bd05c5-f726-4ad3-88bf-b3a351a409bb" containerName="cleanup" Oct 30 13:01:31.817045 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.812857 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5af9446-e0d9-4edc-b01f-d049c4b2d3e7" containerName="cleanup" Oct 30 13:01:31.817045 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.812866 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="b46f76c5-4139-4fbd-af18-5f35fcb1cb05" containerName="cleanup" Oct 30 13:01:31.817045 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.812876 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f4c2c03-089f-4701-87d3-4e0496b08750" containerName="cleanup" Oct 30 13:01:31.817045 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.812885 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="29e5414c-9b69-4713-8b27-80af3af5b4c7" containerName="cleanup" Oct 30 13:01:31.817045 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.812898 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="8165abf7-fbfb-4de2-a94c-5023a834835d" containerName="cleanup" Oct 30 13:01:31.817045 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.812906 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2695f98-7dab-4cf3-aaa3-d1717ecc2d30" containerName="cleanup" Oct 30 13:01:31.817045 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.812916 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f4c2c03-089f-4701-87d3-4e0496b08750" containerName="cleanup" Oct 30 13:01:31.817045 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.812925 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="1daed017-3390-4a05-9457-8d6c97919043" containerName="cleanup" Oct 30 13:01:31.817045 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.812936 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5af9446-e0d9-4edc-b01f-d049c4b2d3e7" containerName="cleanup" Oct 30 13:01:31.817045 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.812945 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7bd05c5-f726-4ad3-88bf-b3a351a409bb" containerName="cleanup" Oct 30 13:01:31.817045 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.812955 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d506c31-90a7-493f-8b73-649d1ffcd0eb" containerName="cleanup" Oct 30 13:01:31.817045 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.812963 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="29e5414c-9b69-4713-8b27-80af3af5b4c7" containerName="cleanup" Oct 30 13:01:31.817045 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.812975 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="ced43030-57a2-418d-bf86-f9c5a3b8cd95" containerName="cleanup" Oct 30 13:01:31.817045 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.812983 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="53e599d0-de7d-4c2f-86f9-97cc39405bc0" containerName="cleanup" Oct 30 13:01:31.817045 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.812993 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="71805e6f-86c5-4659-aa70-849ba38f288c" containerName="cleanup" Oct 30 13:01:31.817045 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.813001 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb87d307-def9-4d16-8e7a-54b531f0d2c5" containerName="cleanup" Oct 30 13:01:31.817045 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.813010 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="76cf17b7-c8b9-48d8-934c-f38aa9d1f008" containerName="cleanup" Oct 30 13:01:31.817045 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.813017 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d506c31-90a7-493f-8b73-649d1ffcd0eb" containerName="cleanup" Oct 30 13:01:31.817045 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.813029 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="2674b41c-6ab9-4222-aa87-4fc2f5d9f468" containerName="cleanup" Oct 30 13:01:31.817045 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.813041 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="76cf17b7-c8b9-48d8-934c-f38aa9d1f008" containerName="cleanup" Oct 30 13:01:31.817045 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.813050 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="2674b41c-6ab9-4222-aa87-4fc2f5d9f468" containerName="cleanup" Oct 30 13:01:31.817045 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.813059 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="8165abf7-fbfb-4de2-a94c-5023a834835d" containerName="cleanup" Oct 30 13:01:31.817045 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.813069 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="2674b41c-6ab9-4222-aa87-4fc2f5d9f468" containerName="cleanup" Oct 30 13:01:31.817045 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.813081 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="94f72c00-8f0f-4bcf-9218-cd64b7773d1f" containerName="cleanup" Oct 30 13:01:31.817045 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.813091 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb87d307-def9-4d16-8e7a-54b531f0d2c5" containerName="cleanup" Oct 30 13:01:31.817045 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.813101 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="1bb2e64a-3fd0-41bb-aafa-8f234fb2ffa2" containerName="cleanup" Oct 30 13:01:31.817045 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.813110 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="71805e6f-86c5-4659-aa70-849ba38f288c" containerName="cleanup" Oct 30 13:01:31.817045 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.813120 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="1baa9cd5-0305-4f77-b7cf-f4751053ace8" containerName="cleanup" Oct 30 13:01:31.819279 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.817568 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd707416-8fx5n" Oct 30 13:01:31.852621 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.852588 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:01:31.863509 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.863256 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 13:01:31.863509 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.863386 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 13:01:31.863509 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.863478 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s75hq\"" Oct 30 13:01:31.870304 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.870273 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd707416-8fx5n"] Oct 30 13:01:31.963965 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.963927 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b64ae455-69c2-4294-b6e3-f892be94b6bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fd707416-8fx5n\" (UID: \"b64ae455-69c2-4294-b6e3-f892be94b6bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd707416-8fx5n" Oct 30 13:01:31.964210 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.964179 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b64ae455-69c2-4294-b6e3-f892be94b6bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fd707416-8fx5n\" (UID: \"b64ae455-69c2-4294-b6e3-f892be94b6bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd707416-8fx5n" Oct 30 13:01:31.964289 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:31.964274 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxtm7\" (UniqueName: \"kubernetes.io/projected/b64ae455-69c2-4294-b6e3-f892be94b6bb-kube-api-access-gxtm7\") pod \"maas-api-revoke-sub-fd707416-8fx5n\" (UID: \"b64ae455-69c2-4294-b6e3-f892be94b6bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd707416-8fx5n" Oct 30 13:01:32.066258 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:32.066159 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b64ae455-69c2-4294-b6e3-f892be94b6bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fd707416-8fx5n\" (UID: \"b64ae455-69c2-4294-b6e3-f892be94b6bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd707416-8fx5n" Oct 30 13:01:32.066258 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:32.066247 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxtm7\" (UniqueName: \"kubernetes.io/projected/b64ae455-69c2-4294-b6e3-f892be94b6bb-kube-api-access-gxtm7\") pod \"maas-api-revoke-sub-fd707416-8fx5n\" (UID: \"b64ae455-69c2-4294-b6e3-f892be94b6bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd707416-8fx5n" Oct 30 13:01:32.066508 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:32.066489 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b64ae455-69c2-4294-b6e3-f892be94b6bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fd707416-8fx5n\" (UID: \"b64ae455-69c2-4294-b6e3-f892be94b6bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd707416-8fx5n" Oct 30 13:01:32.066980 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:32.066952 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b64ae455-69c2-4294-b6e3-f892be94b6bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fd707416-8fx5n\" (UID: \"b64ae455-69c2-4294-b6e3-f892be94b6bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd707416-8fx5n" Oct 30 13:01:32.069050 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:32.069021 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b64ae455-69c2-4294-b6e3-f892be94b6bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fd707416-8fx5n\" (UID: \"b64ae455-69c2-4294-b6e3-f892be94b6bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd707416-8fx5n" Oct 30 13:01:32.090442 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:32.090408 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxtm7\" (UniqueName: \"kubernetes.io/projected/b64ae455-69c2-4294-b6e3-f892be94b6bb-kube-api-access-gxtm7\") pod \"maas-api-revoke-sub-fd707416-8fx5n\" (UID: \"b64ae455-69c2-4294-b6e3-f892be94b6bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd707416-8fx5n" Oct 30 13:01:32.131985 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:32.131947 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd707416-8fx5n" Oct 30 13:01:32.475538 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:32.475502 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd707416-8fx5n"] Oct 30 13:01:32.478014 ip-10-0-141-251 kubenswrapper[4908]: W1030 13:01:32.477972 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb64ae455_69c2_4294_b6e3_f892be94b6bb.slice/crio-8ace38999f3298b0cd69e5fcbb19708e58109e94699fdfb0ecd4279cd299cda3 WatchSource:0}: Error finding container 8ace38999f3298b0cd69e5fcbb19708e58109e94699fdfb0ecd4279cd299cda3: Status 404 returned error can't find the container with id 8ace38999f3298b0cd69e5fcbb19708e58109e94699fdfb0ecd4279cd299cda3 Oct 30 13:01:32.912559 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:32.912518 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd707416-8fx5n" event={"ID":"b64ae455-69c2-4294-b6e3-f892be94b6bb","Type":"ContainerStarted","Data":"a6e4c454d8461d86913f355f19f9349918e1ce555eaedebeaea1433a2f2dc9ca"} Oct 30 13:01:32.913051 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:32.912592 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd707416-8fx5n" event={"ID":"b64ae455-69c2-4294-b6e3-f892be94b6bb","Type":"ContainerStarted","Data":"8ace38999f3298b0cd69e5fcbb19708e58109e94699fdfb0ecd4279cd299cda3"} Oct 30 13:01:32.957919 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:32.957855 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd707416-8fx5n" podStartSLOduration=1.957821113 podStartE2EDuration="1.957821113s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:01:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:01:32.95321929 +0000 UTC m=+4049.467648037" watchObservedRunningTime="2026-10-30 13:01:32.957821113 +0000 UTC m=+4049.472249860" Oct 30 13:01:33.922190 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:33.922094 4908 generic.go:356] "Generic (PLEG): container finished" podID="b64ae455-69c2-4294-b6e3-f892be94b6bb" containerID="a6e4c454d8461d86913f355f19f9349918e1ce555eaedebeaea1433a2f2dc9ca" exitCode=0 Oct 30 13:01:33.922190 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:33.922179 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd707416-8fx5n" event={"ID":"b64ae455-69c2-4294-b6e3-f892be94b6bb","Type":"ContainerDied","Data":"a6e4c454d8461d86913f355f19f9349918e1ce555eaedebeaea1433a2f2dc9ca"} Oct 30 13:01:35.131227 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:35.131191 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd707416-8fx5n" Oct 30 13:01:35.227262 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:35.227229 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b64ae455-69c2-4294-b6e3-f892be94b6bb-kube-api-access-gxtm7\" (UniqueName: \"kubernetes.io/projected/b64ae455-69c2-4294-b6e3-f892be94b6bb-kube-api-access-gxtm7\") pod \"b64ae455-69c2-4294-b6e3-f892be94b6bb\" (UID: \"b64ae455-69c2-4294-b6e3-f892be94b6bb\") " Oct 30 13:01:35.227592 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:35.227572 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b64ae455-69c2-4294-b6e3-f892be94b6bb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b64ae455-69c2-4294-b6e3-f892be94b6bb-maas-api-openshift-ca\") pod \"b64ae455-69c2-4294-b6e3-f892be94b6bb\" (UID: \"b64ae455-69c2-4294-b6e3-f892be94b6bb\") " Oct 30 13:01:35.227786 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:35.227685 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b64ae455-69c2-4294-b6e3-f892be94b6bb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b64ae455-69c2-4294-b6e3-f892be94b6bb-maas-api-xks-ca\") pod \"b64ae455-69c2-4294-b6e3-f892be94b6bb\" (UID: \"b64ae455-69c2-4294-b6e3-f892be94b6bb\") " Oct 30 13:01:35.228055 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:35.228025 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b64ae455-69c2-4294-b6e3-f892be94b6bb-maas-api-openshift-ca" pod "b64ae455-69c2-4294-b6e3-f892be94b6bb" (UID: "b64ae455-69c2-4294-b6e3-f892be94b6bb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:01:35.228726 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:35.228703 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b64ae455-69c2-4294-b6e3-f892be94b6bb-maas-api-openshift-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:01:35.230433 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:35.230405 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b64ae455-69c2-4294-b6e3-f892be94b6bb-kube-api-access-gxtm7" pod "b64ae455-69c2-4294-b6e3-f892be94b6bb" (UID: "b64ae455-69c2-4294-b6e3-f892be94b6bb"). InnerVolumeSpecName "kube-api-access-gxtm7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:01:35.230661 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:35.230625 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b64ae455-69c2-4294-b6e3-f892be94b6bb-maas-api-xks-ca" pod "b64ae455-69c2-4294-b6e3-f892be94b6bb" (UID: "b64ae455-69c2-4294-b6e3-f892be94b6bb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:01:35.331011 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:35.330966 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b64ae455-69c2-4294-b6e3-f892be94b6bb-maas-api-xks-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:01:35.331011 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:35.331009 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gxtm7\" (UniqueName: \"kubernetes.io/projected/b64ae455-69c2-4294-b6e3-f892be94b6bb-kube-api-access-gxtm7\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:01:35.939697 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:35.939608 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd707416-8fx5n" Oct 30 13:01:35.939697 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:35.939647 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd707416-8fx5n" event={"ID":"b64ae455-69c2-4294-b6e3-f892be94b6bb","Type":"ContainerDied","Data":"8ace38999f3298b0cd69e5fcbb19708e58109e94699fdfb0ecd4279cd299cda3"} Oct 30 13:01:35.939953 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:35.939713 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8ace38999f3298b0cd69e5fcbb19708e58109e94699fdfb0ecd4279cd299cda3" Oct 30 13:01:42.626812 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:42.626766 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7f73653-99bl7"] Oct 30 13:01:42.627637 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:42.627610 4908 cpu_manager.go:414] "containerMap: removing container" podUID="71805e6f-86c5-4659-aa70-849ba38f288c" containerName="cleanup" Oct 30 13:01:42.627637 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:42.627637 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71805e6f-86c5-4659-aa70-849ba38f288c" containerName="cleanup" Oct 30 13:01:42.627819 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:42.627667 4908 cpu_manager.go:414] "containerMap: removing container" podUID="76cf17b7-c8b9-48d8-934c-f38aa9d1f008" containerName="cleanup" Oct 30 13:01:42.627819 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:42.627675 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76cf17b7-c8b9-48d8-934c-f38aa9d1f008" containerName="cleanup" Oct 30 13:01:42.627819 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:42.627693 4908 cpu_manager.go:414] "containerMap: removing container" podUID="cb87d307-def9-4d16-8e7a-54b531f0d2c5" containerName="cleanup" Oct 30 13:01:42.627819 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:42.627701 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb87d307-def9-4d16-8e7a-54b531f0d2c5" containerName="cleanup" Oct 30 13:01:42.627819 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:42.627721 4908 cpu_manager.go:414] "containerMap: removing container" podUID="b64ae455-69c2-4294-b6e3-f892be94b6bb" containerName="revoke-keys" Oct 30 13:01:42.627819 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:42.627729 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b64ae455-69c2-4294-b6e3-f892be94b6bb" containerName="revoke-keys" Oct 30 13:01:42.627819 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:42.627758 4908 cpu_manager.go:414] "containerMap: removing container" podUID="53e599d0-de7d-4c2f-86f9-97cc39405bc0" containerName="cleanup" Oct 30 13:01:42.627819 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:42.627765 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53e599d0-de7d-4c2f-86f9-97cc39405bc0" containerName="cleanup" Oct 30 13:01:42.627819 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:42.627785 4908 cpu_manager.go:414] "containerMap: removing container" podUID="94f72c00-8f0f-4bcf-9218-cd64b7773d1f" containerName="cleanup" Oct 30 13:01:42.627819 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:42.627793 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94f72c00-8f0f-4bcf-9218-cd64b7773d1f" containerName="cleanup" Oct 30 13:01:42.627819 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:42.627808 4908 cpu_manager.go:414] "containerMap: removing container" podUID="94bed6ea-d66f-4f63-8859-a5a8d1969a01" containerName="cleanup" Oct 30 13:01:42.627819 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:42.627815 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94bed6ea-d66f-4f63-8859-a5a8d1969a01" containerName="cleanup" Oct 30 13:01:42.627819 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:42.627840 4908 cpu_manager.go:414] "containerMap: removing container" podUID="bf7295f0-efb3-487a-926c-c4f19fddde26" containerName="cleanup" Oct 30 13:01:42.628531 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:42.627854 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf7295f0-efb3-487a-926c-c4f19fddde26" containerName="cleanup" Oct 30 13:01:42.628531 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:42.627868 4908 cpu_manager.go:414] "containerMap: removing container" podUID="f2695f98-7dab-4cf3-aaa3-d1717ecc2d30" containerName="cleanup" Oct 30 13:01:42.628531 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:42.627875 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2695f98-7dab-4cf3-aaa3-d1717ecc2d30" containerName="cleanup" Oct 30 13:01:42.628531 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:42.628047 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="76cf17b7-c8b9-48d8-934c-f38aa9d1f008" containerName="cleanup" Oct 30 13:01:42.628531 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:42.628063 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="8165abf7-fbfb-4de2-a94c-5023a834835d" containerName="cleanup" Oct 30 13:01:42.628531 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:42.628079 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="94f72c00-8f0f-4bcf-9218-cd64b7773d1f" containerName="cleanup" Oct 30 13:01:42.628531 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:42.628093 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="b64ae455-69c2-4294-b6e3-f892be94b6bb" containerName="revoke-keys" Oct 30 13:01:42.628531 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:42.628103 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb87d307-def9-4d16-8e7a-54b531f0d2c5" containerName="cleanup" Oct 30 13:01:42.628531 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:42.628114 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="71805e6f-86c5-4659-aa70-849ba38f288c" containerName="cleanup" Oct 30 13:01:42.628531 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:42.628127 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="1baa9cd5-0305-4f77-b7cf-f4751053ace8" containerName="cleanup" Oct 30 13:01:42.628531 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:42.628136 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="1bb2e64a-3fd0-41bb-aafa-8f234fb2ffa2" containerName="cleanup" Oct 30 13:01:42.628531 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:42.628146 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="347ca826-af8d-496d-bd0b-dca4071b3cd8" containerName="cleanup" Oct 30 13:01:42.628531 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:42.628154 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="a130ae9f-0e32-4b92-b6d3-597640f7e802" containerName="cleanup" Oct 30 13:01:42.628531 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:42.628164 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="94bed6ea-d66f-4f63-8859-a5a8d1969a01" containerName="cleanup" Oct 30 13:01:42.628531 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:42.628173 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf7295f0-efb3-487a-926c-c4f19fddde26" containerName="cleanup" Oct 30 13:01:42.628531 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:42.628185 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5af9446-e0d9-4edc-b01f-d049c4b2d3e7" containerName="cleanup" Oct 30 13:01:42.628531 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:42.628195 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2695f98-7dab-4cf3-aaa3-d1717ecc2d30" containerName="cleanup" Oct 30 13:01:42.628531 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:42.628204 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f4c2c03-089f-4701-87d3-4e0496b08750" containerName="cleanup" Oct 30 13:01:42.628531 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:42.628213 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="1daed017-3390-4a05-9457-8d6c97919043" containerName="cleanup" Oct 30 13:01:42.628531 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:42.628224 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7bd05c5-f726-4ad3-88bf-b3a351a409bb" containerName="cleanup" Oct 30 13:01:42.628531 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:42.628234 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="53e599d0-de7d-4c2f-86f9-97cc39405bc0" containerName="cleanup" Oct 30 13:01:42.628531 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:42.628246 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d506c31-90a7-493f-8b73-649d1ffcd0eb" containerName="cleanup" Oct 30 13:01:42.632080 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:42.632057 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7f73653-99bl7" Oct 30 13:01:42.650339 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:42.650314 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7f73653-99bl7"] Oct 30 13:01:42.656264 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:42.656240 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:01:42.656469 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:42.656444 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 13:01:42.656561 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:42.656488 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 13:01:42.659413 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:42.659391 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s75hq\"" Oct 30 13:01:42.765660 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:42.765618 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxbxp\" (UniqueName: \"kubernetes.io/projected/eec2d564-368e-4413-9ccb-9010f03eb27c-kube-api-access-zxbxp\") pod \"maas-api-revoke-sub-d7f73653-99bl7\" (UID: \"eec2d564-368e-4413-9ccb-9010f03eb27c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7f73653-99bl7" Oct 30 13:01:42.765882 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:42.765804 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eec2d564-368e-4413-9ccb-9010f03eb27c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d7f73653-99bl7\" (UID: \"eec2d564-368e-4413-9ccb-9010f03eb27c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7f73653-99bl7" Oct 30 13:01:42.766005 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:42.765985 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eec2d564-368e-4413-9ccb-9010f03eb27c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d7f73653-99bl7\" (UID: \"eec2d564-368e-4413-9ccb-9010f03eb27c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7f73653-99bl7" Oct 30 13:01:42.867640 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:42.867597 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eec2d564-368e-4413-9ccb-9010f03eb27c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d7f73653-99bl7\" (UID: \"eec2d564-368e-4413-9ccb-9010f03eb27c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7f73653-99bl7" Oct 30 13:01:42.867920 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:42.867776 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxbxp\" (UniqueName: \"kubernetes.io/projected/eec2d564-368e-4413-9ccb-9010f03eb27c-kube-api-access-zxbxp\") pod \"maas-api-revoke-sub-d7f73653-99bl7\" (UID: \"eec2d564-368e-4413-9ccb-9010f03eb27c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7f73653-99bl7" Oct 30 13:01:42.868025 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:42.868002 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eec2d564-368e-4413-9ccb-9010f03eb27c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d7f73653-99bl7\" (UID: \"eec2d564-368e-4413-9ccb-9010f03eb27c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7f73653-99bl7" Oct 30 13:01:42.868477 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:42.868445 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eec2d564-368e-4413-9ccb-9010f03eb27c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d7f73653-99bl7\" (UID: \"eec2d564-368e-4413-9ccb-9010f03eb27c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7f73653-99bl7" Oct 30 13:01:42.870483 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:42.870459 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eec2d564-368e-4413-9ccb-9010f03eb27c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d7f73653-99bl7\" (UID: \"eec2d564-368e-4413-9ccb-9010f03eb27c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7f73653-99bl7" Oct 30 13:01:42.890552 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:42.890462 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxbxp\" (UniqueName: \"kubernetes.io/projected/eec2d564-368e-4413-9ccb-9010f03eb27c-kube-api-access-zxbxp\") pod \"maas-api-revoke-sub-d7f73653-99bl7\" (UID: \"eec2d564-368e-4413-9ccb-9010f03eb27c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7f73653-99bl7" Oct 30 13:01:42.946623 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:42.946587 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7f73653-99bl7" Oct 30 13:01:43.275170 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:43.275134 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7f73653-99bl7"] Oct 30 13:01:43.275317 ip-10-0-141-251 kubenswrapper[4908]: W1030 13:01:43.275195 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeec2d564_368e_4413_9ccb_9010f03eb27c.slice/crio-abd475f320f01b28c40e62198e14b21029583fb305dc0699b5fda11f066b087e WatchSource:0}: Error finding container abd475f320f01b28c40e62198e14b21029583fb305dc0699b5fda11f066b087e: Status 404 returned error can't find the container with id abd475f320f01b28c40e62198e14b21029583fb305dc0699b5fda11f066b087e Oct 30 13:01:44.054516 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:44.054415 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7f73653-99bl7" event={"ID":"eec2d564-368e-4413-9ccb-9010f03eb27c","Type":"ContainerStarted","Data":"4613f11c31a199df39f3648785fb5bf1e75c5a0267566c84f9ce8b29a7ed3729"} Oct 30 13:01:44.054974 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:44.054519 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7f73653-99bl7" event={"ID":"eec2d564-368e-4413-9ccb-9010f03eb27c","Type":"ContainerStarted","Data":"abd475f320f01b28c40e62198e14b21029583fb305dc0699b5fda11f066b087e"} Oct 30 13:01:44.099682 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:44.099210 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7f73653-99bl7" podStartSLOduration=2.099192779 podStartE2EDuration="2.099192779s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:01:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:01:44.098749776 +0000 UTC m=+4060.613178524" watchObservedRunningTime="2026-10-30 13:01:44.099192779 +0000 UTC m=+4060.613621529" Oct 30 13:01:45.140757 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:45.140720 4908 generic.go:356] "Generic (PLEG): container finished" podID="eec2d564-368e-4413-9ccb-9010f03eb27c" containerID="4613f11c31a199df39f3648785fb5bf1e75c5a0267566c84f9ce8b29a7ed3729" exitCode=0 Oct 30 13:01:45.141356 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:45.140787 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7f73653-99bl7" event={"ID":"eec2d564-368e-4413-9ccb-9010f03eb27c","Type":"ContainerDied","Data":"4613f11c31a199df39f3648785fb5bf1e75c5a0267566c84f9ce8b29a7ed3729"} Oct 30 13:01:46.334543 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:46.334514 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7f73653-99bl7" Oct 30 13:01:46.433896 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:46.433791 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eec2d564-368e-4413-9ccb-9010f03eb27c-kube-api-access-zxbxp\" (UniqueName: \"kubernetes.io/projected/eec2d564-368e-4413-9ccb-9010f03eb27c-kube-api-access-zxbxp\") pod \"eec2d564-368e-4413-9ccb-9010f03eb27c\" (UID: \"eec2d564-368e-4413-9ccb-9010f03eb27c\") " Oct 30 13:01:46.434080 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:46.434064 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eec2d564-368e-4413-9ccb-9010f03eb27c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eec2d564-368e-4413-9ccb-9010f03eb27c-maas-api-xks-ca\") pod \"eec2d564-368e-4413-9ccb-9010f03eb27c\" (UID: \"eec2d564-368e-4413-9ccb-9010f03eb27c\") " Oct 30 13:01:46.434336 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:46.434311 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eec2d564-368e-4413-9ccb-9010f03eb27c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eec2d564-368e-4413-9ccb-9010f03eb27c-maas-api-openshift-ca\") pod \"eec2d564-368e-4413-9ccb-9010f03eb27c\" (UID: \"eec2d564-368e-4413-9ccb-9010f03eb27c\") " Oct 30 13:01:46.436209 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:46.435461 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eec2d564-368e-4413-9ccb-9010f03eb27c-maas-api-openshift-ca" pod "eec2d564-368e-4413-9ccb-9010f03eb27c" (UID: "eec2d564-368e-4413-9ccb-9010f03eb27c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:01:46.437105 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:46.437077 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eec2d564-368e-4413-9ccb-9010f03eb27c-maas-api-xks-ca" pod "eec2d564-368e-4413-9ccb-9010f03eb27c" (UID: "eec2d564-368e-4413-9ccb-9010f03eb27c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:01:46.437250 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:46.437158 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eec2d564-368e-4413-9ccb-9010f03eb27c-kube-api-access-zxbxp" pod "eec2d564-368e-4413-9ccb-9010f03eb27c" (UID: "eec2d564-368e-4413-9ccb-9010f03eb27c"). InnerVolumeSpecName "kube-api-access-zxbxp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:01:46.536404 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:46.536366 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eec2d564-368e-4413-9ccb-9010f03eb27c-maas-api-xks-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:01:46.536404 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:46.536398 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eec2d564-368e-4413-9ccb-9010f03eb27c-maas-api-openshift-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:01:46.536404 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:46.536408 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxbxp\" (UniqueName: \"kubernetes.io/projected/eec2d564-368e-4413-9ccb-9010f03eb27c-kube-api-access-zxbxp\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:01:47.160634 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:47.160604 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7f73653-99bl7" Oct 30 13:01:47.160816 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:47.160597 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7f73653-99bl7" event={"ID":"eec2d564-368e-4413-9ccb-9010f03eb27c","Type":"ContainerDied","Data":"abd475f320f01b28c40e62198e14b21029583fb305dc0699b5fda11f066b087e"} Oct 30 13:01:47.160816 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:47.160738 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="abd475f320f01b28c40e62198e14b21029583fb305dc0699b5fda11f066b087e" Oct 30 13:01:49.689117 ip-10-0-141-251 kubenswrapper[4908]: E1030 13:01:49.689068 4908 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vn6vv" containerName="extractor" Oct 30 13:01:54.168814 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:54.168769 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09c34f4e-92lmw"] Oct 30 13:01:54.169457 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:54.169441 4908 cpu_manager.go:414] "containerMap: removing container" podUID="eec2d564-368e-4413-9ccb-9010f03eb27c" containerName="revoke-keys" Oct 30 13:01:54.169504 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:54.169458 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eec2d564-368e-4413-9ccb-9010f03eb27c" containerName="revoke-keys" Oct 30 13:01:54.169634 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:54.169622 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="eec2d564-368e-4413-9ccb-9010f03eb27c" containerName="revoke-keys" Oct 30 13:01:54.173067 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:54.173040 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09c34f4e-92lmw" Oct 30 13:01:54.196653 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:54.196614 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09c34f4e-92lmw"] Oct 30 13:01:54.200699 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:54.200665 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s75hq\"" Oct 30 13:01:54.200699 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:54.200685 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:01:54.200991 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:54.200766 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 13:01:54.201390 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:54.201371 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 13:01:54.292389 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:54.292334 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gsw8\" (UniqueName: \"kubernetes.io/projected/f8888a20-3251-4d90-b5ac-eb8ea3b81245-kube-api-access-4gsw8\") pod \"maas-api-revoke-sub-09c34f4e-92lmw\" (UID: \"f8888a20-3251-4d90-b5ac-eb8ea3b81245\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09c34f4e-92lmw" Oct 30 13:01:54.292661 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:54.292633 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8888a20-3251-4d90-b5ac-eb8ea3b81245-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-09c34f4e-92lmw\" (UID: \"f8888a20-3251-4d90-b5ac-eb8ea3b81245\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09c34f4e-92lmw" Oct 30 13:01:54.292789 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:54.292771 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8888a20-3251-4d90-b5ac-eb8ea3b81245-maas-api-xks-ca\") pod \"maas-api-revoke-sub-09c34f4e-92lmw\" (UID: \"f8888a20-3251-4d90-b5ac-eb8ea3b81245\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09c34f4e-92lmw" Oct 30 13:01:54.394614 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:54.394575 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8888a20-3251-4d90-b5ac-eb8ea3b81245-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-09c34f4e-92lmw\" (UID: \"f8888a20-3251-4d90-b5ac-eb8ea3b81245\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09c34f4e-92lmw" Oct 30 13:01:54.395988 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:54.395959 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8888a20-3251-4d90-b5ac-eb8ea3b81245-maas-api-xks-ca\") pod \"maas-api-revoke-sub-09c34f4e-92lmw\" (UID: \"f8888a20-3251-4d90-b5ac-eb8ea3b81245\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09c34f4e-92lmw" Oct 30 13:01:54.396810 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:54.396783 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gsw8\" (UniqueName: \"kubernetes.io/projected/f8888a20-3251-4d90-b5ac-eb8ea3b81245-kube-api-access-4gsw8\") pod \"maas-api-revoke-sub-09c34f4e-92lmw\" (UID: \"f8888a20-3251-4d90-b5ac-eb8ea3b81245\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09c34f4e-92lmw" Oct 30 13:01:54.397605 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:54.395741 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8888a20-3251-4d90-b5ac-eb8ea3b81245-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-09c34f4e-92lmw\" (UID: \"f8888a20-3251-4d90-b5ac-eb8ea3b81245\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09c34f4e-92lmw" Oct 30 13:01:54.420498 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:54.420396 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8888a20-3251-4d90-b5ac-eb8ea3b81245-maas-api-xks-ca\") pod \"maas-api-revoke-sub-09c34f4e-92lmw\" (UID: \"f8888a20-3251-4d90-b5ac-eb8ea3b81245\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09c34f4e-92lmw" Oct 30 13:01:54.429924 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:54.429841 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gsw8\" (UniqueName: \"kubernetes.io/projected/f8888a20-3251-4d90-b5ac-eb8ea3b81245-kube-api-access-4gsw8\") pod \"maas-api-revoke-sub-09c34f4e-92lmw\" (UID: \"f8888a20-3251-4d90-b5ac-eb8ea3b81245\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09c34f4e-92lmw" Oct 30 13:01:54.491278 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:54.491243 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09c34f4e-92lmw" Oct 30 13:01:55.020112 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:55.020072 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-tjnq5"] Oct 30 13:01:55.020641 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:55.020603 4908 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-tjnq5" podUID="c3e7bd6e-7283-451a-8df8-cc9634908630" containerName="istio-proxy" containerID="cri-o://6c617bec82991ea0e7d318e4505a289d67bffd53172b1f9794c5a407e8884554" gracePeriod=30 Oct 30 13:01:55.136309 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:55.136270 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09c34f4e-92lmw"] Oct 30 13:01:55.240911 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:55.240874 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09c34f4e-92lmw" event={"ID":"f8888a20-3251-4d90-b5ac-eb8ea3b81245","Type":"ContainerStarted","Data":"0fc013a06d2da104bc11d03003470f48ae5449705dce17795554cd3e9ee65ba2"} Oct 30 13:01:56.251755 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:56.251708 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09c34f4e-92lmw" event={"ID":"f8888a20-3251-4d90-b5ac-eb8ea3b81245","Type":"ContainerStarted","Data":"63f32d8866d4780cb8d688d273620c93e7f1c8a99a12774686a02713c32591ec"} Oct 30 13:01:56.317067 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:56.317011 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09c34f4e-92lmw" podStartSLOduration=2.316995947 podStartE2EDuration="2.316995947s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:01:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:01:56.314298096 +0000 UTC m=+4072.828726841" watchObservedRunningTime="2026-10-30 13:01:56.316995947 +0000 UTC m=+4072.831424731" Oct 30 13:01:56.448943 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:56.448901 4908 patch_prober.go:29] interesting pod/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-tjnq5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 13:01:56.449120 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:56.448971 4908 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-tjnq5" podUID="c3e7bd6e-7283-451a-8df8-cc9634908630" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 13:01:57.261713 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:57.261677 4908 generic.go:356] "Generic (PLEG): container finished" podID="f8888a20-3251-4d90-b5ac-eb8ea3b81245" containerID="63f32d8866d4780cb8d688d273620c93e7f1c8a99a12774686a02713c32591ec" exitCode=0 Oct 30 13:01:57.262254 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:57.261761 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09c34f4e-92lmw" event={"ID":"f8888a20-3251-4d90-b5ac-eb8ea3b81245","Type":"ContainerDied","Data":"63f32d8866d4780cb8d688d273620c93e7f1c8a99a12774686a02713c32591ec"} Oct 30 13:01:58.467218 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:58.467184 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09c34f4e-92lmw" Oct 30 13:01:58.553074 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:58.552936 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8888a20-3251-4d90-b5ac-eb8ea3b81245-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8888a20-3251-4d90-b5ac-eb8ea3b81245-maas-api-openshift-ca\") pod \"f8888a20-3251-4d90-b5ac-eb8ea3b81245\" (UID: \"f8888a20-3251-4d90-b5ac-eb8ea3b81245\") " Oct 30 13:01:58.553074 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:58.553027 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8888a20-3251-4d90-b5ac-eb8ea3b81245-kube-api-access-4gsw8\" (UniqueName: \"kubernetes.io/projected/f8888a20-3251-4d90-b5ac-eb8ea3b81245-kube-api-access-4gsw8\") pod \"f8888a20-3251-4d90-b5ac-eb8ea3b81245\" (UID: \"f8888a20-3251-4d90-b5ac-eb8ea3b81245\") " Oct 30 13:01:58.553374 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:58.553259 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f8888a20-3251-4d90-b5ac-eb8ea3b81245-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8888a20-3251-4d90-b5ac-eb8ea3b81245-maas-api-xks-ca\") pod \"f8888a20-3251-4d90-b5ac-eb8ea3b81245\" (UID: \"f8888a20-3251-4d90-b5ac-eb8ea3b81245\") " Oct 30 13:01:58.554073 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:58.554045 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8888a20-3251-4d90-b5ac-eb8ea3b81245-maas-api-openshift-ca" pod "f8888a20-3251-4d90-b5ac-eb8ea3b81245" (UID: "f8888a20-3251-4d90-b5ac-eb8ea3b81245"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:01:58.556711 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:58.556685 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8888a20-3251-4d90-b5ac-eb8ea3b81245-kube-api-access-4gsw8" pod "f8888a20-3251-4d90-b5ac-eb8ea3b81245" (UID: "f8888a20-3251-4d90-b5ac-eb8ea3b81245"). InnerVolumeSpecName "kube-api-access-4gsw8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:01:58.573862 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:58.571355 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8888a20-3251-4d90-b5ac-eb8ea3b81245-maas-api-xks-ca" pod "f8888a20-3251-4d90-b5ac-eb8ea3b81245" (UID: "f8888a20-3251-4d90-b5ac-eb8ea3b81245"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:01:58.656114 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:58.656073 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4gsw8\" (UniqueName: \"kubernetes.io/projected/f8888a20-3251-4d90-b5ac-eb8ea3b81245-kube-api-access-4gsw8\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:01:58.656114 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:58.656108 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8888a20-3251-4d90-b5ac-eb8ea3b81245-maas-api-xks-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:01:58.656114 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:58.656118 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8888a20-3251-4d90-b5ac-eb8ea3b81245-maas-api-openshift-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:01:59.283495 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:59.283454 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09c34f4e-92lmw" Oct 30 13:01:59.283495 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:59.283483 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09c34f4e-92lmw" event={"ID":"f8888a20-3251-4d90-b5ac-eb8ea3b81245","Type":"ContainerDied","Data":"0fc013a06d2da104bc11d03003470f48ae5449705dce17795554cd3e9ee65ba2"} Oct 30 13:01:59.283741 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:01:59.283547 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0fc013a06d2da104bc11d03003470f48ae5449705dce17795554cd3e9ee65ba2" Oct 30 13:02:00.300788 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:00.300741 4908 generic.go:356] "Generic (PLEG): container finished" podID="c3e7bd6e-7283-451a-8df8-cc9634908630" containerID="6c617bec82991ea0e7d318e4505a289d67bffd53172b1f9794c5a407e8884554" exitCode=0 Oct 30 13:02:00.301330 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:00.300909 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-tjnq5" event={"ID":"c3e7bd6e-7283-451a-8df8-cc9634908630","Type":"ContainerDied","Data":"6c617bec82991ea0e7d318e4505a289d67bffd53172b1f9794c5a407e8884554"} Oct 30 13:02:00.349548 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:00.349516 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-tjnq5" Oct 30 13:02:00.497778 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:00.497737 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c3e7bd6e-7283-451a-8df8-cc9634908630-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3e7bd6e-7283-451a-8df8-cc9634908630-istio-podinfo\") pod \"c3e7bd6e-7283-451a-8df8-cc9634908630\" (UID: \"c3e7bd6e-7283-451a-8df8-cc9634908630\") " Oct 30 13:02:00.498043 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:00.497804 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3e7bd6e-7283-451a-8df8-cc9634908630-kube-api-access-stglb\" (UniqueName: \"kubernetes.io/projected/c3e7bd6e-7283-451a-8df8-cc9634908630-kube-api-access-stglb\") pod \"c3e7bd6e-7283-451a-8df8-cc9634908630\" (UID: \"c3e7bd6e-7283-451a-8df8-cc9634908630\") " Oct 30 13:02:00.498043 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:00.497955 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3e7bd6e-7283-451a-8df8-cc9634908630-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3e7bd6e-7283-451a-8df8-cc9634908630-istiod-ca-cert\") pod \"c3e7bd6e-7283-451a-8df8-cc9634908630\" (UID: \"c3e7bd6e-7283-451a-8df8-cc9634908630\") " Oct 30 13:02:00.498399 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:00.498369 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3e7bd6e-7283-451a-8df8-cc9634908630-istio-data" pod "c3e7bd6e-7283-451a-8df8-cc9634908630" (UID: "c3e7bd6e-7283-451a-8df8-cc9634908630"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:02:00.498634 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:00.498615 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3e7bd6e-7283-451a-8df8-cc9634908630-istiod-ca-cert" pod "c3e7bd6e-7283-451a-8df8-cc9634908630" (UID: "c3e7bd6e-7283-451a-8df8-cc9634908630"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:02:00.498714 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:00.498660 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3e7bd6e-7283-451a-8df8-cc9634908630-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3e7bd6e-7283-451a-8df8-cc9634908630-istio-data\") pod \"c3e7bd6e-7283-451a-8df8-cc9634908630\" (UID: \"c3e7bd6e-7283-451a-8df8-cc9634908630\") " Oct 30 13:02:00.498906 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:00.498877 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3e7bd6e-7283-451a-8df8-cc9634908630-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3e7bd6e-7283-451a-8df8-cc9634908630-credential-socket\") pod \"c3e7bd6e-7283-451a-8df8-cc9634908630\" (UID: \"c3e7bd6e-7283-451a-8df8-cc9634908630\") " Oct 30 13:02:00.498998 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:00.498929 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3e7bd6e-7283-451a-8df8-cc9634908630-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3e7bd6e-7283-451a-8df8-cc9634908630-workload-certs\") pod \"c3e7bd6e-7283-451a-8df8-cc9634908630\" (UID: \"c3e7bd6e-7283-451a-8df8-cc9634908630\") " Oct 30 13:02:00.498998 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:00.498975 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3e7bd6e-7283-451a-8df8-cc9634908630-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3e7bd6e-7283-451a-8df8-cc9634908630-workload-socket\") pod \"c3e7bd6e-7283-451a-8df8-cc9634908630\" (UID: \"c3e7bd6e-7283-451a-8df8-cc9634908630\") " Oct 30 13:02:00.499106 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:00.499027 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3e7bd6e-7283-451a-8df8-cc9634908630-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3e7bd6e-7283-451a-8df8-cc9634908630-istio-envoy\") pod \"c3e7bd6e-7283-451a-8df8-cc9634908630\" (UID: \"c3e7bd6e-7283-451a-8df8-cc9634908630\") " Oct 30 13:02:00.499591 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:00.499410 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3e7bd6e-7283-451a-8df8-cc9634908630-workload-certs" pod "c3e7bd6e-7283-451a-8df8-cc9634908630" (UID: "c3e7bd6e-7283-451a-8df8-cc9634908630"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:02:00.499591 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:00.499456 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3e7bd6e-7283-451a-8df8-cc9634908630-credential-socket" pod "c3e7bd6e-7283-451a-8df8-cc9634908630" (UID: "c3e7bd6e-7283-451a-8df8-cc9634908630"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:02:00.499768 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:00.499172 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3e7bd6e-7283-451a-8df8-cc9634908630-istio-token\" (UniqueName: \"kubernetes.io/projected/c3e7bd6e-7283-451a-8df8-cc9634908630-istio-token\") pod \"c3e7bd6e-7283-451a-8df8-cc9634908630\" (UID: \"c3e7bd6e-7283-451a-8df8-cc9634908630\") " Oct 30 13:02:00.500012 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:00.499782 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3e7bd6e-7283-451a-8df8-cc9634908630-workload-socket" pod "c3e7bd6e-7283-451a-8df8-cc9634908630" (UID: "c3e7bd6e-7283-451a-8df8-cc9634908630"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:02:00.500730 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:00.500703 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c3e7bd6e-7283-451a-8df8-cc9634908630-istio-podinfo" pod "c3e7bd6e-7283-451a-8df8-cc9634908630" (UID: "c3e7bd6e-7283-451a-8df8-cc9634908630"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:02:00.501198 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:00.501173 4908 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3e7bd6e-7283-451a-8df8-cc9634908630-istio-podinfo\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:02:00.501309 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:00.501201 4908 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3e7bd6e-7283-451a-8df8-cc9634908630-istiod-ca-cert\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:02:00.501309 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:00.501215 4908 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3e7bd6e-7283-451a-8df8-cc9634908630-istio-data\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:02:00.501309 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:00.501230 4908 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3e7bd6e-7283-451a-8df8-cc9634908630-credential-socket\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:02:00.501309 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:00.501243 4908 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3e7bd6e-7283-451a-8df8-cc9634908630-workload-certs\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:02:00.501309 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:00.501257 4908 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3e7bd6e-7283-451a-8df8-cc9634908630-workload-socket\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:02:00.503103 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:00.503077 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3e7bd6e-7283-451a-8df8-cc9634908630-kube-api-access-stglb" pod "c3e7bd6e-7283-451a-8df8-cc9634908630" (UID: "c3e7bd6e-7283-451a-8df8-cc9634908630"). InnerVolumeSpecName "kube-api-access-stglb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:02:00.503386 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:00.503359 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3e7bd6e-7283-451a-8df8-cc9634908630-istio-envoy" pod "c3e7bd6e-7283-451a-8df8-cc9634908630" (UID: "c3e7bd6e-7283-451a-8df8-cc9634908630"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:02:00.503573 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:00.503551 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3e7bd6e-7283-451a-8df8-cc9634908630-istio-token" pod "c3e7bd6e-7283-451a-8df8-cc9634908630" (UID: "c3e7bd6e-7283-451a-8df8-cc9634908630"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:02:00.603353 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:00.603308 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-stglb\" (UniqueName: \"kubernetes.io/projected/c3e7bd6e-7283-451a-8df8-cc9634908630-kube-api-access-stglb\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:02:00.603353 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:00.603353 4908 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3e7bd6e-7283-451a-8df8-cc9634908630-istio-envoy\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:02:00.603644 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:00.603367 4908 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3e7bd6e-7283-451a-8df8-cc9634908630-istio-token\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:02:01.311292 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:01.311257 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-tjnq5" Oct 30 13:02:01.311759 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:01.311290 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-tjnq5" event={"ID":"c3e7bd6e-7283-451a-8df8-cc9634908630","Type":"ContainerDied","Data":"29e660461b82d900aa1754b7f3b05f576a4a76c8ee3fd53706d79bd0ba3305f5"} Oct 30 13:02:01.311759 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:01.311356 4908 scope.go:122] "RemoveContainer" containerID="6c617bec82991ea0e7d318e4505a289d67bffd53172b1f9794c5a407e8884554" Oct 30 13:02:01.362868 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:01.362814 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-tjnq5"] Oct 30 13:02:01.373873 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:01.373821 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-tjnq5"] Oct 30 13:02:01.613355 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:01.613253 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3e7bd6e-7283-451a-8df8-cc9634908630" path="/var/lib/kubelet/pods/c3e7bd6e-7283-451a-8df8-cc9634908630/volumes" Oct 30 13:02:02.055806 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:02.055772 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-f7464f-qsmck"] Oct 30 13:02:02.056412 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:02.056395 4908 cpu_manager.go:414] "containerMap: removing container" podUID="c3e7bd6e-7283-451a-8df8-cc9634908630" containerName="istio-proxy" Oct 30 13:02:02.056474 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:02.056413 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3e7bd6e-7283-451a-8df8-cc9634908630" containerName="istio-proxy" Oct 30 13:02:02.056474 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:02.056446 4908 cpu_manager.go:414] "containerMap: removing container" podUID="f8888a20-3251-4d90-b5ac-eb8ea3b81245" containerName="revoke-keys" Oct 30 13:02:02.056474 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:02.056451 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8888a20-3251-4d90-b5ac-eb8ea3b81245" containerName="revoke-keys" Oct 30 13:02:02.056585 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:02.056569 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8888a20-3251-4d90-b5ac-eb8ea3b81245" containerName="revoke-keys" Oct 30 13:02:02.056585 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:02.056584 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3e7bd6e-7283-451a-8df8-cc9634908630" containerName="istio-proxy" Oct 30 13:02:02.061325 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:02.061302 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-f7464f-qsmck" Oct 30 13:02:02.076352 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:02.076318 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:02:02.086736 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:02.086710 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 13:02:02.086736 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:02.086728 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 13:02:02.086993 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:02.086718 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s75hq\"" Oct 30 13:02:02.130096 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:02.130058 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5vh2\" (UniqueName: \"kubernetes.io/projected/a81e1728-369a-4022-8bc7-3c3726900efd-kube-api-access-t5vh2\") pod \"maas-api-revoke-keys-e2e-extmod-a-f7464f-qsmck\" (UID: \"a81e1728-369a-4022-8bc7-3c3726900efd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-f7464f-qsmck" Oct 30 13:02:02.130272 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:02.130219 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a81e1728-369a-4022-8bc7-3c3726900efd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-f7464f-qsmck\" (UID: \"a81e1728-369a-4022-8bc7-3c3726900efd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-f7464f-qsmck" Oct 30 13:02:02.130348 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:02.130329 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a81e1728-369a-4022-8bc7-3c3726900efd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-f7464f-qsmck\" (UID: \"a81e1728-369a-4022-8bc7-3c3726900efd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-f7464f-qsmck" Oct 30 13:02:02.169791 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:02.169750 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-f7464f-qsmck"] Oct 30 13:02:02.231592 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:02.231559 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5vh2\" (UniqueName: \"kubernetes.io/projected/a81e1728-369a-4022-8bc7-3c3726900efd-kube-api-access-t5vh2\") pod \"maas-api-revoke-keys-e2e-extmod-a-f7464f-qsmck\" (UID: \"a81e1728-369a-4022-8bc7-3c3726900efd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-f7464f-qsmck" Oct 30 13:02:02.231773 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:02.231663 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a81e1728-369a-4022-8bc7-3c3726900efd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-f7464f-qsmck\" (UID: \"a81e1728-369a-4022-8bc7-3c3726900efd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-f7464f-qsmck" Oct 30 13:02:02.231773 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:02.231745 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a81e1728-369a-4022-8bc7-3c3726900efd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-f7464f-qsmck\" (UID: \"a81e1728-369a-4022-8bc7-3c3726900efd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-f7464f-qsmck" Oct 30 13:02:02.232524 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:02.232504 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a81e1728-369a-4022-8bc7-3c3726900efd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-f7464f-qsmck\" (UID: \"a81e1728-369a-4022-8bc7-3c3726900efd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-f7464f-qsmck" Oct 30 13:02:02.234178 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:02.234156 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a81e1728-369a-4022-8bc7-3c3726900efd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-f7464f-qsmck\" (UID: \"a81e1728-369a-4022-8bc7-3c3726900efd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-f7464f-qsmck" Oct 30 13:02:02.273633 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:02.273598 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5vh2\" (UniqueName: \"kubernetes.io/projected/a81e1728-369a-4022-8bc7-3c3726900efd-kube-api-access-t5vh2\") pod \"maas-api-revoke-keys-e2e-extmod-a-f7464f-qsmck\" (UID: \"a81e1728-369a-4022-8bc7-3c3726900efd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-f7464f-qsmck" Oct 30 13:02:02.379231 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:02.379140 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-f7464f-qsmck" Oct 30 13:02:02.747597 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:02.747567 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-f7464f-qsmck"] Oct 30 13:02:02.750042 ip-10-0-141-251 kubenswrapper[4908]: W1030 13:02:02.750013 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda81e1728_369a_4022_8bc7_3c3726900efd.slice/crio-16dc06c156cceead2eaeab74d989cbf55ee2ff2ff307c3460dd62c7f49884b9a WatchSource:0}: Error finding container 16dc06c156cceead2eaeab74d989cbf55ee2ff2ff307c3460dd62c7f49884b9a: Status 404 returned error can't find the container with id 16dc06c156cceead2eaeab74d989cbf55ee2ff2ff307c3460dd62c7f49884b9a Oct 30 13:02:02.781618 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:02.781587 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-j8kfd"] Oct 30 13:02:02.781901 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:02.781797 4908 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-j8kfd" podUID="2068d5c3-6f62-4bd1-bbf1-20a12a84725c" containerName="istio-proxy" containerID="cri-o://24e4b5280446741f81b6df501d4d1842ba6b2f8b6709fb1c93dc223276414cb8" gracePeriod=30 Oct 30 13:02:03.331216 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:03.331170 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-f7464f-qsmck" event={"ID":"a81e1728-369a-4022-8bc7-3c3726900efd","Type":"ContainerStarted","Data":"16631b86a037458a316965cd02edf6aa3ebe390b449f42844320b9c40349c909"} Oct 30 13:02:03.331375 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:03.331234 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-f7464f-qsmck" event={"ID":"a81e1728-369a-4022-8bc7-3c3726900efd","Type":"ContainerStarted","Data":"16dc06c156cceead2eaeab74d989cbf55ee2ff2ff307c3460dd62c7f49884b9a"} Oct 30 13:02:03.376605 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:03.376501 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-f7464f-qsmck" podStartSLOduration=1.376483207 podStartE2EDuration="1.376483207s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:02:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:02:03.369458728 +0000 UTC m=+4079.883887484" watchObservedRunningTime="2026-10-30 13:02:03.376483207 +0000 UTC m=+4079.890911957" Oct 30 13:02:03.467598 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:03.467560 4908 patch_prober.go:29] interesting pod/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-j8kfd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 13:02:03.468039 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:03.467620 4908 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-j8kfd" podUID="2068d5c3-6f62-4bd1-bbf1-20a12a84725c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 13:02:04.340817 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:04.340775 4908 generic.go:356] "Generic (PLEG): container finished" podID="a81e1728-369a-4022-8bc7-3c3726900efd" containerID="16631b86a037458a316965cd02edf6aa3ebe390b449f42844320b9c40349c909" exitCode=0 Oct 30 13:02:04.341038 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:04.340878 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-f7464f-qsmck" event={"ID":"a81e1728-369a-4022-8bc7-3c3726900efd","Type":"ContainerDied","Data":"16631b86a037458a316965cd02edf6aa3ebe390b449f42844320b9c40349c909"} Oct 30 13:02:05.133717 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:05.133664 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-eab799/test-model-eab799-kserve-7b456659fc-bgzth"] Oct 30 13:02:05.670919 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:05.666303 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf8e7840-pmr87"] Oct 30 13:02:05.672769 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:05.672734 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf8e7840-pmr87" Oct 30 13:02:05.688353 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:05.687063 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf8e7840-pmr87"] Oct 30 13:02:05.718313 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:05.718016 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cs25v\" (UniqueName: \"kubernetes.io/projected/4060653d-76ee-4df6-a077-a6c7bb2a6a5d-kube-api-access-cs25v\") pod \"maas-api-revoke-sub-bf8e7840-pmr87\" (UID: \"4060653d-76ee-4df6-a077-a6c7bb2a6a5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf8e7840-pmr87" Oct 30 13:02:05.718313 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:05.718084 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4060653d-76ee-4df6-a077-a6c7bb2a6a5d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bf8e7840-pmr87\" (UID: \"4060653d-76ee-4df6-a077-a6c7bb2a6a5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf8e7840-pmr87" Oct 30 13:02:05.718313 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:05.718174 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4060653d-76ee-4df6-a077-a6c7bb2a6a5d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bf8e7840-pmr87\" (UID: \"4060653d-76ee-4df6-a077-a6c7bb2a6a5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf8e7840-pmr87" Oct 30 13:02:05.750991 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:05.750916 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-f7464f-qsmck" Oct 30 13:02:05.819439 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:05.819399 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a81e1728-369a-4022-8bc7-3c3726900efd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a81e1728-369a-4022-8bc7-3c3726900efd-maas-api-xks-ca\") pod \"a81e1728-369a-4022-8bc7-3c3726900efd\" (UID: \"a81e1728-369a-4022-8bc7-3c3726900efd\") " Oct 30 13:02:05.819629 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:05.819526 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a81e1728-369a-4022-8bc7-3c3726900efd-kube-api-access-t5vh2\" (UniqueName: \"kubernetes.io/projected/a81e1728-369a-4022-8bc7-3c3726900efd-kube-api-access-t5vh2\") pod \"a81e1728-369a-4022-8bc7-3c3726900efd\" (UID: \"a81e1728-369a-4022-8bc7-3c3726900efd\") " Oct 30 13:02:05.819629 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:05.819584 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a81e1728-369a-4022-8bc7-3c3726900efd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a81e1728-369a-4022-8bc7-3c3726900efd-maas-api-openshift-ca\") pod \"a81e1728-369a-4022-8bc7-3c3726900efd\" (UID: \"a81e1728-369a-4022-8bc7-3c3726900efd\") " Oct 30 13:02:05.819903 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:05.819879 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4060653d-76ee-4df6-a077-a6c7bb2a6a5d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bf8e7840-pmr87\" (UID: \"4060653d-76ee-4df6-a077-a6c7bb2a6a5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf8e7840-pmr87" Oct 30 13:02:05.820204 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:05.820184 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cs25v\" (UniqueName: \"kubernetes.io/projected/4060653d-76ee-4df6-a077-a6c7bb2a6a5d-kube-api-access-cs25v\") pod \"maas-api-revoke-sub-bf8e7840-pmr87\" (UID: \"4060653d-76ee-4df6-a077-a6c7bb2a6a5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf8e7840-pmr87" Oct 30 13:02:05.820300 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:05.820251 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4060653d-76ee-4df6-a077-a6c7bb2a6a5d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bf8e7840-pmr87\" (UID: \"4060653d-76ee-4df6-a077-a6c7bb2a6a5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf8e7840-pmr87" Oct 30 13:02:05.820857 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:05.820803 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4060653d-76ee-4df6-a077-a6c7bb2a6a5d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bf8e7840-pmr87\" (UID: \"4060653d-76ee-4df6-a077-a6c7bb2a6a5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf8e7840-pmr87" Oct 30 13:02:05.821420 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:05.821380 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a81e1728-369a-4022-8bc7-3c3726900efd-maas-api-openshift-ca" pod "a81e1728-369a-4022-8bc7-3c3726900efd" (UID: "a81e1728-369a-4022-8bc7-3c3726900efd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:02:05.822600 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:05.822571 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a81e1728-369a-4022-8bc7-3c3726900efd-kube-api-access-t5vh2" pod "a81e1728-369a-4022-8bc7-3c3726900efd" (UID: "a81e1728-369a-4022-8bc7-3c3726900efd"). InnerVolumeSpecName "kube-api-access-t5vh2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:02:05.823018 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:05.822990 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4060653d-76ee-4df6-a077-a6c7bb2a6a5d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bf8e7840-pmr87\" (UID: \"4060653d-76ee-4df6-a077-a6c7bb2a6a5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf8e7840-pmr87" Oct 30 13:02:05.825000 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:05.824973 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a81e1728-369a-4022-8bc7-3c3726900efd-maas-api-xks-ca" pod "a81e1728-369a-4022-8bc7-3c3726900efd" (UID: "a81e1728-369a-4022-8bc7-3c3726900efd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:02:05.844057 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:05.844031 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cs25v\" (UniqueName: \"kubernetes.io/projected/4060653d-76ee-4df6-a077-a6c7bb2a6a5d-kube-api-access-cs25v\") pod \"maas-api-revoke-sub-bf8e7840-pmr87\" (UID: \"4060653d-76ee-4df6-a077-a6c7bb2a6a5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf8e7840-pmr87" Oct 30 13:02:05.921349 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:05.921311 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a81e1728-369a-4022-8bc7-3c3726900efd-maas-api-xks-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:02:05.921349 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:05.921345 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5vh2\" (UniqueName: \"kubernetes.io/projected/a81e1728-369a-4022-8bc7-3c3726900efd-kube-api-access-t5vh2\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:02:05.921349 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:05.921356 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a81e1728-369a-4022-8bc7-3c3726900efd-maas-api-openshift-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:02:05.997512 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:05.997474 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf8e7840-pmr87" Oct 30 13:02:06.346709 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:06.346678 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf8e7840-pmr87"] Oct 30 13:02:06.348771 ip-10-0-141-251 kubenswrapper[4908]: W1030 13:02:06.348737 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4060653d_76ee_4df6_a077_a6c7bb2a6a5d.slice/crio-3d7f26252beb417033742f4bb43a0d56dce02e3f2cab8e8e82c4223fbd82f708 WatchSource:0}: Error finding container 3d7f26252beb417033742f4bb43a0d56dce02e3f2cab8e8e82c4223fbd82f708: Status 404 returned error can't find the container with id 3d7f26252beb417033742f4bb43a0d56dce02e3f2cab8e8e82c4223fbd82f708 Oct 30 13:02:06.501334 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:06.501271 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-f7464f-qsmck" Oct 30 13:02:06.501729 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:06.501293 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-f7464f-qsmck" event={"ID":"a81e1728-369a-4022-8bc7-3c3726900efd","Type":"ContainerDied","Data":"16dc06c156cceead2eaeab74d989cbf55ee2ff2ff307c3460dd62c7f49884b9a"} Oct 30 13:02:06.501823 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:06.501771 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="16dc06c156cceead2eaeab74d989cbf55ee2ff2ff307c3460dd62c7f49884b9a" Oct 30 13:02:06.503067 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:06.503035 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf8e7840-pmr87" event={"ID":"4060653d-76ee-4df6-a077-a6c7bb2a6a5d","Type":"ContainerStarted","Data":"3d7f26252beb417033742f4bb43a0d56dce02e3f2cab8e8e82c4223fbd82f708"} Oct 30 13:02:07.516783 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:07.516747 4908 generic.go:356] "Generic (PLEG): container finished" podID="4060653d-76ee-4df6-a077-a6c7bb2a6a5d" containerID="9588ea39ea27c5eb940a07541f4f948f1e0e9b26ef7e14aa44fc76498b6dc401" exitCode=0 Oct 30 13:02:07.517168 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:07.516794 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf8e7840-pmr87" event={"ID":"4060653d-76ee-4df6-a077-a6c7bb2a6a5d","Type":"ContainerDied","Data":"9588ea39ea27c5eb940a07541f4f948f1e0e9b26ef7e14aa44fc76498b6dc401"} Oct 30 13:02:08.100532 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:08.100500 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-j8kfd" Oct 30 13:02:08.172988 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:08.172946 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2068d5c3-6f62-4bd1-bbf1-20a12a84725c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2068d5c3-6f62-4bd1-bbf1-20a12a84725c-istio-podinfo\") pod \"2068d5c3-6f62-4bd1-bbf1-20a12a84725c\" (UID: \"2068d5c3-6f62-4bd1-bbf1-20a12a84725c\") " Oct 30 13:02:08.173162 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:08.173015 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2068d5c3-6f62-4bd1-bbf1-20a12a84725c-kube-api-access-6tb68\" (UniqueName: \"kubernetes.io/projected/2068d5c3-6f62-4bd1-bbf1-20a12a84725c-kube-api-access-6tb68\") pod \"2068d5c3-6f62-4bd1-bbf1-20a12a84725c\" (UID: \"2068d5c3-6f62-4bd1-bbf1-20a12a84725c\") " Oct 30 13:02:08.173162 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:08.173072 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2068d5c3-6f62-4bd1-bbf1-20a12a84725c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2068d5c3-6f62-4bd1-bbf1-20a12a84725c-credential-socket\") pod \"2068d5c3-6f62-4bd1-bbf1-20a12a84725c\" (UID: \"2068d5c3-6f62-4bd1-bbf1-20a12a84725c\") " Oct 30 13:02:08.173294 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:08.173188 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2068d5c3-6f62-4bd1-bbf1-20a12a84725c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2068d5c3-6f62-4bd1-bbf1-20a12a84725c-istio-envoy\") pod \"2068d5c3-6f62-4bd1-bbf1-20a12a84725c\" (UID: \"2068d5c3-6f62-4bd1-bbf1-20a12a84725c\") " Oct 30 13:02:08.173294 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:08.173243 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2068d5c3-6f62-4bd1-bbf1-20a12a84725c-istio-token\" (UniqueName: \"kubernetes.io/projected/2068d5c3-6f62-4bd1-bbf1-20a12a84725c-istio-token\") pod \"2068d5c3-6f62-4bd1-bbf1-20a12a84725c\" (UID: \"2068d5c3-6f62-4bd1-bbf1-20a12a84725c\") " Oct 30 13:02:08.173408 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:08.173363 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2068d5c3-6f62-4bd1-bbf1-20a12a84725c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2068d5c3-6f62-4bd1-bbf1-20a12a84725c-workload-socket\") pod \"2068d5c3-6f62-4bd1-bbf1-20a12a84725c\" (UID: \"2068d5c3-6f62-4bd1-bbf1-20a12a84725c\") " Oct 30 13:02:08.173541 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:08.173522 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2068d5c3-6f62-4bd1-bbf1-20a12a84725c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2068d5c3-6f62-4bd1-bbf1-20a12a84725c-istiod-ca-cert\") pod \"2068d5c3-6f62-4bd1-bbf1-20a12a84725c\" (UID: \"2068d5c3-6f62-4bd1-bbf1-20a12a84725c\") " Oct 30 13:02:08.173596 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:08.173560 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2068d5c3-6f62-4bd1-bbf1-20a12a84725c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2068d5c3-6f62-4bd1-bbf1-20a12a84725c-istio-data\") pod \"2068d5c3-6f62-4bd1-bbf1-20a12a84725c\" (UID: \"2068d5c3-6f62-4bd1-bbf1-20a12a84725c\") " Oct 30 13:02:08.173596 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:08.173590 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2068d5c3-6f62-4bd1-bbf1-20a12a84725c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2068d5c3-6f62-4bd1-bbf1-20a12a84725c-workload-certs\") pod \"2068d5c3-6f62-4bd1-bbf1-20a12a84725c\" (UID: \"2068d5c3-6f62-4bd1-bbf1-20a12a84725c\") " Oct 30 13:02:08.176879 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:08.174684 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2068d5c3-6f62-4bd1-bbf1-20a12a84725c-workload-certs" pod "2068d5c3-6f62-4bd1-bbf1-20a12a84725c" (UID: "2068d5c3-6f62-4bd1-bbf1-20a12a84725c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:02:08.176879 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:08.175683 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2068d5c3-6f62-4bd1-bbf1-20a12a84725c-istiod-ca-cert" pod "2068d5c3-6f62-4bd1-bbf1-20a12a84725c" (UID: "2068d5c3-6f62-4bd1-bbf1-20a12a84725c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:02:08.176879 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:08.176478 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2068d5c3-6f62-4bd1-bbf1-20a12a84725c-credential-socket" pod "2068d5c3-6f62-4bd1-bbf1-20a12a84725c" (UID: "2068d5c3-6f62-4bd1-bbf1-20a12a84725c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:02:08.190262 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:08.176900 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2068d5c3-6f62-4bd1-bbf1-20a12a84725c-istio-data" pod "2068d5c3-6f62-4bd1-bbf1-20a12a84725c" (UID: "2068d5c3-6f62-4bd1-bbf1-20a12a84725c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:02:08.190262 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:08.177043 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2068d5c3-6f62-4bd1-bbf1-20a12a84725c-workload-socket" pod "2068d5c3-6f62-4bd1-bbf1-20a12a84725c" (UID: "2068d5c3-6f62-4bd1-bbf1-20a12a84725c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:02:08.190262 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:08.177368 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2068d5c3-6f62-4bd1-bbf1-20a12a84725c-istio-podinfo" pod "2068d5c3-6f62-4bd1-bbf1-20a12a84725c" (UID: "2068d5c3-6f62-4bd1-bbf1-20a12a84725c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:02:08.190262 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:08.177521 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2068d5c3-6f62-4bd1-bbf1-20a12a84725c-istio-token" pod "2068d5c3-6f62-4bd1-bbf1-20a12a84725c" (UID: "2068d5c3-6f62-4bd1-bbf1-20a12a84725c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:02:08.203419 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:08.203375 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2068d5c3-6f62-4bd1-bbf1-20a12a84725c-kube-api-access-6tb68" pod "2068d5c3-6f62-4bd1-bbf1-20a12a84725c" (UID: "2068d5c3-6f62-4bd1-bbf1-20a12a84725c"). InnerVolumeSpecName "kube-api-access-6tb68". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:02:08.205299 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:08.205261 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2068d5c3-6f62-4bd1-bbf1-20a12a84725c-istio-envoy" pod "2068d5c3-6f62-4bd1-bbf1-20a12a84725c" (UID: "2068d5c3-6f62-4bd1-bbf1-20a12a84725c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:02:08.313868 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:08.294611 4908 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2068d5c3-6f62-4bd1-bbf1-20a12a84725c-workload-socket\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:02:08.313868 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:08.294659 4908 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2068d5c3-6f62-4bd1-bbf1-20a12a84725c-istiod-ca-cert\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:02:08.313868 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:08.294674 4908 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2068d5c3-6f62-4bd1-bbf1-20a12a84725c-istio-data\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:02:08.313868 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:08.294686 4908 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2068d5c3-6f62-4bd1-bbf1-20a12a84725c-workload-certs\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:02:08.313868 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:08.294699 4908 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2068d5c3-6f62-4bd1-bbf1-20a12a84725c-istio-podinfo\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:02:08.313868 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:08.294712 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6tb68\" (UniqueName: \"kubernetes.io/projected/2068d5c3-6f62-4bd1-bbf1-20a12a84725c-kube-api-access-6tb68\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:02:08.313868 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:08.294728 4908 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2068d5c3-6f62-4bd1-bbf1-20a12a84725c-credential-socket\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:02:08.313868 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:08.294743 4908 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2068d5c3-6f62-4bd1-bbf1-20a12a84725c-istio-envoy\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:02:08.313868 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:08.294755 4908 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2068d5c3-6f62-4bd1-bbf1-20a12a84725c-istio-token\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:02:08.526937 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:08.526905 4908 generic.go:356] "Generic (PLEG): container finished" podID="2068d5c3-6f62-4bd1-bbf1-20a12a84725c" containerID="24e4b5280446741f81b6df501d4d1842ba6b2f8b6709fb1c93dc223276414cb8" exitCode=0 Oct 30 13:02:08.527469 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:08.526952 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-j8kfd" Oct 30 13:02:08.527469 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:08.526992 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-j8kfd" event={"ID":"2068d5c3-6f62-4bd1-bbf1-20a12a84725c","Type":"ContainerDied","Data":"24e4b5280446741f81b6df501d4d1842ba6b2f8b6709fb1c93dc223276414cb8"} Oct 30 13:02:08.527469 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:08.527055 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-j8kfd" event={"ID":"2068d5c3-6f62-4bd1-bbf1-20a12a84725c","Type":"ContainerDied","Data":"d8ccd10036614d2b5ed3d939aacf9d051f9daf01e2eb1e50b51b034551664818"} Oct 30 13:02:08.527469 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:08.527093 4908 scope.go:122] "RemoveContainer" containerID="24e4b5280446741f81b6df501d4d1842ba6b2f8b6709fb1c93dc223276414cb8" Oct 30 13:02:08.542561 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:08.542538 4908 scope.go:122] "RemoveContainer" containerID="24e4b5280446741f81b6df501d4d1842ba6b2f8b6709fb1c93dc223276414cb8" Oct 30 13:02:08.542955 ip-10-0-141-251 kubenswrapper[4908]: E1030 13:02:08.542926 4908 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24e4b5280446741f81b6df501d4d1842ba6b2f8b6709fb1c93dc223276414cb8\": container with ID starting with 24e4b5280446741f81b6df501d4d1842ba6b2f8b6709fb1c93dc223276414cb8 not found: ID does not exist" containerID="24e4b5280446741f81b6df501d4d1842ba6b2f8b6709fb1c93dc223276414cb8" Oct 30 13:02:08.543034 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:08.542965 4908 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24e4b5280446741f81b6df501d4d1842ba6b2f8b6709fb1c93dc223276414cb8"} err="failed to get container status \"24e4b5280446741f81b6df501d4d1842ba6b2f8b6709fb1c93dc223276414cb8\": rpc error: code = NotFound desc = could not find container \"24e4b5280446741f81b6df501d4d1842ba6b2f8b6709fb1c93dc223276414cb8\": container with ID starting with 24e4b5280446741f81b6df501d4d1842ba6b2f8b6709fb1c93dc223276414cb8 not found: ID does not exist" Oct 30 13:02:08.626929 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:08.626886 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-j8kfd"] Oct 30 13:02:08.635486 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:08.635451 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-j8kfd"] Oct 30 13:02:08.681055 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:08.681027 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf8e7840-pmr87" Oct 30 13:02:08.802176 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:08.802085 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4060653d-76ee-4df6-a077-a6c7bb2a6a5d-kube-api-access-cs25v\" (UniqueName: \"kubernetes.io/projected/4060653d-76ee-4df6-a077-a6c7bb2a6a5d-kube-api-access-cs25v\") pod \"4060653d-76ee-4df6-a077-a6c7bb2a6a5d\" (UID: \"4060653d-76ee-4df6-a077-a6c7bb2a6a5d\") " Oct 30 13:02:08.802346 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:08.802177 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4060653d-76ee-4df6-a077-a6c7bb2a6a5d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4060653d-76ee-4df6-a077-a6c7bb2a6a5d-maas-api-xks-ca\") pod \"4060653d-76ee-4df6-a077-a6c7bb2a6a5d\" (UID: \"4060653d-76ee-4df6-a077-a6c7bb2a6a5d\") " Oct 30 13:02:08.803133 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:08.803078 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4060653d-76ee-4df6-a077-a6c7bb2a6a5d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4060653d-76ee-4df6-a077-a6c7bb2a6a5d-maas-api-openshift-ca\") pod \"4060653d-76ee-4df6-a077-a6c7bb2a6a5d\" (UID: \"4060653d-76ee-4df6-a077-a6c7bb2a6a5d\") " Oct 30 13:02:08.804114 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:08.804088 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4060653d-76ee-4df6-a077-a6c7bb2a6a5d-maas-api-openshift-ca" pod "4060653d-76ee-4df6-a077-a6c7bb2a6a5d" (UID: "4060653d-76ee-4df6-a077-a6c7bb2a6a5d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:02:08.808047 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:08.808018 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4060653d-76ee-4df6-a077-a6c7bb2a6a5d-kube-api-access-cs25v" pod "4060653d-76ee-4df6-a077-a6c7bb2a6a5d" (UID: "4060653d-76ee-4df6-a077-a6c7bb2a6a5d"). InnerVolumeSpecName "kube-api-access-cs25v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:02:08.812045 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:08.812014 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4060653d-76ee-4df6-a077-a6c7bb2a6a5d-maas-api-xks-ca" pod "4060653d-76ee-4df6-a077-a6c7bb2a6a5d" (UID: "4060653d-76ee-4df6-a077-a6c7bb2a6a5d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:02:08.916303 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:08.914318 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4060653d-76ee-4df6-a077-a6c7bb2a6a5d-maas-api-openshift-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:02:08.916303 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:08.914367 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cs25v\" (UniqueName: \"kubernetes.io/projected/4060653d-76ee-4df6-a077-a6c7bb2a6a5d-kube-api-access-cs25v\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:02:08.916303 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:08.914383 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4060653d-76ee-4df6-a077-a6c7bb2a6a5d-maas-api-xks-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:02:09.108221 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:09.108133 4908 scope.go:122] "RemoveContainer" containerID="81a38e655caf32be04534d1f53a6f06d7226d8839289864f5d3a2e71f3d6a55e" Oct 30 13:02:09.536732 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:09.536694 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf8e7840-pmr87" event={"ID":"4060653d-76ee-4df6-a077-a6c7bb2a6a5d","Type":"ContainerDied","Data":"3d7f26252beb417033742f4bb43a0d56dce02e3f2cab8e8e82c4223fbd82f708"} Oct 30 13:02:09.536732 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:09.536723 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf8e7840-pmr87" Oct 30 13:02:09.537214 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:09.536782 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3d7f26252beb417033742f4bb43a0d56dce02e3f2cab8e8e82c4223fbd82f708" Oct 30 13:02:09.611587 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:09.611551 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2068d5c3-6f62-4bd1-bbf1-20a12a84725c" path="/var/lib/kubelet/pods/2068d5c3-6f62-4bd1-bbf1-20a12a84725c/volumes" Oct 30 13:02:11.019884 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:11.019842 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rtxmt"] Oct 30 13:02:11.020331 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:11.020058 4908 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rtxmt" podUID="f36337d3-669a-4d7d-a157-b8a5f78aff17" containerName="istio-proxy" containerID="cri-o://e0a5a8db655d7522436768c1698da13bc7a08d3987e5c844ad7f864ae168b490" gracePeriod=30 Oct 30 13:02:16.323191 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:16.323164 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rtxmt" Oct 30 13:02:16.470057 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:16.470024 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f36337d3-669a-4d7d-a157-b8a5f78aff17-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f36337d3-669a-4d7d-a157-b8a5f78aff17-istio-envoy\") pod \"f36337d3-669a-4d7d-a157-b8a5f78aff17\" (UID: \"f36337d3-669a-4d7d-a157-b8a5f78aff17\") " Oct 30 13:02:16.470243 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:16.470086 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f36337d3-669a-4d7d-a157-b8a5f78aff17-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f36337d3-669a-4d7d-a157-b8a5f78aff17-istiod-ca-cert\") pod \"f36337d3-669a-4d7d-a157-b8a5f78aff17\" (UID: \"f36337d3-669a-4d7d-a157-b8a5f78aff17\") " Oct 30 13:02:16.470243 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:16.470107 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f36337d3-669a-4d7d-a157-b8a5f78aff17-istio-token\" (UniqueName: \"kubernetes.io/projected/f36337d3-669a-4d7d-a157-b8a5f78aff17-istio-token\") pod \"f36337d3-669a-4d7d-a157-b8a5f78aff17\" (UID: \"f36337d3-669a-4d7d-a157-b8a5f78aff17\") " Oct 30 13:02:16.470243 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:16.470211 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f36337d3-669a-4d7d-a157-b8a5f78aff17-kube-api-access-jqrr6\" (UniqueName: \"kubernetes.io/projected/f36337d3-669a-4d7d-a157-b8a5f78aff17-kube-api-access-jqrr6\") pod \"f36337d3-669a-4d7d-a157-b8a5f78aff17\" (UID: \"f36337d3-669a-4d7d-a157-b8a5f78aff17\") " Oct 30 13:02:16.470243 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:16.470239 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f36337d3-669a-4d7d-a157-b8a5f78aff17-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f36337d3-669a-4d7d-a157-b8a5f78aff17-workload-certs\") pod \"f36337d3-669a-4d7d-a157-b8a5f78aff17\" (UID: \"f36337d3-669a-4d7d-a157-b8a5f78aff17\") " Oct 30 13:02:16.470463 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:16.470272 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f36337d3-669a-4d7d-a157-b8a5f78aff17-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f36337d3-669a-4d7d-a157-b8a5f78aff17-istio-data\") pod \"f36337d3-669a-4d7d-a157-b8a5f78aff17\" (UID: \"f36337d3-669a-4d7d-a157-b8a5f78aff17\") " Oct 30 13:02:16.470463 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:16.470314 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f36337d3-669a-4d7d-a157-b8a5f78aff17-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f36337d3-669a-4d7d-a157-b8a5f78aff17-workload-socket\") pod \"f36337d3-669a-4d7d-a157-b8a5f78aff17\" (UID: \"f36337d3-669a-4d7d-a157-b8a5f78aff17\") " Oct 30 13:02:16.470463 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:16.470366 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f36337d3-669a-4d7d-a157-b8a5f78aff17-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f36337d3-669a-4d7d-a157-b8a5f78aff17-istio-podinfo\") pod \"f36337d3-669a-4d7d-a157-b8a5f78aff17\" (UID: \"f36337d3-669a-4d7d-a157-b8a5f78aff17\") " Oct 30 13:02:16.470730 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:16.470500 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f36337d3-669a-4d7d-a157-b8a5f78aff17-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f36337d3-669a-4d7d-a157-b8a5f78aff17-credential-socket\") pod \"f36337d3-669a-4d7d-a157-b8a5f78aff17\" (UID: \"f36337d3-669a-4d7d-a157-b8a5f78aff17\") " Oct 30 13:02:16.470730 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:16.470537 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f36337d3-669a-4d7d-a157-b8a5f78aff17-workload-certs" pod "f36337d3-669a-4d7d-a157-b8a5f78aff17" (UID: "f36337d3-669a-4d7d-a157-b8a5f78aff17"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:02:16.470730 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:16.470578 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f36337d3-669a-4d7d-a157-b8a5f78aff17-istio-data" pod "f36337d3-669a-4d7d-a157-b8a5f78aff17" (UID: "f36337d3-669a-4d7d-a157-b8a5f78aff17"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:02:16.470730 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:16.470598 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f36337d3-669a-4d7d-a157-b8a5f78aff17-istiod-ca-cert" pod "f36337d3-669a-4d7d-a157-b8a5f78aff17" (UID: "f36337d3-669a-4d7d-a157-b8a5f78aff17"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:02:16.471008 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:16.470724 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f36337d3-669a-4d7d-a157-b8a5f78aff17-workload-socket" pod "f36337d3-669a-4d7d-a157-b8a5f78aff17" (UID: "f36337d3-669a-4d7d-a157-b8a5f78aff17"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:02:16.471008 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:16.470939 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f36337d3-669a-4d7d-a157-b8a5f78aff17-credential-socket" pod "f36337d3-669a-4d7d-a157-b8a5f78aff17" (UID: "f36337d3-669a-4d7d-a157-b8a5f78aff17"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:02:16.471293 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:16.471275 4908 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f36337d3-669a-4d7d-a157-b8a5f78aff17-istiod-ca-cert\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:02:16.471370 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:16.471298 4908 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f36337d3-669a-4d7d-a157-b8a5f78aff17-workload-certs\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:02:16.471370 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:16.471310 4908 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f36337d3-669a-4d7d-a157-b8a5f78aff17-istio-data\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:02:16.471370 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:16.471324 4908 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f36337d3-669a-4d7d-a157-b8a5f78aff17-workload-socket\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:02:16.471370 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:16.471338 4908 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f36337d3-669a-4d7d-a157-b8a5f78aff17-credential-socket\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:02:16.472469 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:16.472449 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f36337d3-669a-4d7d-a157-b8a5f78aff17-istio-envoy" pod "f36337d3-669a-4d7d-a157-b8a5f78aff17" (UID: "f36337d3-669a-4d7d-a157-b8a5f78aff17"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:02:16.472872 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:16.472810 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f36337d3-669a-4d7d-a157-b8a5f78aff17-kube-api-access-jqrr6" pod "f36337d3-669a-4d7d-a157-b8a5f78aff17" (UID: "f36337d3-669a-4d7d-a157-b8a5f78aff17"). InnerVolumeSpecName "kube-api-access-jqrr6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:02:16.472952 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:16.472922 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f36337d3-669a-4d7d-a157-b8a5f78aff17-istio-token" pod "f36337d3-669a-4d7d-a157-b8a5f78aff17" (UID: "f36337d3-669a-4d7d-a157-b8a5f78aff17"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:02:16.473017 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:16.472986 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f36337d3-669a-4d7d-a157-b8a5f78aff17-istio-podinfo" pod "f36337d3-669a-4d7d-a157-b8a5f78aff17" (UID: "f36337d3-669a-4d7d-a157-b8a5f78aff17"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:02:16.573275 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:16.573177 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqrr6\" (UniqueName: \"kubernetes.io/projected/f36337d3-669a-4d7d-a157-b8a5f78aff17-kube-api-access-jqrr6\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:02:16.573275 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:16.573224 4908 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f36337d3-669a-4d7d-a157-b8a5f78aff17-istio-podinfo\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:02:16.573275 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:16.573234 4908 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f36337d3-669a-4d7d-a157-b8a5f78aff17-istio-envoy\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:02:16.573275 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:16.573243 4908 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f36337d3-669a-4d7d-a157-b8a5f78aff17-istio-token\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:02:16.600051 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:16.600012 4908 generic.go:356] "Generic (PLEG): container finished" podID="f36337d3-669a-4d7d-a157-b8a5f78aff17" containerID="e0a5a8db655d7522436768c1698da13bc7a08d3987e5c844ad7f864ae168b490" exitCode=0 Oct 30 13:02:16.600229 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:16.600080 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rtxmt" event={"ID":"f36337d3-669a-4d7d-a157-b8a5f78aff17","Type":"ContainerDied","Data":"e0a5a8db655d7522436768c1698da13bc7a08d3987e5c844ad7f864ae168b490"} Oct 30 13:02:16.600229 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:16.600115 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rtxmt" event={"ID":"f36337d3-669a-4d7d-a157-b8a5f78aff17","Type":"ContainerDied","Data":"343d896d6ab1251e837ee528f06dc59c02bf13b4e38fb93d5bbf20715391b254"} Oct 30 13:02:16.600229 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:16.600138 4908 scope.go:122] "RemoveContainer" containerID="e0a5a8db655d7522436768c1698da13bc7a08d3987e5c844ad7f864ae168b490" Oct 30 13:02:16.600229 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:16.600076 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rtxmt" Oct 30 13:02:16.612445 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:16.612425 4908 scope.go:122] "RemoveContainer" containerID="e0a5a8db655d7522436768c1698da13bc7a08d3987e5c844ad7f864ae168b490" Oct 30 13:02:16.612713 ip-10-0-141-251 kubenswrapper[4908]: E1030 13:02:16.612695 4908 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0a5a8db655d7522436768c1698da13bc7a08d3987e5c844ad7f864ae168b490\": container with ID starting with e0a5a8db655d7522436768c1698da13bc7a08d3987e5c844ad7f864ae168b490 not found: ID does not exist" containerID="e0a5a8db655d7522436768c1698da13bc7a08d3987e5c844ad7f864ae168b490" Oct 30 13:02:16.612758 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:16.612723 4908 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0a5a8db655d7522436768c1698da13bc7a08d3987e5c844ad7f864ae168b490"} err="failed to get container status \"e0a5a8db655d7522436768c1698da13bc7a08d3987e5c844ad7f864ae168b490\": rpc error: code = NotFound desc = could not find container \"e0a5a8db655d7522436768c1698da13bc7a08d3987e5c844ad7f864ae168b490\": container with ID starting with e0a5a8db655d7522436768c1698da13bc7a08d3987e5c844ad7f864ae168b490 not found: ID does not exist" Oct 30 13:02:16.717197 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:16.716937 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rtxmt"] Oct 30 13:02:16.723151 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:16.723114 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-rtxmt"] Oct 30 13:02:17.609579 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:17.609537 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f36337d3-669a-4d7d-a157-b8a5f78aff17" path="/var/lib/kubelet/pods/f36337d3-669a-4d7d-a157-b8a5f78aff17/volumes" Oct 30 13:02:19.685587 ip-10-0-141-251 kubenswrapper[4908]: E1030 13:02:19.685547 4908 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vn6vv" containerName="extractor" Oct 30 13:02:20.181728 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:20.181676 4908 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-eab799/test-model-eab799-kserve-7b456659fc-bgzth" podUID="ad52e800-af7e-46e0-b834-0f86ffd55b96" containerName="main" containerID="cri-o://601c535f3ca2f8e911423cae0a4f0ea209d5ccd8da9d10b9edc81d543ce5edeb" gracePeriod=45 Oct 30 13:02:20.483564 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:20.483518 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-eab799/test-model-eab799-kserve-7b456659fc-bgzth" Oct 30 13:02:20.548398 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:20.548357 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad52e800-af7e-46e0-b834-0f86ffd55b96-home\" (UniqueName: \"kubernetes.io/empty-dir/ad52e800-af7e-46e0-b834-0f86ffd55b96-home\") pod \"ad52e800-af7e-46e0-b834-0f86ffd55b96\" (UID: \"ad52e800-af7e-46e0-b834-0f86ffd55b96\") " Oct 30 13:02:20.548594 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:20.548550 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad52e800-af7e-46e0-b834-0f86ffd55b96-kube-api-access-7npr4\" (UniqueName: \"kubernetes.io/projected/ad52e800-af7e-46e0-b834-0f86ffd55b96-kube-api-access-7npr4\") pod \"ad52e800-af7e-46e0-b834-0f86ffd55b96\" (UID: \"ad52e800-af7e-46e0-b834-0f86ffd55b96\") " Oct 30 13:02:20.548674 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:20.548657 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad52e800-af7e-46e0-b834-0f86ffd55b96-tls-certs\" (UniqueName: \"kubernetes.io/secret/ad52e800-af7e-46e0-b834-0f86ffd55b96-tls-certs\") pod \"ad52e800-af7e-46e0-b834-0f86ffd55b96\" (UID: \"ad52e800-af7e-46e0-b834-0f86ffd55b96\") " Oct 30 13:02:20.548734 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:20.548680 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad52e800-af7e-46e0-b834-0f86ffd55b96-home" pod "ad52e800-af7e-46e0-b834-0f86ffd55b96" (UID: "ad52e800-af7e-46e0-b834-0f86ffd55b96"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:02:20.548792 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:20.548773 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad52e800-af7e-46e0-b834-0f86ffd55b96-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad52e800-af7e-46e0-b834-0f86ffd55b96-tmp-dir\") pod \"ad52e800-af7e-46e0-b834-0f86ffd55b96\" (UID: \"ad52e800-af7e-46e0-b834-0f86ffd55b96\") " Oct 30 13:02:20.548889 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:20.548804 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad52e800-af7e-46e0-b834-0f86ffd55b96-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad52e800-af7e-46e0-b834-0f86ffd55b96-dshm\") pod \"ad52e800-af7e-46e0-b834-0f86ffd55b96\" (UID: \"ad52e800-af7e-46e0-b834-0f86ffd55b96\") " Oct 30 13:02:20.548955 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:20.548913 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad52e800-af7e-46e0-b834-0f86ffd55b96-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad52e800-af7e-46e0-b834-0f86ffd55b96-model-cache\") pod \"ad52e800-af7e-46e0-b834-0f86ffd55b96\" (UID: \"ad52e800-af7e-46e0-b834-0f86ffd55b96\") " Oct 30 13:02:20.549337 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:20.549309 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad52e800-af7e-46e0-b834-0f86ffd55b96-tmp-dir" pod "ad52e800-af7e-46e0-b834-0f86ffd55b96" (UID: "ad52e800-af7e-46e0-b834-0f86ffd55b96"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:02:20.549724 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:20.549700 4908 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad52e800-af7e-46e0-b834-0f86ffd55b96-home\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:02:20.549799 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:20.549731 4908 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad52e800-af7e-46e0-b834-0f86ffd55b96-tmp-dir\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:02:20.549799 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:20.549785 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad52e800-af7e-46e0-b834-0f86ffd55b96-model-cache" pod "ad52e800-af7e-46e0-b834-0f86ffd55b96" (UID: "ad52e800-af7e-46e0-b834-0f86ffd55b96"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:02:20.551776 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:20.551739 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad52e800-af7e-46e0-b834-0f86ffd55b96-kube-api-access-7npr4" pod "ad52e800-af7e-46e0-b834-0f86ffd55b96" (UID: "ad52e800-af7e-46e0-b834-0f86ffd55b96"). InnerVolumeSpecName "kube-api-access-7npr4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:02:20.552644 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:20.552613 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad52e800-af7e-46e0-b834-0f86ffd55b96-dshm" pod "ad52e800-af7e-46e0-b834-0f86ffd55b96" (UID: "ad52e800-af7e-46e0-b834-0f86ffd55b96"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:02:20.552746 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:20.552657 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad52e800-af7e-46e0-b834-0f86ffd55b96-tls-certs" pod "ad52e800-af7e-46e0-b834-0f86ffd55b96" (UID: "ad52e800-af7e-46e0-b834-0f86ffd55b96"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:02:20.675640 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:20.668235 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7npr4\" (UniqueName: \"kubernetes.io/projected/ad52e800-af7e-46e0-b834-0f86ffd55b96-kube-api-access-7npr4\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:02:20.675640 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:20.668270 4908 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad52e800-af7e-46e0-b834-0f86ffd55b96-tls-certs\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:02:20.675640 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:20.668286 4908 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad52e800-af7e-46e0-b834-0f86ffd55b96-dshm\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:02:20.675640 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:20.668299 4908 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad52e800-af7e-46e0-b834-0f86ffd55b96-model-cache\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:02:20.679880 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:20.679844 4908 generic.go:356] "Generic (PLEG): container finished" podID="ad52e800-af7e-46e0-b834-0f86ffd55b96" containerID="601c535f3ca2f8e911423cae0a4f0ea209d5ccd8da9d10b9edc81d543ce5edeb" exitCode=0 Oct 30 13:02:20.680004 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:20.679891 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-eab799/test-model-eab799-kserve-7b456659fc-bgzth" event={"ID":"ad52e800-af7e-46e0-b834-0f86ffd55b96","Type":"ContainerDied","Data":"601c535f3ca2f8e911423cae0a4f0ea209d5ccd8da9d10b9edc81d543ce5edeb"} Oct 30 13:02:20.680004 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:20.679944 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-eab799/test-model-eab799-kserve-7b456659fc-bgzth" event={"ID":"ad52e800-af7e-46e0-b834-0f86ffd55b96","Type":"ContainerDied","Data":"9d204b4f2dff3b913ff6383f5c4255a4926aca0ada052f291c6528ddac6ce9fe"} Oct 30 13:02:20.680004 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:20.679903 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-eab799/test-model-eab799-kserve-7b456659fc-bgzth" Oct 30 13:02:20.680004 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:20.679964 4908 scope.go:122] "RemoveContainer" containerID="601c535f3ca2f8e911423cae0a4f0ea209d5ccd8da9d10b9edc81d543ce5edeb" Oct 30 13:02:20.693940 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:20.693912 4908 scope.go:122] "RemoveContainer" containerID="601c535f3ca2f8e911423cae0a4f0ea209d5ccd8da9d10b9edc81d543ce5edeb" Oct 30 13:02:20.694300 ip-10-0-141-251 kubenswrapper[4908]: E1030 13:02:20.694239 4908 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"601c535f3ca2f8e911423cae0a4f0ea209d5ccd8da9d10b9edc81d543ce5edeb\": container with ID starting with 601c535f3ca2f8e911423cae0a4f0ea209d5ccd8da9d10b9edc81d543ce5edeb not found: ID does not exist" containerID="601c535f3ca2f8e911423cae0a4f0ea209d5ccd8da9d10b9edc81d543ce5edeb" Oct 30 13:02:20.694300 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:20.694266 4908 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"601c535f3ca2f8e911423cae0a4f0ea209d5ccd8da9d10b9edc81d543ce5edeb"} err="failed to get container status \"601c535f3ca2f8e911423cae0a4f0ea209d5ccd8da9d10b9edc81d543ce5edeb\": rpc error: code = NotFound desc = could not find container \"601c535f3ca2f8e911423cae0a4f0ea209d5ccd8da9d10b9edc81d543ce5edeb\": container with ID starting with 601c535f3ca2f8e911423cae0a4f0ea209d5ccd8da9d10b9edc81d543ce5edeb not found: ID does not exist" Oct 30 13:02:20.741020 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:20.740981 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-eab799/test-model-eab799-kserve-7b456659fc-bgzth"] Oct 30 13:02:20.751219 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:20.751181 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-eab799/test-model-eab799-kserve-7b456659fc-bgzth"] Oct 30 13:02:21.607168 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:21.607134 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad52e800-af7e-46e0-b834-0f86ffd55b96" path="/var/lib/kubelet/pods/ad52e800-af7e-46e0-b834-0f86ffd55b96/volumes" Oct 30 13:02:22.021151 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:22.021114 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-e9f184/test-model-e9f184-kserve-86dbfc9485-g9lm6"] Oct 30 13:02:22.262004 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:22.261786 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-eab799-hs98g"] Oct 30 13:02:22.263269 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:22.262726 4908 cpu_manager.go:414] "containerMap: removing container" podUID="a81e1728-369a-4022-8bc7-3c3726900efd" containerName="revoke-keys" Oct 30 13:02:22.263269 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:22.262752 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a81e1728-369a-4022-8bc7-3c3726900efd" containerName="revoke-keys" Oct 30 13:02:22.263269 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:22.262788 4908 cpu_manager.go:414] "containerMap: removing container" podUID="4060653d-76ee-4df6-a077-a6c7bb2a6a5d" containerName="revoke-keys" Oct 30 13:02:22.263269 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:22.262797 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4060653d-76ee-4df6-a077-a6c7bb2a6a5d" containerName="revoke-keys" Oct 30 13:02:22.263269 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:22.262862 4908 cpu_manager.go:414] "containerMap: removing container" podUID="2068d5c3-6f62-4bd1-bbf1-20a12a84725c" containerName="istio-proxy" Oct 30 13:02:22.263269 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:22.262872 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2068d5c3-6f62-4bd1-bbf1-20a12a84725c" containerName="istio-proxy" Oct 30 13:02:22.263269 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:22.262886 4908 cpu_manager.go:414] "containerMap: removing container" podUID="f36337d3-669a-4d7d-a157-b8a5f78aff17" containerName="istio-proxy" Oct 30 13:02:22.263269 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:22.262893 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f36337d3-669a-4d7d-a157-b8a5f78aff17" containerName="istio-proxy" Oct 30 13:02:22.263269 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:22.262927 4908 cpu_manager.go:414] "containerMap: removing container" podUID="ad52e800-af7e-46e0-b834-0f86ffd55b96" containerName="main" Oct 30 13:02:22.263269 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:22.262934 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad52e800-af7e-46e0-b834-0f86ffd55b96" containerName="main" Oct 30 13:02:22.263269 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:22.263136 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="2068d5c3-6f62-4bd1-bbf1-20a12a84725c" containerName="istio-proxy" Oct 30 13:02:22.263269 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:22.263150 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="f36337d3-669a-4d7d-a157-b8a5f78aff17" containerName="istio-proxy" Oct 30 13:02:22.263269 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:22.263170 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad52e800-af7e-46e0-b834-0f86ffd55b96" containerName="main" Oct 30 13:02:22.263269 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:22.263182 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="a81e1728-369a-4022-8bc7-3c3726900efd" containerName="revoke-keys" Oct 30 13:02:22.263269 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:22.263195 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="4060653d-76ee-4df6-a077-a6c7bb2a6a5d" containerName="revoke-keys" Oct 30 13:02:22.302113 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:22.301249 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-eab799-hs98g" Oct 30 13:02:22.305231 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:22.304424 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-eab799-hs98g"] Oct 30 13:02:22.341858 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:22.330927 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 13:02:22.341858 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:22.331646 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:02:22.341858 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:22.331748 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s75hq\"" Oct 30 13:02:22.341858 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:22.331804 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 13:02:22.401752 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:22.401719 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zclzg\" (UniqueName: \"kubernetes.io/projected/133a27f0-7837-4fb3-9803-865ec0dd47ba-kube-api-access-zclzg\") pod \"maas-api-revoke-keys-e2e-inf-a-eab799-hs98g\" (UID: \"133a27f0-7837-4fb3-9803-865ec0dd47ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-eab799-hs98g" Oct 30 13:02:22.402021 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:22.401981 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/133a27f0-7837-4fb3-9803-865ec0dd47ba-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-eab799-hs98g\" (UID: \"133a27f0-7837-4fb3-9803-865ec0dd47ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-eab799-hs98g" Oct 30 13:02:22.402121 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:22.402102 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/133a27f0-7837-4fb3-9803-865ec0dd47ba-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-eab799-hs98g\" (UID: \"133a27f0-7837-4fb3-9803-865ec0dd47ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-eab799-hs98g" Oct 30 13:02:22.504107 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:22.504063 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zclzg\" (UniqueName: \"kubernetes.io/projected/133a27f0-7837-4fb3-9803-865ec0dd47ba-kube-api-access-zclzg\") pod \"maas-api-revoke-keys-e2e-inf-a-eab799-hs98g\" (UID: \"133a27f0-7837-4fb3-9803-865ec0dd47ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-eab799-hs98g" Oct 30 13:02:22.504329 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:22.504149 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/133a27f0-7837-4fb3-9803-865ec0dd47ba-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-eab799-hs98g\" (UID: \"133a27f0-7837-4fb3-9803-865ec0dd47ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-eab799-hs98g" Oct 30 13:02:22.504329 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:22.504237 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/133a27f0-7837-4fb3-9803-865ec0dd47ba-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-eab799-hs98g\" (UID: \"133a27f0-7837-4fb3-9803-865ec0dd47ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-eab799-hs98g" Oct 30 13:02:22.505249 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:22.505217 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/133a27f0-7837-4fb3-9803-865ec0dd47ba-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-eab799-hs98g\" (UID: \"133a27f0-7837-4fb3-9803-865ec0dd47ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-eab799-hs98g" Oct 30 13:02:22.506758 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:22.506723 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/133a27f0-7837-4fb3-9803-865ec0dd47ba-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-eab799-hs98g\" (UID: \"133a27f0-7837-4fb3-9803-865ec0dd47ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-eab799-hs98g" Oct 30 13:02:22.551848 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:22.551798 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zclzg\" (UniqueName: \"kubernetes.io/projected/133a27f0-7837-4fb3-9803-865ec0dd47ba-kube-api-access-zclzg\") pod \"maas-api-revoke-keys-e2e-inf-a-eab799-hs98g\" (UID: \"133a27f0-7837-4fb3-9803-865ec0dd47ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-eab799-hs98g" Oct 30 13:02:22.662528 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:22.662490 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-eab799-hs98g" Oct 30 13:02:23.084100 ip-10-0-141-251 kubenswrapper[4908]: W1030 13:02:23.084060 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod133a27f0_7837_4fb3_9803_865ec0dd47ba.slice/crio-86e0786083c916495937111f0d0745b27934ae9cc85c92a95c5c863ca9a3c5ee WatchSource:0}: Error finding container 86e0786083c916495937111f0d0745b27934ae9cc85c92a95c5c863ca9a3c5ee: Status 404 returned error can't find the container with id 86e0786083c916495937111f0d0745b27934ae9cc85c92a95c5c863ca9a3c5ee Oct 30 13:02:23.086426 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:23.086393 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-eab799-hs98g"] Oct 30 13:02:23.787269 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:23.787230 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-eab799-hs98g" event={"ID":"133a27f0-7837-4fb3-9803-865ec0dd47ba","Type":"ContainerStarted","Data":"51d13c6c039e1f4f79a365fe51d17708279d7eb42ee276ca110cf5ff7075f5d6"} Oct 30 13:02:23.787449 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:23.787287 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-eab799-hs98g" event={"ID":"133a27f0-7837-4fb3-9803-865ec0dd47ba","Type":"ContainerStarted","Data":"86e0786083c916495937111f0d0745b27934ae9cc85c92a95c5c863ca9a3c5ee"} Oct 30 13:02:23.833053 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:23.832995 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-eab799-hs98g" podStartSLOduration=1.8329817 podStartE2EDuration="1.8329817s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:02:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:02:23.829375133 +0000 UTC m=+4100.343803880" watchObservedRunningTime="2026-10-30 13:02:23.8329817 +0000 UTC m=+4100.347410447" Oct 30 13:02:24.810532 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:24.810490 4908 generic.go:356] "Generic (PLEG): container finished" podID="133a27f0-7837-4fb3-9803-865ec0dd47ba" containerID="51d13c6c039e1f4f79a365fe51d17708279d7eb42ee276ca110cf5ff7075f5d6" exitCode=0 Oct 30 13:02:24.814030 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:24.813971 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-eab799-hs98g" event={"ID":"133a27f0-7837-4fb3-9803-865ec0dd47ba","Type":"ContainerDied","Data":"51d13c6c039e1f4f79a365fe51d17708279d7eb42ee276ca110cf5ff7075f5d6"} Oct 30 13:02:25.993086 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:25.993054 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-eab799-hs98g" Oct 30 13:02:26.069681 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:26.069630 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/133a27f0-7837-4fb3-9803-865ec0dd47ba-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/133a27f0-7837-4fb3-9803-865ec0dd47ba-maas-api-xks-ca\") pod \"133a27f0-7837-4fb3-9803-865ec0dd47ba\" (UID: \"133a27f0-7837-4fb3-9803-865ec0dd47ba\") " Oct 30 13:02:26.069902 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:26.069780 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/133a27f0-7837-4fb3-9803-865ec0dd47ba-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/133a27f0-7837-4fb3-9803-865ec0dd47ba-maas-api-openshift-ca\") pod \"133a27f0-7837-4fb3-9803-865ec0dd47ba\" (UID: \"133a27f0-7837-4fb3-9803-865ec0dd47ba\") " Oct 30 13:02:26.069980 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:26.069955 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/133a27f0-7837-4fb3-9803-865ec0dd47ba-kube-api-access-zclzg\" (UniqueName: \"kubernetes.io/projected/133a27f0-7837-4fb3-9803-865ec0dd47ba-kube-api-access-zclzg\") pod \"133a27f0-7837-4fb3-9803-865ec0dd47ba\" (UID: \"133a27f0-7837-4fb3-9803-865ec0dd47ba\") " Oct 30 13:02:26.070774 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:26.070498 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/133a27f0-7837-4fb3-9803-865ec0dd47ba-maas-api-openshift-ca" pod "133a27f0-7837-4fb3-9803-865ec0dd47ba" (UID: "133a27f0-7837-4fb3-9803-865ec0dd47ba"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:02:26.070956 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:26.070775 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/133a27f0-7837-4fb3-9803-865ec0dd47ba-maas-api-openshift-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:02:26.072671 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:26.072641 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/133a27f0-7837-4fb3-9803-865ec0dd47ba-maas-api-xks-ca" pod "133a27f0-7837-4fb3-9803-865ec0dd47ba" (UID: "133a27f0-7837-4fb3-9803-865ec0dd47ba"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:02:26.073747 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:26.073719 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/133a27f0-7837-4fb3-9803-865ec0dd47ba-kube-api-access-zclzg" pod "133a27f0-7837-4fb3-9803-865ec0dd47ba" (UID: "133a27f0-7837-4fb3-9803-865ec0dd47ba"). InnerVolumeSpecName "kube-api-access-zclzg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:02:26.171841 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:26.171791 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/133a27f0-7837-4fb3-9803-865ec0dd47ba-maas-api-xks-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:02:26.171841 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:26.171819 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zclzg\" (UniqueName: \"kubernetes.io/projected/133a27f0-7837-4fb3-9803-865ec0dd47ba-kube-api-access-zclzg\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:02:26.835068 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:26.835034 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-eab799-hs98g" Oct 30 13:02:26.835068 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:26.835057 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-eab799-hs98g" event={"ID":"133a27f0-7837-4fb3-9803-865ec0dd47ba","Type":"ContainerDied","Data":"86e0786083c916495937111f0d0745b27934ae9cc85c92a95c5c863ca9a3c5ee"} Oct 30 13:02:26.835302 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:26.835113 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="86e0786083c916495937111f0d0745b27934ae9cc85c92a95c5c863ca9a3c5ee" Oct 30 13:02:34.392716 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:34.392664 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5smp6"] Oct 30 13:02:34.393200 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:34.392937 4908 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5smp6" podUID="2f090df9-3e00-4fec-9008-8088c2ae5f93" containerName="istio-proxy" containerID="cri-o://e8fceedfc1472209f34a8974d35c70a2372deaac130a5f368026bede217b1724" gracePeriod=30 Oct 30 13:02:34.406473 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:34.406430 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xnnfx"] Oct 30 13:02:34.406754 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:34.406706 4908 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xnnfx" podUID="03eea3e0-0b0d-459d-b03b-3a2531f20185" containerName="istio-proxy" containerID="cri-o://1cfa6e9bcd78f5e2646de2ee39cece3863dea75df13a71f952827ee871a002ef" gracePeriod=30 Oct 30 13:02:34.412516 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:34.412481 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-d9b94"] Oct 30 13:02:34.412743 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:34.412712 4908 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-d9b94" podUID="a5b1685b-e185-4b12-9c20-3c87e9837fd7" containerName="istio-proxy" containerID="cri-o://4631e9016d55731633f9fff9d825d54981feba64ae6b2918e84ed19e90831a3b" gracePeriod=30 Oct 30 13:02:34.417678 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:34.417594 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xcqb4"] Oct 30 13:02:34.417962 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:34.417928 4908 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xcqb4" podUID="d2f439fb-154c-4997-b0f8-f18b275d262f" containerName="istio-proxy" containerID="cri-o://ef9aa7c8f24ef453f1e99c8a9ba70dd90e9e893a2982db650c44751916ce037d" gracePeriod=30 Oct 30 13:02:36.334847 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:36.330902 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-f7464f-qsmck"] Oct 30 13:02:36.373735 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:36.367272 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-f7464f-qsmck"] Oct 30 13:02:37.043231 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:37.043185 4908 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-e9f184/test-model-e9f184-kserve-86dbfc9485-g9lm6" podUID="8396834c-234e-431a-8e62-97c193ce6b66" containerName="main" containerID="cri-o://b68646389cae46ea141fd7361c2285df0eaa0dfd5d29ca1a62fea426df161299" gracePeriod=45 Oct 30 13:02:37.143317 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:37.143275 4908 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-d9b94 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 13:02:37.143484 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:37.143327 4908 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-d9b94" podUID="a5b1685b-e185-4b12-9c20-3c87e9837fd7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 13:02:37.343626 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:37.343600 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-e9f184/test-model-e9f184-kserve-86dbfc9485-g9lm6" Oct 30 13:02:37.423474 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:37.423379 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8396834c-234e-431a-8e62-97c193ce6b66-home\" (UniqueName: \"kubernetes.io/empty-dir/8396834c-234e-431a-8e62-97c193ce6b66-home\") pod \"8396834c-234e-431a-8e62-97c193ce6b66\" (UID: \"8396834c-234e-431a-8e62-97c193ce6b66\") " Oct 30 13:02:37.423474 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:37.423446 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8396834c-234e-431a-8e62-97c193ce6b66-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8396834c-234e-431a-8e62-97c193ce6b66-model-cache\") pod \"8396834c-234e-431a-8e62-97c193ce6b66\" (UID: \"8396834c-234e-431a-8e62-97c193ce6b66\") " Oct 30 13:02:37.423799 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:37.423607 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8396834c-234e-431a-8e62-97c193ce6b66-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8396834c-234e-431a-8e62-97c193ce6b66-tmp-dir\") pod \"8396834c-234e-431a-8e62-97c193ce6b66\" (UID: \"8396834c-234e-431a-8e62-97c193ce6b66\") " Oct 30 13:02:37.423799 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:37.423657 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8396834c-234e-431a-8e62-97c193ce6b66-kube-api-access-4kzl8\" (UniqueName: \"kubernetes.io/projected/8396834c-234e-431a-8e62-97c193ce6b66-kube-api-access-4kzl8\") pod \"8396834c-234e-431a-8e62-97c193ce6b66\" (UID: \"8396834c-234e-431a-8e62-97c193ce6b66\") " Oct 30 13:02:37.423799 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:37.423694 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8396834c-234e-431a-8e62-97c193ce6b66-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8396834c-234e-431a-8e62-97c193ce6b66-dshm\") pod \"8396834c-234e-431a-8e62-97c193ce6b66\" (UID: \"8396834c-234e-431a-8e62-97c193ce6b66\") " Oct 30 13:02:37.424020 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:37.423910 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8396834c-234e-431a-8e62-97c193ce6b66-tls-certs\" (UniqueName: \"kubernetes.io/secret/8396834c-234e-431a-8e62-97c193ce6b66-tls-certs\") pod \"8396834c-234e-431a-8e62-97c193ce6b66\" (UID: \"8396834c-234e-431a-8e62-97c193ce6b66\") " Oct 30 13:02:37.424113 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:37.424087 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8396834c-234e-431a-8e62-97c193ce6b66-home" pod "8396834c-234e-431a-8e62-97c193ce6b66" (UID: "8396834c-234e-431a-8e62-97c193ce6b66"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:02:37.424339 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:37.424320 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8396834c-234e-431a-8e62-97c193ce6b66-tmp-dir" pod "8396834c-234e-431a-8e62-97c193ce6b66" (UID: "8396834c-234e-431a-8e62-97c193ce6b66"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:02:37.424790 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:37.424680 4908 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8396834c-234e-431a-8e62-97c193ce6b66-home\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:02:37.424998 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:37.424789 4908 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8396834c-234e-431a-8e62-97c193ce6b66-tmp-dir\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:02:37.425193 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:37.425131 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8396834c-234e-431a-8e62-97c193ce6b66-model-cache" pod "8396834c-234e-431a-8e62-97c193ce6b66" (UID: "8396834c-234e-431a-8e62-97c193ce6b66"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:02:37.427411 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:37.427388 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8396834c-234e-431a-8e62-97c193ce6b66-kube-api-access-4kzl8" pod "8396834c-234e-431a-8e62-97c193ce6b66" (UID: "8396834c-234e-431a-8e62-97c193ce6b66"). InnerVolumeSpecName "kube-api-access-4kzl8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:02:37.427668 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:37.427650 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8396834c-234e-431a-8e62-97c193ce6b66-dshm" pod "8396834c-234e-431a-8e62-97c193ce6b66" (UID: "8396834c-234e-431a-8e62-97c193ce6b66"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:02:37.429790 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:37.429764 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8396834c-234e-431a-8e62-97c193ce6b66-tls-certs" pod "8396834c-234e-431a-8e62-97c193ce6b66" (UID: "8396834c-234e-431a-8e62-97c193ce6b66"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:02:37.526809 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:37.526772 4908 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8396834c-234e-431a-8e62-97c193ce6b66-model-cache\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:02:37.526809 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:37.526807 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4kzl8\" (UniqueName: \"kubernetes.io/projected/8396834c-234e-431a-8e62-97c193ce6b66-kube-api-access-4kzl8\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:02:37.526809 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:37.526819 4908 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8396834c-234e-431a-8e62-97c193ce6b66-dshm\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:02:37.527075 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:37.526842 4908 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8396834c-234e-431a-8e62-97c193ce6b66-tls-certs\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:02:37.608655 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:37.608560 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a81e1728-369a-4022-8bc7-3c3726900efd" path="/var/lib/kubelet/pods/a81e1728-369a-4022-8bc7-3c3726900efd/volumes" Oct 30 13:02:38.140522 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:38.140485 4908 generic.go:356] "Generic (PLEG): container finished" podID="8396834c-234e-431a-8e62-97c193ce6b66" containerID="b68646389cae46ea141fd7361c2285df0eaa0dfd5d29ca1a62fea426df161299" exitCode=0 Oct 30 13:02:38.140522 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:38.140521 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-e9f184/test-model-e9f184-kserve-86dbfc9485-g9lm6" Oct 30 13:02:38.140773 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:38.140559 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-e9f184/test-model-e9f184-kserve-86dbfc9485-g9lm6" event={"ID":"8396834c-234e-431a-8e62-97c193ce6b66","Type":"ContainerDied","Data":"b68646389cae46ea141fd7361c2285df0eaa0dfd5d29ca1a62fea426df161299"} Oct 30 13:02:38.140773 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:38.140602 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-e9f184/test-model-e9f184-kserve-86dbfc9485-g9lm6" event={"ID":"8396834c-234e-431a-8e62-97c193ce6b66","Type":"ContainerDied","Data":"67b9efcbbc9fbcf79346699f505696761695d0d889822cf379e2d1fe36ac62b5"} Oct 30 13:02:38.140773 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:38.140619 4908 scope.go:122] "RemoveContainer" containerID="b68646389cae46ea141fd7361c2285df0eaa0dfd5d29ca1a62fea426df161299" Oct 30 13:02:38.151548 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:38.151527 4908 scope.go:122] "RemoveContainer" containerID="b68646389cae46ea141fd7361c2285df0eaa0dfd5d29ca1a62fea426df161299" Oct 30 13:02:38.151867 ip-10-0-141-251 kubenswrapper[4908]: E1030 13:02:38.151824 4908 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b68646389cae46ea141fd7361c2285df0eaa0dfd5d29ca1a62fea426df161299\": container with ID starting with b68646389cae46ea141fd7361c2285df0eaa0dfd5d29ca1a62fea426df161299 not found: ID does not exist" containerID="b68646389cae46ea141fd7361c2285df0eaa0dfd5d29ca1a62fea426df161299" Oct 30 13:02:38.151938 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:38.151879 4908 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b68646389cae46ea141fd7361c2285df0eaa0dfd5d29ca1a62fea426df161299"} err="failed to get container status \"b68646389cae46ea141fd7361c2285df0eaa0dfd5d29ca1a62fea426df161299\": rpc error: code = NotFound desc = could not find container \"b68646389cae46ea141fd7361c2285df0eaa0dfd5d29ca1a62fea426df161299\": container with ID starting with b68646389cae46ea141fd7361c2285df0eaa0dfd5d29ca1a62fea426df161299 not found: ID does not exist" Oct 30 13:02:38.259848 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:38.259046 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-e9f184/test-model-e9f184-kserve-86dbfc9485-g9lm6"] Oct 30 13:02:38.265740 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:38.265683 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-e9f184/test-model-e9f184-kserve-86dbfc9485-g9lm6"] Oct 30 13:02:39.670575 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:39.670482 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8396834c-234e-431a-8e62-97c193ce6b66" path="/var/lib/kubelet/pods/8396834c-234e-431a-8e62-97c193ce6b66/volumes" Oct 30 13:02:39.827597 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:39.827566 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5smp6" Oct 30 13:02:39.835373 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:39.835343 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xnnfx" Oct 30 13:02:39.845482 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:39.845452 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xcqb4" Oct 30 13:02:39.970401 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:39.970347 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-d9b94" Oct 30 13:02:39.982974 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:39.979704 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f090df9-3e00-4fec-9008-8088c2ae5f93-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f090df9-3e00-4fec-9008-8088c2ae5f93-istio-envoy\") pod \"2f090df9-3e00-4fec-9008-8088c2ae5f93\" (UID: \"2f090df9-3e00-4fec-9008-8088c2ae5f93\") " Oct 30 13:02:39.982974 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:39.979751 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5b1685b-e185-4b12-9c20-3c87e9837fd7-istio-token\" (UniqueName: \"kubernetes.io/projected/a5b1685b-e185-4b12-9c20-3c87e9837fd7-istio-token\") pod \"a5b1685b-e185-4b12-9c20-3c87e9837fd7\" (UID: \"a5b1685b-e185-4b12-9c20-3c87e9837fd7\") " Oct 30 13:02:39.982974 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:39.979804 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5b1685b-e185-4b12-9c20-3c87e9837fd7-kube-api-access-l5zfh\" (UniqueName: \"kubernetes.io/projected/a5b1685b-e185-4b12-9c20-3c87e9837fd7-kube-api-access-l5zfh\") pod \"a5b1685b-e185-4b12-9c20-3c87e9837fd7\" (UID: \"a5b1685b-e185-4b12-9c20-3c87e9837fd7\") " Oct 30 13:02:39.982974 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:39.979886 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f090df9-3e00-4fec-9008-8088c2ae5f93-istio-token\" (UniqueName: \"kubernetes.io/projected/2f090df9-3e00-4fec-9008-8088c2ae5f93-istio-token\") pod \"2f090df9-3e00-4fec-9008-8088c2ae5f93\" (UID: \"2f090df9-3e00-4fec-9008-8088c2ae5f93\") " Oct 30 13:02:39.982974 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:39.979936 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2f439fb-154c-4997-b0f8-f18b275d262f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2f439fb-154c-4997-b0f8-f18b275d262f-workload-socket\") pod \"d2f439fb-154c-4997-b0f8-f18b275d262f\" (UID: \"d2f439fb-154c-4997-b0f8-f18b275d262f\") " Oct 30 13:02:39.982974 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:39.979970 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f090df9-3e00-4fec-9008-8088c2ae5f93-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f090df9-3e00-4fec-9008-8088c2ae5f93-credential-socket\") pod \"2f090df9-3e00-4fec-9008-8088c2ae5f93\" (UID: \"2f090df9-3e00-4fec-9008-8088c2ae5f93\") " Oct 30 13:02:39.982974 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:39.980024 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d2f439fb-154c-4997-b0f8-f18b275d262f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2f439fb-154c-4997-b0f8-f18b275d262f-istio-podinfo\") pod \"d2f439fb-154c-4997-b0f8-f18b275d262f\" (UID: \"d2f439fb-154c-4997-b0f8-f18b275d262f\") " Oct 30 13:02:39.982974 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:39.980086 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03eea3e0-0b0d-459d-b03b-3a2531f20185-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03eea3e0-0b0d-459d-b03b-3a2531f20185-workload-socket\") pod \"03eea3e0-0b0d-459d-b03b-3a2531f20185\" (UID: \"03eea3e0-0b0d-459d-b03b-3a2531f20185\") " Oct 30 13:02:39.982974 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:39.980266 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2f439fb-154c-4997-b0f8-f18b275d262f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2f439fb-154c-4997-b0f8-f18b275d262f-istio-envoy\") pod \"d2f439fb-154c-4997-b0f8-f18b275d262f\" (UID: \"d2f439fb-154c-4997-b0f8-f18b275d262f\") " Oct 30 13:02:39.982974 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:39.980315 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03eea3e0-0b0d-459d-b03b-3a2531f20185-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03eea3e0-0b0d-459d-b03b-3a2531f20185-istio-data\") pod \"03eea3e0-0b0d-459d-b03b-3a2531f20185\" (UID: \"03eea3e0-0b0d-459d-b03b-3a2531f20185\") " Oct 30 13:02:39.982974 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:39.980359 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03eea3e0-0b0d-459d-b03b-3a2531f20185-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03eea3e0-0b0d-459d-b03b-3a2531f20185-credential-socket\") pod \"03eea3e0-0b0d-459d-b03b-3a2531f20185\" (UID: \"03eea3e0-0b0d-459d-b03b-3a2531f20185\") " Oct 30 13:02:39.982974 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:39.980401 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2f439fb-154c-4997-b0f8-f18b275d262f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2f439fb-154c-4997-b0f8-f18b275d262f-istiod-ca-cert\") pod \"d2f439fb-154c-4997-b0f8-f18b275d262f\" (UID: \"d2f439fb-154c-4997-b0f8-f18b275d262f\") " Oct 30 13:02:39.982974 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:39.980444 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/03eea3e0-0b0d-459d-b03b-3a2531f20185-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03eea3e0-0b0d-459d-b03b-3a2531f20185-istio-podinfo\") pod \"03eea3e0-0b0d-459d-b03b-3a2531f20185\" (UID: \"03eea3e0-0b0d-459d-b03b-3a2531f20185\") " Oct 30 13:02:39.982974 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:39.980483 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03eea3e0-0b0d-459d-b03b-3a2531f20185-kube-api-access-7pjf2\" (UniqueName: \"kubernetes.io/projected/03eea3e0-0b0d-459d-b03b-3a2531f20185-kube-api-access-7pjf2\") pod \"03eea3e0-0b0d-459d-b03b-3a2531f20185\" (UID: \"03eea3e0-0b0d-459d-b03b-3a2531f20185\") " Oct 30 13:02:39.982974 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:39.980526 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03eea3e0-0b0d-459d-b03b-3a2531f20185-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03eea3e0-0b0d-459d-b03b-3a2531f20185-workload-certs\") pod \"03eea3e0-0b0d-459d-b03b-3a2531f20185\" (UID: \"03eea3e0-0b0d-459d-b03b-3a2531f20185\") " Oct 30 13:02:39.982974 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:39.980585 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f090df9-3e00-4fec-9008-8088c2ae5f93-kube-api-access-fkpbp\" (UniqueName: \"kubernetes.io/projected/2f090df9-3e00-4fec-9008-8088c2ae5f93-kube-api-access-fkpbp\") pod \"2f090df9-3e00-4fec-9008-8088c2ae5f93\" (UID: \"2f090df9-3e00-4fec-9008-8088c2ae5f93\") " Oct 30 13:02:39.982974 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:39.980612 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f090df9-3e00-4fec-9008-8088c2ae5f93-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f090df9-3e00-4fec-9008-8088c2ae5f93-workload-socket\") pod \"2f090df9-3e00-4fec-9008-8088c2ae5f93\" (UID: \"2f090df9-3e00-4fec-9008-8088c2ae5f93\") " Oct 30 13:02:39.982974 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:39.980663 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03eea3e0-0b0d-459d-b03b-3a2531f20185-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03eea3e0-0b0d-459d-b03b-3a2531f20185-istio-envoy\") pod \"03eea3e0-0b0d-459d-b03b-3a2531f20185\" (UID: \"03eea3e0-0b0d-459d-b03b-3a2531f20185\") " Oct 30 13:02:39.982974 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:39.980730 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f090df9-3e00-4fec-9008-8088c2ae5f93-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f090df9-3e00-4fec-9008-8088c2ae5f93-workload-certs\") pod \"2f090df9-3e00-4fec-9008-8088c2ae5f93\" (UID: \"2f090df9-3e00-4fec-9008-8088c2ae5f93\") " Oct 30 13:02:39.982974 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:39.980811 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5b1685b-e185-4b12-9c20-3c87e9837fd7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5b1685b-e185-4b12-9c20-3c87e9837fd7-credential-socket\") pod \"a5b1685b-e185-4b12-9c20-3c87e9837fd7\" (UID: \"a5b1685b-e185-4b12-9c20-3c87e9837fd7\") " Oct 30 13:02:39.982974 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:39.980896 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5b1685b-e185-4b12-9c20-3c87e9837fd7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5b1685b-e185-4b12-9c20-3c87e9837fd7-istiod-ca-cert\") pod \"a5b1685b-e185-4b12-9c20-3c87e9837fd7\" (UID: \"a5b1685b-e185-4b12-9c20-3c87e9837fd7\") " Oct 30 13:02:39.982974 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:39.980939 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5b1685b-e185-4b12-9c20-3c87e9837fd7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5b1685b-e185-4b12-9c20-3c87e9837fd7-istio-envoy\") pod \"a5b1685b-e185-4b12-9c20-3c87e9837fd7\" (UID: \"a5b1685b-e185-4b12-9c20-3c87e9837fd7\") " Oct 30 13:02:39.982974 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:39.981075 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03eea3e0-0b0d-459d-b03b-3a2531f20185-workload-socket" pod "03eea3e0-0b0d-459d-b03b-3a2531f20185" (UID: "03eea3e0-0b0d-459d-b03b-3a2531f20185"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:02:39.982974 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:39.981313 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2f439fb-154c-4997-b0f8-f18b275d262f-workload-socket" pod "d2f439fb-154c-4997-b0f8-f18b275d262f" (UID: "d2f439fb-154c-4997-b0f8-f18b275d262f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:02:39.982974 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:39.981519 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f090df9-3e00-4fec-9008-8088c2ae5f93-credential-socket" pod "2f090df9-3e00-4fec-9008-8088c2ae5f93" (UID: "2f090df9-3e00-4fec-9008-8088c2ae5f93"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:02:39.982974 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:39.982145 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03eea3e0-0b0d-459d-b03b-3a2531f20185-istio-data" pod "03eea3e0-0b0d-459d-b03b-3a2531f20185" (UID: "03eea3e0-0b0d-459d-b03b-3a2531f20185"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:02:39.982974 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:39.982427 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03eea3e0-0b0d-459d-b03b-3a2531f20185-credential-socket" pod "03eea3e0-0b0d-459d-b03b-3a2531f20185" (UID: "03eea3e0-0b0d-459d-b03b-3a2531f20185"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:02:39.982974 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:39.982666 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2f439fb-154c-4997-b0f8-f18b275d262f-istiod-ca-cert" pod "d2f439fb-154c-4997-b0f8-f18b275d262f" (UID: "d2f439fb-154c-4997-b0f8-f18b275d262f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:02:39.982974 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:39.980985 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5b1685b-e185-4b12-9c20-3c87e9837fd7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5b1685b-e185-4b12-9c20-3c87e9837fd7-workload-certs\") pod \"a5b1685b-e185-4b12-9c20-3c87e9837fd7\" (UID: \"a5b1685b-e185-4b12-9c20-3c87e9837fd7\") " Oct 30 13:02:39.982974 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:39.982981 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5b1685b-e185-4b12-9c20-3c87e9837fd7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5b1685b-e185-4b12-9c20-3c87e9837fd7-istio-data\") pod \"a5b1685b-e185-4b12-9c20-3c87e9837fd7\" (UID: \"a5b1685b-e185-4b12-9c20-3c87e9837fd7\") " Oct 30 13:02:39.985648 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:39.983067 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5b1685b-e185-4b12-9c20-3c87e9837fd7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5b1685b-e185-4b12-9c20-3c87e9837fd7-workload-socket\") pod \"a5b1685b-e185-4b12-9c20-3c87e9837fd7\" (UID: \"a5b1685b-e185-4b12-9c20-3c87e9837fd7\") " Oct 30 13:02:39.985648 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:39.983108 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2f439fb-154c-4997-b0f8-f18b275d262f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2f439fb-154c-4997-b0f8-f18b275d262f-workload-certs\") pod \"d2f439fb-154c-4997-b0f8-f18b275d262f\" (UID: \"d2f439fb-154c-4997-b0f8-f18b275d262f\") " Oct 30 13:02:39.985648 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:39.983161 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03eea3e0-0b0d-459d-b03b-3a2531f20185-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03eea3e0-0b0d-459d-b03b-3a2531f20185-istiod-ca-cert\") pod \"03eea3e0-0b0d-459d-b03b-3a2531f20185\" (UID: \"03eea3e0-0b0d-459d-b03b-3a2531f20185\") " Oct 30 13:02:39.985648 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:39.983215 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2f439fb-154c-4997-b0f8-f18b275d262f-istio-token\" (UniqueName: \"kubernetes.io/projected/d2f439fb-154c-4997-b0f8-f18b275d262f-istio-token\") pod \"d2f439fb-154c-4997-b0f8-f18b275d262f\" (UID: \"d2f439fb-154c-4997-b0f8-f18b275d262f\") " Oct 30 13:02:39.985648 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:39.983260 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a5b1685b-e185-4b12-9c20-3c87e9837fd7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5b1685b-e185-4b12-9c20-3c87e9837fd7-istio-podinfo\") pod \"a5b1685b-e185-4b12-9c20-3c87e9837fd7\" (UID: \"a5b1685b-e185-4b12-9c20-3c87e9837fd7\") " Oct 30 13:02:39.985648 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:39.983326 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2f439fb-154c-4997-b0f8-f18b275d262f-kube-api-access-4pgm7\" (UniqueName: \"kubernetes.io/projected/d2f439fb-154c-4997-b0f8-f18b275d262f-kube-api-access-4pgm7\") pod \"d2f439fb-154c-4997-b0f8-f18b275d262f\" (UID: \"d2f439fb-154c-4997-b0f8-f18b275d262f\") " Oct 30 13:02:39.985648 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:39.983372 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2f090df9-3e00-4fec-9008-8088c2ae5f93-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f090df9-3e00-4fec-9008-8088c2ae5f93-istio-podinfo\") pod \"2f090df9-3e00-4fec-9008-8088c2ae5f93\" (UID: \"2f090df9-3e00-4fec-9008-8088c2ae5f93\") " Oct 30 13:02:39.985648 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:39.983403 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2f439fb-154c-4997-b0f8-f18b275d262f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2f439fb-154c-4997-b0f8-f18b275d262f-credential-socket\") pod \"d2f439fb-154c-4997-b0f8-f18b275d262f\" (UID: \"d2f439fb-154c-4997-b0f8-f18b275d262f\") " Oct 30 13:02:39.985648 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:39.983434 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03eea3e0-0b0d-459d-b03b-3a2531f20185-istio-token\" (UniqueName: \"kubernetes.io/projected/03eea3e0-0b0d-459d-b03b-3a2531f20185-istio-token\") pod \"03eea3e0-0b0d-459d-b03b-3a2531f20185\" (UID: \"03eea3e0-0b0d-459d-b03b-3a2531f20185\") " Oct 30 13:02:39.985648 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:39.983471 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f090df9-3e00-4fec-9008-8088c2ae5f93-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f090df9-3e00-4fec-9008-8088c2ae5f93-istio-data\") pod \"2f090df9-3e00-4fec-9008-8088c2ae5f93\" (UID: \"2f090df9-3e00-4fec-9008-8088c2ae5f93\") " Oct 30 13:02:39.985648 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:39.983513 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f090df9-3e00-4fec-9008-8088c2ae5f93-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f090df9-3e00-4fec-9008-8088c2ae5f93-istiod-ca-cert\") pod \"2f090df9-3e00-4fec-9008-8088c2ae5f93\" (UID: \"2f090df9-3e00-4fec-9008-8088c2ae5f93\") " Oct 30 13:02:39.985648 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:39.983566 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2f439fb-154c-4997-b0f8-f18b275d262f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2f439fb-154c-4997-b0f8-f18b275d262f-istio-data\") pod \"d2f439fb-154c-4997-b0f8-f18b275d262f\" (UID: \"d2f439fb-154c-4997-b0f8-f18b275d262f\") " Oct 30 13:02:39.985648 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:39.984502 4908 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03eea3e0-0b0d-459d-b03b-3a2531f20185-istio-data\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:02:39.985648 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:39.984528 4908 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03eea3e0-0b0d-459d-b03b-3a2531f20185-credential-socket\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:02:39.985648 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:39.984545 4908 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2f439fb-154c-4997-b0f8-f18b275d262f-istiod-ca-cert\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:02:39.985648 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:39.984561 4908 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2f439fb-154c-4997-b0f8-f18b275d262f-workload-socket\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:02:39.985648 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:39.984575 4908 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f090df9-3e00-4fec-9008-8088c2ae5f93-credential-socket\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:02:39.985648 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:39.984590 4908 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03eea3e0-0b0d-459d-b03b-3a2531f20185-workload-socket\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:02:39.985648 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:39.984845 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2f439fb-154c-4997-b0f8-f18b275d262f-istio-data" pod "d2f439fb-154c-4997-b0f8-f18b275d262f" (UID: "d2f439fb-154c-4997-b0f8-f18b275d262f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:02:39.985648 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:39.985034 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f090df9-3e00-4fec-9008-8088c2ae5f93-workload-certs" pod "2f090df9-3e00-4fec-9008-8088c2ae5f93" (UID: "2f090df9-3e00-4fec-9008-8088c2ae5f93"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:02:39.985648 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:39.985237 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5b1685b-e185-4b12-9c20-3c87e9837fd7-credential-socket" pod "a5b1685b-e185-4b12-9c20-3c87e9837fd7" (UID: "a5b1685b-e185-4b12-9c20-3c87e9837fd7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:02:39.985648 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:39.985271 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03eea3e0-0b0d-459d-b03b-3a2531f20185-istio-envoy" pod "03eea3e0-0b0d-459d-b03b-3a2531f20185" (UID: "03eea3e0-0b0d-459d-b03b-3a2531f20185"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:02:39.985648 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:39.985555 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5b1685b-e185-4b12-9c20-3c87e9837fd7-istiod-ca-cert" pod "a5b1685b-e185-4b12-9c20-3c87e9837fd7" (UID: "a5b1685b-e185-4b12-9c20-3c87e9837fd7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:02:39.987815 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:39.987787 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f090df9-3e00-4fec-9008-8088c2ae5f93-istio-token" pod "2f090df9-3e00-4fec-9008-8088c2ae5f93" (UID: "2f090df9-3e00-4fec-9008-8088c2ae5f93"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:02:39.988005 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:39.987867 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d2f439fb-154c-4997-b0f8-f18b275d262f-istio-podinfo" pod "d2f439fb-154c-4997-b0f8-f18b275d262f" (UID: "d2f439fb-154c-4997-b0f8-f18b275d262f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:02:39.988868 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:39.988799 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03eea3e0-0b0d-459d-b03b-3a2531f20185-kube-api-access-7pjf2" pod "03eea3e0-0b0d-459d-b03b-3a2531f20185" (UID: "03eea3e0-0b0d-459d-b03b-3a2531f20185"). InnerVolumeSpecName "kube-api-access-7pjf2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:02:39.989060 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:39.989042 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03eea3e0-0b0d-459d-b03b-3a2531f20185-workload-certs" pod "03eea3e0-0b0d-459d-b03b-3a2531f20185" (UID: "03eea3e0-0b0d-459d-b03b-3a2531f20185"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:02:39.989619 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:39.989566 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2f439fb-154c-4997-b0f8-f18b275d262f-istio-envoy" pod "d2f439fb-154c-4997-b0f8-f18b275d262f" (UID: "d2f439fb-154c-4997-b0f8-f18b275d262f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:02:39.989940 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:39.989902 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f090df9-3e00-4fec-9008-8088c2ae5f93-workload-socket" pod "2f090df9-3e00-4fec-9008-8088c2ae5f93" (UID: "2f090df9-3e00-4fec-9008-8088c2ae5f93"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:02:39.990073 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:39.990051 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5b1685b-e185-4b12-9c20-3c87e9837fd7-istio-envoy" pod "a5b1685b-e185-4b12-9c20-3c87e9837fd7" (UID: "a5b1685b-e185-4b12-9c20-3c87e9837fd7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:02:39.990381 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:39.990358 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5b1685b-e185-4b12-9c20-3c87e9837fd7-workload-socket" pod "a5b1685b-e185-4b12-9c20-3c87e9837fd7" (UID: "a5b1685b-e185-4b12-9c20-3c87e9837fd7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:02:39.990580 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:39.990557 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5b1685b-e185-4b12-9c20-3c87e9837fd7-workload-certs" pod "a5b1685b-e185-4b12-9c20-3c87e9837fd7" (UID: "a5b1685b-e185-4b12-9c20-3c87e9837fd7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:02:39.990778 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:39.990753 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5b1685b-e185-4b12-9c20-3c87e9837fd7-istio-data" pod "a5b1685b-e185-4b12-9c20-3c87e9837fd7" (UID: "a5b1685b-e185-4b12-9c20-3c87e9837fd7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:02:39.991896 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:39.989920 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/03eea3e0-0b0d-459d-b03b-3a2531f20185-istio-podinfo" pod "03eea3e0-0b0d-459d-b03b-3a2531f20185" (UID: "03eea3e0-0b0d-459d-b03b-3a2531f20185"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:02:39.992009 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:39.991994 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f090df9-3e00-4fec-9008-8088c2ae5f93-istio-envoy" pod "2f090df9-3e00-4fec-9008-8088c2ae5f93" (UID: "2f090df9-3e00-4fec-9008-8088c2ae5f93"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:02:39.992501 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:39.992255 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2f439fb-154c-4997-b0f8-f18b275d262f-credential-socket" pod "d2f439fb-154c-4997-b0f8-f18b275d262f" (UID: "d2f439fb-154c-4997-b0f8-f18b275d262f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:02:39.992605 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:39.992571 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03eea3e0-0b0d-459d-b03b-3a2531f20185-istiod-ca-cert" pod "03eea3e0-0b0d-459d-b03b-3a2531f20185" (UID: "03eea3e0-0b0d-459d-b03b-3a2531f20185"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:02:39.993594 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:39.992871 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2f439fb-154c-4997-b0f8-f18b275d262f-workload-certs" pod "d2f439fb-154c-4997-b0f8-f18b275d262f" (UID: "d2f439fb-154c-4997-b0f8-f18b275d262f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:02:39.993594 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:39.993076 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f090df9-3e00-4fec-9008-8088c2ae5f93-istio-data" pod "2f090df9-3e00-4fec-9008-8088c2ae5f93" (UID: "2f090df9-3e00-4fec-9008-8088c2ae5f93"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:02:39.993594 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:39.993390 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f090df9-3e00-4fec-9008-8088c2ae5f93-istiod-ca-cert" pod "2f090df9-3e00-4fec-9008-8088c2ae5f93" (UID: "2f090df9-3e00-4fec-9008-8088c2ae5f93"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:02:39.993594 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:39.993581 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5b1685b-e185-4b12-9c20-3c87e9837fd7-istio-token" pod "a5b1685b-e185-4b12-9c20-3c87e9837fd7" (UID: "a5b1685b-e185-4b12-9c20-3c87e9837fd7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:02:39.993888 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:39.993672 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f090df9-3e00-4fec-9008-8088c2ae5f93-kube-api-access-fkpbp" pod "2f090df9-3e00-4fec-9008-8088c2ae5f93" (UID: "2f090df9-3e00-4fec-9008-8088c2ae5f93"). InnerVolumeSpecName "kube-api-access-fkpbp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:02:39.993888 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:39.993719 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2f439fb-154c-4997-b0f8-f18b275d262f-kube-api-access-4pgm7" pod "d2f439fb-154c-4997-b0f8-f18b275d262f" (UID: "d2f439fb-154c-4997-b0f8-f18b275d262f"). InnerVolumeSpecName "kube-api-access-4pgm7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:02:39.995092 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:39.994084 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03eea3e0-0b0d-459d-b03b-3a2531f20185-istio-token" pod "03eea3e0-0b0d-459d-b03b-3a2531f20185" (UID: "03eea3e0-0b0d-459d-b03b-3a2531f20185"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:02:39.995092 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:39.994091 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a5b1685b-e185-4b12-9c20-3c87e9837fd7-istio-podinfo" pod "a5b1685b-e185-4b12-9c20-3c87e9837fd7" (UID: "a5b1685b-e185-4b12-9c20-3c87e9837fd7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:02:39.995092 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:39.994481 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5b1685b-e185-4b12-9c20-3c87e9837fd7-kube-api-access-l5zfh" pod "a5b1685b-e185-4b12-9c20-3c87e9837fd7" (UID: "a5b1685b-e185-4b12-9c20-3c87e9837fd7"). InnerVolumeSpecName "kube-api-access-l5zfh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:02:39.995660 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:39.995632 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2f439fb-154c-4997-b0f8-f18b275d262f-istio-token" pod "d2f439fb-154c-4997-b0f8-f18b275d262f" (UID: "d2f439fb-154c-4997-b0f8-f18b275d262f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:02:39.995987 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:39.995958 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2f090df9-3e00-4fec-9008-8088c2ae5f93-istio-podinfo" pod "2f090df9-3e00-4fec-9008-8088c2ae5f93" (UID: "2f090df9-3e00-4fec-9008-8088c2ae5f93"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:02:40.086266 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:40.086224 4908 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f090df9-3e00-4fec-9008-8088c2ae5f93-istio-podinfo\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:02:40.086266 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:40.086265 4908 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2f439fb-154c-4997-b0f8-f18b275d262f-credential-socket\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:02:40.086530 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:40.086282 4908 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03eea3e0-0b0d-459d-b03b-3a2531f20185-istio-token\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:02:40.086530 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:40.086295 4908 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f090df9-3e00-4fec-9008-8088c2ae5f93-istio-data\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:02:40.086530 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:40.086308 4908 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f090df9-3e00-4fec-9008-8088c2ae5f93-istiod-ca-cert\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:02:40.086530 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:40.086320 4908 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2f439fb-154c-4997-b0f8-f18b275d262f-istio-data\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:02:40.086530 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:40.086332 4908 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f090df9-3e00-4fec-9008-8088c2ae5f93-istio-envoy\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:02:40.086530 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:40.086345 4908 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5b1685b-e185-4b12-9c20-3c87e9837fd7-istio-token\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:02:40.086530 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:40.086358 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5zfh\" (UniqueName: \"kubernetes.io/projected/a5b1685b-e185-4b12-9c20-3c87e9837fd7-kube-api-access-l5zfh\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:02:40.086530 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:40.086370 4908 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f090df9-3e00-4fec-9008-8088c2ae5f93-istio-token\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:02:40.086530 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:40.086385 4908 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2f439fb-154c-4997-b0f8-f18b275d262f-istio-podinfo\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:02:40.086530 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:40.086399 4908 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2f439fb-154c-4997-b0f8-f18b275d262f-istio-envoy\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:02:40.086530 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:40.086411 4908 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03eea3e0-0b0d-459d-b03b-3a2531f20185-istio-podinfo\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:02:40.086530 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:40.086424 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7pjf2\" (UniqueName: \"kubernetes.io/projected/03eea3e0-0b0d-459d-b03b-3a2531f20185-kube-api-access-7pjf2\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:02:40.086530 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:40.086437 4908 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03eea3e0-0b0d-459d-b03b-3a2531f20185-workload-certs\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:02:40.086530 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:40.086450 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fkpbp\" (UniqueName: \"kubernetes.io/projected/2f090df9-3e00-4fec-9008-8088c2ae5f93-kube-api-access-fkpbp\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:02:40.086530 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:40.086464 4908 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f090df9-3e00-4fec-9008-8088c2ae5f93-workload-socket\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:02:40.086530 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:40.086480 4908 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03eea3e0-0b0d-459d-b03b-3a2531f20185-istio-envoy\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:02:40.086530 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:40.086495 4908 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f090df9-3e00-4fec-9008-8088c2ae5f93-workload-certs\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:02:40.086530 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:40.086507 4908 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5b1685b-e185-4b12-9c20-3c87e9837fd7-credential-socket\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:02:40.086530 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:40.086520 4908 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5b1685b-e185-4b12-9c20-3c87e9837fd7-istiod-ca-cert\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:02:40.086530 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:40.086533 4908 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5b1685b-e185-4b12-9c20-3c87e9837fd7-istio-envoy\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:02:40.087221 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:40.086546 4908 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5b1685b-e185-4b12-9c20-3c87e9837fd7-workload-certs\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:02:40.087221 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:40.086557 4908 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5b1685b-e185-4b12-9c20-3c87e9837fd7-istio-data\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:02:40.087221 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:40.086569 4908 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5b1685b-e185-4b12-9c20-3c87e9837fd7-workload-socket\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:02:40.087221 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:40.086583 4908 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2f439fb-154c-4997-b0f8-f18b275d262f-workload-certs\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:02:40.087221 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:40.086595 4908 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03eea3e0-0b0d-459d-b03b-3a2531f20185-istiod-ca-cert\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:02:40.087221 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:40.086608 4908 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2f439fb-154c-4997-b0f8-f18b275d262f-istio-token\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:02:40.087221 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:40.086620 4908 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5b1685b-e185-4b12-9c20-3c87e9837fd7-istio-podinfo\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:02:40.087221 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:40.086634 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4pgm7\" (UniqueName: \"kubernetes.io/projected/d2f439fb-154c-4997-b0f8-f18b275d262f-kube-api-access-4pgm7\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:02:40.199412 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:40.196886 4908 generic.go:356] "Generic (PLEG): container finished" podID="a5b1685b-e185-4b12-9c20-3c87e9837fd7" containerID="4631e9016d55731633f9fff9d825d54981feba64ae6b2918e84ed19e90831a3b" exitCode=0 Oct 30 13:02:40.199412 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:40.197100 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-d9b94" event={"ID":"a5b1685b-e185-4b12-9c20-3c87e9837fd7","Type":"ContainerDied","Data":"4631e9016d55731633f9fff9d825d54981feba64ae6b2918e84ed19e90831a3b"} Oct 30 13:02:40.199412 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:40.197157 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-d9b94" event={"ID":"a5b1685b-e185-4b12-9c20-3c87e9837fd7","Type":"ContainerDied","Data":"c33eae1ff5a10b776e2333b31251d70d4d881e0a6f6be490896eca127a32e637"} Oct 30 13:02:40.199412 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:40.197188 4908 scope.go:122] "RemoveContainer" containerID="4631e9016d55731633f9fff9d825d54981feba64ae6b2918e84ed19e90831a3b" Oct 30 13:02:40.199412 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:40.197392 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-d9b94" Oct 30 13:02:40.210260 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:40.204870 4908 generic.go:356] "Generic (PLEG): container finished" podID="2f090df9-3e00-4fec-9008-8088c2ae5f93" containerID="e8fceedfc1472209f34a8974d35c70a2372deaac130a5f368026bede217b1724" exitCode=0 Oct 30 13:02:40.210260 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:40.205023 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5smp6" event={"ID":"2f090df9-3e00-4fec-9008-8088c2ae5f93","Type":"ContainerDied","Data":"e8fceedfc1472209f34a8974d35c70a2372deaac130a5f368026bede217b1724"} Oct 30 13:02:40.210260 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:40.205072 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5smp6" event={"ID":"2f090df9-3e00-4fec-9008-8088c2ae5f93","Type":"ContainerDied","Data":"65b7eaf086f882bbe96093d3d15f54746225735cd1db2e5a91b76758ead26a86"} Oct 30 13:02:40.210260 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:40.205171 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5smp6" Oct 30 13:02:40.215697 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:40.214859 4908 generic.go:356] "Generic (PLEG): container finished" podID="03eea3e0-0b0d-459d-b03b-3a2531f20185" containerID="1cfa6e9bcd78f5e2646de2ee39cece3863dea75df13a71f952827ee871a002ef" exitCode=0 Oct 30 13:02:40.215697 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:40.214982 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xnnfx" event={"ID":"03eea3e0-0b0d-459d-b03b-3a2531f20185","Type":"ContainerDied","Data":"1cfa6e9bcd78f5e2646de2ee39cece3863dea75df13a71f952827ee871a002ef"} Oct 30 13:02:40.215697 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:40.215040 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xnnfx" event={"ID":"03eea3e0-0b0d-459d-b03b-3a2531f20185","Type":"ContainerDied","Data":"28bcaf273eb62279701b3936685f4e7fb548ff7cbe7b48c44f0677f4088de427"} Oct 30 13:02:40.215697 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:40.215133 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xnnfx" Oct 30 13:02:40.224477 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:40.224452 4908 generic.go:356] "Generic (PLEG): container finished" podID="d2f439fb-154c-4997-b0f8-f18b275d262f" containerID="ef9aa7c8f24ef453f1e99c8a9ba70dd90e9e893a2982db650c44751916ce037d" exitCode=0 Oct 30 13:02:40.224652 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:40.224586 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xcqb4" event={"ID":"d2f439fb-154c-4997-b0f8-f18b275d262f","Type":"ContainerDied","Data":"ef9aa7c8f24ef453f1e99c8a9ba70dd90e9e893a2982db650c44751916ce037d"} Oct 30 13:02:40.224652 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:40.224590 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xcqb4" Oct 30 13:02:40.224652 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:40.224621 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xcqb4" event={"ID":"d2f439fb-154c-4997-b0f8-f18b275d262f","Type":"ContainerDied","Data":"ebefebca60fe69df83f85b5f7377dec7a5ef5ebdbc97af773e9c1729849890dc"} Oct 30 13:02:40.228179 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:40.228155 4908 scope.go:122] "RemoveContainer" containerID="e490ba37cb4c0f66a103fec7579c2a868db8760d8211a33dd6400e0d8c452b93" Oct 30 13:02:40.245416 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:40.245390 4908 scope.go:122] "RemoveContainer" containerID="4631e9016d55731633f9fff9d825d54981feba64ae6b2918e84ed19e90831a3b" Oct 30 13:02:40.245773 ip-10-0-141-251 kubenswrapper[4908]: E1030 13:02:40.245742 4908 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4631e9016d55731633f9fff9d825d54981feba64ae6b2918e84ed19e90831a3b\": container with ID starting with 4631e9016d55731633f9fff9d825d54981feba64ae6b2918e84ed19e90831a3b not found: ID does not exist" containerID="4631e9016d55731633f9fff9d825d54981feba64ae6b2918e84ed19e90831a3b" Oct 30 13:02:40.245883 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:40.245788 4908 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4631e9016d55731633f9fff9d825d54981feba64ae6b2918e84ed19e90831a3b"} err="failed to get container status \"4631e9016d55731633f9fff9d825d54981feba64ae6b2918e84ed19e90831a3b\": rpc error: code = NotFound desc = could not find container \"4631e9016d55731633f9fff9d825d54981feba64ae6b2918e84ed19e90831a3b\": container with ID starting with 4631e9016d55731633f9fff9d825d54981feba64ae6b2918e84ed19e90831a3b not found: ID does not exist" Oct 30 13:02:40.245883 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:40.245811 4908 scope.go:122] "RemoveContainer" containerID="e490ba37cb4c0f66a103fec7579c2a868db8760d8211a33dd6400e0d8c452b93" Oct 30 13:02:40.246170 ip-10-0-141-251 kubenswrapper[4908]: E1030 13:02:40.246146 4908 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e490ba37cb4c0f66a103fec7579c2a868db8760d8211a33dd6400e0d8c452b93\": container with ID starting with e490ba37cb4c0f66a103fec7579c2a868db8760d8211a33dd6400e0d8c452b93 not found: ID does not exist" containerID="e490ba37cb4c0f66a103fec7579c2a868db8760d8211a33dd6400e0d8c452b93" Oct 30 13:02:40.246258 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:40.246180 4908 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e490ba37cb4c0f66a103fec7579c2a868db8760d8211a33dd6400e0d8c452b93"} err="failed to get container status \"e490ba37cb4c0f66a103fec7579c2a868db8760d8211a33dd6400e0d8c452b93\": rpc error: code = NotFound desc = could not find container \"e490ba37cb4c0f66a103fec7579c2a868db8760d8211a33dd6400e0d8c452b93\": container with ID starting with e490ba37cb4c0f66a103fec7579c2a868db8760d8211a33dd6400e0d8c452b93 not found: ID does not exist" Oct 30 13:02:40.246258 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:40.246206 4908 scope.go:122] "RemoveContainer" containerID="e8fceedfc1472209f34a8974d35c70a2372deaac130a5f368026bede217b1724" Oct 30 13:02:40.258244 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:40.258217 4908 scope.go:122] "RemoveContainer" containerID="e8fceedfc1472209f34a8974d35c70a2372deaac130a5f368026bede217b1724" Oct 30 13:02:40.258582 ip-10-0-141-251 kubenswrapper[4908]: E1030 13:02:40.258559 4908 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8fceedfc1472209f34a8974d35c70a2372deaac130a5f368026bede217b1724\": container with ID starting with e8fceedfc1472209f34a8974d35c70a2372deaac130a5f368026bede217b1724 not found: ID does not exist" containerID="e8fceedfc1472209f34a8974d35c70a2372deaac130a5f368026bede217b1724" Oct 30 13:02:40.258676 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:40.258590 4908 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8fceedfc1472209f34a8974d35c70a2372deaac130a5f368026bede217b1724"} err="failed to get container status \"e8fceedfc1472209f34a8974d35c70a2372deaac130a5f368026bede217b1724\": rpc error: code = NotFound desc = could not find container \"e8fceedfc1472209f34a8974d35c70a2372deaac130a5f368026bede217b1724\": container with ID starting with e8fceedfc1472209f34a8974d35c70a2372deaac130a5f368026bede217b1724 not found: ID does not exist" Oct 30 13:02:40.258676 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:40.258607 4908 scope.go:122] "RemoveContainer" containerID="1cfa6e9bcd78f5e2646de2ee39cece3863dea75df13a71f952827ee871a002ef" Oct 30 13:02:40.272274 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:40.272252 4908 scope.go:122] "RemoveContainer" containerID="1cfa6e9bcd78f5e2646de2ee39cece3863dea75df13a71f952827ee871a002ef" Oct 30 13:02:40.272597 ip-10-0-141-251 kubenswrapper[4908]: E1030 13:02:40.272570 4908 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1cfa6e9bcd78f5e2646de2ee39cece3863dea75df13a71f952827ee871a002ef\": container with ID starting with 1cfa6e9bcd78f5e2646de2ee39cece3863dea75df13a71f952827ee871a002ef not found: ID does not exist" containerID="1cfa6e9bcd78f5e2646de2ee39cece3863dea75df13a71f952827ee871a002ef" Oct 30 13:02:40.272679 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:40.272605 4908 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1cfa6e9bcd78f5e2646de2ee39cece3863dea75df13a71f952827ee871a002ef"} err="failed to get container status \"1cfa6e9bcd78f5e2646de2ee39cece3863dea75df13a71f952827ee871a002ef\": rpc error: code = NotFound desc = could not find container \"1cfa6e9bcd78f5e2646de2ee39cece3863dea75df13a71f952827ee871a002ef\": container with ID starting with 1cfa6e9bcd78f5e2646de2ee39cece3863dea75df13a71f952827ee871a002ef not found: ID does not exist" Oct 30 13:02:40.272679 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:40.272624 4908 scope.go:122] "RemoveContainer" containerID="ef9aa7c8f24ef453f1e99c8a9ba70dd90e9e893a2982db650c44751916ce037d" Oct 30 13:02:40.283344 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:40.283316 4908 scope.go:122] "RemoveContainer" containerID="ef9aa7c8f24ef453f1e99c8a9ba70dd90e9e893a2982db650c44751916ce037d" Oct 30 13:02:40.283680 ip-10-0-141-251 kubenswrapper[4908]: E1030 13:02:40.283657 4908 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef9aa7c8f24ef453f1e99c8a9ba70dd90e9e893a2982db650c44751916ce037d\": container with ID starting with ef9aa7c8f24ef453f1e99c8a9ba70dd90e9e893a2982db650c44751916ce037d not found: ID does not exist" containerID="ef9aa7c8f24ef453f1e99c8a9ba70dd90e9e893a2982db650c44751916ce037d" Oct 30 13:02:40.283766 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:40.283691 4908 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef9aa7c8f24ef453f1e99c8a9ba70dd90e9e893a2982db650c44751916ce037d"} err="failed to get container status \"ef9aa7c8f24ef453f1e99c8a9ba70dd90e9e893a2982db650c44751916ce037d\": rpc error: code = NotFound desc = could not find container \"ef9aa7c8f24ef453f1e99c8a9ba70dd90e9e893a2982db650c44751916ce037d\": container with ID starting with ef9aa7c8f24ef453f1e99c8a9ba70dd90e9e893a2982db650c44751916ce037d not found: ID does not exist" Oct 30 13:02:40.290905 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:40.290867 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xnnfx"] Oct 30 13:02:40.295122 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:40.295090 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xnnfx"] Oct 30 13:02:40.326496 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:40.326460 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xcqb4"] Oct 30 13:02:40.330631 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:40.330598 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xcqb4"] Oct 30 13:02:40.365659 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:40.365627 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5smp6"] Oct 30 13:02:40.369530 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:40.369501 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5smp6"] Oct 30 13:02:40.400736 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:40.400702 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-d9b94"] Oct 30 13:02:40.407347 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:40.407311 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-d9b94"] Oct 30 13:02:41.615080 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:41.615041 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03eea3e0-0b0d-459d-b03b-3a2531f20185" path="/var/lib/kubelet/pods/03eea3e0-0b0d-459d-b03b-3a2531f20185/volumes" Oct 30 13:02:41.615649 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:41.615631 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f090df9-3e00-4fec-9008-8088c2ae5f93" path="/var/lib/kubelet/pods/2f090df9-3e00-4fec-9008-8088c2ae5f93/volumes" Oct 30 13:02:41.616213 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:41.616196 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5b1685b-e185-4b12-9c20-3c87e9837fd7" path="/var/lib/kubelet/pods/a5b1685b-e185-4b12-9c20-3c87e9837fd7/volumes" Oct 30 13:02:41.617085 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:41.617067 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2f439fb-154c-4997-b0f8-f18b275d262f" path="/var/lib/kubelet/pods/d2f439fb-154c-4997-b0f8-f18b275d262f/volumes" Oct 30 13:02:42.131646 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:42.131604 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9wwfc"] Oct 30 13:02:45.321849 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:45.321656 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-ecb3aa-mtn5d"] Oct 30 13:02:45.334855 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:45.333768 4908 cpu_manager.go:414] "containerMap: removing container" podUID="03eea3e0-0b0d-459d-b03b-3a2531f20185" containerName="istio-proxy" Oct 30 13:02:45.334855 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:45.333803 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03eea3e0-0b0d-459d-b03b-3a2531f20185" containerName="istio-proxy" Oct 30 13:02:45.334855 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:45.333819 4908 cpu_manager.go:414] "containerMap: removing container" podUID="133a27f0-7837-4fb3-9803-865ec0dd47ba" containerName="revoke-keys" Oct 30 13:02:45.334855 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:45.333847 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="133a27f0-7837-4fb3-9803-865ec0dd47ba" containerName="revoke-keys" Oct 30 13:02:45.334855 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:45.333874 4908 cpu_manager.go:414] "containerMap: removing container" podUID="2f090df9-3e00-4fec-9008-8088c2ae5f93" containerName="istio-proxy" Oct 30 13:02:45.334855 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:45.333881 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f090df9-3e00-4fec-9008-8088c2ae5f93" containerName="istio-proxy" Oct 30 13:02:45.334855 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:45.333904 4908 cpu_manager.go:414] "containerMap: removing container" podUID="d2f439fb-154c-4997-b0f8-f18b275d262f" containerName="istio-proxy" Oct 30 13:02:45.334855 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:45.333911 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2f439fb-154c-4997-b0f8-f18b275d262f" containerName="istio-proxy" Oct 30 13:02:45.334855 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:45.333952 4908 cpu_manager.go:414] "containerMap: removing container" podUID="8396834c-234e-431a-8e62-97c193ce6b66" containerName="main" Oct 30 13:02:45.334855 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:45.333960 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8396834c-234e-431a-8e62-97c193ce6b66" containerName="main" Oct 30 13:02:45.334855 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:45.333989 4908 cpu_manager.go:414] "containerMap: removing container" podUID="a5b1685b-e185-4b12-9c20-3c87e9837fd7" containerName="istio-proxy" Oct 30 13:02:45.334855 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:45.333996 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5b1685b-e185-4b12-9c20-3c87e9837fd7" containerName="istio-proxy" Oct 30 13:02:45.334855 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:45.334016 4908 cpu_manager.go:414] "containerMap: removing container" podUID="a5b1685b-e185-4b12-9c20-3c87e9837fd7" containerName="istio-proxy" Oct 30 13:02:45.334855 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:45.334023 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5b1685b-e185-4b12-9c20-3c87e9837fd7" containerName="istio-proxy" Oct 30 13:02:45.334855 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:45.334180 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="8396834c-234e-431a-8e62-97c193ce6b66" containerName="main" Oct 30 13:02:45.334855 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:45.334195 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5b1685b-e185-4b12-9c20-3c87e9837fd7" containerName="istio-proxy" Oct 30 13:02:45.334855 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:45.334205 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5b1685b-e185-4b12-9c20-3c87e9837fd7" containerName="istio-proxy" Oct 30 13:02:45.334855 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:45.334216 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="03eea3e0-0b0d-459d-b03b-3a2531f20185" containerName="istio-proxy" Oct 30 13:02:45.334855 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:45.334224 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="133a27f0-7837-4fb3-9803-865ec0dd47ba" containerName="revoke-keys" Oct 30 13:02:45.334855 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:45.334235 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f090df9-3e00-4fec-9008-8088c2ae5f93" containerName="istio-proxy" Oct 30 13:02:45.334855 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:45.334245 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2f439fb-154c-4997-b0f8-f18b275d262f" containerName="istio-proxy" Oct 30 13:02:45.351505 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:45.349690 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-ecb3aa-mtn5d"] Oct 30 13:02:45.351505 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:45.349819 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-ecb3aa-mtn5d" Oct 30 13:02:45.368599 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:45.368561 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:02:45.369011 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:45.368988 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s75hq\"" Oct 30 13:02:45.380226 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:45.378249 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rm56\" (UniqueName: \"kubernetes.io/projected/3c0b299c-2b48-4b4f-a1c3-daf725126170-kube-api-access-2rm56\") pod \"maas-api-revoke-keys-e2e-extmod-b-ecb3aa-mtn5d\" (UID: \"3c0b299c-2b48-4b4f-a1c3-daf725126170\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-ecb3aa-mtn5d" Oct 30 13:02:45.380226 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:45.378587 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c0b299c-2b48-4b4f-a1c3-daf725126170-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-ecb3aa-mtn5d\" (UID: \"3c0b299c-2b48-4b4f-a1c3-daf725126170\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-ecb3aa-mtn5d" Oct 30 13:02:45.380226 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:45.378631 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c0b299c-2b48-4b4f-a1c3-daf725126170-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-ecb3aa-mtn5d\" (UID: \"3c0b299c-2b48-4b4f-a1c3-daf725126170\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-ecb3aa-mtn5d" Oct 30 13:02:45.386877 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:45.381804 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 13:02:45.386877 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:45.381806 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 13:02:45.479514 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:45.479476 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rm56\" (UniqueName: \"kubernetes.io/projected/3c0b299c-2b48-4b4f-a1c3-daf725126170-kube-api-access-2rm56\") pod \"maas-api-revoke-keys-e2e-extmod-b-ecb3aa-mtn5d\" (UID: \"3c0b299c-2b48-4b4f-a1c3-daf725126170\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-ecb3aa-mtn5d" Oct 30 13:02:45.479746 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:45.479605 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c0b299c-2b48-4b4f-a1c3-daf725126170-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-ecb3aa-mtn5d\" (UID: \"3c0b299c-2b48-4b4f-a1c3-daf725126170\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-ecb3aa-mtn5d" Oct 30 13:02:45.479746 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:45.479628 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c0b299c-2b48-4b4f-a1c3-daf725126170-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-ecb3aa-mtn5d\" (UID: \"3c0b299c-2b48-4b4f-a1c3-daf725126170\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-ecb3aa-mtn5d" Oct 30 13:02:45.480377 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:45.480351 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c0b299c-2b48-4b4f-a1c3-daf725126170-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-ecb3aa-mtn5d\" (UID: \"3c0b299c-2b48-4b4f-a1c3-daf725126170\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-ecb3aa-mtn5d" Oct 30 13:02:45.482198 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:45.482171 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c0b299c-2b48-4b4f-a1c3-daf725126170-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-ecb3aa-mtn5d\" (UID: \"3c0b299c-2b48-4b4f-a1c3-daf725126170\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-ecb3aa-mtn5d" Oct 30 13:02:45.502013 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:45.501980 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rm56\" (UniqueName: \"kubernetes.io/projected/3c0b299c-2b48-4b4f-a1c3-daf725126170-kube-api-access-2rm56\") pod \"maas-api-revoke-keys-e2e-extmod-b-ecb3aa-mtn5d\" (UID: \"3c0b299c-2b48-4b4f-a1c3-daf725126170\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-ecb3aa-mtn5d" Oct 30 13:02:45.691056 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:45.690970 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-ecb3aa-mtn5d" Oct 30 13:02:45.999026 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:45.998988 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-ecb3aa-mtn5d"] Oct 30 13:02:46.003116 ip-10-0-141-251 kubenswrapper[4908]: W1030 13:02:46.003085 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c0b299c_2b48_4b4f_a1c3_daf725126170.slice/crio-05a0554a4a81a32b9858999cfd42c9d7a2916550b2706734da76bc28cb86fc3e WatchSource:0}: Error finding container 05a0554a4a81a32b9858999cfd42c9d7a2916550b2706734da76bc28cb86fc3e: Status 404 returned error can't find the container with id 05a0554a4a81a32b9858999cfd42c9d7a2916550b2706734da76bc28cb86fc3e Oct 30 13:02:46.285923 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:46.285802 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-ecb3aa-mtn5d" event={"ID":"3c0b299c-2b48-4b4f-a1c3-daf725126170","Type":"ContainerStarted","Data":"05a0554a4a81a32b9858999cfd42c9d7a2916550b2706734da76bc28cb86fc3e"} Oct 30 13:02:47.296194 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:47.296158 4908 generic.go:356] "Generic (PLEG): container finished" podID="3c0b299c-2b48-4b4f-a1c3-daf725126170" containerID="152a06bad4922828997c7c83c521980706205440d1023dd02f0c3597eeec3b07" exitCode=0 Oct 30 13:02:47.296797 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:47.296252 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-ecb3aa-mtn5d" event={"ID":"3c0b299c-2b48-4b4f-a1c3-daf725126170","Type":"ContainerDied","Data":"152a06bad4922828997c7c83c521980706205440d1023dd02f0c3597eeec3b07"} Oct 30 13:02:48.487357 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:48.487322 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-ecb3aa-mtn5d" Oct 30 13:02:48.541742 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:48.541704 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3c0b299c-2b48-4b4f-a1c3-daf725126170-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c0b299c-2b48-4b4f-a1c3-daf725126170-maas-api-xks-ca\") pod \"3c0b299c-2b48-4b4f-a1c3-daf725126170\" (UID: \"3c0b299c-2b48-4b4f-a1c3-daf725126170\") " Oct 30 13:02:48.541934 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:48.541881 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c0b299c-2b48-4b4f-a1c3-daf725126170-kube-api-access-2rm56\" (UniqueName: \"kubernetes.io/projected/3c0b299c-2b48-4b4f-a1c3-daf725126170-kube-api-access-2rm56\") pod \"3c0b299c-2b48-4b4f-a1c3-daf725126170\" (UID: \"3c0b299c-2b48-4b4f-a1c3-daf725126170\") " Oct 30 13:02:48.542038 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:48.541997 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c0b299c-2b48-4b4f-a1c3-daf725126170-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c0b299c-2b48-4b4f-a1c3-daf725126170-maas-api-openshift-ca\") pod \"3c0b299c-2b48-4b4f-a1c3-daf725126170\" (UID: \"3c0b299c-2b48-4b4f-a1c3-daf725126170\") " Oct 30 13:02:48.542462 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:48.542434 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c0b299c-2b48-4b4f-a1c3-daf725126170-maas-api-openshift-ca" pod "3c0b299c-2b48-4b4f-a1c3-daf725126170" (UID: "3c0b299c-2b48-4b4f-a1c3-daf725126170"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:02:48.542692 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:48.542665 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c0b299c-2b48-4b4f-a1c3-daf725126170-maas-api-openshift-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:02:48.544071 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:48.544048 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c0b299c-2b48-4b4f-a1c3-daf725126170-kube-api-access-2rm56" pod "3c0b299c-2b48-4b4f-a1c3-daf725126170" (UID: "3c0b299c-2b48-4b4f-a1c3-daf725126170"). InnerVolumeSpecName "kube-api-access-2rm56". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:02:48.544250 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:48.544211 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c0b299c-2b48-4b4f-a1c3-daf725126170-maas-api-xks-ca" pod "3c0b299c-2b48-4b4f-a1c3-daf725126170" (UID: "3c0b299c-2b48-4b4f-a1c3-daf725126170"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:02:48.644103 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:48.644012 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rm56\" (UniqueName: \"kubernetes.io/projected/3c0b299c-2b48-4b4f-a1c3-daf725126170-kube-api-access-2rm56\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:02:48.644103 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:48.644044 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c0b299c-2b48-4b4f-a1c3-daf725126170-maas-api-xks-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:02:49.315363 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:49.315327 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-ecb3aa-mtn5d" Oct 30 13:02:49.315569 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:49.315364 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-ecb3aa-mtn5d" event={"ID":"3c0b299c-2b48-4b4f-a1c3-daf725126170","Type":"ContainerDied","Data":"05a0554a4a81a32b9858999cfd42c9d7a2916550b2706734da76bc28cb86fc3e"} Oct 30 13:02:49.315569 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:02:49.315424 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="05a0554a4a81a32b9858999cfd42c9d7a2916550b2706734da76bc28cb86fc3e" Oct 30 13:02:49.688070 ip-10-0-141-251 kubenswrapper[4908]: E1030 13:02:49.687963 4908 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vn6vv" containerName="extractor" Oct 30 13:03:07.594630 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:07.594590 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-eab799-hs98g"] Oct 30 13:03:07.607619 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:07.607591 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-eab799-hs98g"] Oct 30 13:03:09.150719 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:09.150674 4908 scope.go:122] "RemoveContainer" containerID="3296a51a15a59accfb37f0639fb4a0b2097a7a55ec9cd6f757e0d46812afdb57" Oct 30 13:03:09.609040 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:09.608988 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="133a27f0-7837-4fb3-9803-865ec0dd47ba" path="/var/lib/kubelet/pods/133a27f0-7837-4fb3-9803-865ec0dd47ba/volumes" Oct 30 13:03:14.444068 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:14.444023 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-eab799-openshift-default-574ddfd59c-tfw7h"] Oct 30 13:03:14.444524 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:14.444304 4908 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-eab799-openshift-default-574ddfd59c-tfw7h" podUID="61f8676a-8552-4066-a292-26bbd8b64d6b" containerName="istio-proxy" containerID="cri-o://84fee92755c2dd385d20823190e7b6344f9f725ba5dd89e3ad8630a2419be57d" gracePeriod=30 Oct 30 13:03:15.131696 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:15.131651 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e9f184-lz9mg"] Oct 30 13:03:15.132358 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:15.132335 4908 cpu_manager.go:414] "containerMap: removing container" podUID="3c0b299c-2b48-4b4f-a1c3-daf725126170" containerName="revoke-keys" Oct 30 13:03:15.132479 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:15.132359 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c0b299c-2b48-4b4f-a1c3-daf725126170" containerName="revoke-keys" Oct 30 13:03:15.132541 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:15.132517 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c0b299c-2b48-4b4f-a1c3-daf725126170" containerName="revoke-keys" Oct 30 13:03:15.136165 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:15.136144 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e9f184-lz9mg" Oct 30 13:03:15.154965 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:15.154932 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:03:15.154965 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:15.154949 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 13:03:15.155144 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:15.155016 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 13:03:15.155144 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:15.155116 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s75hq\"" Oct 30 13:03:15.173147 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:15.173108 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e9f184-lz9mg"] Oct 30 13:03:15.257398 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:15.257360 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/151948d5-7d82-4a4b-8db1-feed4b587749-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-e9f184-lz9mg\" (UID: \"151948d5-7d82-4a4b-8db1-feed4b587749\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e9f184-lz9mg" Oct 30 13:03:15.257587 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:15.257559 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/151948d5-7d82-4a4b-8db1-feed4b587749-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-e9f184-lz9mg\" (UID: \"151948d5-7d82-4a4b-8db1-feed4b587749\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e9f184-lz9mg" Oct 30 13:03:15.257650 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:15.257611 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74d5r\" (UniqueName: \"kubernetes.io/projected/151948d5-7d82-4a4b-8db1-feed4b587749-kube-api-access-74d5r\") pod \"maas-api-revoke-keys-e2e-inf-b-e9f184-lz9mg\" (UID: \"151948d5-7d82-4a4b-8db1-feed4b587749\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e9f184-lz9mg" Oct 30 13:03:15.359534 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:15.359490 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/151948d5-7d82-4a4b-8db1-feed4b587749-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-e9f184-lz9mg\" (UID: \"151948d5-7d82-4a4b-8db1-feed4b587749\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e9f184-lz9mg" Oct 30 13:03:15.359789 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:15.359600 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74d5r\" (UniqueName: \"kubernetes.io/projected/151948d5-7d82-4a4b-8db1-feed4b587749-kube-api-access-74d5r\") pod \"maas-api-revoke-keys-e2e-inf-b-e9f184-lz9mg\" (UID: \"151948d5-7d82-4a4b-8db1-feed4b587749\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e9f184-lz9mg" Oct 30 13:03:15.359789 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:15.359680 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/151948d5-7d82-4a4b-8db1-feed4b587749-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-e9f184-lz9mg\" (UID: \"151948d5-7d82-4a4b-8db1-feed4b587749\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e9f184-lz9mg" Oct 30 13:03:15.360516 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:15.360488 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/151948d5-7d82-4a4b-8db1-feed4b587749-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-e9f184-lz9mg\" (UID: \"151948d5-7d82-4a4b-8db1-feed4b587749\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e9f184-lz9mg" Oct 30 13:03:15.362876 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:15.362815 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/151948d5-7d82-4a4b-8db1-feed4b587749-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-e9f184-lz9mg\" (UID: \"151948d5-7d82-4a4b-8db1-feed4b587749\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e9f184-lz9mg" Oct 30 13:03:15.382368 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:15.382292 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74d5r\" (UniqueName: \"kubernetes.io/projected/151948d5-7d82-4a4b-8db1-feed4b587749-kube-api-access-74d5r\") pod \"maas-api-revoke-keys-e2e-inf-b-e9f184-lz9mg\" (UID: \"151948d5-7d82-4a4b-8db1-feed4b587749\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e9f184-lz9mg" Oct 30 13:03:15.450289 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:15.450253 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e9f184-lz9mg" Oct 30 13:03:15.762090 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:15.762059 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e9f184-lz9mg"] Oct 30 13:03:15.766063 ip-10-0-141-251 kubenswrapper[4908]: W1030 13:03:15.766014 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod151948d5_7d82_4a4b_8db1_feed4b587749.slice/crio-98a9fe129ad3e4276f93a979264db1d7f8924eff2a1231d97e24106cc37233cc WatchSource:0}: Error finding container 98a9fe129ad3e4276f93a979264db1d7f8924eff2a1231d97e24106cc37233cc: Status 404 returned error can't find the container with id 98a9fe129ad3e4276f93a979264db1d7f8924eff2a1231d97e24106cc37233cc Oct 30 13:03:15.926085 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:15.926051 4908 patch_prober.go:29] interesting pod/e2e-inf-a-eab799-openshift-default-574ddfd59c-tfw7h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 13:03:15.926258 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:15.926118 4908 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-eab799-openshift-default-574ddfd59c-tfw7h" podUID="61f8676a-8552-4066-a292-26bbd8b64d6b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 13:03:16.700098 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:16.700063 4908 generic.go:356] "Generic (PLEG): container finished" podID="151948d5-7d82-4a4b-8db1-feed4b587749" containerID="610e9f6e3570a524674d74893bfc0ef5a8996a025ca0fb0327714d799eda674a" exitCode=0 Oct 30 13:03:16.700499 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:16.700144 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e9f184-lz9mg" event={"ID":"151948d5-7d82-4a4b-8db1-feed4b587749","Type":"ContainerDied","Data":"610e9f6e3570a524674d74893bfc0ef5a8996a025ca0fb0327714d799eda674a"} Oct 30 13:03:16.700499 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:16.700193 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e9f184-lz9mg" event={"ID":"151948d5-7d82-4a4b-8db1-feed4b587749","Type":"ContainerStarted","Data":"98a9fe129ad3e4276f93a979264db1d7f8924eff2a1231d97e24106cc37233cc"} Oct 30 13:03:17.959422 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:17.959387 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e9f184-lz9mg" Oct 30 13:03:18.130371 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:18.130340 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/151948d5-7d82-4a4b-8db1-feed4b587749-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/151948d5-7d82-4a4b-8db1-feed4b587749-maas-api-xks-ca\") pod \"151948d5-7d82-4a4b-8db1-feed4b587749\" (UID: \"151948d5-7d82-4a4b-8db1-feed4b587749\") " Oct 30 13:03:18.130573 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:18.130383 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/151948d5-7d82-4a4b-8db1-feed4b587749-kube-api-access-74d5r\" (UniqueName: \"kubernetes.io/projected/151948d5-7d82-4a4b-8db1-feed4b587749-kube-api-access-74d5r\") pod \"151948d5-7d82-4a4b-8db1-feed4b587749\" (UID: \"151948d5-7d82-4a4b-8db1-feed4b587749\") " Oct 30 13:03:18.130624 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:18.130592 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/151948d5-7d82-4a4b-8db1-feed4b587749-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/151948d5-7d82-4a4b-8db1-feed4b587749-maas-api-openshift-ca\") pod \"151948d5-7d82-4a4b-8db1-feed4b587749\" (UID: \"151948d5-7d82-4a4b-8db1-feed4b587749\") " Oct 30 13:03:18.131090 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:18.131059 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/151948d5-7d82-4a4b-8db1-feed4b587749-maas-api-openshift-ca" pod "151948d5-7d82-4a4b-8db1-feed4b587749" (UID: "151948d5-7d82-4a4b-8db1-feed4b587749"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:03:18.131219 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:18.131191 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/151948d5-7d82-4a4b-8db1-feed4b587749-maas-api-openshift-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:03:18.132595 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:18.132567 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/151948d5-7d82-4a4b-8db1-feed4b587749-maas-api-xks-ca" pod "151948d5-7d82-4a4b-8db1-feed4b587749" (UID: "151948d5-7d82-4a4b-8db1-feed4b587749"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:03:18.132697 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:18.132674 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/151948d5-7d82-4a4b-8db1-feed4b587749-kube-api-access-74d5r" pod "151948d5-7d82-4a4b-8db1-feed4b587749" (UID: "151948d5-7d82-4a4b-8db1-feed4b587749"). InnerVolumeSpecName "kube-api-access-74d5r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:03:18.233091 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:18.233055 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/151948d5-7d82-4a4b-8db1-feed4b587749-maas-api-xks-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:03:18.233091 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:18.233087 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74d5r\" (UniqueName: \"kubernetes.io/projected/151948d5-7d82-4a4b-8db1-feed4b587749-kube-api-access-74d5r\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:03:18.719332 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:18.719298 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e9f184-lz9mg" Oct 30 13:03:18.719538 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:18.719330 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e9f184-lz9mg" event={"ID":"151948d5-7d82-4a4b-8db1-feed4b587749","Type":"ContainerDied","Data":"98a9fe129ad3e4276f93a979264db1d7f8924eff2a1231d97e24106cc37233cc"} Oct 30 13:03:18.719538 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:18.719379 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="98a9fe129ad3e4276f93a979264db1d7f8924eff2a1231d97e24106cc37233cc" Oct 30 13:03:19.706951 ip-10-0-141-251 kubenswrapper[4908]: E1030 13:03:19.706903 4908 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vn6vv" containerName="extractor" Oct 30 13:03:19.731937 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:19.731902 4908 generic.go:356] "Generic (PLEG): container finished" podID="61f8676a-8552-4066-a292-26bbd8b64d6b" containerID="84fee92755c2dd385d20823190e7b6344f9f725ba5dd89e3ad8630a2419be57d" exitCode=0 Oct 30 13:03:19.732120 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:19.731948 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-eab799-openshift-default-574ddfd59c-tfw7h" event={"ID":"61f8676a-8552-4066-a292-26bbd8b64d6b","Type":"ContainerDied","Data":"84fee92755c2dd385d20823190e7b6344f9f725ba5dd89e3ad8630a2419be57d"} Oct 30 13:03:19.732120 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:19.732005 4908 scope.go:122] "RemoveContainer" containerID="13fa47b6b1bc9c45ac4920d0db4a61a189ed3bd195fc322079723c64eb6e4f0f" Oct 30 13:03:19.780269 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:19.780232 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-eab799-openshift-default-574ddfd59c-tfw7h" Oct 30 13:03:19.867623 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:19.867586 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61f8676a-8552-4066-a292-26bbd8b64d6b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61f8676a-8552-4066-a292-26bbd8b64d6b-credential-socket\") pod \"61f8676a-8552-4066-a292-26bbd8b64d6b\" (UID: \"61f8676a-8552-4066-a292-26bbd8b64d6b\") " Oct 30 13:03:19.867623 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:19.867636 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61f8676a-8552-4066-a292-26bbd8b64d6b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61f8676a-8552-4066-a292-26bbd8b64d6b-istiod-ca-cert\") pod \"61f8676a-8552-4066-a292-26bbd8b64d6b\" (UID: \"61f8676a-8552-4066-a292-26bbd8b64d6b\") " Oct 30 13:03:19.867939 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:19.867680 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61f8676a-8552-4066-a292-26bbd8b64d6b-kube-api-access-425jt\" (UniqueName: \"kubernetes.io/projected/61f8676a-8552-4066-a292-26bbd8b64d6b-kube-api-access-425jt\") pod \"61f8676a-8552-4066-a292-26bbd8b64d6b\" (UID: \"61f8676a-8552-4066-a292-26bbd8b64d6b\") " Oct 30 13:03:19.867939 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:19.867735 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61f8676a-8552-4066-a292-26bbd8b64d6b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61f8676a-8552-4066-a292-26bbd8b64d6b-workload-socket\") pod \"61f8676a-8552-4066-a292-26bbd8b64d6b\" (UID: \"61f8676a-8552-4066-a292-26bbd8b64d6b\") " Oct 30 13:03:19.867939 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:19.867911 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61f8676a-8552-4066-a292-26bbd8b64d6b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61f8676a-8552-4066-a292-26bbd8b64d6b-istio-envoy\") pod \"61f8676a-8552-4066-a292-26bbd8b64d6b\" (UID: \"61f8676a-8552-4066-a292-26bbd8b64d6b\") " Oct 30 13:03:19.868191 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:19.868160 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61f8676a-8552-4066-a292-26bbd8b64d6b-credential-socket" pod "61f8676a-8552-4066-a292-26bbd8b64d6b" (UID: "61f8676a-8552-4066-a292-26bbd8b64d6b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:03:19.868253 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:19.868174 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61f8676a-8552-4066-a292-26bbd8b64d6b-workload-socket" pod "61f8676a-8552-4066-a292-26bbd8b64d6b" (UID: "61f8676a-8552-4066-a292-26bbd8b64d6b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:03:19.868253 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:19.868166 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61f8676a-8552-4066-a292-26bbd8b64d6b-istiod-ca-cert" pod "61f8676a-8552-4066-a292-26bbd8b64d6b" (UID: "61f8676a-8552-4066-a292-26bbd8b64d6b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:03:19.869726 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:19.868526 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61f8676a-8552-4066-a292-26bbd8b64d6b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61f8676a-8552-4066-a292-26bbd8b64d6b-istio-data\") pod \"61f8676a-8552-4066-a292-26bbd8b64d6b\" (UID: \"61f8676a-8552-4066-a292-26bbd8b64d6b\") " Oct 30 13:03:19.869726 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:19.868581 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/61f8676a-8552-4066-a292-26bbd8b64d6b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61f8676a-8552-4066-a292-26bbd8b64d6b-istio-podinfo\") pod \"61f8676a-8552-4066-a292-26bbd8b64d6b\" (UID: \"61f8676a-8552-4066-a292-26bbd8b64d6b\") " Oct 30 13:03:19.869726 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:19.868627 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61f8676a-8552-4066-a292-26bbd8b64d6b-istio-token\" (UniqueName: \"kubernetes.io/projected/61f8676a-8552-4066-a292-26bbd8b64d6b-istio-token\") pod \"61f8676a-8552-4066-a292-26bbd8b64d6b\" (UID: \"61f8676a-8552-4066-a292-26bbd8b64d6b\") " Oct 30 13:03:19.869726 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:19.868664 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61f8676a-8552-4066-a292-26bbd8b64d6b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61f8676a-8552-4066-a292-26bbd8b64d6b-workload-certs\") pod \"61f8676a-8552-4066-a292-26bbd8b64d6b\" (UID: \"61f8676a-8552-4066-a292-26bbd8b64d6b\") " Oct 30 13:03:19.869726 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:19.869471 4908 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61f8676a-8552-4066-a292-26bbd8b64d6b-credential-socket\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:03:19.869726 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:19.869495 4908 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61f8676a-8552-4066-a292-26bbd8b64d6b-istiod-ca-cert\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:03:19.869726 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:19.869510 4908 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61f8676a-8552-4066-a292-26bbd8b64d6b-workload-socket\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:03:19.870399 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:19.870365 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61f8676a-8552-4066-a292-26bbd8b64d6b-workload-certs" pod "61f8676a-8552-4066-a292-26bbd8b64d6b" (UID: "61f8676a-8552-4066-a292-26bbd8b64d6b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:03:19.870885 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:19.870848 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61f8676a-8552-4066-a292-26bbd8b64d6b-istio-data" pod "61f8676a-8552-4066-a292-26bbd8b64d6b" (UID: "61f8676a-8552-4066-a292-26bbd8b64d6b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:03:19.870995 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:19.870939 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61f8676a-8552-4066-a292-26bbd8b64d6b-istio-envoy" pod "61f8676a-8552-4066-a292-26bbd8b64d6b" (UID: "61f8676a-8552-4066-a292-26bbd8b64d6b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:03:19.872258 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:19.872231 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61f8676a-8552-4066-a292-26bbd8b64d6b-kube-api-access-425jt" pod "61f8676a-8552-4066-a292-26bbd8b64d6b" (UID: "61f8676a-8552-4066-a292-26bbd8b64d6b"). InnerVolumeSpecName "kube-api-access-425jt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:03:19.872453 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:19.872431 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/61f8676a-8552-4066-a292-26bbd8b64d6b-istio-podinfo" pod "61f8676a-8552-4066-a292-26bbd8b64d6b" (UID: "61f8676a-8552-4066-a292-26bbd8b64d6b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:03:19.873993 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:19.873964 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61f8676a-8552-4066-a292-26bbd8b64d6b-istio-token" pod "61f8676a-8552-4066-a292-26bbd8b64d6b" (UID: "61f8676a-8552-4066-a292-26bbd8b64d6b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:03:19.971408 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:19.971370 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-425jt\" (UniqueName: \"kubernetes.io/projected/61f8676a-8552-4066-a292-26bbd8b64d6b-kube-api-access-425jt\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:03:19.971408 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:19.971403 4908 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61f8676a-8552-4066-a292-26bbd8b64d6b-istio-envoy\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:03:19.971408 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:19.971412 4908 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61f8676a-8552-4066-a292-26bbd8b64d6b-istio-data\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:03:19.971408 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:19.971420 4908 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61f8676a-8552-4066-a292-26bbd8b64d6b-istio-podinfo\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:03:19.971814 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:19.971428 4908 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61f8676a-8552-4066-a292-26bbd8b64d6b-istio-token\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:03:19.971814 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:19.971438 4908 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61f8676a-8552-4066-a292-26bbd8b64d6b-workload-certs\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:03:20.741143 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:20.741108 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-eab799-openshift-default-574ddfd59c-tfw7h" event={"ID":"61f8676a-8552-4066-a292-26bbd8b64d6b","Type":"ContainerDied","Data":"69087d3e4b72b8ba3236f8d4e449a182aa9faf5deabb91fc0e102b8485645496"} Oct 30 13:03:20.741636 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:20.741173 4908 scope.go:122] "RemoveContainer" containerID="84fee92755c2dd385d20823190e7b6344f9f725ba5dd89e3ad8630a2419be57d" Oct 30 13:03:20.741636 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:20.741117 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-eab799-openshift-default-574ddfd59c-tfw7h" Oct 30 13:03:20.805639 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:20.805604 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-eab799-openshift-default-574ddfd59c-tfw7h"] Oct 30 13:03:20.816262 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:20.816233 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-eab799-openshift-default-574ddfd59c-tfw7h"] Oct 30 13:03:21.610257 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:21.610222 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61f8676a-8552-4066-a292-26bbd8b64d6b" path="/var/lib/kubelet/pods/61f8676a-8552-4066-a292-26bbd8b64d6b/volumes" Oct 30 13:03:43.877501 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:43.877456 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-ecb3aa-mtn5d"] Oct 30 13:03:43.888727 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:43.887902 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-ecb3aa-mtn5d"] Oct 30 13:03:45.630243 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:45.630194 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c0b299c-2b48-4b4f-a1c3-daf725126170" path="/var/lib/kubelet/pods/3c0b299c-2b48-4b4f-a1c3-daf725126170/volumes" Oct 30 13:03:49.879731 ip-10-0-141-251 kubenswrapper[4908]: E1030 13:03:49.879697 4908 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vn6vv" containerName="extractor" Oct 30 13:03:51.382346 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:51.382307 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-ecb3aa-openshift-default-68d9fff7cc-xb5ct"] Oct 30 13:03:51.382806 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:51.382548 4908 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-ecb3aa-openshift-default-68d9fff7cc-xb5ct" podUID="e59c9971-42c6-4df7-94a0-fd6753d847ca" containerName="istio-proxy" containerID="cri-o://8ce08c9d4597660bc5284fdf10c7b97e8d93f9c4e7db07260f38a845bb89a568" gracePeriod=30 Oct 30 13:03:51.927384 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:51.927339 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-0bda67-zhgmh"] Oct 30 13:03:51.928895 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:51.928864 4908 cpu_manager.go:414] "containerMap: removing container" podUID="61f8676a-8552-4066-a292-26bbd8b64d6b" containerName="istio-proxy" Oct 30 13:03:51.929064 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:51.929051 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61f8676a-8552-4066-a292-26bbd8b64d6b" containerName="istio-proxy" Oct 30 13:03:51.929177 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:51.929166 4908 cpu_manager.go:414] "containerMap: removing container" podUID="61f8676a-8552-4066-a292-26bbd8b64d6b" containerName="istio-proxy" Oct 30 13:03:51.929254 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:51.929244 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61f8676a-8552-4066-a292-26bbd8b64d6b" containerName="istio-proxy" Oct 30 13:03:51.929363 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:51.929352 4908 cpu_manager.go:414] "containerMap: removing container" podUID="151948d5-7d82-4a4b-8db1-feed4b587749" containerName="revoke-keys" Oct 30 13:03:51.929441 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:51.929431 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="151948d5-7d82-4a4b-8db1-feed4b587749" containerName="revoke-keys" Oct 30 13:03:51.929855 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:51.929818 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="61f8676a-8552-4066-a292-26bbd8b64d6b" containerName="istio-proxy" Oct 30 13:03:51.930035 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:51.930021 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="151948d5-7d82-4a4b-8db1-feed4b587749" containerName="revoke-keys" Oct 30 13:03:51.936528 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:51.936504 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-0bda67-zhgmh" Oct 30 13:03:51.953823 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:51.953790 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-0bda67-zhgmh"] Oct 30 13:03:51.958332 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:51.958302 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:03:51.958489 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:51.958397 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s75hq\"" Oct 30 13:03:51.958608 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:51.958588 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 13:03:51.963220 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:51.963192 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 13:03:52.053350 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:52.053310 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ml6ps\" (UniqueName: \"kubernetes.io/projected/d19ef1fd-2af5-46f9-a0dd-59af198e9b6f-kube-api-access-ml6ps\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-0bda67-zhgmh\" (UID: \"d19ef1fd-2af5-46f9-a0dd-59af198e9b6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-0bda67-zhgmh" Oct 30 13:03:52.053547 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:52.053357 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d19ef1fd-2af5-46f9-a0dd-59af198e9b6f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-0bda67-zhgmh\" (UID: \"d19ef1fd-2af5-46f9-a0dd-59af198e9b6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-0bda67-zhgmh" Oct 30 13:03:52.053592 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:52.053535 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d19ef1fd-2af5-46f9-a0dd-59af198e9b6f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-0bda67-zhgmh\" (UID: \"d19ef1fd-2af5-46f9-a0dd-59af198e9b6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-0bda67-zhgmh" Oct 30 13:03:52.154438 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:52.154399 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d19ef1fd-2af5-46f9-a0dd-59af198e9b6f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-0bda67-zhgmh\" (UID: \"d19ef1fd-2af5-46f9-a0dd-59af198e9b6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-0bda67-zhgmh" Oct 30 13:03:52.154617 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:52.154555 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ml6ps\" (UniqueName: \"kubernetes.io/projected/d19ef1fd-2af5-46f9-a0dd-59af198e9b6f-kube-api-access-ml6ps\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-0bda67-zhgmh\" (UID: \"d19ef1fd-2af5-46f9-a0dd-59af198e9b6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-0bda67-zhgmh" Oct 30 13:03:52.154617 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:52.154584 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d19ef1fd-2af5-46f9-a0dd-59af198e9b6f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-0bda67-zhgmh\" (UID: \"d19ef1fd-2af5-46f9-a0dd-59af198e9b6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-0bda67-zhgmh" Oct 30 13:03:52.155157 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:52.155133 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d19ef1fd-2af5-46f9-a0dd-59af198e9b6f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-0bda67-zhgmh\" (UID: \"d19ef1fd-2af5-46f9-a0dd-59af198e9b6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-0bda67-zhgmh" Oct 30 13:03:52.157147 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:52.157126 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d19ef1fd-2af5-46f9-a0dd-59af198e9b6f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-0bda67-zhgmh\" (UID: \"d19ef1fd-2af5-46f9-a0dd-59af198e9b6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-0bda67-zhgmh" Oct 30 13:03:52.183053 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:52.182983 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ml6ps\" (UniqueName: \"kubernetes.io/projected/d19ef1fd-2af5-46f9-a0dd-59af198e9b6f-kube-api-access-ml6ps\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-0bda67-zhgmh\" (UID: \"d19ef1fd-2af5-46f9-a0dd-59af198e9b6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-0bda67-zhgmh" Oct 30 13:03:52.253058 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:52.253012 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-0bda67-zhgmh" Oct 30 13:03:52.550692 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:52.550657 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c8c8523-lwhzc"] Oct 30 13:03:52.551375 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:52.551358 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="61f8676a-8552-4066-a292-26bbd8b64d6b" containerName="istio-proxy" Oct 30 13:03:52.555657 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:52.555639 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c8c8523-lwhzc" Oct 30 13:03:52.564899 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:52.564633 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-0bda67-zhgmh"] Oct 30 13:03:52.588105 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:52.588073 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c8c8523-lwhzc"] Oct 30 13:03:52.661261 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:52.661222 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00b2ae7c-2c85-4f84-b5a3-a1d3ef7ca4e6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2c8c8523-lwhzc\" (UID: \"00b2ae7c-2c85-4f84-b5a3-a1d3ef7ca4e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c8c8523-lwhzc" Oct 30 13:03:52.661470 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:52.661433 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00b2ae7c-2c85-4f84-b5a3-a1d3ef7ca4e6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2c8c8523-lwhzc\" (UID: \"00b2ae7c-2c85-4f84-b5a3-a1d3ef7ca4e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c8c8523-lwhzc" Oct 30 13:03:52.661608 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:52.661590 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4d9n\" (UniqueName: \"kubernetes.io/projected/00b2ae7c-2c85-4f84-b5a3-a1d3ef7ca4e6-kube-api-access-r4d9n\") pod \"maas-api-revoke-sub-2c8c8523-lwhzc\" (UID: \"00b2ae7c-2c85-4f84-b5a3-a1d3ef7ca4e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c8c8523-lwhzc" Oct 30 13:03:52.763606 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:52.763564 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00b2ae7c-2c85-4f84-b5a3-a1d3ef7ca4e6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2c8c8523-lwhzc\" (UID: \"00b2ae7c-2c85-4f84-b5a3-a1d3ef7ca4e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c8c8523-lwhzc" Oct 30 13:03:52.763846 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:52.763743 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00b2ae7c-2c85-4f84-b5a3-a1d3ef7ca4e6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2c8c8523-lwhzc\" (UID: \"00b2ae7c-2c85-4f84-b5a3-a1d3ef7ca4e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c8c8523-lwhzc" Oct 30 13:03:52.763979 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:52.763954 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4d9n\" (UniqueName: \"kubernetes.io/projected/00b2ae7c-2c85-4f84-b5a3-a1d3ef7ca4e6-kube-api-access-r4d9n\") pod \"maas-api-revoke-sub-2c8c8523-lwhzc\" (UID: \"00b2ae7c-2c85-4f84-b5a3-a1d3ef7ca4e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c8c8523-lwhzc" Oct 30 13:03:52.764431 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:52.764406 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00b2ae7c-2c85-4f84-b5a3-a1d3ef7ca4e6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2c8c8523-lwhzc\" (UID: \"00b2ae7c-2c85-4f84-b5a3-a1d3ef7ca4e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c8c8523-lwhzc" Oct 30 13:03:52.766091 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:52.766070 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00b2ae7c-2c85-4f84-b5a3-a1d3ef7ca4e6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2c8c8523-lwhzc\" (UID: \"00b2ae7c-2c85-4f84-b5a3-a1d3ef7ca4e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c8c8523-lwhzc" Oct 30 13:03:52.786551 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:52.786528 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4d9n\" (UniqueName: \"kubernetes.io/projected/00b2ae7c-2c85-4f84-b5a3-a1d3ef7ca4e6-kube-api-access-r4d9n\") pod \"maas-api-revoke-sub-2c8c8523-lwhzc\" (UID: \"00b2ae7c-2c85-4f84-b5a3-a1d3ef7ca4e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c8c8523-lwhzc" Oct 30 13:03:52.891860 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:52.891811 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c8c8523-lwhzc" Oct 30 13:03:53.049686 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:53.049643 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-0bda67-zhgmh" event={"ID":"d19ef1fd-2af5-46f9-a0dd-59af198e9b6f","Type":"ContainerStarted","Data":"3d2cf01d94a04429b1efa5eb7b77e903288f114bce1ac251e446dbe445501071"} Oct 30 13:03:53.049864 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:53.049702 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-0bda67-zhgmh" event={"ID":"d19ef1fd-2af5-46f9-a0dd-59af198e9b6f","Type":"ContainerStarted","Data":"81119df3e0e2628dfdd56eb8ec1a9709cc4a0671d7607c805ff940ad4c2f269b"} Oct 30 13:03:53.110061 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:53.109988 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-0bda67-zhgmh" podStartSLOduration=2.109966377 podStartE2EDuration="2.109966377s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:03:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:03:53.09350592 +0000 UTC m=+4189.607934690" watchObservedRunningTime="2026-10-30 13:03:53.109966377 +0000 UTC m=+4189.624395125" Oct 30 13:03:53.204805 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:53.204774 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c8c8523-lwhzc"] Oct 30 13:03:53.207203 ip-10-0-141-251 kubenswrapper[4908]: W1030 13:03:53.207172 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00b2ae7c_2c85_4f84_b5a3_a1d3ef7ca4e6.slice/crio-b30e262ebe6ca105b18937bbce1c5016146a63813a25b7888dce81b324cadc2b WatchSource:0}: Error finding container b30e262ebe6ca105b18937bbce1c5016146a63813a25b7888dce81b324cadc2b: Status 404 returned error can't find the container with id b30e262ebe6ca105b18937bbce1c5016146a63813a25b7888dce81b324cadc2b Oct 30 13:03:54.060249 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:54.060211 4908 generic.go:356] "Generic (PLEG): container finished" podID="d19ef1fd-2af5-46f9-a0dd-59af198e9b6f" containerID="3d2cf01d94a04429b1efa5eb7b77e903288f114bce1ac251e446dbe445501071" exitCode=0 Oct 30 13:03:54.060701 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:54.060309 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-0bda67-zhgmh" event={"ID":"d19ef1fd-2af5-46f9-a0dd-59af198e9b6f","Type":"ContainerDied","Data":"3d2cf01d94a04429b1efa5eb7b77e903288f114bce1ac251e446dbe445501071"} Oct 30 13:03:54.062586 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:54.062555 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c8c8523-lwhzc" event={"ID":"00b2ae7c-2c85-4f84-b5a3-a1d3ef7ca4e6","Type":"ContainerStarted","Data":"b45da0a61ff596744b6b70fa4ca7c566b2793b22603fe65ac2d3b633a6ec44e7"} Oct 30 13:03:54.062745 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:54.062605 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c8c8523-lwhzc" event={"ID":"00b2ae7c-2c85-4f84-b5a3-a1d3ef7ca4e6","Type":"ContainerStarted","Data":"b30e262ebe6ca105b18937bbce1c5016146a63813a25b7888dce81b324cadc2b"} Oct 30 13:03:54.147510 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:54.147439 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c8c8523-lwhzc" podStartSLOduration=2.14741961 podStartE2EDuration="2.14741961s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:03:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:03:54.143978788 +0000 UTC m=+4190.658407534" watchObservedRunningTime="2026-10-30 13:03:54.14741961 +0000 UTC m=+4190.661848359" Oct 30 13:03:55.072772 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:55.072731 4908 generic.go:356] "Generic (PLEG): container finished" podID="00b2ae7c-2c85-4f84-b5a3-a1d3ef7ca4e6" containerID="b45da0a61ff596744b6b70fa4ca7c566b2793b22603fe65ac2d3b633a6ec44e7" exitCode=0 Oct 30 13:03:55.073289 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:55.072812 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c8c8523-lwhzc" event={"ID":"00b2ae7c-2c85-4f84-b5a3-a1d3ef7ca4e6","Type":"ContainerDied","Data":"b45da0a61ff596744b6b70fa4ca7c566b2793b22603fe65ac2d3b633a6ec44e7"} Oct 30 13:03:55.263917 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:55.263885 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-0bda67-zhgmh" Oct 30 13:03:55.403463 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:55.403363 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d19ef1fd-2af5-46f9-a0dd-59af198e9b6f-kube-api-access-ml6ps\" (UniqueName: \"kubernetes.io/projected/d19ef1fd-2af5-46f9-a0dd-59af198e9b6f-kube-api-access-ml6ps\") pod \"d19ef1fd-2af5-46f9-a0dd-59af198e9b6f\" (UID: \"d19ef1fd-2af5-46f9-a0dd-59af198e9b6f\") " Oct 30 13:03:55.403463 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:55.403454 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d19ef1fd-2af5-46f9-a0dd-59af198e9b6f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d19ef1fd-2af5-46f9-a0dd-59af198e9b6f-maas-api-openshift-ca\") pod \"d19ef1fd-2af5-46f9-a0dd-59af198e9b6f\" (UID: \"d19ef1fd-2af5-46f9-a0dd-59af198e9b6f\") " Oct 30 13:03:55.403719 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:55.403597 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d19ef1fd-2af5-46f9-a0dd-59af198e9b6f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d19ef1fd-2af5-46f9-a0dd-59af198e9b6f-maas-api-xks-ca\") pod \"d19ef1fd-2af5-46f9-a0dd-59af198e9b6f\" (UID: \"d19ef1fd-2af5-46f9-a0dd-59af198e9b6f\") " Oct 30 13:03:55.403810 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:55.403787 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d19ef1fd-2af5-46f9-a0dd-59af198e9b6f-maas-api-openshift-ca" pod "d19ef1fd-2af5-46f9-a0dd-59af198e9b6f" (UID: "d19ef1fd-2af5-46f9-a0dd-59af198e9b6f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:03:55.404165 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:55.404146 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d19ef1fd-2af5-46f9-a0dd-59af198e9b6f-maas-api-openshift-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:03:55.405712 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:55.405682 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d19ef1fd-2af5-46f9-a0dd-59af198e9b6f-maas-api-xks-ca" pod "d19ef1fd-2af5-46f9-a0dd-59af198e9b6f" (UID: "d19ef1fd-2af5-46f9-a0dd-59af198e9b6f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:03:55.405881 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:55.405723 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d19ef1fd-2af5-46f9-a0dd-59af198e9b6f-kube-api-access-ml6ps" pod "d19ef1fd-2af5-46f9-a0dd-59af198e9b6f" (UID: "d19ef1fd-2af5-46f9-a0dd-59af198e9b6f"). InnerVolumeSpecName "kube-api-access-ml6ps". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:03:55.513859 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:55.505947 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d19ef1fd-2af5-46f9-a0dd-59af198e9b6f-maas-api-xks-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:03:55.513859 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:55.505993 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ml6ps\" (UniqueName: \"kubernetes.io/projected/d19ef1fd-2af5-46f9-a0dd-59af198e9b6f-kube-api-access-ml6ps\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:03:56.081444 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:56.081408 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-0bda67-zhgmh" Oct 30 13:03:56.081444 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:56.081429 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-0bda67-zhgmh" event={"ID":"d19ef1fd-2af5-46f9-a0dd-59af198e9b6f","Type":"ContainerDied","Data":"81119df3e0e2628dfdd56eb8ec1a9709cc4a0671d7607c805ff940ad4c2f269b"} Oct 30 13:03:56.081944 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:56.081477 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="81119df3e0e2628dfdd56eb8ec1a9709cc4a0671d7607c805ff940ad4c2f269b" Oct 30 13:03:56.261623 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:56.261591 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c8c8523-lwhzc" Oct 30 13:03:56.418654 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:56.418618 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/00b2ae7c-2c85-4f84-b5a3-a1d3ef7ca4e6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00b2ae7c-2c85-4f84-b5a3-a1d3ef7ca4e6-maas-api-xks-ca\") pod \"00b2ae7c-2c85-4f84-b5a3-a1d3ef7ca4e6\" (UID: \"00b2ae7c-2c85-4f84-b5a3-a1d3ef7ca4e6\") " Oct 30 13:03:56.418909 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:56.418890 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00b2ae7c-2c85-4f84-b5a3-a1d3ef7ca4e6-kube-api-access-r4d9n\" (UniqueName: \"kubernetes.io/projected/00b2ae7c-2c85-4f84-b5a3-a1d3ef7ca4e6-kube-api-access-r4d9n\") pod \"00b2ae7c-2c85-4f84-b5a3-a1d3ef7ca4e6\" (UID: \"00b2ae7c-2c85-4f84-b5a3-a1d3ef7ca4e6\") " Oct 30 13:03:56.419028 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:56.419011 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00b2ae7c-2c85-4f84-b5a3-a1d3ef7ca4e6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00b2ae7c-2c85-4f84-b5a3-a1d3ef7ca4e6-maas-api-openshift-ca\") pod \"00b2ae7c-2c85-4f84-b5a3-a1d3ef7ca4e6\" (UID: \"00b2ae7c-2c85-4f84-b5a3-a1d3ef7ca4e6\") " Oct 30 13:03:56.419370 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:56.419344 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00b2ae7c-2c85-4f84-b5a3-a1d3ef7ca4e6-maas-api-openshift-ca" pod "00b2ae7c-2c85-4f84-b5a3-a1d3ef7ca4e6" (UID: "00b2ae7c-2c85-4f84-b5a3-a1d3ef7ca4e6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:03:56.419656 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:56.419636 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00b2ae7c-2c85-4f84-b5a3-a1d3ef7ca4e6-maas-api-openshift-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:03:56.420728 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:56.420702 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00b2ae7c-2c85-4f84-b5a3-a1d3ef7ca4e6-maas-api-xks-ca" pod "00b2ae7c-2c85-4f84-b5a3-a1d3ef7ca4e6" (UID: "00b2ae7c-2c85-4f84-b5a3-a1d3ef7ca4e6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:03:56.421000 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:56.420975 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00b2ae7c-2c85-4f84-b5a3-a1d3ef7ca4e6-kube-api-access-r4d9n" pod "00b2ae7c-2c85-4f84-b5a3-a1d3ef7ca4e6" (UID: "00b2ae7c-2c85-4f84-b5a3-a1d3ef7ca4e6"). InnerVolumeSpecName "kube-api-access-r4d9n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:03:56.539343 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:56.539304 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00b2ae7c-2c85-4f84-b5a3-a1d3ef7ca4e6-maas-api-xks-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:03:56.539578 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:56.539560 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4d9n\" (UniqueName: \"kubernetes.io/projected/00b2ae7c-2c85-4f84-b5a3-a1d3ef7ca4e6-kube-api-access-r4d9n\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:03:56.722987 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:56.722958 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-ecb3aa-openshift-default-68d9fff7cc-xb5ct" Oct 30 13:03:56.844791 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:56.844757 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e59c9971-42c6-4df7-94a0-fd6753d847ca-istio-token\" (UniqueName: \"kubernetes.io/projected/e59c9971-42c6-4df7-94a0-fd6753d847ca-istio-token\") pod \"e59c9971-42c6-4df7-94a0-fd6753d847ca\" (UID: \"e59c9971-42c6-4df7-94a0-fd6753d847ca\") " Oct 30 13:03:56.845039 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:56.844823 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e59c9971-42c6-4df7-94a0-fd6753d847ca-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e59c9971-42c6-4df7-94a0-fd6753d847ca-istio-data\") pod \"e59c9971-42c6-4df7-94a0-fd6753d847ca\" (UID: \"e59c9971-42c6-4df7-94a0-fd6753d847ca\") " Oct 30 13:03:56.845039 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:56.844923 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e59c9971-42c6-4df7-94a0-fd6753d847ca-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e59c9971-42c6-4df7-94a0-fd6753d847ca-workload-socket\") pod \"e59c9971-42c6-4df7-94a0-fd6753d847ca\" (UID: \"e59c9971-42c6-4df7-94a0-fd6753d847ca\") " Oct 30 13:03:56.845039 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:56.844955 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e59c9971-42c6-4df7-94a0-fd6753d847ca-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e59c9971-42c6-4df7-94a0-fd6753d847ca-istio-podinfo\") pod \"e59c9971-42c6-4df7-94a0-fd6753d847ca\" (UID: \"e59c9971-42c6-4df7-94a0-fd6753d847ca\") " Oct 30 13:03:56.845232 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:56.845047 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e59c9971-42c6-4df7-94a0-fd6753d847ca-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e59c9971-42c6-4df7-94a0-fd6753d847ca-credential-socket\") pod \"e59c9971-42c6-4df7-94a0-fd6753d847ca\" (UID: \"e59c9971-42c6-4df7-94a0-fd6753d847ca\") " Oct 30 13:03:56.845232 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:56.845072 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e59c9971-42c6-4df7-94a0-fd6753d847ca-kube-api-access-sztmp\" (UniqueName: \"kubernetes.io/projected/e59c9971-42c6-4df7-94a0-fd6753d847ca-kube-api-access-sztmp\") pod \"e59c9971-42c6-4df7-94a0-fd6753d847ca\" (UID: \"e59c9971-42c6-4df7-94a0-fd6753d847ca\") " Oct 30 13:03:56.845232 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:56.845152 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e59c9971-42c6-4df7-94a0-fd6753d847ca-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e59c9971-42c6-4df7-94a0-fd6753d847ca-istiod-ca-cert\") pod \"e59c9971-42c6-4df7-94a0-fd6753d847ca\" (UID: \"e59c9971-42c6-4df7-94a0-fd6753d847ca\") " Oct 30 13:03:56.845232 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:56.845161 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e59c9971-42c6-4df7-94a0-fd6753d847ca-istio-data" pod "e59c9971-42c6-4df7-94a0-fd6753d847ca" (UID: "e59c9971-42c6-4df7-94a0-fd6753d847ca"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:03:56.845507 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:56.845232 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e59c9971-42c6-4df7-94a0-fd6753d847ca-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e59c9971-42c6-4df7-94a0-fd6753d847ca-workload-certs\") pod \"e59c9971-42c6-4df7-94a0-fd6753d847ca\" (UID: \"e59c9971-42c6-4df7-94a0-fd6753d847ca\") " Oct 30 13:03:56.845507 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:56.845263 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e59c9971-42c6-4df7-94a0-fd6753d847ca-workload-socket" pod "e59c9971-42c6-4df7-94a0-fd6753d847ca" (UID: "e59c9971-42c6-4df7-94a0-fd6753d847ca"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:03:56.845507 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:56.845345 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e59c9971-42c6-4df7-94a0-fd6753d847ca-credential-socket" pod "e59c9971-42c6-4df7-94a0-fd6753d847ca" (UID: "e59c9971-42c6-4df7-94a0-fd6753d847ca"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:03:56.845507 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:56.845364 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e59c9971-42c6-4df7-94a0-fd6753d847ca-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e59c9971-42c6-4df7-94a0-fd6753d847ca-istio-envoy\") pod \"e59c9971-42c6-4df7-94a0-fd6753d847ca\" (UID: \"e59c9971-42c6-4df7-94a0-fd6753d847ca\") " Oct 30 13:03:56.845734 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:56.845553 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e59c9971-42c6-4df7-94a0-fd6753d847ca-istiod-ca-cert" pod "e59c9971-42c6-4df7-94a0-fd6753d847ca" (UID: "e59c9971-42c6-4df7-94a0-fd6753d847ca"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:03:56.845734 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:56.845574 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e59c9971-42c6-4df7-94a0-fd6753d847ca-workload-certs" pod "e59c9971-42c6-4df7-94a0-fd6753d847ca" (UID: "e59c9971-42c6-4df7-94a0-fd6753d847ca"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:03:56.846056 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:56.846034 4908 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e59c9971-42c6-4df7-94a0-fd6753d847ca-istio-data\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:03:56.846125 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:56.846063 4908 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e59c9971-42c6-4df7-94a0-fd6753d847ca-workload-socket\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:03:56.846125 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:56.846078 4908 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e59c9971-42c6-4df7-94a0-fd6753d847ca-credential-socket\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:03:56.846125 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:56.846091 4908 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e59c9971-42c6-4df7-94a0-fd6753d847ca-istiod-ca-cert\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:03:56.846125 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:56.846106 4908 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e59c9971-42c6-4df7-94a0-fd6753d847ca-workload-certs\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:03:56.847334 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:56.847313 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e59c9971-42c6-4df7-94a0-fd6753d847ca-kube-api-access-sztmp" pod "e59c9971-42c6-4df7-94a0-fd6753d847ca" (UID: "e59c9971-42c6-4df7-94a0-fd6753d847ca"). InnerVolumeSpecName "kube-api-access-sztmp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:03:56.847401 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:56.847364 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e59c9971-42c6-4df7-94a0-fd6753d847ca-istio-envoy" pod "e59c9971-42c6-4df7-94a0-fd6753d847ca" (UID: "e59c9971-42c6-4df7-94a0-fd6753d847ca"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:03:56.847576 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:56.847555 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e59c9971-42c6-4df7-94a0-fd6753d847ca-istio-podinfo" pod "e59c9971-42c6-4df7-94a0-fd6753d847ca" (UID: "e59c9971-42c6-4df7-94a0-fd6753d847ca"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:03:56.847617 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:56.847576 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e59c9971-42c6-4df7-94a0-fd6753d847ca-istio-token" pod "e59c9971-42c6-4df7-94a0-fd6753d847ca" (UID: "e59c9971-42c6-4df7-94a0-fd6753d847ca"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:03:56.947965 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:56.947875 4908 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e59c9971-42c6-4df7-94a0-fd6753d847ca-istio-envoy\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:03:56.947965 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:56.947908 4908 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e59c9971-42c6-4df7-94a0-fd6753d847ca-istio-token\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:03:56.947965 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:56.947918 4908 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e59c9971-42c6-4df7-94a0-fd6753d847ca-istio-podinfo\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:03:56.947965 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:56.947927 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sztmp\" (UniqueName: \"kubernetes.io/projected/e59c9971-42c6-4df7-94a0-fd6753d847ca-kube-api-access-sztmp\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:03:57.091252 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:57.091217 4908 generic.go:356] "Generic (PLEG): container finished" podID="e59c9971-42c6-4df7-94a0-fd6753d847ca" containerID="8ce08c9d4597660bc5284fdf10c7b97e8d93f9c4e7db07260f38a845bb89a568" exitCode=0 Oct 30 13:03:57.091733 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:57.091270 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-ecb3aa-openshift-default-68d9fff7cc-xb5ct" Oct 30 13:03:57.091733 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:57.091303 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-ecb3aa-openshift-default-68d9fff7cc-xb5ct" event={"ID":"e59c9971-42c6-4df7-94a0-fd6753d847ca","Type":"ContainerDied","Data":"8ce08c9d4597660bc5284fdf10c7b97e8d93f9c4e7db07260f38a845bb89a568"} Oct 30 13:03:57.091733 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:57.091364 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-ecb3aa-openshift-default-68d9fff7cc-xb5ct" event={"ID":"e59c9971-42c6-4df7-94a0-fd6753d847ca","Type":"ContainerDied","Data":"31c8b00537e28e5b9740ff12221a0dbfc2a3f94bbc7ad6717d2bec3c73513b68"} Oct 30 13:03:57.091733 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:57.091395 4908 scope.go:122] "RemoveContainer" containerID="8ce08c9d4597660bc5284fdf10c7b97e8d93f9c4e7db07260f38a845bb89a568" Oct 30 13:03:57.092794 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:57.092752 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c8c8523-lwhzc" event={"ID":"00b2ae7c-2c85-4f84-b5a3-a1d3ef7ca4e6","Type":"ContainerDied","Data":"b30e262ebe6ca105b18937bbce1c5016146a63813a25b7888dce81b324cadc2b"} Oct 30 13:03:57.092794 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:57.092766 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c8c8523-lwhzc" Oct 30 13:03:57.092794 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:57.092793 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b30e262ebe6ca105b18937bbce1c5016146a63813a25b7888dce81b324cadc2b" Oct 30 13:03:57.109739 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:57.109711 4908 scope.go:122] "RemoveContainer" containerID="2964357075b0307575fbc5e3023d9e141572ec36c9be49b795754dcf26839c80" Oct 30 13:03:57.120534 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:57.120514 4908 scope.go:122] "RemoveContainer" containerID="8ce08c9d4597660bc5284fdf10c7b97e8d93f9c4e7db07260f38a845bb89a568" Oct 30 13:03:57.120864 ip-10-0-141-251 kubenswrapper[4908]: E1030 13:03:57.120822 4908 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ce08c9d4597660bc5284fdf10c7b97e8d93f9c4e7db07260f38a845bb89a568\": container with ID starting with 8ce08c9d4597660bc5284fdf10c7b97e8d93f9c4e7db07260f38a845bb89a568 not found: ID does not exist" containerID="8ce08c9d4597660bc5284fdf10c7b97e8d93f9c4e7db07260f38a845bb89a568" Oct 30 13:03:57.120911 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:57.120875 4908 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ce08c9d4597660bc5284fdf10c7b97e8d93f9c4e7db07260f38a845bb89a568"} err="failed to get container status \"8ce08c9d4597660bc5284fdf10c7b97e8d93f9c4e7db07260f38a845bb89a568\": rpc error: code = NotFound desc = could not find container \"8ce08c9d4597660bc5284fdf10c7b97e8d93f9c4e7db07260f38a845bb89a568\": container with ID starting with 8ce08c9d4597660bc5284fdf10c7b97e8d93f9c4e7db07260f38a845bb89a568 not found: ID does not exist" Oct 30 13:03:57.120911 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:57.120894 4908 scope.go:122] "RemoveContainer" containerID="2964357075b0307575fbc5e3023d9e141572ec36c9be49b795754dcf26839c80" Oct 30 13:03:57.121141 ip-10-0-141-251 kubenswrapper[4908]: E1030 13:03:57.121123 4908 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2964357075b0307575fbc5e3023d9e141572ec36c9be49b795754dcf26839c80\": container with ID starting with 2964357075b0307575fbc5e3023d9e141572ec36c9be49b795754dcf26839c80 not found: ID does not exist" containerID="2964357075b0307575fbc5e3023d9e141572ec36c9be49b795754dcf26839c80" Oct 30 13:03:57.121201 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:57.121144 4908 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2964357075b0307575fbc5e3023d9e141572ec36c9be49b795754dcf26839c80"} err="failed to get container status \"2964357075b0307575fbc5e3023d9e141572ec36c9be49b795754dcf26839c80\": rpc error: code = NotFound desc = could not find container \"2964357075b0307575fbc5e3023d9e141572ec36c9be49b795754dcf26839c80\": container with ID starting with 2964357075b0307575fbc5e3023d9e141572ec36c9be49b795754dcf26839c80 not found: ID does not exist" Oct 30 13:03:57.172534 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:57.172497 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-ecb3aa-openshift-default-68d9fff7cc-xb5ct"] Oct 30 13:03:57.183884 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:57.183854 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-ecb3aa-openshift-default-68d9fff7cc-xb5ct"] Oct 30 13:03:57.611615 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:03:57.611533 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e59c9971-42c6-4df7-94a0-fd6753d847ca" path="/var/lib/kubelet/pods/e59c9971-42c6-4df7-94a0-fd6753d847ca/volumes" Oct 30 13:04:00.518401 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:00.518356 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-992edf68-5x8qs"] Oct 30 13:04:00.518945 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:00.518931 4908 cpu_manager.go:414] "containerMap: removing container" podUID="d19ef1fd-2af5-46f9-a0dd-59af198e9b6f" containerName="revoke-keys" Oct 30 13:04:00.518992 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:00.518946 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d19ef1fd-2af5-46f9-a0dd-59af198e9b6f" containerName="revoke-keys" Oct 30 13:04:00.518992 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:00.518968 4908 cpu_manager.go:414] "containerMap: removing container" podUID="e59c9971-42c6-4df7-94a0-fd6753d847ca" containerName="istio-proxy" Oct 30 13:04:00.518992 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:00.518974 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e59c9971-42c6-4df7-94a0-fd6753d847ca" containerName="istio-proxy" Oct 30 13:04:00.519092 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:00.518992 4908 cpu_manager.go:414] "containerMap: removing container" podUID="e59c9971-42c6-4df7-94a0-fd6753d847ca" containerName="istio-proxy" Oct 30 13:04:00.519092 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:00.518998 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e59c9971-42c6-4df7-94a0-fd6753d847ca" containerName="istio-proxy" Oct 30 13:04:00.519092 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:00.519019 4908 cpu_manager.go:414] "containerMap: removing container" podUID="00b2ae7c-2c85-4f84-b5a3-a1d3ef7ca4e6" containerName="revoke-keys" Oct 30 13:04:00.519092 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:00.519025 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00b2ae7c-2c85-4f84-b5a3-a1d3ef7ca4e6" containerName="revoke-keys" Oct 30 13:04:00.519221 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:00.519115 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="e59c9971-42c6-4df7-94a0-fd6753d847ca" containerName="istio-proxy" Oct 30 13:04:00.519221 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:00.519126 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="e59c9971-42c6-4df7-94a0-fd6753d847ca" containerName="istio-proxy" Oct 30 13:04:00.519221 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:00.519134 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="00b2ae7c-2c85-4f84-b5a3-a1d3ef7ca4e6" containerName="revoke-keys" Oct 30 13:04:00.519221 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:00.519143 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="d19ef1fd-2af5-46f9-a0dd-59af198e9b6f" containerName="revoke-keys" Oct 30 13:04:00.524434 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:00.524408 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-992edf68-5x8qs" Oct 30 13:04:00.538750 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:00.538716 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s75hq\"" Oct 30 13:04:00.543128 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:00.543095 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-992edf68-5x8qs"] Oct 30 13:04:00.549220 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:00.549190 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:04:00.549332 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:00.549194 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 13:04:00.549332 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:00.549238 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 13:04:00.605228 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:00.605191 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t27cp\" (UniqueName: \"kubernetes.io/projected/1c19d2e0-b5e4-4e44-a2cf-6724f0f1e0b1-kube-api-access-t27cp\") pod \"maas-api-revoke-sub-992edf68-5x8qs\" (UID: \"1c19d2e0-b5e4-4e44-a2cf-6724f0f1e0b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-992edf68-5x8qs" Oct 30 13:04:00.605434 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:00.605241 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c19d2e0-b5e4-4e44-a2cf-6724f0f1e0b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-992edf68-5x8qs\" (UID: \"1c19d2e0-b5e4-4e44-a2cf-6724f0f1e0b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-992edf68-5x8qs" Oct 30 13:04:00.605434 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:00.605388 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c19d2e0-b5e4-4e44-a2cf-6724f0f1e0b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-992edf68-5x8qs\" (UID: \"1c19d2e0-b5e4-4e44-a2cf-6724f0f1e0b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-992edf68-5x8qs" Oct 30 13:04:00.706913 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:00.706869 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t27cp\" (UniqueName: \"kubernetes.io/projected/1c19d2e0-b5e4-4e44-a2cf-6724f0f1e0b1-kube-api-access-t27cp\") pod \"maas-api-revoke-sub-992edf68-5x8qs\" (UID: \"1c19d2e0-b5e4-4e44-a2cf-6724f0f1e0b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-992edf68-5x8qs" Oct 30 13:04:00.707113 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:00.706952 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c19d2e0-b5e4-4e44-a2cf-6724f0f1e0b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-992edf68-5x8qs\" (UID: \"1c19d2e0-b5e4-4e44-a2cf-6724f0f1e0b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-992edf68-5x8qs" Oct 30 13:04:00.707113 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:00.707078 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c19d2e0-b5e4-4e44-a2cf-6724f0f1e0b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-992edf68-5x8qs\" (UID: \"1c19d2e0-b5e4-4e44-a2cf-6724f0f1e0b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-992edf68-5x8qs" Oct 30 13:04:00.707955 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:00.707925 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c19d2e0-b5e4-4e44-a2cf-6724f0f1e0b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-992edf68-5x8qs\" (UID: \"1c19d2e0-b5e4-4e44-a2cf-6724f0f1e0b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-992edf68-5x8qs" Oct 30 13:04:00.709500 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:00.709475 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c19d2e0-b5e4-4e44-a2cf-6724f0f1e0b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-992edf68-5x8qs\" (UID: \"1c19d2e0-b5e4-4e44-a2cf-6724f0f1e0b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-992edf68-5x8qs" Oct 30 13:04:00.729531 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:00.729498 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t27cp\" (UniqueName: \"kubernetes.io/projected/1c19d2e0-b5e4-4e44-a2cf-6724f0f1e0b1-kube-api-access-t27cp\") pod \"maas-api-revoke-sub-992edf68-5x8qs\" (UID: \"1c19d2e0-b5e4-4e44-a2cf-6724f0f1e0b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-992edf68-5x8qs" Oct 30 13:04:00.839690 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:00.839607 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-992edf68-5x8qs" Oct 30 13:04:01.153246 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:01.153212 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-992edf68-5x8qs"] Oct 30 13:04:01.155787 ip-10-0-141-251 kubenswrapper[4908]: W1030 13:04:01.155753 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c19d2e0_b5e4_4e44_a2cf_6724f0f1e0b1.slice/crio-074a36bf41549761d4e07a07dc5c5e6b932c25d3fa6ccd2071965ccc0856fe68 WatchSource:0}: Error finding container 074a36bf41549761d4e07a07dc5c5e6b932c25d3fa6ccd2071965ccc0856fe68: Status 404 returned error can't find the container with id 074a36bf41549761d4e07a07dc5c5e6b932c25d3fa6ccd2071965ccc0856fe68 Oct 30 13:04:02.133146 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:02.133111 4908 generic.go:356] "Generic (PLEG): container finished" podID="1c19d2e0-b5e4-4e44-a2cf-6724f0f1e0b1" containerID="453f3a1ce0c37b291835f57325d6789c0977328bac829442c8d6a273e12ddbe4" exitCode=0 Oct 30 13:04:02.133548 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:02.133197 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-992edf68-5x8qs" event={"ID":"1c19d2e0-b5e4-4e44-a2cf-6724f0f1e0b1","Type":"ContainerDied","Data":"453f3a1ce0c37b291835f57325d6789c0977328bac829442c8d6a273e12ddbe4"} Oct 30 13:04:02.133548 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:02.133255 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-992edf68-5x8qs" event={"ID":"1c19d2e0-b5e4-4e44-a2cf-6724f0f1e0b1","Type":"ContainerStarted","Data":"074a36bf41549761d4e07a07dc5c5e6b932c25d3fa6ccd2071965ccc0856fe68"} Oct 30 13:04:03.432318 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:03.432249 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-992edf68-5x8qs" Oct 30 13:04:03.551274 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:03.546352 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-6b67f8/test-model-6b67f8-kserve-6cdd49c85c-2ccp8"] Oct 30 13:04:03.582468 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:03.582435 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c19d2e0-b5e4-4e44-a2cf-6724f0f1e0b1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c19d2e0-b5e4-4e44-a2cf-6724f0f1e0b1-maas-api-xks-ca\") pod \"1c19d2e0-b5e4-4e44-a2cf-6724f0f1e0b1\" (UID: \"1c19d2e0-b5e4-4e44-a2cf-6724f0f1e0b1\") " Oct 30 13:04:03.583528 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:03.583502 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c19d2e0-b5e4-4e44-a2cf-6724f0f1e0b1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c19d2e0-b5e4-4e44-a2cf-6724f0f1e0b1-maas-api-openshift-ca\") pod \"1c19d2e0-b5e4-4e44-a2cf-6724f0f1e0b1\" (UID: \"1c19d2e0-b5e4-4e44-a2cf-6724f0f1e0b1\") " Oct 30 13:04:03.583669 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:03.583647 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c19d2e0-b5e4-4e44-a2cf-6724f0f1e0b1-kube-api-access-t27cp\" (UniqueName: \"kubernetes.io/projected/1c19d2e0-b5e4-4e44-a2cf-6724f0f1e0b1-kube-api-access-t27cp\") pod \"1c19d2e0-b5e4-4e44-a2cf-6724f0f1e0b1\" (UID: \"1c19d2e0-b5e4-4e44-a2cf-6724f0f1e0b1\") " Oct 30 13:04:03.587087 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:03.584474 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c19d2e0-b5e4-4e44-a2cf-6724f0f1e0b1-maas-api-openshift-ca" pod "1c19d2e0-b5e4-4e44-a2cf-6724f0f1e0b1" (UID: "1c19d2e0-b5e4-4e44-a2cf-6724f0f1e0b1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:04:03.587087 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:03.586723 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c19d2e0-b5e4-4e44-a2cf-6724f0f1e0b1-maas-api-xks-ca" pod "1c19d2e0-b5e4-4e44-a2cf-6724f0f1e0b1" (UID: "1c19d2e0-b5e4-4e44-a2cf-6724f0f1e0b1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:04:03.588334 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:03.588299 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c19d2e0-b5e4-4e44-a2cf-6724f0f1e0b1-kube-api-access-t27cp" pod "1c19d2e0-b5e4-4e44-a2cf-6724f0f1e0b1" (UID: "1c19d2e0-b5e4-4e44-a2cf-6724f0f1e0b1"). InnerVolumeSpecName "kube-api-access-t27cp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:04:03.716916 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:03.713285 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c19d2e0-b5e4-4e44-a2cf-6724f0f1e0b1-maas-api-xks-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:04:03.717172 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:03.717148 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c19d2e0-b5e4-4e44-a2cf-6724f0f1e0b1-maas-api-openshift-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:04:03.717290 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:03.717272 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t27cp\" (UniqueName: \"kubernetes.io/projected/1c19d2e0-b5e4-4e44-a2cf-6724f0f1e0b1-kube-api-access-t27cp\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:04:04.095559 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:04.095521 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd8413a1-9x4km"] Oct 30 13:04:04.096111 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:04.096096 4908 cpu_manager.go:414] "containerMap: removing container" podUID="1c19d2e0-b5e4-4e44-a2cf-6724f0f1e0b1" containerName="revoke-keys" Oct 30 13:04:04.096164 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:04.096112 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c19d2e0-b5e4-4e44-a2cf-6724f0f1e0b1" containerName="revoke-keys" Oct 30 13:04:04.096241 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:04.096232 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c19d2e0-b5e4-4e44-a2cf-6724f0f1e0b1" containerName="revoke-keys" Oct 30 13:04:04.100065 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:04.100036 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd8413a1-9x4km" Oct 30 13:04:04.120232 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:04.120198 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd8413a1-9x4km"] Oct 30 13:04:04.130667 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:04.130622 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a88b3635-eba4-41bf-b9d3-a1a14c5305f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fd8413a1-9x4km\" (UID: \"a88b3635-eba4-41bf-b9d3-a1a14c5305f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd8413a1-9x4km" Oct 30 13:04:04.130900 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:04.130679 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a88b3635-eba4-41bf-b9d3-a1a14c5305f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fd8413a1-9x4km\" (UID: \"a88b3635-eba4-41bf-b9d3-a1a14c5305f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd8413a1-9x4km" Oct 30 13:04:04.130900 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:04.130747 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97jsx\" (UniqueName: \"kubernetes.io/projected/a88b3635-eba4-41bf-b9d3-a1a14c5305f9-kube-api-access-97jsx\") pod \"maas-api-revoke-sub-fd8413a1-9x4km\" (UID: \"a88b3635-eba4-41bf-b9d3-a1a14c5305f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd8413a1-9x4km" Oct 30 13:04:04.176670 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:04.176632 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-992edf68-5x8qs" Oct 30 13:04:04.176670 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:04.176663 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-992edf68-5x8qs" event={"ID":"1c19d2e0-b5e4-4e44-a2cf-6724f0f1e0b1","Type":"ContainerDied","Data":"074a36bf41549761d4e07a07dc5c5e6b932c25d3fa6ccd2071965ccc0856fe68"} Oct 30 13:04:04.176960 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:04.176714 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="074a36bf41549761d4e07a07dc5c5e6b932c25d3fa6ccd2071965ccc0856fe68" Oct 30 13:04:04.232611 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:04.232573 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a88b3635-eba4-41bf-b9d3-a1a14c5305f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fd8413a1-9x4km\" (UID: \"a88b3635-eba4-41bf-b9d3-a1a14c5305f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd8413a1-9x4km" Oct 30 13:04:04.232611 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:04.232609 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a88b3635-eba4-41bf-b9d3-a1a14c5305f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fd8413a1-9x4km\" (UID: \"a88b3635-eba4-41bf-b9d3-a1a14c5305f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd8413a1-9x4km" Oct 30 13:04:04.232869 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:04.232692 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97jsx\" (UniqueName: \"kubernetes.io/projected/a88b3635-eba4-41bf-b9d3-a1a14c5305f9-kube-api-access-97jsx\") pod \"maas-api-revoke-sub-fd8413a1-9x4km\" (UID: \"a88b3635-eba4-41bf-b9d3-a1a14c5305f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd8413a1-9x4km" Oct 30 13:04:04.233436 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:04.233412 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a88b3635-eba4-41bf-b9d3-a1a14c5305f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fd8413a1-9x4km\" (UID: \"a88b3635-eba4-41bf-b9d3-a1a14c5305f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd8413a1-9x4km" Oct 30 13:04:04.235164 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:04.235137 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a88b3635-eba4-41bf-b9d3-a1a14c5305f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fd8413a1-9x4km\" (UID: \"a88b3635-eba4-41bf-b9d3-a1a14c5305f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd8413a1-9x4km" Oct 30 13:04:04.255276 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:04.255241 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-97jsx\" (UniqueName: \"kubernetes.io/projected/a88b3635-eba4-41bf-b9d3-a1a14c5305f9-kube-api-access-97jsx\") pod \"maas-api-revoke-sub-fd8413a1-9x4km\" (UID: \"a88b3635-eba4-41bf-b9d3-a1a14c5305f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd8413a1-9x4km" Oct 30 13:04:04.420556 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:04.420444 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd8413a1-9x4km" Oct 30 13:04:04.745088 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:04.745059 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd8413a1-9x4km"] Oct 30 13:04:04.747304 ip-10-0-141-251 kubenswrapper[4908]: W1030 13:04:04.747261 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda88b3635_eba4_41bf_b9d3_a1a14c5305f9.slice/crio-3a58bacf250910d694e341ce7c40bc758cacb0244caa59ac0ba22d289561ecfa WatchSource:0}: Error finding container 3a58bacf250910d694e341ce7c40bc758cacb0244caa59ac0ba22d289561ecfa: Status 404 returned error can't find the container with id 3a58bacf250910d694e341ce7c40bc758cacb0244caa59ac0ba22d289561ecfa Oct 30 13:04:05.194770 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:05.194726 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd8413a1-9x4km" event={"ID":"a88b3635-eba4-41bf-b9d3-a1a14c5305f9","Type":"ContainerStarted","Data":"394b5d61c097f9b712a7522773ee65dda6be9ad76415ad0079a3504a679aac7f"} Oct 30 13:04:05.195010 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:05.194797 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd8413a1-9x4km" event={"ID":"a88b3635-eba4-41bf-b9d3-a1a14c5305f9","Type":"ContainerStarted","Data":"3a58bacf250910d694e341ce7c40bc758cacb0244caa59ac0ba22d289561ecfa"} Oct 30 13:04:06.206251 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:06.206163 4908 generic.go:356] "Generic (PLEG): container finished" podID="a88b3635-eba4-41bf-b9d3-a1a14c5305f9" containerID="394b5d61c097f9b712a7522773ee65dda6be9ad76415ad0079a3504a679aac7f" exitCode=0 Oct 30 13:04:06.206684 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:06.206292 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd8413a1-9x4km" event={"ID":"a88b3635-eba4-41bf-b9d3-a1a14c5305f9","Type":"ContainerDied","Data":"394b5d61c097f9b712a7522773ee65dda6be9ad76415ad0079a3504a679aac7f"} Oct 30 13:04:06.549000 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:06.548971 4908 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-plczt_a70ee11c-c20e-4a4f-8476-ac5eb02816d0/console-operator/3.log" Oct 30 13:04:06.554256 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:06.554234 4908 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b8bxz_644e19b3-10ad-4145-9ed3-595d19ed9b07/ovn-acl-logging/0.log" Oct 30 13:04:07.329923 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:07.326748 4908 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-plczt_a70ee11c-c20e-4a4f-8476-ac5eb02816d0/console-operator/3.log" Oct 30 13:04:07.337529 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:07.337502 4908 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b8bxz_644e19b3-10ad-4145-9ed3-595d19ed9b07/ovn-acl-logging/0.log" Oct 30 13:04:07.398444 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:07.398413 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd8413a1-9x4km" Oct 30 13:04:07.479387 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:07.479350 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a88b3635-eba4-41bf-b9d3-a1a14c5305f9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a88b3635-eba4-41bf-b9d3-a1a14c5305f9-maas-api-openshift-ca\") pod \"a88b3635-eba4-41bf-b9d3-a1a14c5305f9\" (UID: \"a88b3635-eba4-41bf-b9d3-a1a14c5305f9\") " Oct 30 13:04:07.479589 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:07.479556 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a88b3635-eba4-41bf-b9d3-a1a14c5305f9-kube-api-access-97jsx\" (UniqueName: \"kubernetes.io/projected/a88b3635-eba4-41bf-b9d3-a1a14c5305f9-kube-api-access-97jsx\") pod \"a88b3635-eba4-41bf-b9d3-a1a14c5305f9\" (UID: \"a88b3635-eba4-41bf-b9d3-a1a14c5305f9\") " Oct 30 13:04:07.479663 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:07.479605 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a88b3635-eba4-41bf-b9d3-a1a14c5305f9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a88b3635-eba4-41bf-b9d3-a1a14c5305f9-maas-api-xks-ca\") pod \"a88b3635-eba4-41bf-b9d3-a1a14c5305f9\" (UID: \"a88b3635-eba4-41bf-b9d3-a1a14c5305f9\") " Oct 30 13:04:07.479812 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:07.479744 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a88b3635-eba4-41bf-b9d3-a1a14c5305f9-maas-api-openshift-ca" pod "a88b3635-eba4-41bf-b9d3-a1a14c5305f9" (UID: "a88b3635-eba4-41bf-b9d3-a1a14c5305f9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:04:07.480273 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:07.480248 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a88b3635-eba4-41bf-b9d3-a1a14c5305f9-maas-api-openshift-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:04:07.481995 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:07.481971 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a88b3635-eba4-41bf-b9d3-a1a14c5305f9-maas-api-xks-ca" pod "a88b3635-eba4-41bf-b9d3-a1a14c5305f9" (UID: "a88b3635-eba4-41bf-b9d3-a1a14c5305f9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:04:07.482475 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:07.482456 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a88b3635-eba4-41bf-b9d3-a1a14c5305f9-kube-api-access-97jsx" pod "a88b3635-eba4-41bf-b9d3-a1a14c5305f9" (UID: "a88b3635-eba4-41bf-b9d3-a1a14c5305f9"). InnerVolumeSpecName "kube-api-access-97jsx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:04:07.582011 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:07.581968 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97jsx\" (UniqueName: \"kubernetes.io/projected/a88b3635-eba4-41bf-b9d3-a1a14c5305f9-kube-api-access-97jsx\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:04:07.582011 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:07.582010 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a88b3635-eba4-41bf-b9d3-a1a14c5305f9-maas-api-xks-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:04:07.836471 ip-10-0-141-251 kubenswrapper[4908]: E1030 13:04:07.836380 4908 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c19d2e0_b5e4_4e44_a2cf_6724f0f1e0b1.slice\": RecentStats: unable to find data in memory cache]" Oct 30 13:04:08.224896 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:08.224771 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd8413a1-9x4km" Oct 30 13:04:08.224896 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:08.224801 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd8413a1-9x4km" event={"ID":"a88b3635-eba4-41bf-b9d3-a1a14c5305f9","Type":"ContainerDied","Data":"3a58bacf250910d694e341ce7c40bc758cacb0244caa59ac0ba22d289561ecfa"} Oct 30 13:04:08.225099 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:08.224903 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3a58bacf250910d694e341ce7c40bc758cacb0244caa59ac0ba22d289561ecfa" Oct 30 13:04:09.163677 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:09.163640 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e9f184-lz9mg"] Oct 30 13:04:09.174092 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:09.174064 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e9f184-lz9mg"] Oct 30 13:04:09.609898 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:09.609813 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="151948d5-7d82-4a4b-8db1-feed4b587749" path="/var/lib/kubelet/pods/151948d5-7d82-4a4b-8db1-feed4b587749/volumes" Oct 30 13:04:10.507512 ip-10-0-141-251 kubenswrapper[4908]: E1030 13:04:10.507462 4908 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c19d2e0_b5e4_4e44_a2cf_6724f0f1e0b1.slice\": RecentStats: unable to find data in memory cache]" Oct 30 13:04:14.731785 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:14.731734 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-63f4e4/test-model-63f4e4-kserve-57bdcf4b79-cgctl"] Oct 30 13:04:15.224925 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:15.224839 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc7cac0f-6mqbg"] Oct 30 13:04:15.225420 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:15.225406 4908 cpu_manager.go:414] "containerMap: removing container" podUID="a88b3635-eba4-41bf-b9d3-a1a14c5305f9" containerName="revoke-keys" Oct 30 13:04:15.225467 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:15.225421 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a88b3635-eba4-41bf-b9d3-a1a14c5305f9" containerName="revoke-keys" Oct 30 13:04:15.225547 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:15.225538 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="a88b3635-eba4-41bf-b9d3-a1a14c5305f9" containerName="revoke-keys" Oct 30 13:04:15.228647 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:15.228628 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc7cac0f-6mqbg" Oct 30 13:04:15.245202 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:15.245170 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:04:15.245348 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:15.245247 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 13:04:15.245348 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:15.245323 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 13:04:15.263674 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:15.263571 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc7cac0f-6mqbg"] Oct 30 13:04:15.264986 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:15.263747 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s75hq\"" Oct 30 13:04:15.344333 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:15.344298 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7838da57-6c71-4e7a-a830-a2a69759e28d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc7cac0f-6mqbg\" (UID: \"7838da57-6c71-4e7a-a830-a2a69759e28d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc7cac0f-6mqbg" Oct 30 13:04:15.344544 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:15.344502 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdq2d\" (UniqueName: \"kubernetes.io/projected/7838da57-6c71-4e7a-a830-a2a69759e28d-kube-api-access-sdq2d\") pod \"maas-api-revoke-sub-bc7cac0f-6mqbg\" (UID: \"7838da57-6c71-4e7a-a830-a2a69759e28d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc7cac0f-6mqbg" Oct 30 13:04:15.344699 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:15.344680 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7838da57-6c71-4e7a-a830-a2a69759e28d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc7cac0f-6mqbg\" (UID: \"7838da57-6c71-4e7a-a830-a2a69759e28d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc7cac0f-6mqbg" Oct 30 13:04:15.446228 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:15.446189 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7838da57-6c71-4e7a-a830-a2a69759e28d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc7cac0f-6mqbg\" (UID: \"7838da57-6c71-4e7a-a830-a2a69759e28d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc7cac0f-6mqbg" Oct 30 13:04:15.446435 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:15.446272 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7838da57-6c71-4e7a-a830-a2a69759e28d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc7cac0f-6mqbg\" (UID: \"7838da57-6c71-4e7a-a830-a2a69759e28d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc7cac0f-6mqbg" Oct 30 13:04:15.446435 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:15.446380 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdq2d\" (UniqueName: \"kubernetes.io/projected/7838da57-6c71-4e7a-a830-a2a69759e28d-kube-api-access-sdq2d\") pod \"maas-api-revoke-sub-bc7cac0f-6mqbg\" (UID: \"7838da57-6c71-4e7a-a830-a2a69759e28d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc7cac0f-6mqbg" Oct 30 13:04:15.447055 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:15.447026 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7838da57-6c71-4e7a-a830-a2a69759e28d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc7cac0f-6mqbg\" (UID: \"7838da57-6c71-4e7a-a830-a2a69759e28d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc7cac0f-6mqbg" Oct 30 13:04:15.448754 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:15.448732 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7838da57-6c71-4e7a-a830-a2a69759e28d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc7cac0f-6mqbg\" (UID: \"7838da57-6c71-4e7a-a830-a2a69759e28d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc7cac0f-6mqbg" Oct 30 13:04:15.469769 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:15.469739 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdq2d\" (UniqueName: \"kubernetes.io/projected/7838da57-6c71-4e7a-a830-a2a69759e28d-kube-api-access-sdq2d\") pod \"maas-api-revoke-sub-bc7cac0f-6mqbg\" (UID: \"7838da57-6c71-4e7a-a830-a2a69759e28d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc7cac0f-6mqbg" Oct 30 13:04:15.542592 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:15.542549 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc7cac0f-6mqbg" Oct 30 13:04:15.861280 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:15.861251 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc7cac0f-6mqbg"] Oct 30 13:04:15.864095 ip-10-0-141-251 kubenswrapper[4908]: W1030 13:04:15.864062 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7838da57_6c71_4e7a_a830_a2a69759e28d.slice/crio-2c7fbdbd6ec78f5910a48c922fe516c2dc1380bac40648b118775a57c022bdbc WatchSource:0}: Error finding container 2c7fbdbd6ec78f5910a48c922fe516c2dc1380bac40648b118775a57c022bdbc: Status 404 returned error can't find the container with id 2c7fbdbd6ec78f5910a48c922fe516c2dc1380bac40648b118775a57c022bdbc Oct 30 13:04:16.213254 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:16.213142 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9wwfc"] Oct 30 13:04:16.321314 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:16.321265 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc7cac0f-6mqbg" event={"ID":"7838da57-6c71-4e7a-a830-a2a69759e28d","Type":"ContainerStarted","Data":"bd66b32c1c7493da6e62815b13c2e7cf234ea713966edb5d812a75a807258d04"} Oct 30 13:04:16.321577 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:16.321335 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc7cac0f-6mqbg" event={"ID":"7838da57-6c71-4e7a-a830-a2a69759e28d","Type":"ContainerStarted","Data":"2c7fbdbd6ec78f5910a48c922fe516c2dc1380bac40648b118775a57c022bdbc"} Oct 30 13:04:16.367701 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:16.367639 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc7cac0f-6mqbg" podStartSLOduration=1.36762192 podStartE2EDuration="1.36762192s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:04:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:04:16.365119794 +0000 UTC m=+4212.879548566" watchObservedRunningTime="2026-10-30 13:04:16.36762192 +0000 UTC m=+4212.882050666" Oct 30 13:04:17.333228 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:17.333191 4908 generic.go:356] "Generic (PLEG): container finished" podID="7838da57-6c71-4e7a-a830-a2a69759e28d" containerID="bd66b32c1c7493da6e62815b13c2e7cf234ea713966edb5d812a75a807258d04" exitCode=0 Oct 30 13:04:17.333655 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:17.333281 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc7cac0f-6mqbg" event={"ID":"7838da57-6c71-4e7a-a830-a2a69759e28d","Type":"ContainerDied","Data":"bd66b32c1c7493da6e62815b13c2e7cf234ea713966edb5d812a75a807258d04"} Oct 30 13:04:18.332005 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:18.331965 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-e9f184-openshift-default-55dd875b9f-jjt74"] Oct 30 13:04:18.332238 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:18.332211 4908 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-e9f184-openshift-default-55dd875b9f-jjt74" podUID="b1b34ded-198e-48a2-bd18-303865ac83f3" containerName="istio-proxy" containerID="cri-o://2b7f3531e930a52fbe1b088154c9e9fb14cb455b18cd81f2b6e4838fc2a06d84" gracePeriod=30 Oct 30 13:04:18.598250 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:18.593734 4908 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-6b67f8/test-model-6b67f8-kserve-6cdd49c85c-2ccp8" podUID="cd0cc8f8-87d0-49fe-a25f-51653f60dbdb" containerName="main" containerID="cri-o://f83f152572f8b2f95387e5d9c89701a204a807d4c9f18dd0b46140acae559e29" gracePeriod=45 Oct 30 13:04:18.686545 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:18.686459 4908 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-6b67f8/test-model-6b67f8-kserve-6cdd49c85c-2ccp8" podUID="cd0cc8f8-87d0-49fe-a25f-51653f60dbdb" containerName="main" probeResult="failure" output="Get \"https://10.132.1.43:8000/ready\": EOF" Oct 30 13:04:18.814737 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:18.814702 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc7cac0f-6mqbg" Oct 30 13:04:18.912108 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:18.912018 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6ad1da-8rb5r"] Oct 30 13:04:18.912816 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:18.912784 4908 cpu_manager.go:414] "containerMap: removing container" podUID="7838da57-6c71-4e7a-a830-a2a69759e28d" containerName="revoke-keys" Oct 30 13:04:18.912816 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:18.912811 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7838da57-6c71-4e7a-a830-a2a69759e28d" containerName="revoke-keys" Oct 30 13:04:18.913043 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:18.912965 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="7838da57-6c71-4e7a-a830-a2a69759e28d" containerName="revoke-keys" Oct 30 13:04:18.916926 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:18.916902 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6ad1da-8rb5r" Oct 30 13:04:18.935687 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:18.935649 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7838da57-6c71-4e7a-a830-a2a69759e28d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7838da57-6c71-4e7a-a830-a2a69759e28d-maas-api-openshift-ca\") pod \"7838da57-6c71-4e7a-a830-a2a69759e28d\" (UID: \"7838da57-6c71-4e7a-a830-a2a69759e28d\") " Oct 30 13:04:18.935867 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:18.935757 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7838da57-6c71-4e7a-a830-a2a69759e28d-kube-api-access-sdq2d\" (UniqueName: \"kubernetes.io/projected/7838da57-6c71-4e7a-a830-a2a69759e28d-kube-api-access-sdq2d\") pod \"7838da57-6c71-4e7a-a830-a2a69759e28d\" (UID: \"7838da57-6c71-4e7a-a830-a2a69759e28d\") " Oct 30 13:04:18.935955 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:18.935939 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7838da57-6c71-4e7a-a830-a2a69759e28d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7838da57-6c71-4e7a-a830-a2a69759e28d-maas-api-xks-ca\") pod \"7838da57-6c71-4e7a-a830-a2a69759e28d\" (UID: \"7838da57-6c71-4e7a-a830-a2a69759e28d\") " Oct 30 13:04:18.936220 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:18.936191 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7838da57-6c71-4e7a-a830-a2a69759e28d-maas-api-openshift-ca" pod "7838da57-6c71-4e7a-a830-a2a69759e28d" (UID: "7838da57-6c71-4e7a-a830-a2a69759e28d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:04:18.938098 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:18.936560 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7838da57-6c71-4e7a-a830-a2a69759e28d-maas-api-openshift-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:04:18.938098 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:18.936885 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6ad1da-8rb5r"] Oct 30 13:04:18.938291 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:18.938197 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7838da57-6c71-4e7a-a830-a2a69759e28d-kube-api-access-sdq2d" pod "7838da57-6c71-4e7a-a830-a2a69759e28d" (UID: "7838da57-6c71-4e7a-a830-a2a69759e28d"). InnerVolumeSpecName "kube-api-access-sdq2d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:04:18.938605 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:18.938583 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7838da57-6c71-4e7a-a830-a2a69759e28d-maas-api-xks-ca" pod "7838da57-6c71-4e7a-a830-a2a69759e28d" (UID: "7838da57-6c71-4e7a-a830-a2a69759e28d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:04:18.972239 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:18.972199 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-6b67f8/test-model-6b67f8-kserve-6cdd49c85c-2ccp8" Oct 30 13:04:19.037895 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:19.037796 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2124dd77-f0aa-4e09-87a8-1a0e62b17cde-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-6ad1da-8rb5r\" (UID: \"2124dd77-f0aa-4e09-87a8-1a0e62b17cde\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6ad1da-8rb5r" Oct 30 13:04:19.038176 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:19.038147 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65f9h\" (UniqueName: \"kubernetes.io/projected/2124dd77-f0aa-4e09-87a8-1a0e62b17cde-kube-api-access-65f9h\") pod \"maas-api-revoke-keys-e2e-extmod-a-6ad1da-8rb5r\" (UID: \"2124dd77-f0aa-4e09-87a8-1a0e62b17cde\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6ad1da-8rb5r" Oct 30 13:04:19.038307 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:19.038199 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2124dd77-f0aa-4e09-87a8-1a0e62b17cde-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-6ad1da-8rb5r\" (UID: \"2124dd77-f0aa-4e09-87a8-1a0e62b17cde\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6ad1da-8rb5r" Oct 30 13:04:19.038380 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:19.038340 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdq2d\" (UniqueName: \"kubernetes.io/projected/7838da57-6c71-4e7a-a830-a2a69759e28d-kube-api-access-sdq2d\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:04:19.038380 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:19.038357 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7838da57-6c71-4e7a-a830-a2a69759e28d-maas-api-xks-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:04:19.139254 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:19.139221 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd0cc8f8-87d0-49fe-a25f-51653f60dbdb-home\" (UniqueName: \"kubernetes.io/empty-dir/cd0cc8f8-87d0-49fe-a25f-51653f60dbdb-home\") pod \"cd0cc8f8-87d0-49fe-a25f-51653f60dbdb\" (UID: \"cd0cc8f8-87d0-49fe-a25f-51653f60dbdb\") " Oct 30 13:04:19.139445 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:19.139344 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd0cc8f8-87d0-49fe-a25f-51653f60dbdb-kube-api-access-pgnw8\" (UniqueName: \"kubernetes.io/projected/cd0cc8f8-87d0-49fe-a25f-51653f60dbdb-kube-api-access-pgnw8\") pod \"cd0cc8f8-87d0-49fe-a25f-51653f60dbdb\" (UID: \"cd0cc8f8-87d0-49fe-a25f-51653f60dbdb\") " Oct 30 13:04:19.139445 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:19.139431 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd0cc8f8-87d0-49fe-a25f-51653f60dbdb-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd0cc8f8-87d0-49fe-a25f-51653f60dbdb-tmp-dir\") pod \"cd0cc8f8-87d0-49fe-a25f-51653f60dbdb\" (UID: \"cd0cc8f8-87d0-49fe-a25f-51653f60dbdb\") " Oct 30 13:04:19.139563 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:19.139547 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd0cc8f8-87d0-49fe-a25f-51653f60dbdb-dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd0cc8f8-87d0-49fe-a25f-51653f60dbdb-dshm\") pod \"cd0cc8f8-87d0-49fe-a25f-51653f60dbdb\" (UID: \"cd0cc8f8-87d0-49fe-a25f-51653f60dbdb\") " Oct 30 13:04:19.139613 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:19.139554 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd0cc8f8-87d0-49fe-a25f-51653f60dbdb-home" pod "cd0cc8f8-87d0-49fe-a25f-51653f60dbdb" (UID: "cd0cc8f8-87d0-49fe-a25f-51653f60dbdb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:04:19.139613 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:19.139595 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd0cc8f8-87d0-49fe-a25f-51653f60dbdb-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd0cc8f8-87d0-49fe-a25f-51653f60dbdb-model-cache\") pod \"cd0cc8f8-87d0-49fe-a25f-51653f60dbdb\" (UID: \"cd0cc8f8-87d0-49fe-a25f-51653f60dbdb\") " Oct 30 13:04:19.139693 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:19.139655 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd0cc8f8-87d0-49fe-a25f-51653f60dbdb-tls-certs\" (UniqueName: \"kubernetes.io/secret/cd0cc8f8-87d0-49fe-a25f-51653f60dbdb-tls-certs\") pod \"cd0cc8f8-87d0-49fe-a25f-51653f60dbdb\" (UID: \"cd0cc8f8-87d0-49fe-a25f-51653f60dbdb\") " Oct 30 13:04:19.139745 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:19.139722 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd0cc8f8-87d0-49fe-a25f-51653f60dbdb-tmp-dir" pod "cd0cc8f8-87d0-49fe-a25f-51653f60dbdb" (UID: "cd0cc8f8-87d0-49fe-a25f-51653f60dbdb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:04:19.139904 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:19.139877 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2124dd77-f0aa-4e09-87a8-1a0e62b17cde-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-6ad1da-8rb5r\" (UID: \"2124dd77-f0aa-4e09-87a8-1a0e62b17cde\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6ad1da-8rb5r" Oct 30 13:04:19.140007 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:19.139932 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd0cc8f8-87d0-49fe-a25f-51653f60dbdb-model-cache" pod "cd0cc8f8-87d0-49fe-a25f-51653f60dbdb" (UID: "cd0cc8f8-87d0-49fe-a25f-51653f60dbdb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:04:19.140070 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:19.140025 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2124dd77-f0aa-4e09-87a8-1a0e62b17cde-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-6ad1da-8rb5r\" (UID: \"2124dd77-f0aa-4e09-87a8-1a0e62b17cde\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6ad1da-8rb5r" Oct 30 13:04:19.140317 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:19.140274 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65f9h\" (UniqueName: \"kubernetes.io/projected/2124dd77-f0aa-4e09-87a8-1a0e62b17cde-kube-api-access-65f9h\") pod \"maas-api-revoke-keys-e2e-extmod-a-6ad1da-8rb5r\" (UID: \"2124dd77-f0aa-4e09-87a8-1a0e62b17cde\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6ad1da-8rb5r" Oct 30 13:04:19.140844 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:19.140352 4908 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd0cc8f8-87d0-49fe-a25f-51653f60dbdb-tmp-dir\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:04:19.140844 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:19.140371 4908 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd0cc8f8-87d0-49fe-a25f-51653f60dbdb-model-cache\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:04:19.140844 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:19.140386 4908 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd0cc8f8-87d0-49fe-a25f-51653f60dbdb-home\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:04:19.140844 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:19.140791 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2124dd77-f0aa-4e09-87a8-1a0e62b17cde-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-6ad1da-8rb5r\" (UID: \"2124dd77-f0aa-4e09-87a8-1a0e62b17cde\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6ad1da-8rb5r" Oct 30 13:04:19.142110 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:19.142087 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd0cc8f8-87d0-49fe-a25f-51653f60dbdb-tls-certs" pod "cd0cc8f8-87d0-49fe-a25f-51653f60dbdb" (UID: "cd0cc8f8-87d0-49fe-a25f-51653f60dbdb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:04:19.142110 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:19.142101 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd0cc8f8-87d0-49fe-a25f-51653f60dbdb-dshm" pod "cd0cc8f8-87d0-49fe-a25f-51653f60dbdb" (UID: "cd0cc8f8-87d0-49fe-a25f-51653f60dbdb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:04:19.142288 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:19.142100 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd0cc8f8-87d0-49fe-a25f-51653f60dbdb-kube-api-access-pgnw8" pod "cd0cc8f8-87d0-49fe-a25f-51653f60dbdb" (UID: "cd0cc8f8-87d0-49fe-a25f-51653f60dbdb"). InnerVolumeSpecName "kube-api-access-pgnw8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:04:19.142416 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:19.142396 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2124dd77-f0aa-4e09-87a8-1a0e62b17cde-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-6ad1da-8rb5r\" (UID: \"2124dd77-f0aa-4e09-87a8-1a0e62b17cde\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6ad1da-8rb5r" Oct 30 13:04:19.164956 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:19.164893 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65f9h\" (UniqueName: \"kubernetes.io/projected/2124dd77-f0aa-4e09-87a8-1a0e62b17cde-kube-api-access-65f9h\") pod \"maas-api-revoke-keys-e2e-extmod-a-6ad1da-8rb5r\" (UID: \"2124dd77-f0aa-4e09-87a8-1a0e62b17cde\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6ad1da-8rb5r" Oct 30 13:04:19.236588 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:19.236551 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6ad1da-8rb5r" Oct 30 13:04:19.242269 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:19.242235 4908 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd0cc8f8-87d0-49fe-a25f-51653f60dbdb-tls-certs\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:04:19.242269 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:19.242266 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pgnw8\" (UniqueName: \"kubernetes.io/projected/cd0cc8f8-87d0-49fe-a25f-51653f60dbdb-kube-api-access-pgnw8\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:04:19.242461 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:19.242277 4908 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd0cc8f8-87d0-49fe-a25f-51653f60dbdb-dshm\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:04:19.356378 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:19.356223 4908 generic.go:356] "Generic (PLEG): container finished" podID="cd0cc8f8-87d0-49fe-a25f-51653f60dbdb" containerID="f83f152572f8b2f95387e5d9c89701a204a807d4c9f18dd0b46140acae559e29" exitCode=0 Oct 30 13:04:19.356378 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:19.356322 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-6b67f8/test-model-6b67f8-kserve-6cdd49c85c-2ccp8" event={"ID":"cd0cc8f8-87d0-49fe-a25f-51653f60dbdb","Type":"ContainerDied","Data":"f83f152572f8b2f95387e5d9c89701a204a807d4c9f18dd0b46140acae559e29"} Oct 30 13:04:19.359893 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:19.357635 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-6b67f8/test-model-6b67f8-kserve-6cdd49c85c-2ccp8" event={"ID":"cd0cc8f8-87d0-49fe-a25f-51653f60dbdb","Type":"ContainerDied","Data":"9ad127f88812530bce29cf6d51f1236ca7db28e94a652f3d41bae4d9ea228fd4"} Oct 30 13:04:19.359893 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:19.357681 4908 scope.go:122] "RemoveContainer" containerID="f83f152572f8b2f95387e5d9c89701a204a807d4c9f18dd0b46140acae559e29" Oct 30 13:04:19.359893 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:19.357871 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-6b67f8/test-model-6b67f8-kserve-6cdd49c85c-2ccp8" Oct 30 13:04:19.368675 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:19.366226 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc7cac0f-6mqbg" event={"ID":"7838da57-6c71-4e7a-a830-a2a69759e28d","Type":"ContainerDied","Data":"2c7fbdbd6ec78f5910a48c922fe516c2dc1380bac40648b118775a57c022bdbc"} Oct 30 13:04:19.368675 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:19.366289 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2c7fbdbd6ec78f5910a48c922fe516c2dc1380bac40648b118775a57c022bdbc" Oct 30 13:04:19.368675 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:19.366357 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc7cac0f-6mqbg" Oct 30 13:04:19.380332 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:19.380305 4908 scope.go:122] "RemoveContainer" containerID="f83f152572f8b2f95387e5d9c89701a204a807d4c9f18dd0b46140acae559e29" Oct 30 13:04:19.380724 ip-10-0-141-251 kubenswrapper[4908]: E1030 13:04:19.380698 4908 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f83f152572f8b2f95387e5d9c89701a204a807d4c9f18dd0b46140acae559e29\": container with ID starting with f83f152572f8b2f95387e5d9c89701a204a807d4c9f18dd0b46140acae559e29 not found: ID does not exist" containerID="f83f152572f8b2f95387e5d9c89701a204a807d4c9f18dd0b46140acae559e29" Oct 30 13:04:19.380788 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:19.380740 4908 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f83f152572f8b2f95387e5d9c89701a204a807d4c9f18dd0b46140acae559e29"} err="failed to get container status \"f83f152572f8b2f95387e5d9c89701a204a807d4c9f18dd0b46140acae559e29\": rpc error: code = NotFound desc = could not find container \"f83f152572f8b2f95387e5d9c89701a204a807d4c9f18dd0b46140acae559e29\": container with ID starting with f83f152572f8b2f95387e5d9c89701a204a807d4c9f18dd0b46140acae559e29 not found: ID does not exist" Oct 30 13:04:19.421940 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:19.421848 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-6b67f8/test-model-6b67f8-kserve-6cdd49c85c-2ccp8"] Oct 30 13:04:19.428586 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:19.428542 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-6b67f8/test-model-6b67f8-kserve-6cdd49c85c-2ccp8"] Oct 30 13:04:19.549220 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:19.549188 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6ad1da-8rb5r"] Oct 30 13:04:19.551685 ip-10-0-141-251 kubenswrapper[4908]: W1030 13:04:19.551657 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2124dd77_f0aa_4e09_87a8_1a0e62b17cde.slice/crio-d8f7300d4824058f163dd83942e1f420e883341454fbef06b0c3993a0398a908 WatchSource:0}: Error finding container d8f7300d4824058f163dd83942e1f420e883341454fbef06b0c3993a0398a908: Status 404 returned error can't find the container with id d8f7300d4824058f163dd83942e1f420e883341454fbef06b0c3993a0398a908 Oct 30 13:04:19.608773 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:19.608736 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd0cc8f8-87d0-49fe-a25f-51653f60dbdb" path="/var/lib/kubelet/pods/cd0cc8f8-87d0-49fe-a25f-51653f60dbdb/volumes" Oct 30 13:04:19.685909 ip-10-0-141-251 kubenswrapper[4908]: E1030 13:04:19.685811 4908 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vn6vv" containerName="extractor" Oct 30 13:04:20.389418 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:20.389371 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6ad1da-8rb5r" event={"ID":"2124dd77-f0aa-4e09-87a8-1a0e62b17cde","Type":"ContainerStarted","Data":"699382bafd0745bb6bfa068c1e24c814d52fae4237b7320378153a43d12f92e6"} Oct 30 13:04:20.389604 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:20.389480 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6ad1da-8rb5r" event={"ID":"2124dd77-f0aa-4e09-87a8-1a0e62b17cde","Type":"ContainerStarted","Data":"d8f7300d4824058f163dd83942e1f420e883341454fbef06b0c3993a0398a908"} Oct 30 13:04:20.434626 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:20.434567 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6ad1da-8rb5r" podStartSLOduration=2.434550739 podStartE2EDuration="2.434550739s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:04:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:04:20.431265661 +0000 UTC m=+4216.945694408" watchObservedRunningTime="2026-10-30 13:04:20.434550739 +0000 UTC m=+4216.948979484" Oct 30 13:04:20.543957 ip-10-0-141-251 kubenswrapper[4908]: E1030 13:04:20.527021 4908 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c19d2e0_b5e4_4e44_a2cf_6724f0f1e0b1.slice\": RecentStats: unable to find data in memory cache]" Oct 30 13:04:21.399841 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:21.399790 4908 generic.go:356] "Generic (PLEG): container finished" podID="2124dd77-f0aa-4e09-87a8-1a0e62b17cde" containerID="699382bafd0745bb6bfa068c1e24c814d52fae4237b7320378153a43d12f92e6" exitCode=0 Oct 30 13:04:21.400350 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:21.399859 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6ad1da-8rb5r" event={"ID":"2124dd77-f0aa-4e09-87a8-1a0e62b17cde","Type":"ContainerDied","Data":"699382bafd0745bb6bfa068c1e24c814d52fae4237b7320378153a43d12f92e6"} Oct 30 13:04:22.751898 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:22.751859 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-0bda67-zhgmh"] Oct 30 13:04:22.769853 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:22.763102 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-0bda67-zhgmh"] Oct 30 13:04:22.879064 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:22.879022 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6ad1da-8rb5r" Oct 30 13:04:22.897926 ip-10-0-141-251 kubenswrapper[4908]: E1030 13:04:22.897879 4908 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c19d2e0_b5e4_4e44_a2cf_6724f0f1e0b1.slice\": RecentStats: unable to find data in memory cache]" Oct 30 13:04:22.954050 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:22.954016 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2124dd77-f0aa-4e09-87a8-1a0e62b17cde-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2124dd77-f0aa-4e09-87a8-1a0e62b17cde-maas-api-xks-ca\") pod \"2124dd77-f0aa-4e09-87a8-1a0e62b17cde\" (UID: \"2124dd77-f0aa-4e09-87a8-1a0e62b17cde\") " Oct 30 13:04:22.954247 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:22.954228 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2124dd77-f0aa-4e09-87a8-1a0e62b17cde-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2124dd77-f0aa-4e09-87a8-1a0e62b17cde-maas-api-openshift-ca\") pod \"2124dd77-f0aa-4e09-87a8-1a0e62b17cde\" (UID: \"2124dd77-f0aa-4e09-87a8-1a0e62b17cde\") " Oct 30 13:04:22.954329 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:22.954314 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2124dd77-f0aa-4e09-87a8-1a0e62b17cde-kube-api-access-65f9h\" (UniqueName: \"kubernetes.io/projected/2124dd77-f0aa-4e09-87a8-1a0e62b17cde-kube-api-access-65f9h\") pod \"2124dd77-f0aa-4e09-87a8-1a0e62b17cde\" (UID: \"2124dd77-f0aa-4e09-87a8-1a0e62b17cde\") " Oct 30 13:04:22.954677 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:22.954644 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2124dd77-f0aa-4e09-87a8-1a0e62b17cde-maas-api-openshift-ca" pod "2124dd77-f0aa-4e09-87a8-1a0e62b17cde" (UID: "2124dd77-f0aa-4e09-87a8-1a0e62b17cde"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:04:22.954976 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:22.954914 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2124dd77-f0aa-4e09-87a8-1a0e62b17cde-maas-api-openshift-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:04:22.959878 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:22.956672 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2124dd77-f0aa-4e09-87a8-1a0e62b17cde-kube-api-access-65f9h" pod "2124dd77-f0aa-4e09-87a8-1a0e62b17cde" (UID: "2124dd77-f0aa-4e09-87a8-1a0e62b17cde"). InnerVolumeSpecName "kube-api-access-65f9h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:04:22.959878 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:22.956771 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2124dd77-f0aa-4e09-87a8-1a0e62b17cde-maas-api-xks-ca" pod "2124dd77-f0aa-4e09-87a8-1a0e62b17cde" (UID: "2124dd77-f0aa-4e09-87a8-1a0e62b17cde"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:04:23.056770 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:23.056733 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2124dd77-f0aa-4e09-87a8-1a0e62b17cde-maas-api-xks-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:04:23.056770 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:23.056768 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65f9h\" (UniqueName: \"kubernetes.io/projected/2124dd77-f0aa-4e09-87a8-1a0e62b17cde-kube-api-access-65f9h\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:04:23.602944 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:23.602856 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e9f184-openshift-default-55dd875b9f-jjt74" Oct 30 13:04:23.606926 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:23.606895 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d19ef1fd-2af5-46f9-a0dd-59af198e9b6f" path="/var/lib/kubelet/pods/d19ef1fd-2af5-46f9-a0dd-59af198e9b6f/volumes" Oct 30 13:04:23.615992 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:23.615968 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6ad1da-8rb5r" Oct 30 13:04:23.616176 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:23.616141 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6ad1da-8rb5r" event={"ID":"2124dd77-f0aa-4e09-87a8-1a0e62b17cde","Type":"ContainerDied","Data":"d8f7300d4824058f163dd83942e1f420e883341454fbef06b0c3993a0398a908"} Oct 30 13:04:23.616305 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:23.616203 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d8f7300d4824058f163dd83942e1f420e883341454fbef06b0c3993a0398a908" Oct 30 13:04:23.618657 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:23.618622 4908 generic.go:356] "Generic (PLEG): container finished" podID="b1b34ded-198e-48a2-bd18-303865ac83f3" containerID="2b7f3531e930a52fbe1b088154c9e9fb14cb455b18cd81f2b6e4838fc2a06d84" exitCode=0 Oct 30 13:04:23.618763 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:23.618704 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e9f184-openshift-default-55dd875b9f-jjt74" Oct 30 13:04:23.618763 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:23.618725 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e9f184-openshift-default-55dd875b9f-jjt74" event={"ID":"b1b34ded-198e-48a2-bd18-303865ac83f3","Type":"ContainerDied","Data":"2b7f3531e930a52fbe1b088154c9e9fb14cb455b18cd81f2b6e4838fc2a06d84"} Oct 30 13:04:23.618940 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:23.618766 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e9f184-openshift-default-55dd875b9f-jjt74" event={"ID":"b1b34ded-198e-48a2-bd18-303865ac83f3","Type":"ContainerDied","Data":"7e4dd14598c2e6c2b6775b150517e904160928faf5c67b22c7859665112ff9e5"} Oct 30 13:04:23.618940 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:23.618794 4908 scope.go:122] "RemoveContainer" containerID="2b7f3531e930a52fbe1b088154c9e9fb14cb455b18cd81f2b6e4838fc2a06d84" Oct 30 13:04:23.629945 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:23.629925 4908 scope.go:122] "RemoveContainer" containerID="bc985fb1bf4ea590b514fce128c27da55759701718af051ba7efcee15a3e13a8" Oct 30 13:04:23.640278 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:23.640258 4908 scope.go:122] "RemoveContainer" containerID="2b7f3531e930a52fbe1b088154c9e9fb14cb455b18cd81f2b6e4838fc2a06d84" Oct 30 13:04:23.640567 ip-10-0-141-251 kubenswrapper[4908]: E1030 13:04:23.640544 4908 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b7f3531e930a52fbe1b088154c9e9fb14cb455b18cd81f2b6e4838fc2a06d84\": container with ID starting with 2b7f3531e930a52fbe1b088154c9e9fb14cb455b18cd81f2b6e4838fc2a06d84 not found: ID does not exist" containerID="2b7f3531e930a52fbe1b088154c9e9fb14cb455b18cd81f2b6e4838fc2a06d84" Oct 30 13:04:23.640679 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:23.640574 4908 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b7f3531e930a52fbe1b088154c9e9fb14cb455b18cd81f2b6e4838fc2a06d84"} err="failed to get container status \"2b7f3531e930a52fbe1b088154c9e9fb14cb455b18cd81f2b6e4838fc2a06d84\": rpc error: code = NotFound desc = could not find container \"2b7f3531e930a52fbe1b088154c9e9fb14cb455b18cd81f2b6e4838fc2a06d84\": container with ID starting with 2b7f3531e930a52fbe1b088154c9e9fb14cb455b18cd81f2b6e4838fc2a06d84 not found: ID does not exist" Oct 30 13:04:23.640679 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:23.640593 4908 scope.go:122] "RemoveContainer" containerID="bc985fb1bf4ea590b514fce128c27da55759701718af051ba7efcee15a3e13a8" Oct 30 13:04:23.640803 ip-10-0-141-251 kubenswrapper[4908]: E1030 13:04:23.640789 4908 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc985fb1bf4ea590b514fce128c27da55759701718af051ba7efcee15a3e13a8\": container with ID starting with bc985fb1bf4ea590b514fce128c27da55759701718af051ba7efcee15a3e13a8 not found: ID does not exist" containerID="bc985fb1bf4ea590b514fce128c27da55759701718af051ba7efcee15a3e13a8" Oct 30 13:04:23.640902 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:23.640812 4908 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc985fb1bf4ea590b514fce128c27da55759701718af051ba7efcee15a3e13a8"} err="failed to get container status \"bc985fb1bf4ea590b514fce128c27da55759701718af051ba7efcee15a3e13a8\": rpc error: code = NotFound desc = could not find container \"bc985fb1bf4ea590b514fce128c27da55759701718af051ba7efcee15a3e13a8\": container with ID starting with bc985fb1bf4ea590b514fce128c27da55759701718af051ba7efcee15a3e13a8 not found: ID does not exist" Oct 30 13:04:23.665941 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:23.665915 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1b34ded-198e-48a2-bd18-303865ac83f3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1b34ded-198e-48a2-bd18-303865ac83f3-credential-socket\") pod \"b1b34ded-198e-48a2-bd18-303865ac83f3\" (UID: \"b1b34ded-198e-48a2-bd18-303865ac83f3\") " Oct 30 13:04:23.666113 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:23.665956 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1b34ded-198e-48a2-bd18-303865ac83f3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1b34ded-198e-48a2-bd18-303865ac83f3-workload-socket\") pod \"b1b34ded-198e-48a2-bd18-303865ac83f3\" (UID: \"b1b34ded-198e-48a2-bd18-303865ac83f3\") " Oct 30 13:04:23.666113 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:23.666054 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b1b34ded-198e-48a2-bd18-303865ac83f3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1b34ded-198e-48a2-bd18-303865ac83f3-istio-podinfo\") pod \"b1b34ded-198e-48a2-bd18-303865ac83f3\" (UID: \"b1b34ded-198e-48a2-bd18-303865ac83f3\") " Oct 30 13:04:23.666113 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:23.666083 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1b34ded-198e-48a2-bd18-303865ac83f3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1b34ded-198e-48a2-bd18-303865ac83f3-istio-envoy\") pod \"b1b34ded-198e-48a2-bd18-303865ac83f3\" (UID: \"b1b34ded-198e-48a2-bd18-303865ac83f3\") " Oct 30 13:04:23.666113 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:23.666107 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1b34ded-198e-48a2-bd18-303865ac83f3-kube-api-access-45dts\" (UniqueName: \"kubernetes.io/projected/b1b34ded-198e-48a2-bd18-303865ac83f3-kube-api-access-45dts\") pod \"b1b34ded-198e-48a2-bd18-303865ac83f3\" (UID: \"b1b34ded-198e-48a2-bd18-303865ac83f3\") " Oct 30 13:04:23.666348 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:23.666148 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1b34ded-198e-48a2-bd18-303865ac83f3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1b34ded-198e-48a2-bd18-303865ac83f3-istio-data\") pod \"b1b34ded-198e-48a2-bd18-303865ac83f3\" (UID: \"b1b34ded-198e-48a2-bd18-303865ac83f3\") " Oct 30 13:04:23.666348 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:23.666174 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1b34ded-198e-48a2-bd18-303865ac83f3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1b34ded-198e-48a2-bd18-303865ac83f3-workload-certs\") pod \"b1b34ded-198e-48a2-bd18-303865ac83f3\" (UID: \"b1b34ded-198e-48a2-bd18-303865ac83f3\") " Oct 30 13:04:23.666348 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:23.666272 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1b34ded-198e-48a2-bd18-303865ac83f3-credential-socket" pod "b1b34ded-198e-48a2-bd18-303865ac83f3" (UID: "b1b34ded-198e-48a2-bd18-303865ac83f3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:04:23.666348 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:23.666307 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1b34ded-198e-48a2-bd18-303865ac83f3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1b34ded-198e-48a2-bd18-303865ac83f3-istiod-ca-cert\") pod \"b1b34ded-198e-48a2-bd18-303865ac83f3\" (UID: \"b1b34ded-198e-48a2-bd18-303865ac83f3\") " Oct 30 13:04:23.666563 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:23.666352 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1b34ded-198e-48a2-bd18-303865ac83f3-workload-socket" pod "b1b34ded-198e-48a2-bd18-303865ac83f3" (UID: "b1b34ded-198e-48a2-bd18-303865ac83f3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:04:23.666563 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:23.666361 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1b34ded-198e-48a2-bd18-303865ac83f3-istio-token\" (UniqueName: \"kubernetes.io/projected/b1b34ded-198e-48a2-bd18-303865ac83f3-istio-token\") pod \"b1b34ded-198e-48a2-bd18-303865ac83f3\" (UID: \"b1b34ded-198e-48a2-bd18-303865ac83f3\") " Oct 30 13:04:23.667047 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:23.666660 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1b34ded-198e-48a2-bd18-303865ac83f3-workload-certs" pod "b1b34ded-198e-48a2-bd18-303865ac83f3" (UID: "b1b34ded-198e-48a2-bd18-303865ac83f3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:04:23.667047 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:23.666917 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1b34ded-198e-48a2-bd18-303865ac83f3-istio-data" pod "b1b34ded-198e-48a2-bd18-303865ac83f3" (UID: "b1b34ded-198e-48a2-bd18-303865ac83f3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:04:23.667047 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:23.667004 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1b34ded-198e-48a2-bd18-303865ac83f3-istiod-ca-cert" pod "b1b34ded-198e-48a2-bd18-303865ac83f3" (UID: "b1b34ded-198e-48a2-bd18-303865ac83f3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:04:23.667271 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:23.667151 4908 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1b34ded-198e-48a2-bd18-303865ac83f3-istio-data\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:04:23.667271 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:23.667170 4908 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1b34ded-198e-48a2-bd18-303865ac83f3-workload-certs\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:04:23.667271 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:23.667184 4908 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1b34ded-198e-48a2-bd18-303865ac83f3-istiod-ca-cert\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:04:23.667271 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:23.667197 4908 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1b34ded-198e-48a2-bd18-303865ac83f3-credential-socket\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:04:23.667271 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:23.667211 4908 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1b34ded-198e-48a2-bd18-303865ac83f3-workload-socket\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:04:23.669025 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:23.668999 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1b34ded-198e-48a2-bd18-303865ac83f3-istio-token" pod "b1b34ded-198e-48a2-bd18-303865ac83f3" (UID: "b1b34ded-198e-48a2-bd18-303865ac83f3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:04:23.669175 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:23.669094 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1b34ded-198e-48a2-bd18-303865ac83f3-kube-api-access-45dts" pod "b1b34ded-198e-48a2-bd18-303865ac83f3" (UID: "b1b34ded-198e-48a2-bd18-303865ac83f3"). InnerVolumeSpecName "kube-api-access-45dts". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:04:23.669239 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:23.669177 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1b34ded-198e-48a2-bd18-303865ac83f3-istio-envoy" pod "b1b34ded-198e-48a2-bd18-303865ac83f3" (UID: "b1b34ded-198e-48a2-bd18-303865ac83f3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:04:23.669526 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:23.669506 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b1b34ded-198e-48a2-bd18-303865ac83f3-istio-podinfo" pod "b1b34ded-198e-48a2-bd18-303865ac83f3" (UID: "b1b34ded-198e-48a2-bd18-303865ac83f3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:04:23.768951 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:23.768911 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45dts\" (UniqueName: \"kubernetes.io/projected/b1b34ded-198e-48a2-bd18-303865ac83f3-kube-api-access-45dts\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:04:23.768951 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:23.768946 4908 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1b34ded-198e-48a2-bd18-303865ac83f3-istio-token\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:04:23.768951 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:23.768957 4908 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1b34ded-198e-48a2-bd18-303865ac83f3-istio-podinfo\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:04:23.769416 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:23.768974 4908 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1b34ded-198e-48a2-bd18-303865ac83f3-istio-envoy\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:04:23.984409 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:23.984371 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-e9f184-openshift-default-55dd875b9f-jjt74"] Oct 30 13:04:24.001586 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:24.001545 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-e9f184-openshift-default-55dd875b9f-jjt74"] Oct 30 13:04:25.607433 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:25.607398 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1b34ded-198e-48a2-bd18-303865ac83f3" path="/var/lib/kubelet/pods/b1b34ded-198e-48a2-bd18-303865ac83f3/volumes" Oct 30 13:04:27.698604 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:27.698557 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6b67f8-2g65x"] Oct 30 13:04:27.699237 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:27.699216 4908 cpu_manager.go:414] "containerMap: removing container" podUID="b1b34ded-198e-48a2-bd18-303865ac83f3" containerName="istio-proxy" Oct 30 13:04:27.699290 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:27.699238 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1b34ded-198e-48a2-bd18-303865ac83f3" containerName="istio-proxy" Oct 30 13:04:27.699290 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:27.699248 4908 cpu_manager.go:414] "containerMap: removing container" podUID="cd0cc8f8-87d0-49fe-a25f-51653f60dbdb" containerName="main" Oct 30 13:04:27.699290 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:27.699255 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd0cc8f8-87d0-49fe-a25f-51653f60dbdb" containerName="main" Oct 30 13:04:27.699290 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:27.699278 4908 cpu_manager.go:414] "containerMap: removing container" podUID="b1b34ded-198e-48a2-bd18-303865ac83f3" containerName="istio-proxy" Oct 30 13:04:27.699290 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:27.699283 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1b34ded-198e-48a2-bd18-303865ac83f3" containerName="istio-proxy" Oct 30 13:04:27.699450 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:27.699314 4908 cpu_manager.go:414] "containerMap: removing container" podUID="2124dd77-f0aa-4e09-87a8-1a0e62b17cde" containerName="revoke-keys" Oct 30 13:04:27.699450 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:27.699319 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2124dd77-f0aa-4e09-87a8-1a0e62b17cde" containerName="revoke-keys" Oct 30 13:04:27.699450 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:27.699403 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="2124dd77-f0aa-4e09-87a8-1a0e62b17cde" containerName="revoke-keys" Oct 30 13:04:27.699450 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:27.699414 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1b34ded-198e-48a2-bd18-303865ac83f3" containerName="istio-proxy" Oct 30 13:04:27.699450 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:27.699420 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd0cc8f8-87d0-49fe-a25f-51653f60dbdb" containerName="main" Oct 30 13:04:27.699450 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:27.699430 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1b34ded-198e-48a2-bd18-303865ac83f3" containerName="istio-proxy" Oct 30 13:04:27.704547 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:27.704521 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6b67f8-2g65x" Oct 30 13:04:27.720117 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:27.720093 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6b67f8-2g65x"] Oct 30 13:04:27.726489 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:27.726463 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:04:27.726658 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:27.726603 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s75hq\"" Oct 30 13:04:27.742883 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:27.737711 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrtzg\" (UniqueName: \"kubernetes.io/projected/cabd4f34-efec-4a7e-89e7-b58692f3b37a-kube-api-access-nrtzg\") pod \"maas-api-revoke-keys-e2e-inf-a-6b67f8-2g65x\" (UID: \"cabd4f34-efec-4a7e-89e7-b58692f3b37a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6b67f8-2g65x" Oct 30 13:04:27.742883 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:27.737873 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cabd4f34-efec-4a7e-89e7-b58692f3b37a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-6b67f8-2g65x\" (UID: \"cabd4f34-efec-4a7e-89e7-b58692f3b37a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6b67f8-2g65x" Oct 30 13:04:27.742883 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:27.738065 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cabd4f34-efec-4a7e-89e7-b58692f3b37a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-6b67f8-2g65x\" (UID: \"cabd4f34-efec-4a7e-89e7-b58692f3b37a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6b67f8-2g65x" Oct 30 13:04:27.745131 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:27.745110 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 13:04:27.745297 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:27.745237 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 13:04:27.839108 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:27.839070 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cabd4f34-efec-4a7e-89e7-b58692f3b37a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-6b67f8-2g65x\" (UID: \"cabd4f34-efec-4a7e-89e7-b58692f3b37a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6b67f8-2g65x" Oct 30 13:04:27.839282 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:27.839225 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cabd4f34-efec-4a7e-89e7-b58692f3b37a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-6b67f8-2g65x\" (UID: \"cabd4f34-efec-4a7e-89e7-b58692f3b37a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6b67f8-2g65x" Oct 30 13:04:27.839393 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:27.839376 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrtzg\" (UniqueName: \"kubernetes.io/projected/cabd4f34-efec-4a7e-89e7-b58692f3b37a-kube-api-access-nrtzg\") pod \"maas-api-revoke-keys-e2e-inf-a-6b67f8-2g65x\" (UID: \"cabd4f34-efec-4a7e-89e7-b58692f3b37a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6b67f8-2g65x" Oct 30 13:04:27.839994 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:27.839965 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cabd4f34-efec-4a7e-89e7-b58692f3b37a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-6b67f8-2g65x\" (UID: \"cabd4f34-efec-4a7e-89e7-b58692f3b37a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6b67f8-2g65x" Oct 30 13:04:27.842235 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:27.842205 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cabd4f34-efec-4a7e-89e7-b58692f3b37a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-6b67f8-2g65x\" (UID: \"cabd4f34-efec-4a7e-89e7-b58692f3b37a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6b67f8-2g65x" Oct 30 13:04:27.872633 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:27.872596 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrtzg\" (UniqueName: \"kubernetes.io/projected/cabd4f34-efec-4a7e-89e7-b58692f3b37a-kube-api-access-nrtzg\") pod \"maas-api-revoke-keys-e2e-inf-a-6b67f8-2g65x\" (UID: \"cabd4f34-efec-4a7e-89e7-b58692f3b37a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6b67f8-2g65x" Oct 30 13:04:28.023305 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:28.023268 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6b67f8-2g65x" Oct 30 13:04:28.382681 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:28.381237 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6b67f8-2g65x"] Oct 30 13:04:28.385747 ip-10-0-141-251 kubenswrapper[4908]: W1030 13:04:28.385707 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcabd4f34_efec_4a7e_89e7_b58692f3b37a.slice/crio-066baad4df284a1030709604b61bfeaea1e20db0894abdad9e2baf3e8c3e02b0 WatchSource:0}: Error finding container 066baad4df284a1030709604b61bfeaea1e20db0894abdad9e2baf3e8c3e02b0: Status 404 returned error can't find the container with id 066baad4df284a1030709604b61bfeaea1e20db0894abdad9e2baf3e8c3e02b0 Oct 30 13:04:28.681521 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:28.681485 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6b67f8-2g65x" event={"ID":"cabd4f34-efec-4a7e-89e7-b58692f3b37a","Type":"ContainerStarted","Data":"066baad4df284a1030709604b61bfeaea1e20db0894abdad9e2baf3e8c3e02b0"} Oct 30 13:04:29.692526 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:29.692425 4908 generic.go:356] "Generic (PLEG): container finished" podID="cabd4f34-efec-4a7e-89e7-b58692f3b37a" containerID="9aeb27ca296d5a21220756ae2b62f61e1ef0f9f2ccfd6f3479cd646a39ea9fe4" exitCode=0 Oct 30 13:04:29.692979 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:29.692510 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6b67f8-2g65x" event={"ID":"cabd4f34-efec-4a7e-89e7-b58692f3b37a","Type":"ContainerDied","Data":"9aeb27ca296d5a21220756ae2b62f61e1ef0f9f2ccfd6f3479cd646a39ea9fe4"} Oct 30 13:04:29.753967 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:29.753906 4908 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-63f4e4/test-model-63f4e4-kserve-57bdcf4b79-cgctl" podUID="b3a9d487-206d-4027-b563-81d2db998422" containerName="main" containerID="cri-o://a8acc0048246a2aff4ebe1c30827792d5fb04c84f22516061f37988fc01081e5" gracePeriod=45 Oct 30 13:04:29.793303 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:29.793245 4908 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-63f4e4/test-model-63f4e4-kserve-57bdcf4b79-cgctl" podUID="b3a9d487-206d-4027-b563-81d2db998422" containerName="main" probeResult="failure" output="Get \"https://10.132.1.41:8000/ready\": dial tcp 10.132.1.41:8000: connect: connection refused" Oct 30 13:04:30.209894 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:30.209863 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-63f4e4/test-model-63f4e4-kserve-57bdcf4b79-cgctl" Oct 30 13:04:30.285539 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:30.285499 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3a9d487-206d-4027-b563-81d2db998422-home\" (UniqueName: \"kubernetes.io/empty-dir/b3a9d487-206d-4027-b563-81d2db998422-home\") pod \"b3a9d487-206d-4027-b563-81d2db998422\" (UID: \"b3a9d487-206d-4027-b563-81d2db998422\") " Oct 30 13:04:30.285762 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:30.285621 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3a9d487-206d-4027-b563-81d2db998422-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3a9d487-206d-4027-b563-81d2db998422-dshm\") pod \"b3a9d487-206d-4027-b563-81d2db998422\" (UID: \"b3a9d487-206d-4027-b563-81d2db998422\") " Oct 30 13:04:30.285762 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:30.285742 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3a9d487-206d-4027-b563-81d2db998422-kube-api-access-mf8jw\" (UniqueName: \"kubernetes.io/projected/b3a9d487-206d-4027-b563-81d2db998422-kube-api-access-mf8jw\") pod \"b3a9d487-206d-4027-b563-81d2db998422\" (UID: \"b3a9d487-206d-4027-b563-81d2db998422\") " Oct 30 13:04:30.285903 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:30.285787 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b3a9d487-206d-4027-b563-81d2db998422-tls-certs\" (UniqueName: \"kubernetes.io/secret/b3a9d487-206d-4027-b563-81d2db998422-tls-certs\") pod \"b3a9d487-206d-4027-b563-81d2db998422\" (UID: \"b3a9d487-206d-4027-b563-81d2db998422\") " Oct 30 13:04:30.285903 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:30.285850 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3a9d487-206d-4027-b563-81d2db998422-home" pod "b3a9d487-206d-4027-b563-81d2db998422" (UID: "b3a9d487-206d-4027-b563-81d2db998422"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:04:30.285903 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:30.285888 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3a9d487-206d-4027-b563-81d2db998422-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3a9d487-206d-4027-b563-81d2db998422-tmp-dir\") pod \"b3a9d487-206d-4027-b563-81d2db998422\" (UID: \"b3a9d487-206d-4027-b563-81d2db998422\") " Oct 30 13:04:30.286087 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:30.285952 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3a9d487-206d-4027-b563-81d2db998422-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3a9d487-206d-4027-b563-81d2db998422-model-cache\") pod \"b3a9d487-206d-4027-b563-81d2db998422\" (UID: \"b3a9d487-206d-4027-b563-81d2db998422\") " Oct 30 13:04:30.286735 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:30.286343 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3a9d487-206d-4027-b563-81d2db998422-tmp-dir" pod "b3a9d487-206d-4027-b563-81d2db998422" (UID: "b3a9d487-206d-4027-b563-81d2db998422"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:04:30.286735 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:30.286593 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3a9d487-206d-4027-b563-81d2db998422-model-cache" pod "b3a9d487-206d-4027-b563-81d2db998422" (UID: "b3a9d487-206d-4027-b563-81d2db998422"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:04:30.286735 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:30.286686 4908 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3a9d487-206d-4027-b563-81d2db998422-tmp-dir\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:04:30.286735 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:30.286710 4908 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3a9d487-206d-4027-b563-81d2db998422-home\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:04:30.288626 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:30.288597 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3a9d487-206d-4027-b563-81d2db998422-dshm" pod "b3a9d487-206d-4027-b563-81d2db998422" (UID: "b3a9d487-206d-4027-b563-81d2db998422"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:04:30.288723 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:30.288666 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3a9d487-206d-4027-b563-81d2db998422-tls-certs" pod "b3a9d487-206d-4027-b563-81d2db998422" (UID: "b3a9d487-206d-4027-b563-81d2db998422"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:04:30.289100 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:30.289076 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3a9d487-206d-4027-b563-81d2db998422-kube-api-access-mf8jw" pod "b3a9d487-206d-4027-b563-81d2db998422" (UID: "b3a9d487-206d-4027-b563-81d2db998422"). InnerVolumeSpecName "kube-api-access-mf8jw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:04:30.388438 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:30.388350 4908 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3a9d487-206d-4027-b563-81d2db998422-model-cache\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:04:30.388438 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:30.388380 4908 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3a9d487-206d-4027-b563-81d2db998422-dshm\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:04:30.388438 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:30.388390 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mf8jw\" (UniqueName: \"kubernetes.io/projected/b3a9d487-206d-4027-b563-81d2db998422-kube-api-access-mf8jw\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:04:30.388438 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:30.388402 4908 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3a9d487-206d-4027-b563-81d2db998422-tls-certs\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:04:30.622003 ip-10-0-141-251 kubenswrapper[4908]: E1030 13:04:30.621969 4908 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c19d2e0_b5e4_4e44_a2cf_6724f0f1e0b1.slice\": RecentStats: unable to find data in memory cache]" Oct 30 13:04:30.703478 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:30.703392 4908 generic.go:356] "Generic (PLEG): container finished" podID="b3a9d487-206d-4027-b563-81d2db998422" containerID="a8acc0048246a2aff4ebe1c30827792d5fb04c84f22516061f37988fc01081e5" exitCode=0 Oct 30 13:04:30.703478 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:30.703448 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-63f4e4/test-model-63f4e4-kserve-57bdcf4b79-cgctl" Oct 30 13:04:30.704086 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:30.703478 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-63f4e4/test-model-63f4e4-kserve-57bdcf4b79-cgctl" event={"ID":"b3a9d487-206d-4027-b563-81d2db998422","Type":"ContainerDied","Data":"a8acc0048246a2aff4ebe1c30827792d5fb04c84f22516061f37988fc01081e5"} Oct 30 13:04:30.704086 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:30.703528 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-63f4e4/test-model-63f4e4-kserve-57bdcf4b79-cgctl" event={"ID":"b3a9d487-206d-4027-b563-81d2db998422","Type":"ContainerDied","Data":"b483af6ce64f9a474b463379c4797455faf2aa0d31c2df42369c82e167b176ff"} Oct 30 13:04:30.704086 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:30.703550 4908 scope.go:122] "RemoveContainer" containerID="a8acc0048246a2aff4ebe1c30827792d5fb04c84f22516061f37988fc01081e5" Oct 30 13:04:30.716991 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:30.716966 4908 scope.go:122] "RemoveContainer" containerID="a8acc0048246a2aff4ebe1c30827792d5fb04c84f22516061f37988fc01081e5" Oct 30 13:04:30.717300 ip-10-0-141-251 kubenswrapper[4908]: E1030 13:04:30.717278 4908 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8acc0048246a2aff4ebe1c30827792d5fb04c84f22516061f37988fc01081e5\": container with ID starting with a8acc0048246a2aff4ebe1c30827792d5fb04c84f22516061f37988fc01081e5 not found: ID does not exist" containerID="a8acc0048246a2aff4ebe1c30827792d5fb04c84f22516061f37988fc01081e5" Oct 30 13:04:30.717380 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:30.717311 4908 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8acc0048246a2aff4ebe1c30827792d5fb04c84f22516061f37988fc01081e5"} err="failed to get container status \"a8acc0048246a2aff4ebe1c30827792d5fb04c84f22516061f37988fc01081e5\": rpc error: code = NotFound desc = could not find container \"a8acc0048246a2aff4ebe1c30827792d5fb04c84f22516061f37988fc01081e5\": container with ID starting with a8acc0048246a2aff4ebe1c30827792d5fb04c84f22516061f37988fc01081e5 not found: ID does not exist" Oct 30 13:04:30.755578 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:30.755510 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-63f4e4/test-model-63f4e4-kserve-57bdcf4b79-cgctl"] Oct 30 13:04:30.766713 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:30.766677 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-63f4e4/test-model-63f4e4-kserve-57bdcf4b79-cgctl"] Oct 30 13:04:30.860174 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:30.860141 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6b67f8-2g65x" Oct 30 13:04:30.894285 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:30.894253 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cabd4f34-efec-4a7e-89e7-b58692f3b37a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cabd4f34-efec-4a7e-89e7-b58692f3b37a-maas-api-openshift-ca\") pod \"cabd4f34-efec-4a7e-89e7-b58692f3b37a\" (UID: \"cabd4f34-efec-4a7e-89e7-b58692f3b37a\") " Oct 30 13:04:30.894489 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:30.894305 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cabd4f34-efec-4a7e-89e7-b58692f3b37a-kube-api-access-nrtzg\" (UniqueName: \"kubernetes.io/projected/cabd4f34-efec-4a7e-89e7-b58692f3b37a-kube-api-access-nrtzg\") pod \"cabd4f34-efec-4a7e-89e7-b58692f3b37a\" (UID: \"cabd4f34-efec-4a7e-89e7-b58692f3b37a\") " Oct 30 13:04:30.894489 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:30.894437 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cabd4f34-efec-4a7e-89e7-b58692f3b37a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cabd4f34-efec-4a7e-89e7-b58692f3b37a-maas-api-xks-ca\") pod \"cabd4f34-efec-4a7e-89e7-b58692f3b37a\" (UID: \"cabd4f34-efec-4a7e-89e7-b58692f3b37a\") " Oct 30 13:04:30.894692 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:30.894667 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cabd4f34-efec-4a7e-89e7-b58692f3b37a-maas-api-openshift-ca" pod "cabd4f34-efec-4a7e-89e7-b58692f3b37a" (UID: "cabd4f34-efec-4a7e-89e7-b58692f3b37a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:04:30.894978 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:30.894957 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cabd4f34-efec-4a7e-89e7-b58692f3b37a-maas-api-openshift-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:04:30.896555 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:30.896527 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cabd4f34-efec-4a7e-89e7-b58692f3b37a-kube-api-access-nrtzg" pod "cabd4f34-efec-4a7e-89e7-b58692f3b37a" (UID: "cabd4f34-efec-4a7e-89e7-b58692f3b37a"). InnerVolumeSpecName "kube-api-access-nrtzg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:04:30.896555 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:30.896528 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cabd4f34-efec-4a7e-89e7-b58692f3b37a-maas-api-xks-ca" pod "cabd4f34-efec-4a7e-89e7-b58692f3b37a" (UID: "cabd4f34-efec-4a7e-89e7-b58692f3b37a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:04:31.017862 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:31.005111 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cabd4f34-efec-4a7e-89e7-b58692f3b37a-maas-api-xks-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:04:31.017862 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:31.005246 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrtzg\" (UniqueName: \"kubernetes.io/projected/cabd4f34-efec-4a7e-89e7-b58692f3b37a-kube-api-access-nrtzg\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:04:31.620846 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:31.619890 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3a9d487-206d-4027-b563-81d2db998422" path="/var/lib/kubelet/pods/b3a9d487-206d-4027-b563-81d2db998422/volumes" Oct 30 13:04:31.714061 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:31.714027 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6b67f8-2g65x" Oct 30 13:04:31.714528 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:31.714054 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6b67f8-2g65x" event={"ID":"cabd4f34-efec-4a7e-89e7-b58692f3b37a","Type":"ContainerDied","Data":"066baad4df284a1030709604b61bfeaea1e20db0894abdad9e2baf3e8c3e02b0"} Oct 30 13:04:31.714528 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:31.714113 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="066baad4df284a1030709604b61bfeaea1e20db0894abdad9e2baf3e8c3e02b0" Oct 30 13:04:31.892868 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:31.892764 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-dbe123-mc4f4"] Oct 30 13:04:31.893480 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:31.893456 4908 cpu_manager.go:414] "containerMap: removing container" podUID="b3a9d487-206d-4027-b563-81d2db998422" containerName="main" Oct 30 13:04:31.893590 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:31.893481 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3a9d487-206d-4027-b563-81d2db998422" containerName="main" Oct 30 13:04:31.893590 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:31.893506 4908 cpu_manager.go:414] "containerMap: removing container" podUID="cabd4f34-efec-4a7e-89e7-b58692f3b37a" containerName="revoke-keys" Oct 30 13:04:31.893590 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:31.893514 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cabd4f34-efec-4a7e-89e7-b58692f3b37a" containerName="revoke-keys" Oct 30 13:04:31.893765 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:31.893682 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3a9d487-206d-4027-b563-81d2db998422" containerName="main" Oct 30 13:04:31.893765 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:31.893699 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="cabd4f34-efec-4a7e-89e7-b58692f3b37a" containerName="revoke-keys" Oct 30 13:04:31.898612 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:31.898588 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-dbe123-mc4f4" Oct 30 13:04:31.931281 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:31.930966 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:04:31.970856 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:31.965988 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 13:04:31.970856 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:31.967540 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 13:04:31.970856 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:31.968498 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s75hq\"" Oct 30 13:04:32.000317 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:32.000276 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-dbe123-mc4f4"] Oct 30 13:04:32.025349 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:32.025307 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdrww\" (UniqueName: \"kubernetes.io/projected/aedc62a9-914d-4e3c-a7fd-f1c0093de3cc-kube-api-access-fdrww\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-dbe123-mc4f4\" (UID: \"aedc62a9-914d-4e3c-a7fd-f1c0093de3cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-dbe123-mc4f4" Oct 30 13:04:32.025563 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:32.025375 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aedc62a9-914d-4e3c-a7fd-f1c0093de3cc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-dbe123-mc4f4\" (UID: \"aedc62a9-914d-4e3c-a7fd-f1c0093de3cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-dbe123-mc4f4" Oct 30 13:04:32.025563 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:32.025515 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aedc62a9-914d-4e3c-a7fd-f1c0093de3cc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-dbe123-mc4f4\" (UID: \"aedc62a9-914d-4e3c-a7fd-f1c0093de3cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-dbe123-mc4f4" Oct 30 13:04:32.127212 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:32.127160 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdrww\" (UniqueName: \"kubernetes.io/projected/aedc62a9-914d-4e3c-a7fd-f1c0093de3cc-kube-api-access-fdrww\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-dbe123-mc4f4\" (UID: \"aedc62a9-914d-4e3c-a7fd-f1c0093de3cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-dbe123-mc4f4" Oct 30 13:04:32.127435 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:32.127238 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aedc62a9-914d-4e3c-a7fd-f1c0093de3cc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-dbe123-mc4f4\" (UID: \"aedc62a9-914d-4e3c-a7fd-f1c0093de3cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-dbe123-mc4f4" Oct 30 13:04:32.127435 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:32.127297 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aedc62a9-914d-4e3c-a7fd-f1c0093de3cc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-dbe123-mc4f4\" (UID: \"aedc62a9-914d-4e3c-a7fd-f1c0093de3cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-dbe123-mc4f4" Oct 30 13:04:32.128077 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:32.128051 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aedc62a9-914d-4e3c-a7fd-f1c0093de3cc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-dbe123-mc4f4\" (UID: \"aedc62a9-914d-4e3c-a7fd-f1c0093de3cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-dbe123-mc4f4" Oct 30 13:04:32.129648 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:32.129628 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aedc62a9-914d-4e3c-a7fd-f1c0093de3cc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-dbe123-mc4f4\" (UID: \"aedc62a9-914d-4e3c-a7fd-f1c0093de3cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-dbe123-mc4f4" Oct 30 13:04:32.150215 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:32.150136 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdrww\" (UniqueName: \"kubernetes.io/projected/aedc62a9-914d-4e3c-a7fd-f1c0093de3cc-kube-api-access-fdrww\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-dbe123-mc4f4\" (UID: \"aedc62a9-914d-4e3c-a7fd-f1c0093de3cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-dbe123-mc4f4" Oct 30 13:04:32.224328 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:32.224294 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-dbe123-mc4f4" Oct 30 13:04:32.539452 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:32.539416 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-dbe123-mc4f4"] Oct 30 13:04:32.539882 ip-10-0-141-251 kubenswrapper[4908]: W1030 13:04:32.539848 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaedc62a9_914d_4e3c_a7fd_f1c0093de3cc.slice/crio-e11d9fd6c5cde99936f0698a41f8afbbe4c434fa8ee7ef6ab8dbe5966e82594f WatchSource:0}: Error finding container e11d9fd6c5cde99936f0698a41f8afbbe4c434fa8ee7ef6ab8dbe5966e82594f: Status 404 returned error can't find the container with id e11d9fd6c5cde99936f0698a41f8afbbe4c434fa8ee7ef6ab8dbe5966e82594f Oct 30 13:04:32.724657 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:32.724619 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-dbe123-mc4f4" event={"ID":"aedc62a9-914d-4e3c-a7fd-f1c0093de3cc","Type":"ContainerStarted","Data":"e11d9fd6c5cde99936f0698a41f8afbbe4c434fa8ee7ef6ab8dbe5966e82594f"} Oct 30 13:04:33.780552 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:33.779840 4908 generic.go:356] "Generic (PLEG): container finished" podID="aedc62a9-914d-4e3c-a7fd-f1c0093de3cc" containerID="41b3145b5de9e0c9d132302a7d51d2d735c7b58057ddabdccc6786257af5dea1" exitCode=0 Oct 30 13:04:33.780552 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:33.780144 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-dbe123-mc4f4" event={"ID":"aedc62a9-914d-4e3c-a7fd-f1c0093de3cc","Type":"ContainerDied","Data":"41b3145b5de9e0c9d132302a7d51d2d735c7b58057ddabdccc6786257af5dea1"} Oct 30 13:04:34.196776 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:34.196691 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6ad1da-8rb5r"] Oct 30 13:04:34.212012 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:34.211688 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6ad1da-8rb5r"] Oct 30 13:04:34.960986 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:34.960949 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-dbe123-mc4f4" Oct 30 13:04:34.992980 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:34.992937 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aedc62a9-914d-4e3c-a7fd-f1c0093de3cc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aedc62a9-914d-4e3c-a7fd-f1c0093de3cc-maas-api-xks-ca\") pod \"aedc62a9-914d-4e3c-a7fd-f1c0093de3cc\" (UID: \"aedc62a9-914d-4e3c-a7fd-f1c0093de3cc\") " Oct 30 13:04:34.993205 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:34.992995 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aedc62a9-914d-4e3c-a7fd-f1c0093de3cc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aedc62a9-914d-4e3c-a7fd-f1c0093de3cc-maas-api-openshift-ca\") pod \"aedc62a9-914d-4e3c-a7fd-f1c0093de3cc\" (UID: \"aedc62a9-914d-4e3c-a7fd-f1c0093de3cc\") " Oct 30 13:04:34.993205 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:34.993180 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aedc62a9-914d-4e3c-a7fd-f1c0093de3cc-kube-api-access-fdrww\" (UniqueName: \"kubernetes.io/projected/aedc62a9-914d-4e3c-a7fd-f1c0093de3cc-kube-api-access-fdrww\") pod \"aedc62a9-914d-4e3c-a7fd-f1c0093de3cc\" (UID: \"aedc62a9-914d-4e3c-a7fd-f1c0093de3cc\") " Oct 30 13:04:34.993397 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:34.993373 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aedc62a9-914d-4e3c-a7fd-f1c0093de3cc-maas-api-openshift-ca" pod "aedc62a9-914d-4e3c-a7fd-f1c0093de3cc" (UID: "aedc62a9-914d-4e3c-a7fd-f1c0093de3cc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:04:34.993762 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:34.993737 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aedc62a9-914d-4e3c-a7fd-f1c0093de3cc-maas-api-openshift-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:04:34.995335 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:34.995310 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aedc62a9-914d-4e3c-a7fd-f1c0093de3cc-kube-api-access-fdrww" pod "aedc62a9-914d-4e3c-a7fd-f1c0093de3cc" (UID: "aedc62a9-914d-4e3c-a7fd-f1c0093de3cc"). InnerVolumeSpecName "kube-api-access-fdrww". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:04:34.995451 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:34.995354 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aedc62a9-914d-4e3c-a7fd-f1c0093de3cc-maas-api-xks-ca" pod "aedc62a9-914d-4e3c-a7fd-f1c0093de3cc" (UID: "aedc62a9-914d-4e3c-a7fd-f1c0093de3cc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:04:35.095641 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:35.095537 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aedc62a9-914d-4e3c-a7fd-f1c0093de3cc-maas-api-xks-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:04:35.095641 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:35.095576 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdrww\" (UniqueName: \"kubernetes.io/projected/aedc62a9-914d-4e3c-a7fd-f1c0093de3cc-kube-api-access-fdrww\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:04:35.609152 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:35.609113 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2124dd77-f0aa-4e09-87a8-1a0e62b17cde" path="/var/lib/kubelet/pods/2124dd77-f0aa-4e09-87a8-1a0e62b17cde/volumes" Oct 30 13:04:35.801087 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:35.801053 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-dbe123-mc4f4" Oct 30 13:04:35.801087 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:35.801069 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-dbe123-mc4f4" event={"ID":"aedc62a9-914d-4e3c-a7fd-f1c0093de3cc","Type":"ContainerDied","Data":"e11d9fd6c5cde99936f0698a41f8afbbe4c434fa8ee7ef6ab8dbe5966e82594f"} Oct 30 13:04:35.801384 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:35.801134 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e11d9fd6c5cde99936f0698a41f8afbbe4c434fa8ee7ef6ab8dbe5966e82594f" Oct 30 13:04:37.577011 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:37.576968 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-734ffe/test-model-734ffe-kserve-7c69bdf48f-46tft"] Oct 30 13:04:37.898985 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:37.898169 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-63f4e4-nm6f4"] Oct 30 13:04:37.902886 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:37.899232 4908 cpu_manager.go:414] "containerMap: removing container" podUID="aedc62a9-914d-4e3c-a7fd-f1c0093de3cc" containerName="revoke-keys" Oct 30 13:04:37.902886 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:37.899262 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aedc62a9-914d-4e3c-a7fd-f1c0093de3cc" containerName="revoke-keys" Oct 30 13:04:37.902886 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:37.899461 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="aedc62a9-914d-4e3c-a7fd-f1c0093de3cc" containerName="revoke-keys" Oct 30 13:04:37.914509 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:37.909029 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-63f4e4-nm6f4" Oct 30 13:04:37.922851 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:37.919402 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-63f4e4-nm6f4"] Oct 30 13:04:37.938575 ip-10-0-141-251 kubenswrapper[4908]: E1030 13:04:37.938542 4908 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c19d2e0_b5e4_4e44_a2cf_6724f0f1e0b1.slice\": RecentStats: unable to find data in memory cache]" Oct 30 13:04:37.941221 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:37.941197 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:04:37.941360 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:37.941337 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 13:04:37.941425 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:37.941384 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 13:04:37.941843 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:37.941812 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s75hq\"" Oct 30 13:04:38.046374 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:38.046333 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrtjc\" (UniqueName: \"kubernetes.io/projected/9801b39b-9af9-4004-ae48-3025ca5bde8f-kube-api-access-jrtjc\") pod \"maas-api-revoke-keys-e2e-inf-a-63f4e4-nm6f4\" (UID: \"9801b39b-9af9-4004-ae48-3025ca5bde8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-63f4e4-nm6f4" Oct 30 13:04:38.046597 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:38.046490 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9801b39b-9af9-4004-ae48-3025ca5bde8f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-63f4e4-nm6f4\" (UID: \"9801b39b-9af9-4004-ae48-3025ca5bde8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-63f4e4-nm6f4" Oct 30 13:04:38.046597 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:38.046579 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9801b39b-9af9-4004-ae48-3025ca5bde8f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-63f4e4-nm6f4\" (UID: \"9801b39b-9af9-4004-ae48-3025ca5bde8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-63f4e4-nm6f4" Oct 30 13:04:38.148185 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:38.148142 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9801b39b-9af9-4004-ae48-3025ca5bde8f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-63f4e4-nm6f4\" (UID: \"9801b39b-9af9-4004-ae48-3025ca5bde8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-63f4e4-nm6f4" Oct 30 13:04:38.148382 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:38.148214 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9801b39b-9af9-4004-ae48-3025ca5bde8f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-63f4e4-nm6f4\" (UID: \"9801b39b-9af9-4004-ae48-3025ca5bde8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-63f4e4-nm6f4" Oct 30 13:04:38.148435 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:38.148386 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrtjc\" (UniqueName: \"kubernetes.io/projected/9801b39b-9af9-4004-ae48-3025ca5bde8f-kube-api-access-jrtjc\") pod \"maas-api-revoke-keys-e2e-inf-a-63f4e4-nm6f4\" (UID: \"9801b39b-9af9-4004-ae48-3025ca5bde8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-63f4e4-nm6f4" Oct 30 13:04:38.148966 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:38.148947 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9801b39b-9af9-4004-ae48-3025ca5bde8f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-63f4e4-nm6f4\" (UID: \"9801b39b-9af9-4004-ae48-3025ca5bde8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-63f4e4-nm6f4" Oct 30 13:04:38.150752 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:38.150702 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9801b39b-9af9-4004-ae48-3025ca5bde8f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-63f4e4-nm6f4\" (UID: \"9801b39b-9af9-4004-ae48-3025ca5bde8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-63f4e4-nm6f4" Oct 30 13:04:38.176303 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:38.176271 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrtjc\" (UniqueName: \"kubernetes.io/projected/9801b39b-9af9-4004-ae48-3025ca5bde8f-kube-api-access-jrtjc\") pod \"maas-api-revoke-keys-e2e-inf-a-63f4e4-nm6f4\" (UID: \"9801b39b-9af9-4004-ae48-3025ca5bde8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-63f4e4-nm6f4" Oct 30 13:04:38.253190 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:38.253157 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-63f4e4-nm6f4" Oct 30 13:04:38.571270 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:38.571234 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-63f4e4-nm6f4"] Oct 30 13:04:38.573215 ip-10-0-141-251 kubenswrapper[4908]: W1030 13:04:38.573178 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9801b39b_9af9_4004_ae48_3025ca5bde8f.slice/crio-6b09b0cd4b529ca62901328f6d0701b6540dee8a94bfa213b1d3922b10f99765 WatchSource:0}: Error finding container 6b09b0cd4b529ca62901328f6d0701b6540dee8a94bfa213b1d3922b10f99765: Status 404 returned error can't find the container with id 6b09b0cd4b529ca62901328f6d0701b6540dee8a94bfa213b1d3922b10f99765 Oct 30 13:04:38.897522 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:38.897410 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-63f4e4-nm6f4" event={"ID":"9801b39b-9af9-4004-ae48-3025ca5bde8f","Type":"ContainerStarted","Data":"6b09b0cd4b529ca62901328f6d0701b6540dee8a94bfa213b1d3922b10f99765"} Oct 30 13:04:39.945723 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:39.945631 4908 generic.go:356] "Generic (PLEG): container finished" podID="9801b39b-9af9-4004-ae48-3025ca5bde8f" containerID="49290d1120a90a4a4b5601151b635d54bdf429165045f50edf373a412c32d10b" exitCode=0 Oct 30 13:04:39.945723 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:39.945689 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-63f4e4-nm6f4" event={"ID":"9801b39b-9af9-4004-ae48-3025ca5bde8f","Type":"ContainerDied","Data":"49290d1120a90a4a4b5601151b635d54bdf429165045f50edf373a412c32d10b"} Oct 30 13:04:40.637584 ip-10-0-141-251 kubenswrapper[4908]: E1030 13:04:40.637546 4908 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c19d2e0_b5e4_4e44_a2cf_6724f0f1e0b1.slice\": RecentStats: unable to find data in memory cache]" Oct 30 13:04:41.133517 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:41.133493 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-63f4e4-nm6f4" Oct 30 13:04:41.205766 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:41.204811 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9801b39b-9af9-4004-ae48-3025ca5bde8f-kube-api-access-jrtjc\" (UniqueName: \"kubernetes.io/projected/9801b39b-9af9-4004-ae48-3025ca5bde8f-kube-api-access-jrtjc\") pod \"9801b39b-9af9-4004-ae48-3025ca5bde8f\" (UID: \"9801b39b-9af9-4004-ae48-3025ca5bde8f\") " Oct 30 13:04:41.205766 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:41.204999 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9801b39b-9af9-4004-ae48-3025ca5bde8f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9801b39b-9af9-4004-ae48-3025ca5bde8f-maas-api-openshift-ca\") pod \"9801b39b-9af9-4004-ae48-3025ca5bde8f\" (UID: \"9801b39b-9af9-4004-ae48-3025ca5bde8f\") " Oct 30 13:04:41.205766 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:41.205039 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9801b39b-9af9-4004-ae48-3025ca5bde8f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9801b39b-9af9-4004-ae48-3025ca5bde8f-maas-api-xks-ca\") pod \"9801b39b-9af9-4004-ae48-3025ca5bde8f\" (UID: \"9801b39b-9af9-4004-ae48-3025ca5bde8f\") " Oct 30 13:04:41.205766 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:41.205476 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9801b39b-9af9-4004-ae48-3025ca5bde8f-maas-api-openshift-ca" pod "9801b39b-9af9-4004-ae48-3025ca5bde8f" (UID: "9801b39b-9af9-4004-ae48-3025ca5bde8f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:04:41.205766 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:41.205737 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9801b39b-9af9-4004-ae48-3025ca5bde8f-maas-api-openshift-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:04:41.209080 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:41.209040 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9801b39b-9af9-4004-ae48-3025ca5bde8f-maas-api-xks-ca" pod "9801b39b-9af9-4004-ae48-3025ca5bde8f" (UID: "9801b39b-9af9-4004-ae48-3025ca5bde8f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:04:41.210371 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:41.210342 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9801b39b-9af9-4004-ae48-3025ca5bde8f-kube-api-access-jrtjc" pod "9801b39b-9af9-4004-ae48-3025ca5bde8f" (UID: "9801b39b-9af9-4004-ae48-3025ca5bde8f"). InnerVolumeSpecName "kube-api-access-jrtjc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:04:41.306960 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:41.306923 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrtjc\" (UniqueName: \"kubernetes.io/projected/9801b39b-9af9-4004-ae48-3025ca5bde8f-kube-api-access-jrtjc\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:04:41.306960 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:41.306957 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9801b39b-9af9-4004-ae48-3025ca5bde8f-maas-api-xks-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:04:42.005978 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:42.005947 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-63f4e4-nm6f4" Oct 30 13:04:42.011567 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:42.011450 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-63f4e4-nm6f4" event={"ID":"9801b39b-9af9-4004-ae48-3025ca5bde8f","Type":"ContainerDied","Data":"6b09b0cd4b529ca62901328f6d0701b6540dee8a94bfa213b1d3922b10f99765"} Oct 30 13:04:42.011567 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:42.011525 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6b09b0cd4b529ca62901328f6d0701b6540dee8a94bfa213b1d3922b10f99765" Oct 30 13:04:42.306141 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:42.306058 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0de4b9-zfpt4"] Oct 30 13:04:42.306643 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:42.306628 4908 cpu_manager.go:414] "containerMap: removing container" podUID="9801b39b-9af9-4004-ae48-3025ca5bde8f" containerName="revoke-keys" Oct 30 13:04:42.306701 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:42.306644 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9801b39b-9af9-4004-ae48-3025ca5bde8f" containerName="revoke-keys" Oct 30 13:04:42.306764 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:42.306755 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="9801b39b-9af9-4004-ae48-3025ca5bde8f" containerName="revoke-keys" Oct 30 13:04:42.310684 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:42.310661 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0de4b9-zfpt4" Oct 30 13:04:42.328025 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:42.328003 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 13:04:42.332037 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:42.332005 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0de4b9-zfpt4"] Oct 30 13:04:42.338109 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:42.338067 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s75hq\"" Oct 30 13:04:42.338109 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:42.338070 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:04:42.338308 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:42.338097 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 13:04:42.433746 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:42.433707 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cca21fd0-97fa-42cc-b7d9-a514773ecd97-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-0de4b9-zfpt4\" (UID: \"cca21fd0-97fa-42cc-b7d9-a514773ecd97\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0de4b9-zfpt4" Oct 30 13:04:42.433954 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:42.433778 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cca21fd0-97fa-42cc-b7d9-a514773ecd97-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-0de4b9-zfpt4\" (UID: \"cca21fd0-97fa-42cc-b7d9-a514773ecd97\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0de4b9-zfpt4" Oct 30 13:04:42.433954 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:42.433872 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ssmbt\" (UniqueName: \"kubernetes.io/projected/cca21fd0-97fa-42cc-b7d9-a514773ecd97-kube-api-access-ssmbt\") pod \"maas-api-revoke-keys-e2e-extmod-b-0de4b9-zfpt4\" (UID: \"cca21fd0-97fa-42cc-b7d9-a514773ecd97\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0de4b9-zfpt4" Oct 30 13:04:42.535683 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:42.535634 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cca21fd0-97fa-42cc-b7d9-a514773ecd97-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-0de4b9-zfpt4\" (UID: \"cca21fd0-97fa-42cc-b7d9-a514773ecd97\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0de4b9-zfpt4" Oct 30 13:04:42.535892 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:42.535707 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cca21fd0-97fa-42cc-b7d9-a514773ecd97-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-0de4b9-zfpt4\" (UID: \"cca21fd0-97fa-42cc-b7d9-a514773ecd97\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0de4b9-zfpt4" Oct 30 13:04:42.535892 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:42.535732 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ssmbt\" (UniqueName: \"kubernetes.io/projected/cca21fd0-97fa-42cc-b7d9-a514773ecd97-kube-api-access-ssmbt\") pod \"maas-api-revoke-keys-e2e-extmod-b-0de4b9-zfpt4\" (UID: \"cca21fd0-97fa-42cc-b7d9-a514773ecd97\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0de4b9-zfpt4" Oct 30 13:04:42.536464 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:42.536438 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cca21fd0-97fa-42cc-b7d9-a514773ecd97-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-0de4b9-zfpt4\" (UID: \"cca21fd0-97fa-42cc-b7d9-a514773ecd97\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0de4b9-zfpt4" Oct 30 13:04:42.538243 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:42.538219 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cca21fd0-97fa-42cc-b7d9-a514773ecd97-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-0de4b9-zfpt4\" (UID: \"cca21fd0-97fa-42cc-b7d9-a514773ecd97\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0de4b9-zfpt4" Oct 30 13:04:42.561707 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:42.561636 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssmbt\" (UniqueName: \"kubernetes.io/projected/cca21fd0-97fa-42cc-b7d9-a514773ecd97-kube-api-access-ssmbt\") pod \"maas-api-revoke-keys-e2e-extmod-b-0de4b9-zfpt4\" (UID: \"cca21fd0-97fa-42cc-b7d9-a514773ecd97\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0de4b9-zfpt4" Oct 30 13:04:42.624559 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:42.624518 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0de4b9-zfpt4" Oct 30 13:04:42.940749 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:42.940642 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0de4b9-zfpt4"] Oct 30 13:04:43.021780 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:43.021738 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0de4b9-zfpt4" event={"ID":"cca21fd0-97fa-42cc-b7d9-a514773ecd97","Type":"ContainerStarted","Data":"16987ad6f9a6648860eb693008c76b0d4fa2a5924bb604486f7b2821f7b98d07"} Oct 30 13:04:44.031757 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:44.031707 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0de4b9-zfpt4" event={"ID":"cca21fd0-97fa-42cc-b7d9-a514773ecd97","Type":"ContainerStarted","Data":"e9a59a42b0cd5577dae6f112d3a0cc652ae344de6686411f601d1bb1ef0919b2"} Oct 30 13:04:44.072189 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:44.072127 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0de4b9-zfpt4" podStartSLOduration=2.072109922 podStartE2EDuration="2.072109922s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:04:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:04:44.070650018 +0000 UTC m=+4240.585078765" watchObservedRunningTime="2026-10-30 13:04:44.072109922 +0000 UTC m=+4240.586538668" Oct 30 13:04:45.045547 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:45.045503 4908 generic.go:356] "Generic (PLEG): container finished" podID="cca21fd0-97fa-42cc-b7d9-a514773ecd97" containerID="e9a59a42b0cd5577dae6f112d3a0cc652ae344de6686411f601d1bb1ef0919b2" exitCode=0 Oct 30 13:04:45.046138 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:45.045592 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0de4b9-zfpt4" event={"ID":"cca21fd0-97fa-42cc-b7d9-a514773ecd97","Type":"ContainerDied","Data":"e9a59a42b0cd5577dae6f112d3a0cc652ae344de6686411f601d1bb1ef0919b2"} Oct 30 13:04:45.686185 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:45.686141 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-dbe123-mc4f4"] Oct 30 13:04:45.696223 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:45.696187 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-dbe123-mc4f4"] Oct 30 13:04:46.230417 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:46.230386 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0de4b9-zfpt4" Oct 30 13:04:46.263410 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:46.263365 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cca21fd0-97fa-42cc-b7d9-a514773ecd97-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cca21fd0-97fa-42cc-b7d9-a514773ecd97-maas-api-openshift-ca\") pod \"cca21fd0-97fa-42cc-b7d9-a514773ecd97\" (UID: \"cca21fd0-97fa-42cc-b7d9-a514773ecd97\") " Oct 30 13:04:46.263598 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:46.263536 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cca21fd0-97fa-42cc-b7d9-a514773ecd97-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cca21fd0-97fa-42cc-b7d9-a514773ecd97-maas-api-xks-ca\") pod \"cca21fd0-97fa-42cc-b7d9-a514773ecd97\" (UID: \"cca21fd0-97fa-42cc-b7d9-a514773ecd97\") " Oct 30 13:04:46.264216 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:46.263730 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cca21fd0-97fa-42cc-b7d9-a514773ecd97-kube-api-access-ssmbt\" (UniqueName: \"kubernetes.io/projected/cca21fd0-97fa-42cc-b7d9-a514773ecd97-kube-api-access-ssmbt\") pod \"cca21fd0-97fa-42cc-b7d9-a514773ecd97\" (UID: \"cca21fd0-97fa-42cc-b7d9-a514773ecd97\") " Oct 30 13:04:46.264216 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:46.263786 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cca21fd0-97fa-42cc-b7d9-a514773ecd97-maas-api-openshift-ca" pod "cca21fd0-97fa-42cc-b7d9-a514773ecd97" (UID: "cca21fd0-97fa-42cc-b7d9-a514773ecd97"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:04:46.264216 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:46.264191 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cca21fd0-97fa-42cc-b7d9-a514773ecd97-maas-api-openshift-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:04:46.266488 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:46.266270 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cca21fd0-97fa-42cc-b7d9-a514773ecd97-maas-api-xks-ca" pod "cca21fd0-97fa-42cc-b7d9-a514773ecd97" (UID: "cca21fd0-97fa-42cc-b7d9-a514773ecd97"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:04:46.266673 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:46.266485 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cca21fd0-97fa-42cc-b7d9-a514773ecd97-kube-api-access-ssmbt" pod "cca21fd0-97fa-42cc-b7d9-a514773ecd97" (UID: "cca21fd0-97fa-42cc-b7d9-a514773ecd97"). InnerVolumeSpecName "kube-api-access-ssmbt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:04:46.368076 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:46.367979 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cca21fd0-97fa-42cc-b7d9-a514773ecd97-maas-api-xks-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:04:46.368076 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:46.368041 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ssmbt\" (UniqueName: \"kubernetes.io/projected/cca21fd0-97fa-42cc-b7d9-a514773ecd97-kube-api-access-ssmbt\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:04:47.068276 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:47.068231 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0de4b9-zfpt4" event={"ID":"cca21fd0-97fa-42cc-b7d9-a514773ecd97","Type":"ContainerDied","Data":"16987ad6f9a6648860eb693008c76b0d4fa2a5924bb604486f7b2821f7b98d07"} Oct 30 13:04:47.068610 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:47.068569 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="16987ad6f9a6648860eb693008c76b0d4fa2a5924bb604486f7b2821f7b98d07" Oct 30 13:04:47.068610 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:47.068368 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0de4b9-zfpt4" Oct 30 13:04:47.627584 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:47.627546 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aedc62a9-914d-4e3c-a7fd-f1c0093de3cc" path="/var/lib/kubelet/pods/aedc62a9-914d-4e3c-a7fd-f1c0093de3cc/volumes" Oct 30 13:04:49.686093 ip-10-0-141-251 kubenswrapper[4908]: E1030 13:04:49.686048 4908 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vn6vv" containerName="extractor" Oct 30 13:04:50.712571 ip-10-0-141-251 kubenswrapper[4908]: E1030 13:04:50.712530 4908 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c19d2e0_b5e4_4e44_a2cf_6724f0f1e0b1.slice\": RecentStats: unable to find data in memory cache]" Oct 30 13:04:52.608680 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:52.608638 4908 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-734ffe/test-model-734ffe-kserve-7c69bdf48f-46tft" podUID="56318ef3-7e27-4b11-9344-46778652e73a" containerName="main" containerID="cri-o://8789e596d04f46b5ff698a60c60b6545de34538a8f7a2a76047f67b430ed3f4e" gracePeriod=45 Oct 30 13:04:52.782731 ip-10-0-141-251 kubenswrapper[4908]: E1030 13:04:52.782697 4908 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c19d2e0_b5e4_4e44_a2cf_6724f0f1e0b1.slice\": RecentStats: unable to find data in memory cache]" Oct 30 13:04:52.930748 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:52.930720 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-734ffe/test-model-734ffe-kserve-7c69bdf48f-46tft" Oct 30 13:04:52.991195 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:52.991153 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/56318ef3-7e27-4b11-9344-46778652e73a-tls-certs\" (UniqueName: \"kubernetes.io/secret/56318ef3-7e27-4b11-9344-46778652e73a-tls-certs\") pod \"56318ef3-7e27-4b11-9344-46778652e73a\" (UID: \"56318ef3-7e27-4b11-9344-46778652e73a\") " Oct 30 13:04:52.991493 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:52.991463 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56318ef3-7e27-4b11-9344-46778652e73a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56318ef3-7e27-4b11-9344-46778652e73a-model-cache\") pod \"56318ef3-7e27-4b11-9344-46778652e73a\" (UID: \"56318ef3-7e27-4b11-9344-46778652e73a\") " Oct 30 13:04:52.991581 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:52.991534 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56318ef3-7e27-4b11-9344-46778652e73a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56318ef3-7e27-4b11-9344-46778652e73a-tmp-dir\") pod \"56318ef3-7e27-4b11-9344-46778652e73a\" (UID: \"56318ef3-7e27-4b11-9344-46778652e73a\") " Oct 30 13:04:52.991639 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:52.991614 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56318ef3-7e27-4b11-9344-46778652e73a-home\" (UniqueName: \"kubernetes.io/empty-dir/56318ef3-7e27-4b11-9344-46778652e73a-home\") pod \"56318ef3-7e27-4b11-9344-46778652e73a\" (UID: \"56318ef3-7e27-4b11-9344-46778652e73a\") " Oct 30 13:04:52.991703 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:52.991643 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56318ef3-7e27-4b11-9344-46778652e73a-kube-api-access-6g5nf\" (UniqueName: \"kubernetes.io/projected/56318ef3-7e27-4b11-9344-46778652e73a-kube-api-access-6g5nf\") pod \"56318ef3-7e27-4b11-9344-46778652e73a\" (UID: \"56318ef3-7e27-4b11-9344-46778652e73a\") " Oct 30 13:04:52.991760 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:52.991737 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56318ef3-7e27-4b11-9344-46778652e73a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/56318ef3-7e27-4b11-9344-46778652e73a-dshm\") pod \"56318ef3-7e27-4b11-9344-46778652e73a\" (UID: \"56318ef3-7e27-4b11-9344-46778652e73a\") " Oct 30 13:04:53.015123 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:53.001623 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56318ef3-7e27-4b11-9344-46778652e73a-model-cache" pod "56318ef3-7e27-4b11-9344-46778652e73a" (UID: "56318ef3-7e27-4b11-9344-46778652e73a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:04:53.015123 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:53.001903 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56318ef3-7e27-4b11-9344-46778652e73a-tmp-dir" pod "56318ef3-7e27-4b11-9344-46778652e73a" (UID: "56318ef3-7e27-4b11-9344-46778652e73a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:04:53.015123 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:53.002114 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56318ef3-7e27-4b11-9344-46778652e73a-home" pod "56318ef3-7e27-4b11-9344-46778652e73a" (UID: "56318ef3-7e27-4b11-9344-46778652e73a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:04:53.015123 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:53.003000 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56318ef3-7e27-4b11-9344-46778652e73a-dshm" pod "56318ef3-7e27-4b11-9344-46778652e73a" (UID: "56318ef3-7e27-4b11-9344-46778652e73a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:04:53.015123 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:53.003182 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56318ef3-7e27-4b11-9344-46778652e73a-tls-certs" pod "56318ef3-7e27-4b11-9344-46778652e73a" (UID: "56318ef3-7e27-4b11-9344-46778652e73a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:04:53.019086 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:53.019046 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56318ef3-7e27-4b11-9344-46778652e73a-kube-api-access-6g5nf" pod "56318ef3-7e27-4b11-9344-46778652e73a" (UID: "56318ef3-7e27-4b11-9344-46778652e73a"). InnerVolumeSpecName "kube-api-access-6g5nf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:04:53.097649 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:53.097613 4908 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56318ef3-7e27-4b11-9344-46778652e73a-model-cache\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:04:53.097649 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:53.097646 4908 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56318ef3-7e27-4b11-9344-46778652e73a-tmp-dir\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:04:53.097649 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:53.097657 4908 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56318ef3-7e27-4b11-9344-46778652e73a-home\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:04:53.098063 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:53.097669 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6g5nf\" (UniqueName: \"kubernetes.io/projected/56318ef3-7e27-4b11-9344-46778652e73a-kube-api-access-6g5nf\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:04:53.098063 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:53.097681 4908 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56318ef3-7e27-4b11-9344-46778652e73a-dshm\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:04:53.098063 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:53.097695 4908 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56318ef3-7e27-4b11-9344-46778652e73a-tls-certs\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:04:53.124394 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:53.124356 4908 generic.go:356] "Generic (PLEG): container finished" podID="56318ef3-7e27-4b11-9344-46778652e73a" containerID="8789e596d04f46b5ff698a60c60b6545de34538a8f7a2a76047f67b430ed3f4e" exitCode=0 Oct 30 13:04:53.124595 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:53.124410 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-734ffe/test-model-734ffe-kserve-7c69bdf48f-46tft" Oct 30 13:04:53.124595 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:53.124463 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-734ffe/test-model-734ffe-kserve-7c69bdf48f-46tft" event={"ID":"56318ef3-7e27-4b11-9344-46778652e73a","Type":"ContainerDied","Data":"8789e596d04f46b5ff698a60c60b6545de34538a8f7a2a76047f67b430ed3f4e"} Oct 30 13:04:53.124595 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:53.124497 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-734ffe/test-model-734ffe-kserve-7c69bdf48f-46tft" event={"ID":"56318ef3-7e27-4b11-9344-46778652e73a","Type":"ContainerDied","Data":"3035f38cbe79aac798f375efaba687b92371c72ccca2b69ead71499ba621935c"} Oct 30 13:04:53.124595 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:53.124519 4908 scope.go:122] "RemoveContainer" containerID="8789e596d04f46b5ff698a60c60b6545de34538a8f7a2a76047f67b430ed3f4e" Oct 30 13:04:53.141218 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:53.141187 4908 scope.go:122] "RemoveContainer" containerID="8789e596d04f46b5ff698a60c60b6545de34538a8f7a2a76047f67b430ed3f4e" Oct 30 13:04:53.141580 ip-10-0-141-251 kubenswrapper[4908]: E1030 13:04:53.141551 4908 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8789e596d04f46b5ff698a60c60b6545de34538a8f7a2a76047f67b430ed3f4e\": container with ID starting with 8789e596d04f46b5ff698a60c60b6545de34538a8f7a2a76047f67b430ed3f4e not found: ID does not exist" containerID="8789e596d04f46b5ff698a60c60b6545de34538a8f7a2a76047f67b430ed3f4e" Oct 30 13:04:53.141695 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:53.141593 4908 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8789e596d04f46b5ff698a60c60b6545de34538a8f7a2a76047f67b430ed3f4e"} err="failed to get container status \"8789e596d04f46b5ff698a60c60b6545de34538a8f7a2a76047f67b430ed3f4e\": rpc error: code = NotFound desc = could not find container \"8789e596d04f46b5ff698a60c60b6545de34538a8f7a2a76047f67b430ed3f4e\": container with ID starting with 8789e596d04f46b5ff698a60c60b6545de34538a8f7a2a76047f67b430ed3f4e not found: ID does not exist" Oct 30 13:04:53.215899 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:53.211909 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-nntkd"] Oct 30 13:04:53.215899 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:53.212102 4908 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-nntkd" podUID="e6810256-6b25-44b9-a7f1-fb0cb023a5b9" containerName="istio-proxy" containerID="cri-o://61e639a16ad650e85438962111c2da99430d9e83ce51902b2bed8c1b8d716516" gracePeriod=30 Oct 30 13:04:53.215899 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:53.214760 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-734ffe/test-model-734ffe-kserve-7c69bdf48f-46tft"] Oct 30 13:04:53.220951 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:53.220926 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-734ffe/test-model-734ffe-kserve-7c69bdf48f-46tft"] Oct 30 13:04:53.643106 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:53.643044 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56318ef3-7e27-4b11-9344-46778652e73a" path="/var/lib/kubelet/pods/56318ef3-7e27-4b11-9344-46778652e73a/volumes" Oct 30 13:04:53.896222 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:53.896124 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c1b1e8-tdxb5"] Oct 30 13:04:53.896809 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:53.896785 4908 cpu_manager.go:414] "containerMap: removing container" podUID="56318ef3-7e27-4b11-9344-46778652e73a" containerName="main" Oct 30 13:04:53.896957 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:53.896811 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56318ef3-7e27-4b11-9344-46778652e73a" containerName="main" Oct 30 13:04:53.896957 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:53.896861 4908 cpu_manager.go:414] "containerMap: removing container" podUID="cca21fd0-97fa-42cc-b7d9-a514773ecd97" containerName="revoke-keys" Oct 30 13:04:53.896957 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:53.896869 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cca21fd0-97fa-42cc-b7d9-a514773ecd97" containerName="revoke-keys" Oct 30 13:04:53.897126 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:53.897052 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="56318ef3-7e27-4b11-9344-46778652e73a" containerName="main" Oct 30 13:04:53.897126 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:53.897074 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="cca21fd0-97fa-42cc-b7d9-a514773ecd97" containerName="revoke-keys" Oct 30 13:04:53.902122 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:53.902097 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c1b1e8-tdxb5" Oct 30 13:04:53.919527 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:53.919485 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c1b1e8-tdxb5"] Oct 30 13:04:53.925592 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:53.925562 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 13:04:53.925767 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:53.925608 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 13:04:53.959127 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:53.959055 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s75hq\"" Oct 30 13:04:53.959127 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:53.959076 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:04:54.027061 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:54.027025 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92dbbda1-816b-4692-8b8f-85441a6609d0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-c1b1e8-tdxb5\" (UID: \"92dbbda1-816b-4692-8b8f-85441a6609d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c1b1e8-tdxb5" Oct 30 13:04:54.027311 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:54.027228 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92dbbda1-816b-4692-8b8f-85441a6609d0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-c1b1e8-tdxb5\" (UID: \"92dbbda1-816b-4692-8b8f-85441a6609d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c1b1e8-tdxb5" Oct 30 13:04:54.027311 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:54.027266 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kn4ft\" (UniqueName: \"kubernetes.io/projected/92dbbda1-816b-4692-8b8f-85441a6609d0-kube-api-access-kn4ft\") pod \"maas-api-revoke-keys-e2e-worker-w1-c1b1e8-tdxb5\" (UID: \"92dbbda1-816b-4692-8b8f-85441a6609d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c1b1e8-tdxb5" Oct 30 13:04:54.129072 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:54.129026 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92dbbda1-816b-4692-8b8f-85441a6609d0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-c1b1e8-tdxb5\" (UID: \"92dbbda1-816b-4692-8b8f-85441a6609d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c1b1e8-tdxb5" Oct 30 13:04:54.129293 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:54.129160 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92dbbda1-816b-4692-8b8f-85441a6609d0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-c1b1e8-tdxb5\" (UID: \"92dbbda1-816b-4692-8b8f-85441a6609d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c1b1e8-tdxb5" Oct 30 13:04:54.129293 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:54.129180 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kn4ft\" (UniqueName: \"kubernetes.io/projected/92dbbda1-816b-4692-8b8f-85441a6609d0-kube-api-access-kn4ft\") pod \"maas-api-revoke-keys-e2e-worker-w1-c1b1e8-tdxb5\" (UID: \"92dbbda1-816b-4692-8b8f-85441a6609d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c1b1e8-tdxb5" Oct 30 13:04:54.130065 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:54.130033 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92dbbda1-816b-4692-8b8f-85441a6609d0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-c1b1e8-tdxb5\" (UID: \"92dbbda1-816b-4692-8b8f-85441a6609d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c1b1e8-tdxb5" Oct 30 13:04:54.131714 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:54.131686 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92dbbda1-816b-4692-8b8f-85441a6609d0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-c1b1e8-tdxb5\" (UID: \"92dbbda1-816b-4692-8b8f-85441a6609d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c1b1e8-tdxb5" Oct 30 13:04:54.151920 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:54.151853 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kn4ft\" (UniqueName: \"kubernetes.io/projected/92dbbda1-816b-4692-8b8f-85441a6609d0-kube-api-access-kn4ft\") pod \"maas-api-revoke-keys-e2e-worker-w1-c1b1e8-tdxb5\" (UID: \"92dbbda1-816b-4692-8b8f-85441a6609d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c1b1e8-tdxb5" Oct 30 13:04:54.217251 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:54.217210 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c1b1e8-tdxb5" Oct 30 13:04:54.531937 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:54.531901 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c1b1e8-tdxb5"] Oct 30 13:04:54.534173 ip-10-0-141-251 kubenswrapper[4908]: W1030 13:04:54.534134 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92dbbda1_816b_4692_8b8f_85441a6609d0.slice/crio-f6c0d2f488560c271b241000a56d4cbf7022c6aba055ab3e3c6ba222a7a22bba WatchSource:0}: Error finding container f6c0d2f488560c271b241000a56d4cbf7022c6aba055ab3e3c6ba222a7a22bba: Status 404 returned error can't find the container with id f6c0d2f488560c271b241000a56d4cbf7022c6aba055ab3e3c6ba222a7a22bba Oct 30 13:04:55.143922 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:55.143824 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c1b1e8-tdxb5" event={"ID":"92dbbda1-816b-4692-8b8f-85441a6609d0","Type":"ContainerStarted","Data":"3a70a6c6bf1db2ea899117bbed9fbf175f11436b7e878d7a3ca855cf4c0e60ee"} Oct 30 13:04:55.143922 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:55.143888 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c1b1e8-tdxb5" event={"ID":"92dbbda1-816b-4692-8b8f-85441a6609d0","Type":"ContainerStarted","Data":"f6c0d2f488560c271b241000a56d4cbf7022c6aba055ab3e3c6ba222a7a22bba"} Oct 30 13:04:55.182770 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:55.182712 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c1b1e8-tdxb5" podStartSLOduration=2.182697654 podStartE2EDuration="2.182697654s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:04:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:04:55.181873483 +0000 UTC m=+4251.696302257" watchObservedRunningTime="2026-10-30 13:04:55.182697654 +0000 UTC m=+4251.697126396" Oct 30 13:04:56.152749 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:56.152707 4908 generic.go:356] "Generic (PLEG): container finished" podID="92dbbda1-816b-4692-8b8f-85441a6609d0" containerID="3a70a6c6bf1db2ea899117bbed9fbf175f11436b7e878d7a3ca855cf4c0e60ee" exitCode=0 Oct 30 13:04:56.153292 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:56.152789 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c1b1e8-tdxb5" event={"ID":"92dbbda1-816b-4692-8b8f-85441a6609d0","Type":"ContainerDied","Data":"3a70a6c6bf1db2ea899117bbed9fbf175f11436b7e878d7a3ca855cf4c0e60ee"} Oct 30 13:04:57.344985 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:57.344949 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c1b1e8-tdxb5" Oct 30 13:04:57.482911 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:57.482877 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92dbbda1-816b-4692-8b8f-85441a6609d0-kube-api-access-kn4ft\" (UniqueName: \"kubernetes.io/projected/92dbbda1-816b-4692-8b8f-85441a6609d0-kube-api-access-kn4ft\") pod \"92dbbda1-816b-4692-8b8f-85441a6609d0\" (UID: \"92dbbda1-816b-4692-8b8f-85441a6609d0\") " Oct 30 13:04:57.483144 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:57.483017 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92dbbda1-816b-4692-8b8f-85441a6609d0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92dbbda1-816b-4692-8b8f-85441a6609d0-maas-api-openshift-ca\") pod \"92dbbda1-816b-4692-8b8f-85441a6609d0\" (UID: \"92dbbda1-816b-4692-8b8f-85441a6609d0\") " Oct 30 13:04:57.483243 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:57.483207 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/92dbbda1-816b-4692-8b8f-85441a6609d0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92dbbda1-816b-4692-8b8f-85441a6609d0-maas-api-xks-ca\") pod \"92dbbda1-816b-4692-8b8f-85441a6609d0\" (UID: \"92dbbda1-816b-4692-8b8f-85441a6609d0\") " Oct 30 13:04:57.483406 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:57.483378 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92dbbda1-816b-4692-8b8f-85441a6609d0-maas-api-openshift-ca" pod "92dbbda1-816b-4692-8b8f-85441a6609d0" (UID: "92dbbda1-816b-4692-8b8f-85441a6609d0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:04:57.483727 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:57.483706 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92dbbda1-816b-4692-8b8f-85441a6609d0-maas-api-openshift-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:04:57.485339 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:57.485313 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92dbbda1-816b-4692-8b8f-85441a6609d0-maas-api-xks-ca" pod "92dbbda1-816b-4692-8b8f-85441a6609d0" (UID: "92dbbda1-816b-4692-8b8f-85441a6609d0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:04:57.485462 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:57.485341 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92dbbda1-816b-4692-8b8f-85441a6609d0-kube-api-access-kn4ft" pod "92dbbda1-816b-4692-8b8f-85441a6609d0" (UID: "92dbbda1-816b-4692-8b8f-85441a6609d0"). InnerVolumeSpecName "kube-api-access-kn4ft". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:04:57.584713 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:57.584669 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92dbbda1-816b-4692-8b8f-85441a6609d0-maas-api-xks-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:04:57.584713 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:57.584705 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kn4ft\" (UniqueName: \"kubernetes.io/projected/92dbbda1-816b-4692-8b8f-85441a6609d0-kube-api-access-kn4ft\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:04:58.172611 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:58.172576 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c1b1e8-tdxb5" Oct 30 13:04:58.172810 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:58.172617 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c1b1e8-tdxb5" event={"ID":"92dbbda1-816b-4692-8b8f-85441a6609d0","Type":"ContainerDied","Data":"f6c0d2f488560c271b241000a56d4cbf7022c6aba055ab3e3c6ba222a7a22bba"} Oct 30 13:04:58.172810 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:58.172672 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f6c0d2f488560c271b241000a56d4cbf7022c6aba055ab3e3c6ba222a7a22bba" Oct 30 13:04:58.473674 ip-10-0-141-251 kubenswrapper[4908]: E1030 13:04:58.473596 4908 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c19d2e0_b5e4_4e44_a2cf_6724f0f1e0b1.slice\": RecentStats: unable to find data in memory cache]" Oct 30 13:04:58.473674 ip-10-0-141-251 kubenswrapper[4908]: E1030 13:04:58.473619 4908 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c19d2e0_b5e4_4e44_a2cf_6724f0f1e0b1.slice\": RecentStats: unable to find data in memory cache]" Oct 30 13:04:58.505159 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:58.505129 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-nntkd" Oct 30 13:04:58.600180 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:58.600146 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6810256-6b25-44b9-a7f1-fb0cb023a5b9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6810256-6b25-44b9-a7f1-fb0cb023a5b9-istio-data\") pod \"e6810256-6b25-44b9-a7f1-fb0cb023a5b9\" (UID: \"e6810256-6b25-44b9-a7f1-fb0cb023a5b9\") " Oct 30 13:04:58.600357 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:58.600194 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6810256-6b25-44b9-a7f1-fb0cb023a5b9-istio-token\" (UniqueName: \"kubernetes.io/projected/e6810256-6b25-44b9-a7f1-fb0cb023a5b9-istio-token\") pod \"e6810256-6b25-44b9-a7f1-fb0cb023a5b9\" (UID: \"e6810256-6b25-44b9-a7f1-fb0cb023a5b9\") " Oct 30 13:04:58.600357 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:58.600231 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6810256-6b25-44b9-a7f1-fb0cb023a5b9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6810256-6b25-44b9-a7f1-fb0cb023a5b9-istio-envoy\") pod \"e6810256-6b25-44b9-a7f1-fb0cb023a5b9\" (UID: \"e6810256-6b25-44b9-a7f1-fb0cb023a5b9\") " Oct 30 13:04:58.600357 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:58.600269 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6810256-6b25-44b9-a7f1-fb0cb023a5b9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6810256-6b25-44b9-a7f1-fb0cb023a5b9-workload-socket\") pod \"e6810256-6b25-44b9-a7f1-fb0cb023a5b9\" (UID: \"e6810256-6b25-44b9-a7f1-fb0cb023a5b9\") " Oct 30 13:04:58.600357 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:58.600303 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6810256-6b25-44b9-a7f1-fb0cb023a5b9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6810256-6b25-44b9-a7f1-fb0cb023a5b9-istiod-ca-cert\") pod \"e6810256-6b25-44b9-a7f1-fb0cb023a5b9\" (UID: \"e6810256-6b25-44b9-a7f1-fb0cb023a5b9\") " Oct 30 13:04:58.600601 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:58.600454 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6810256-6b25-44b9-a7f1-fb0cb023a5b9-istio-data" pod "e6810256-6b25-44b9-a7f1-fb0cb023a5b9" (UID: "e6810256-6b25-44b9-a7f1-fb0cb023a5b9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:04:58.600601 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:58.600466 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6810256-6b25-44b9-a7f1-fb0cb023a5b9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6810256-6b25-44b9-a7f1-fb0cb023a5b9-workload-certs\") pod \"e6810256-6b25-44b9-a7f1-fb0cb023a5b9\" (UID: \"e6810256-6b25-44b9-a7f1-fb0cb023a5b9\") " Oct 30 13:04:58.600601 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:58.600540 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e6810256-6b25-44b9-a7f1-fb0cb023a5b9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6810256-6b25-44b9-a7f1-fb0cb023a5b9-istio-podinfo\") pod \"e6810256-6b25-44b9-a7f1-fb0cb023a5b9\" (UID: \"e6810256-6b25-44b9-a7f1-fb0cb023a5b9\") " Oct 30 13:04:58.600754 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:58.600648 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6810256-6b25-44b9-a7f1-fb0cb023a5b9-workload-certs" pod "e6810256-6b25-44b9-a7f1-fb0cb023a5b9" (UID: "e6810256-6b25-44b9-a7f1-fb0cb023a5b9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:04:58.600816 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:58.600772 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6810256-6b25-44b9-a7f1-fb0cb023a5b9-kube-api-access-wv2v6\" (UniqueName: \"kubernetes.io/projected/e6810256-6b25-44b9-a7f1-fb0cb023a5b9-kube-api-access-wv2v6\") pod \"e6810256-6b25-44b9-a7f1-fb0cb023a5b9\" (UID: \"e6810256-6b25-44b9-a7f1-fb0cb023a5b9\") " Oct 30 13:04:58.601498 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:58.600921 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6810256-6b25-44b9-a7f1-fb0cb023a5b9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6810256-6b25-44b9-a7f1-fb0cb023a5b9-credential-socket\") pod \"e6810256-6b25-44b9-a7f1-fb0cb023a5b9\" (UID: \"e6810256-6b25-44b9-a7f1-fb0cb023a5b9\") " Oct 30 13:04:58.601498 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:58.601396 4908 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6810256-6b25-44b9-a7f1-fb0cb023a5b9-workload-certs\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:04:58.601498 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:58.601418 4908 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6810256-6b25-44b9-a7f1-fb0cb023a5b9-istio-data\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:04:58.601816 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:58.601747 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6810256-6b25-44b9-a7f1-fb0cb023a5b9-istiod-ca-cert" pod "e6810256-6b25-44b9-a7f1-fb0cb023a5b9" (UID: "e6810256-6b25-44b9-a7f1-fb0cb023a5b9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:04:58.601934 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:58.601821 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6810256-6b25-44b9-a7f1-fb0cb023a5b9-workload-socket" pod "e6810256-6b25-44b9-a7f1-fb0cb023a5b9" (UID: "e6810256-6b25-44b9-a7f1-fb0cb023a5b9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:04:58.602114 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:58.602084 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6810256-6b25-44b9-a7f1-fb0cb023a5b9-credential-socket" pod "e6810256-6b25-44b9-a7f1-fb0cb023a5b9" (UID: "e6810256-6b25-44b9-a7f1-fb0cb023a5b9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:04:58.603411 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:58.603309 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6810256-6b25-44b9-a7f1-fb0cb023a5b9-kube-api-access-wv2v6" pod "e6810256-6b25-44b9-a7f1-fb0cb023a5b9" (UID: "e6810256-6b25-44b9-a7f1-fb0cb023a5b9"). InnerVolumeSpecName "kube-api-access-wv2v6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:04:58.615162 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:58.615134 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6810256-6b25-44b9-a7f1-fb0cb023a5b9-istio-envoy" pod "e6810256-6b25-44b9-a7f1-fb0cb023a5b9" (UID: "e6810256-6b25-44b9-a7f1-fb0cb023a5b9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:04:58.615162 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:58.615150 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e6810256-6b25-44b9-a7f1-fb0cb023a5b9-istio-podinfo" pod "e6810256-6b25-44b9-a7f1-fb0cb023a5b9" (UID: "e6810256-6b25-44b9-a7f1-fb0cb023a5b9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:04:58.616587 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:58.616418 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6810256-6b25-44b9-a7f1-fb0cb023a5b9-istio-token" pod "e6810256-6b25-44b9-a7f1-fb0cb023a5b9" (UID: "e6810256-6b25-44b9-a7f1-fb0cb023a5b9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:04:58.703227 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:58.703096 4908 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6810256-6b25-44b9-a7f1-fb0cb023a5b9-istio-podinfo\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:04:58.703227 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:58.703136 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wv2v6\" (UniqueName: \"kubernetes.io/projected/e6810256-6b25-44b9-a7f1-fb0cb023a5b9-kube-api-access-wv2v6\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:04:58.703227 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:58.703153 4908 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6810256-6b25-44b9-a7f1-fb0cb023a5b9-credential-socket\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:04:58.703227 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:58.703169 4908 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6810256-6b25-44b9-a7f1-fb0cb023a5b9-istio-token\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:04:58.703227 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:58.703185 4908 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6810256-6b25-44b9-a7f1-fb0cb023a5b9-istio-envoy\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:04:58.703227 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:58.703198 4908 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6810256-6b25-44b9-a7f1-fb0cb023a5b9-workload-socket\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:04:58.703227 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:58.703221 4908 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6810256-6b25-44b9-a7f1-fb0cb023a5b9-istiod-ca-cert\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:04:59.186438 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:59.186394 4908 generic.go:356] "Generic (PLEG): container finished" podID="e6810256-6b25-44b9-a7f1-fb0cb023a5b9" containerID="61e639a16ad650e85438962111c2da99430d9e83ce51902b2bed8c1b8d716516" exitCode=0 Oct 30 13:04:59.186670 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:59.186459 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-nntkd" Oct 30 13:04:59.186670 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:59.186572 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-nntkd" event={"ID":"e6810256-6b25-44b9-a7f1-fb0cb023a5b9","Type":"ContainerDied","Data":"61e639a16ad650e85438962111c2da99430d9e83ce51902b2bed8c1b8d716516"} Oct 30 13:04:59.186670 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:59.186641 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-nntkd" event={"ID":"e6810256-6b25-44b9-a7f1-fb0cb023a5b9","Type":"ContainerDied","Data":"bdaf6dec313f3748acecbf10ec608f8dc6ee578209416d5bfea52900d0a6aa46"} Oct 30 13:04:59.186910 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:59.186685 4908 scope.go:122] "RemoveContainer" containerID="61e639a16ad650e85438962111c2da99430d9e83ce51902b2bed8c1b8d716516" Oct 30 13:04:59.209037 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:59.209011 4908 scope.go:122] "RemoveContainer" containerID="61e639a16ad650e85438962111c2da99430d9e83ce51902b2bed8c1b8d716516" Oct 30 13:04:59.209376 ip-10-0-141-251 kubenswrapper[4908]: E1030 13:04:59.209351 4908 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61e639a16ad650e85438962111c2da99430d9e83ce51902b2bed8c1b8d716516\": container with ID starting with 61e639a16ad650e85438962111c2da99430d9e83ce51902b2bed8c1b8d716516 not found: ID does not exist" containerID="61e639a16ad650e85438962111c2da99430d9e83ce51902b2bed8c1b8d716516" Oct 30 13:04:59.209449 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:59.209388 4908 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61e639a16ad650e85438962111c2da99430d9e83ce51902b2bed8c1b8d716516"} err="failed to get container status \"61e639a16ad650e85438962111c2da99430d9e83ce51902b2bed8c1b8d716516\": rpc error: code = NotFound desc = could not find container \"61e639a16ad650e85438962111c2da99430d9e83ce51902b2bed8c1b8d716516\": container with ID starting with 61e639a16ad650e85438962111c2da99430d9e83ce51902b2bed8c1b8d716516 not found: ID does not exist" Oct 30 13:04:59.252259 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:59.252223 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-nntkd"] Oct 30 13:04:59.255852 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:59.255815 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-nntkd"] Oct 30 13:04:59.731855 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:04:59.728125 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6810256-6b25-44b9-a7f1-fb0cb023a5b9" path="/var/lib/kubelet/pods/e6810256-6b25-44b9-a7f1-fb0cb023a5b9/volumes" Oct 30 13:05:00.062518 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:00.062415 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-63f4e4-nm6f4"] Oct 30 13:05:00.066728 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:00.066691 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-63f4e4-nm6f4"] Oct 30 13:05:00.729162 ip-10-0-141-251 kubenswrapper[4908]: E1030 13:05:00.729125 4908 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c19d2e0_b5e4_4e44_a2cf_6724f0f1e0b1.slice\": RecentStats: unable to find data in memory cache]" Oct 30 13:05:01.609617 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:01.609577 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9801b39b-9af9-4004-ae48-3025ca5bde8f" path="/var/lib/kubelet/pods/9801b39b-9af9-4004-ae48-3025ca5bde8f/volumes" Oct 30 13:05:07.638320 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:07.638281 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-63f4e4-openshift-default-786cd844c9-lxsbx"] Oct 30 13:05:07.638778 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:07.638550 4908 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-63f4e4-openshift-default-786cd844c9-lxsbx" podUID="f1ca551b-2beb-407b-9e9c-e410199f6f33" containerName="istio-proxy" containerID="cri-o://b38b1af2792d7db04419d6757727cf8f5967968075a7a33b871fcfbe36c3fa02" gracePeriod=30 Oct 30 13:05:08.235814 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:08.235772 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-734ffe-x6b97"] Oct 30 13:05:08.236359 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:08.236341 4908 cpu_manager.go:414] "containerMap: removing container" podUID="e6810256-6b25-44b9-a7f1-fb0cb023a5b9" containerName="istio-proxy" Oct 30 13:05:08.236449 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:08.236359 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6810256-6b25-44b9-a7f1-fb0cb023a5b9" containerName="istio-proxy" Oct 30 13:05:08.236449 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:08.236375 4908 cpu_manager.go:414] "containerMap: removing container" podUID="92dbbda1-816b-4692-8b8f-85441a6609d0" containerName="revoke-keys" Oct 30 13:05:08.236449 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:08.236380 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92dbbda1-816b-4692-8b8f-85441a6609d0" containerName="revoke-keys" Oct 30 13:05:08.236625 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:08.236481 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6810256-6b25-44b9-a7f1-fb0cb023a5b9" containerName="istio-proxy" Oct 30 13:05:08.236625 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:08.236491 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="92dbbda1-816b-4692-8b8f-85441a6609d0" containerName="revoke-keys" Oct 30 13:05:08.241424 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:08.241400 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-734ffe-x6b97" Oct 30 13:05:08.257669 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:08.257065 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 13:05:08.257669 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:08.257149 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 13:05:08.257669 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:08.257249 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s75hq\"" Oct 30 13:05:08.257669 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:08.257273 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:05:08.260633 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:08.260601 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-734ffe-x6b97"] Oct 30 13:05:08.336178 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:08.336140 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pm7dx\" (UniqueName: \"kubernetes.io/projected/01a6f969-4660-4001-b9e7-190f4cfd7633-kube-api-access-pm7dx\") pod \"maas-api-revoke-keys-e2e-inf-b-734ffe-x6b97\" (UID: \"01a6f969-4660-4001-b9e7-190f4cfd7633\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-734ffe-x6b97" Oct 30 13:05:08.336401 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:08.336323 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01a6f969-4660-4001-b9e7-190f4cfd7633-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-734ffe-x6b97\" (UID: \"01a6f969-4660-4001-b9e7-190f4cfd7633\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-734ffe-x6b97" Oct 30 13:05:08.336456 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:08.336434 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01a6f969-4660-4001-b9e7-190f4cfd7633-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-734ffe-x6b97\" (UID: \"01a6f969-4660-4001-b9e7-190f4cfd7633\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-734ffe-x6b97" Oct 30 13:05:08.437281 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:08.437241 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01a6f969-4660-4001-b9e7-190f4cfd7633-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-734ffe-x6b97\" (UID: \"01a6f969-4660-4001-b9e7-190f4cfd7633\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-734ffe-x6b97" Oct 30 13:05:08.437469 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:08.437331 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01a6f969-4660-4001-b9e7-190f4cfd7633-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-734ffe-x6b97\" (UID: \"01a6f969-4660-4001-b9e7-190f4cfd7633\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-734ffe-x6b97" Oct 30 13:05:08.437469 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:08.437432 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pm7dx\" (UniqueName: \"kubernetes.io/projected/01a6f969-4660-4001-b9e7-190f4cfd7633-kube-api-access-pm7dx\") pod \"maas-api-revoke-keys-e2e-inf-b-734ffe-x6b97\" (UID: \"01a6f969-4660-4001-b9e7-190f4cfd7633\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-734ffe-x6b97" Oct 30 13:05:08.438151 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:08.438119 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01a6f969-4660-4001-b9e7-190f4cfd7633-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-734ffe-x6b97\" (UID: \"01a6f969-4660-4001-b9e7-190f4cfd7633\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-734ffe-x6b97" Oct 30 13:05:08.439899 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:08.439874 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01a6f969-4660-4001-b9e7-190f4cfd7633-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-734ffe-x6b97\" (UID: \"01a6f969-4660-4001-b9e7-190f4cfd7633\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-734ffe-x6b97" Oct 30 13:05:08.460619 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:08.460586 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pm7dx\" (UniqueName: \"kubernetes.io/projected/01a6f969-4660-4001-b9e7-190f4cfd7633-kube-api-access-pm7dx\") pod \"maas-api-revoke-keys-e2e-inf-b-734ffe-x6b97\" (UID: \"01a6f969-4660-4001-b9e7-190f4cfd7633\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-734ffe-x6b97" Oct 30 13:05:08.558449 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:08.558368 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-734ffe-x6b97" Oct 30 13:05:08.875677 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:08.875639 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-734ffe-x6b97"] Oct 30 13:05:08.877116 ip-10-0-141-251 kubenswrapper[4908]: W1030 13:05:08.877085 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01a6f969_4660_4001_b9e7_190f4cfd7633.slice/crio-8ba82bfa95806480fb17808e3a1fb6ab98b6bbabbde633c554999d9bb10d98ee WatchSource:0}: Error finding container 8ba82bfa95806480fb17808e3a1fb6ab98b6bbabbde633c554999d9bb10d98ee: Status 404 returned error can't find the container with id 8ba82bfa95806480fb17808e3a1fb6ab98b6bbabbde633c554999d9bb10d98ee Oct 30 13:05:09.276431 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:09.276387 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-734ffe-x6b97" event={"ID":"01a6f969-4660-4001-b9e7-190f4cfd7633","Type":"ContainerStarted","Data":"7cddb14d0ca255503291d55777923024c4afdcafdcd6d9a4a9ee93915bc5b713"} Oct 30 13:05:09.276648 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:09.276455 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-734ffe-x6b97" event={"ID":"01a6f969-4660-4001-b9e7-190f4cfd7633","Type":"ContainerStarted","Data":"8ba82bfa95806480fb17808e3a1fb6ab98b6bbabbde633c554999d9bb10d98ee"} Oct 30 13:05:09.317459 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:09.317246 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-734ffe-x6b97" podStartSLOduration=1.317228333 podStartE2EDuration="1.317228333s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:05:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:05:09.315183768 +0000 UTC m=+4265.829612514" watchObservedRunningTime="2026-10-30 13:05:09.317228333 +0000 UTC m=+4265.831657079" Oct 30 13:05:10.290595 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:10.290550 4908 generic.go:356] "Generic (PLEG): container finished" podID="01a6f969-4660-4001-b9e7-190f4cfd7633" containerID="7cddb14d0ca255503291d55777923024c4afdcafdcd6d9a4a9ee93915bc5b713" exitCode=0 Oct 30 13:05:10.291162 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:10.290675 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-734ffe-x6b97" event={"ID":"01a6f969-4660-4001-b9e7-190f4cfd7633","Type":"ContainerDied","Data":"7cddb14d0ca255503291d55777923024c4afdcafdcd6d9a4a9ee93915bc5b713"} Oct 30 13:05:10.838456 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:10.838417 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6b67f8-2g65x"] Oct 30 13:05:10.842250 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:10.842220 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6b67f8-2g65x"] Oct 30 13:05:11.500751 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:11.500717 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-734ffe-x6b97" Oct 30 13:05:11.606304 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:11.606212 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cabd4f34-efec-4a7e-89e7-b58692f3b37a" path="/var/lib/kubelet/pods/cabd4f34-efec-4a7e-89e7-b58692f3b37a/volumes" Oct 30 13:05:11.632966 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:11.632925 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01a6f969-4660-4001-b9e7-190f4cfd7633-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01a6f969-4660-4001-b9e7-190f4cfd7633-maas-api-openshift-ca\") pod \"01a6f969-4660-4001-b9e7-190f4cfd7633\" (UID: \"01a6f969-4660-4001-b9e7-190f4cfd7633\") " Oct 30 13:05:11.633149 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:11.632980 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01a6f969-4660-4001-b9e7-190f4cfd7633-kube-api-access-pm7dx\" (UniqueName: \"kubernetes.io/projected/01a6f969-4660-4001-b9e7-190f4cfd7633-kube-api-access-pm7dx\") pod \"01a6f969-4660-4001-b9e7-190f4cfd7633\" (UID: \"01a6f969-4660-4001-b9e7-190f4cfd7633\") " Oct 30 13:05:11.633256 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:11.633234 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/01a6f969-4660-4001-b9e7-190f4cfd7633-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01a6f969-4660-4001-b9e7-190f4cfd7633-maas-api-xks-ca\") pod \"01a6f969-4660-4001-b9e7-190f4cfd7633\" (UID: \"01a6f969-4660-4001-b9e7-190f4cfd7633\") " Oct 30 13:05:11.633361 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:11.633337 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01a6f969-4660-4001-b9e7-190f4cfd7633-maas-api-openshift-ca" pod "01a6f969-4660-4001-b9e7-190f4cfd7633" (UID: "01a6f969-4660-4001-b9e7-190f4cfd7633"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:05:11.633693 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:11.633672 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01a6f969-4660-4001-b9e7-190f4cfd7633-maas-api-openshift-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:05:11.635356 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:11.635330 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01a6f969-4660-4001-b9e7-190f4cfd7633-kube-api-access-pm7dx" pod "01a6f969-4660-4001-b9e7-190f4cfd7633" (UID: "01a6f969-4660-4001-b9e7-190f4cfd7633"). InnerVolumeSpecName "kube-api-access-pm7dx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:05:11.635467 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:11.635348 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01a6f969-4660-4001-b9e7-190f4cfd7633-maas-api-xks-ca" pod "01a6f969-4660-4001-b9e7-190f4cfd7633" (UID: "01a6f969-4660-4001-b9e7-190f4cfd7633"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:05:11.734725 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:11.734679 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01a6f969-4660-4001-b9e7-190f4cfd7633-maas-api-xks-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:05:11.734725 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:11.734722 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pm7dx\" (UniqueName: \"kubernetes.io/projected/01a6f969-4660-4001-b9e7-190f4cfd7633-kube-api-access-pm7dx\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:05:12.094551 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:12.094509 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cfba1837-8pwmd"] Oct 30 13:05:12.095146 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:12.095130 4908 cpu_manager.go:414] "containerMap: removing container" podUID="01a6f969-4660-4001-b9e7-190f4cfd7633" containerName="revoke-keys" Oct 30 13:05:12.095215 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:12.095148 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01a6f969-4660-4001-b9e7-190f4cfd7633" containerName="revoke-keys" Oct 30 13:05:12.095259 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:12.095253 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="01a6f969-4660-4001-b9e7-190f4cfd7633" containerName="revoke-keys" Oct 30 13:05:12.099043 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:12.099025 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfba1837-8pwmd" Oct 30 13:05:12.118249 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:12.118211 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cfba1837-8pwmd"] Oct 30 13:05:12.241156 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:12.241109 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e77e666-f695-41d9-9243-d845872f235c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cfba1837-8pwmd\" (UID: \"8e77e666-f695-41d9-9243-d845872f235c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfba1837-8pwmd" Oct 30 13:05:12.241361 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:12.241282 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jw5cr\" (UniqueName: \"kubernetes.io/projected/8e77e666-f695-41d9-9243-d845872f235c-kube-api-access-jw5cr\") pod \"maas-api-revoke-sub-cfba1837-8pwmd\" (UID: \"8e77e666-f695-41d9-9243-d845872f235c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfba1837-8pwmd" Oct 30 13:05:12.241361 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:12.241320 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e77e666-f695-41d9-9243-d845872f235c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cfba1837-8pwmd\" (UID: \"8e77e666-f695-41d9-9243-d845872f235c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfba1837-8pwmd" Oct 30 13:05:12.308261 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:12.308225 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-734ffe-x6b97" event={"ID":"01a6f969-4660-4001-b9e7-190f4cfd7633","Type":"ContainerDied","Data":"8ba82bfa95806480fb17808e3a1fb6ab98b6bbabbde633c554999d9bb10d98ee"} Oct 30 13:05:12.308261 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:12.308260 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-734ffe-x6b97" Oct 30 13:05:12.308527 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:12.308274 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8ba82bfa95806480fb17808e3a1fb6ab98b6bbabbde633c554999d9bb10d98ee" Oct 30 13:05:12.342481 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:12.342429 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e77e666-f695-41d9-9243-d845872f235c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cfba1837-8pwmd\" (UID: \"8e77e666-f695-41d9-9243-d845872f235c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfba1837-8pwmd" Oct 30 13:05:12.342714 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:12.342571 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jw5cr\" (UniqueName: \"kubernetes.io/projected/8e77e666-f695-41d9-9243-d845872f235c-kube-api-access-jw5cr\") pod \"maas-api-revoke-sub-cfba1837-8pwmd\" (UID: \"8e77e666-f695-41d9-9243-d845872f235c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfba1837-8pwmd" Oct 30 13:05:12.342714 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:12.342617 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e77e666-f695-41d9-9243-d845872f235c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cfba1837-8pwmd\" (UID: \"8e77e666-f695-41d9-9243-d845872f235c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfba1837-8pwmd" Oct 30 13:05:12.343233 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:12.343207 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e77e666-f695-41d9-9243-d845872f235c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cfba1837-8pwmd\" (UID: \"8e77e666-f695-41d9-9243-d845872f235c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfba1837-8pwmd" Oct 30 13:05:12.345253 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:12.345189 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e77e666-f695-41d9-9243-d845872f235c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cfba1837-8pwmd\" (UID: \"8e77e666-f695-41d9-9243-d845872f235c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfba1837-8pwmd" Oct 30 13:05:12.379871 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:12.379822 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jw5cr\" (UniqueName: \"kubernetes.io/projected/8e77e666-f695-41d9-9243-d845872f235c-kube-api-access-jw5cr\") pod \"maas-api-revoke-sub-cfba1837-8pwmd\" (UID: \"8e77e666-f695-41d9-9243-d845872f235c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfba1837-8pwmd" Oct 30 13:05:12.413466 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:12.413416 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfba1837-8pwmd" Oct 30 13:05:12.746204 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:12.746172 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cfba1837-8pwmd"] Oct 30 13:05:12.748044 ip-10-0-141-251 kubenswrapper[4908]: W1030 13:05:12.748014 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e77e666_f695_41d9_9243_d845872f235c.slice/crio-4730b9d9100ef8bb8d130c5fd070617608db47b354fb9e3f3d5984d14c70ea45 WatchSource:0}: Error finding container 4730b9d9100ef8bb8d130c5fd070617608db47b354fb9e3f3d5984d14c70ea45: Status 404 returned error can't find the container with id 4730b9d9100ef8bb8d130c5fd070617608db47b354fb9e3f3d5984d14c70ea45 Oct 30 13:05:12.999644 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:12.999616 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-63f4e4-openshift-default-786cd844c9-lxsbx" Oct 30 13:05:13.067874 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:13.067798 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1ca551b-2beb-407b-9e9c-e410199f6f33-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1ca551b-2beb-407b-9e9c-e410199f6f33-workload-socket\") pod \"f1ca551b-2beb-407b-9e9c-e410199f6f33\" (UID: \"f1ca551b-2beb-407b-9e9c-e410199f6f33\") " Oct 30 13:05:13.068068 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:13.067910 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1ca551b-2beb-407b-9e9c-e410199f6f33-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1ca551b-2beb-407b-9e9c-e410199f6f33-istio-envoy\") pod \"f1ca551b-2beb-407b-9e9c-e410199f6f33\" (UID: \"f1ca551b-2beb-407b-9e9c-e410199f6f33\") " Oct 30 13:05:13.068068 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:13.068008 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1ca551b-2beb-407b-9e9c-e410199f6f33-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1ca551b-2beb-407b-9e9c-e410199f6f33-credential-socket\") pod \"f1ca551b-2beb-407b-9e9c-e410199f6f33\" (UID: \"f1ca551b-2beb-407b-9e9c-e410199f6f33\") " Oct 30 13:05:13.068198 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:13.068123 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f1ca551b-2beb-407b-9e9c-e410199f6f33-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1ca551b-2beb-407b-9e9c-e410199f6f33-istio-podinfo\") pod \"f1ca551b-2beb-407b-9e9c-e410199f6f33\" (UID: \"f1ca551b-2beb-407b-9e9c-e410199f6f33\") " Oct 30 13:05:13.068198 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:13.068156 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1ca551b-2beb-407b-9e9c-e410199f6f33-workload-socket" pod "f1ca551b-2beb-407b-9e9c-e410199f6f33" (UID: "f1ca551b-2beb-407b-9e9c-e410199f6f33"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:05:13.068313 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:13.068223 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1ca551b-2beb-407b-9e9c-e410199f6f33-istio-token\" (UniqueName: \"kubernetes.io/projected/f1ca551b-2beb-407b-9e9c-e410199f6f33-istio-token\") pod \"f1ca551b-2beb-407b-9e9c-e410199f6f33\" (UID: \"f1ca551b-2beb-407b-9e9c-e410199f6f33\") " Oct 30 13:05:13.068382 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:13.068323 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1ca551b-2beb-407b-9e9c-e410199f6f33-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1ca551b-2beb-407b-9e9c-e410199f6f33-istio-data\") pod \"f1ca551b-2beb-407b-9e9c-e410199f6f33\" (UID: \"f1ca551b-2beb-407b-9e9c-e410199f6f33\") " Oct 30 13:05:13.068382 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:13.068353 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1ca551b-2beb-407b-9e9c-e410199f6f33-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1ca551b-2beb-407b-9e9c-e410199f6f33-workload-certs\") pod \"f1ca551b-2beb-407b-9e9c-e410199f6f33\" (UID: \"f1ca551b-2beb-407b-9e9c-e410199f6f33\") " Oct 30 13:05:13.068493 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:13.068396 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1ca551b-2beb-407b-9e9c-e410199f6f33-kube-api-access-nb9tb\" (UniqueName: \"kubernetes.io/projected/f1ca551b-2beb-407b-9e9c-e410199f6f33-kube-api-access-nb9tb\") pod \"f1ca551b-2beb-407b-9e9c-e410199f6f33\" (UID: \"f1ca551b-2beb-407b-9e9c-e410199f6f33\") " Oct 30 13:05:13.068493 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:13.068429 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1ca551b-2beb-407b-9e9c-e410199f6f33-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1ca551b-2beb-407b-9e9c-e410199f6f33-istiod-ca-cert\") pod \"f1ca551b-2beb-407b-9e9c-e410199f6f33\" (UID: \"f1ca551b-2beb-407b-9e9c-e410199f6f33\") " Oct 30 13:05:13.068493 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:13.068464 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1ca551b-2beb-407b-9e9c-e410199f6f33-credential-socket" pod "f1ca551b-2beb-407b-9e9c-e410199f6f33" (UID: "f1ca551b-2beb-407b-9e9c-e410199f6f33"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:05:13.068667 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:13.068600 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1ca551b-2beb-407b-9e9c-e410199f6f33-istio-data" pod "f1ca551b-2beb-407b-9e9c-e410199f6f33" (UID: "f1ca551b-2beb-407b-9e9c-e410199f6f33"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:05:13.068743 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:13.068721 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1ca551b-2beb-407b-9e9c-e410199f6f33-workload-certs" pod "f1ca551b-2beb-407b-9e9c-e410199f6f33" (UID: "f1ca551b-2beb-407b-9e9c-e410199f6f33"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:05:13.069133 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:13.069102 4908 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1ca551b-2beb-407b-9e9c-e410199f6f33-workload-socket\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:05:13.069133 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:13.069126 4908 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1ca551b-2beb-407b-9e9c-e410199f6f33-credential-socket\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:05:13.069301 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:13.069140 4908 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1ca551b-2beb-407b-9e9c-e410199f6f33-istio-data\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:05:13.069301 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:13.069152 4908 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1ca551b-2beb-407b-9e9c-e410199f6f33-workload-certs\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:05:13.069587 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:13.069562 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1ca551b-2beb-407b-9e9c-e410199f6f33-istiod-ca-cert" pod "f1ca551b-2beb-407b-9e9c-e410199f6f33" (UID: "f1ca551b-2beb-407b-9e9c-e410199f6f33"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:05:13.071166 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:13.071137 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1ca551b-2beb-407b-9e9c-e410199f6f33-istio-envoy" pod "f1ca551b-2beb-407b-9e9c-e410199f6f33" (UID: "f1ca551b-2beb-407b-9e9c-e410199f6f33"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:05:13.071557 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:13.071535 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1ca551b-2beb-407b-9e9c-e410199f6f33-kube-api-access-nb9tb" pod "f1ca551b-2beb-407b-9e9c-e410199f6f33" (UID: "f1ca551b-2beb-407b-9e9c-e410199f6f33"). InnerVolumeSpecName "kube-api-access-nb9tb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:05:13.071667 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:13.071628 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1ca551b-2beb-407b-9e9c-e410199f6f33-istio-token" pod "f1ca551b-2beb-407b-9e9c-e410199f6f33" (UID: "f1ca551b-2beb-407b-9e9c-e410199f6f33"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:05:13.072303 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:13.072269 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f1ca551b-2beb-407b-9e9c-e410199f6f33-istio-podinfo" pod "f1ca551b-2beb-407b-9e9c-e410199f6f33" (UID: "f1ca551b-2beb-407b-9e9c-e410199f6f33"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:05:13.170607 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:13.170564 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nb9tb\" (UniqueName: \"kubernetes.io/projected/f1ca551b-2beb-407b-9e9c-e410199f6f33-kube-api-access-nb9tb\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:05:13.170607 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:13.170605 4908 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1ca551b-2beb-407b-9e9c-e410199f6f33-istiod-ca-cert\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:05:13.170607 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:13.170616 4908 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1ca551b-2beb-407b-9e9c-e410199f6f33-istio-envoy\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:05:13.170902 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:13.170624 4908 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1ca551b-2beb-407b-9e9c-e410199f6f33-istio-podinfo\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:05:13.170902 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:13.170634 4908 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1ca551b-2beb-407b-9e9c-e410199f6f33-istio-token\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:05:13.322753 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:13.322677 4908 generic.go:356] "Generic (PLEG): container finished" podID="f1ca551b-2beb-407b-9e9c-e410199f6f33" containerID="b38b1af2792d7db04419d6757727cf8f5967968075a7a33b871fcfbe36c3fa02" exitCode=0 Oct 30 13:05:13.322753 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:13.322735 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-63f4e4-openshift-default-786cd844c9-lxsbx" Oct 30 13:05:13.323117 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:13.322734 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-63f4e4-openshift-default-786cd844c9-lxsbx" event={"ID":"f1ca551b-2beb-407b-9e9c-e410199f6f33","Type":"ContainerDied","Data":"b38b1af2792d7db04419d6757727cf8f5967968075a7a33b871fcfbe36c3fa02"} Oct 30 13:05:13.323117 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:13.322936 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-63f4e4-openshift-default-786cd844c9-lxsbx" event={"ID":"f1ca551b-2beb-407b-9e9c-e410199f6f33","Type":"ContainerDied","Data":"38aa741646389e8e45b5488d0d81d94c6f925cee0cfb9856a0ad52cfdabebcff"} Oct 30 13:05:13.323117 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:13.322990 4908 scope.go:122] "RemoveContainer" containerID="b38b1af2792d7db04419d6757727cf8f5967968075a7a33b871fcfbe36c3fa02" Oct 30 13:05:13.325092 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:13.325061 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfba1837-8pwmd" event={"ID":"8e77e666-f695-41d9-9243-d845872f235c","Type":"ContainerStarted","Data":"5ab285593f719a6dce287e347205aef7bc6056f1d93cbf09d1d5b71deb1feeb5"} Oct 30 13:05:13.325237 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:13.325125 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfba1837-8pwmd" event={"ID":"8e77e666-f695-41d9-9243-d845872f235c","Type":"ContainerStarted","Data":"4730b9d9100ef8bb8d130c5fd070617608db47b354fb9e3f3d5984d14c70ea45"} Oct 30 13:05:13.336109 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:13.336087 4908 scope.go:122] "RemoveContainer" containerID="48dc6ab44528d0f5a5162fc8b89d173676851668e94c506a2ab0d5f16d0cb238" Oct 30 13:05:13.347320 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:13.347287 4908 scope.go:122] "RemoveContainer" containerID="b38b1af2792d7db04419d6757727cf8f5967968075a7a33b871fcfbe36c3fa02" Oct 30 13:05:13.347618 ip-10-0-141-251 kubenswrapper[4908]: E1030 13:05:13.347594 4908 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b38b1af2792d7db04419d6757727cf8f5967968075a7a33b871fcfbe36c3fa02\": container with ID starting with b38b1af2792d7db04419d6757727cf8f5967968075a7a33b871fcfbe36c3fa02 not found: ID does not exist" containerID="b38b1af2792d7db04419d6757727cf8f5967968075a7a33b871fcfbe36c3fa02" Oct 30 13:05:13.347687 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:13.347629 4908 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b38b1af2792d7db04419d6757727cf8f5967968075a7a33b871fcfbe36c3fa02"} err="failed to get container status \"b38b1af2792d7db04419d6757727cf8f5967968075a7a33b871fcfbe36c3fa02\": rpc error: code = NotFound desc = could not find container \"b38b1af2792d7db04419d6757727cf8f5967968075a7a33b871fcfbe36c3fa02\": container with ID starting with b38b1af2792d7db04419d6757727cf8f5967968075a7a33b871fcfbe36c3fa02 not found: ID does not exist" Oct 30 13:05:13.347687 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:13.347649 4908 scope.go:122] "RemoveContainer" containerID="48dc6ab44528d0f5a5162fc8b89d173676851668e94c506a2ab0d5f16d0cb238" Oct 30 13:05:13.347977 ip-10-0-141-251 kubenswrapper[4908]: E1030 13:05:13.347939 4908 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48dc6ab44528d0f5a5162fc8b89d173676851668e94c506a2ab0d5f16d0cb238\": container with ID starting with 48dc6ab44528d0f5a5162fc8b89d173676851668e94c506a2ab0d5f16d0cb238 not found: ID does not exist" containerID="48dc6ab44528d0f5a5162fc8b89d173676851668e94c506a2ab0d5f16d0cb238" Oct 30 13:05:13.348042 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:13.347984 4908 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48dc6ab44528d0f5a5162fc8b89d173676851668e94c506a2ab0d5f16d0cb238"} err="failed to get container status \"48dc6ab44528d0f5a5162fc8b89d173676851668e94c506a2ab0d5f16d0cb238\": rpc error: code = NotFound desc = could not find container \"48dc6ab44528d0f5a5162fc8b89d173676851668e94c506a2ab0d5f16d0cb238\": container with ID starting with 48dc6ab44528d0f5a5162fc8b89d173676851668e94c506a2ab0d5f16d0cb238 not found: ID does not exist" Oct 30 13:05:13.389163 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:13.389107 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfba1837-8pwmd" podStartSLOduration=1.389093039 podStartE2EDuration="1.389093039s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:05:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:05:13.3852499 +0000 UTC m=+4269.899678643" watchObservedRunningTime="2026-10-30 13:05:13.389093039 +0000 UTC m=+4269.903522044" Oct 30 13:05:13.425231 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:13.425195 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-63f4e4-openshift-default-786cd844c9-lxsbx"] Oct 30 13:05:13.435816 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:13.435789 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-63f4e4-openshift-default-786cd844c9-lxsbx"] Oct 30 13:05:13.616513 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:13.616429 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1ca551b-2beb-407b-9e9c-e410199f6f33" path="/var/lib/kubelet/pods/f1ca551b-2beb-407b-9e9c-e410199f6f33/volumes" Oct 30 13:05:14.335810 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:14.335771 4908 generic.go:356] "Generic (PLEG): container finished" podID="8e77e666-f695-41d9-9243-d845872f235c" containerID="5ab285593f719a6dce287e347205aef7bc6056f1d93cbf09d1d5b71deb1feeb5" exitCode=0 Oct 30 13:05:14.336265 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:14.335862 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfba1837-8pwmd" event={"ID":"8e77e666-f695-41d9-9243-d845872f235c","Type":"ContainerDied","Data":"5ab285593f719a6dce287e347205aef7bc6056f1d93cbf09d1d5b71deb1feeb5"} Oct 30 13:05:15.520925 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:15.520897 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfba1837-8pwmd" Oct 30 13:05:15.618957 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:15.618819 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e77e666-f695-41d9-9243-d845872f235c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e77e666-f695-41d9-9243-d845872f235c-maas-api-openshift-ca\") pod \"8e77e666-f695-41d9-9243-d845872f235c\" (UID: \"8e77e666-f695-41d9-9243-d845872f235c\") " Oct 30 13:05:15.619139 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:15.618990 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8e77e666-f695-41d9-9243-d845872f235c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e77e666-f695-41d9-9243-d845872f235c-maas-api-xks-ca\") pod \"8e77e666-f695-41d9-9243-d845872f235c\" (UID: \"8e77e666-f695-41d9-9243-d845872f235c\") " Oct 30 13:05:15.619139 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:15.619092 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e77e666-f695-41d9-9243-d845872f235c-kube-api-access-jw5cr\" (UniqueName: \"kubernetes.io/projected/8e77e666-f695-41d9-9243-d845872f235c-kube-api-access-jw5cr\") pod \"8e77e666-f695-41d9-9243-d845872f235c\" (UID: \"8e77e666-f695-41d9-9243-d845872f235c\") " Oct 30 13:05:15.619268 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:15.619201 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e77e666-f695-41d9-9243-d845872f235c-maas-api-openshift-ca" pod "8e77e666-f695-41d9-9243-d845872f235c" (UID: "8e77e666-f695-41d9-9243-d845872f235c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:05:15.619616 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:15.619589 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e77e666-f695-41d9-9243-d845872f235c-maas-api-openshift-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:05:15.621292 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:15.621266 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e77e666-f695-41d9-9243-d845872f235c-maas-api-xks-ca" pod "8e77e666-f695-41d9-9243-d845872f235c" (UID: "8e77e666-f695-41d9-9243-d845872f235c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:05:15.621417 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:15.621306 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e77e666-f695-41d9-9243-d845872f235c-kube-api-access-jw5cr" pod "8e77e666-f695-41d9-9243-d845872f235c" (UID: "8e77e666-f695-41d9-9243-d845872f235c"). InnerVolumeSpecName "kube-api-access-jw5cr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:05:15.720674 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:15.720637 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e77e666-f695-41d9-9243-d845872f235c-maas-api-xks-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:05:15.720674 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:15.720670 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jw5cr\" (UniqueName: \"kubernetes.io/projected/8e77e666-f695-41d9-9243-d845872f235c-kube-api-access-jw5cr\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:05:16.353166 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:16.353123 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfba1837-8pwmd" event={"ID":"8e77e666-f695-41d9-9243-d845872f235c","Type":"ContainerDied","Data":"4730b9d9100ef8bb8d130c5fd070617608db47b354fb9e3f3d5984d14c70ea45"} Oct 30 13:05:16.353333 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:16.353185 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4730b9d9100ef8bb8d130c5fd070617608db47b354fb9e3f3d5984d14c70ea45" Oct 30 13:05:16.353333 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:16.353138 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfba1837-8pwmd" Oct 30 13:05:18.951995 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:18.951955 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-6b67f8-openshift-default-696ff8b6f-6t5q2"] Oct 30 13:05:18.952439 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:18.952224 4908 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-6b67f8-openshift-default-696ff8b6f-6t5q2" podUID="0ab626b9-027f-4799-ba80-e47276070fdb" containerName="istio-proxy" containerID="cri-o://f5aca2adf11b10aa6418e97b0c907a75af90a650dd5fe67b5c443a53b26f7af3" gracePeriod=30 Oct 30 13:05:19.563964 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:19.563921 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-befaa6-qp8mv"] Oct 30 13:05:19.564773 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:19.564744 4908 cpu_manager.go:414] "containerMap: removing container" podUID="f1ca551b-2beb-407b-9e9c-e410199f6f33" containerName="istio-proxy" Oct 30 13:05:19.564864 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:19.564778 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1ca551b-2beb-407b-9e9c-e410199f6f33" containerName="istio-proxy" Oct 30 13:05:19.564942 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:19.564887 4908 cpu_manager.go:414] "containerMap: removing container" podUID="8e77e666-f695-41d9-9243-d845872f235c" containerName="revoke-keys" Oct 30 13:05:19.564942 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:19.564898 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e77e666-f695-41d9-9243-d845872f235c" containerName="revoke-keys" Oct 30 13:05:19.565148 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:19.565132 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1ca551b-2beb-407b-9e9c-e410199f6f33" containerName="istio-proxy" Oct 30 13:05:19.565195 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:19.565153 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e77e666-f695-41d9-9243-d845872f235c" containerName="revoke-keys" Oct 30 13:05:19.565195 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:19.565168 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1ca551b-2beb-407b-9e9c-e410199f6f33" containerName="istio-proxy" Oct 30 13:05:19.569471 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:19.569443 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-befaa6-qp8mv" Oct 30 13:05:19.594434 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:19.593907 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-befaa6-qp8mv"] Oct 30 13:05:19.617424 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:19.617384 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 13:05:19.617723 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:19.617552 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 13:05:19.648082 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:19.648038 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:05:19.648082 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:19.648082 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s75hq\"" Oct 30 13:05:19.687422 ip-10-0-141-251 kubenswrapper[4908]: E1030 13:05:19.687386 4908 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vn6vv" containerName="extractor" Oct 30 13:05:19.706872 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:19.706806 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a22eb470-fdda-42a0-8204-3a218401efba-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-befaa6-qp8mv\" (UID: \"a22eb470-fdda-42a0-8204-3a218401efba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-befaa6-qp8mv" Oct 30 13:05:19.707080 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:19.706987 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a22eb470-fdda-42a0-8204-3a218401efba-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-befaa6-qp8mv\" (UID: \"a22eb470-fdda-42a0-8204-3a218401efba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-befaa6-qp8mv" Oct 30 13:05:19.707154 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:19.707135 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26z9k\" (UniqueName: \"kubernetes.io/projected/a22eb470-fdda-42a0-8204-3a218401efba-kube-api-access-26z9k\") pod \"maas-api-revoke-keys-e2e-inf-b-befaa6-qp8mv\" (UID: \"a22eb470-fdda-42a0-8204-3a218401efba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-befaa6-qp8mv" Oct 30 13:05:19.826460 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:19.825655 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a22eb470-fdda-42a0-8204-3a218401efba-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-befaa6-qp8mv\" (UID: \"a22eb470-fdda-42a0-8204-3a218401efba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-befaa6-qp8mv" Oct 30 13:05:19.826460 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:19.825753 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a22eb470-fdda-42a0-8204-3a218401efba-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-befaa6-qp8mv\" (UID: \"a22eb470-fdda-42a0-8204-3a218401efba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-befaa6-qp8mv" Oct 30 13:05:19.826460 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:19.825903 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26z9k\" (UniqueName: \"kubernetes.io/projected/a22eb470-fdda-42a0-8204-3a218401efba-kube-api-access-26z9k\") pod \"maas-api-revoke-keys-e2e-inf-b-befaa6-qp8mv\" (UID: \"a22eb470-fdda-42a0-8204-3a218401efba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-befaa6-qp8mv" Oct 30 13:05:19.831081 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:19.827626 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a22eb470-fdda-42a0-8204-3a218401efba-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-befaa6-qp8mv\" (UID: \"a22eb470-fdda-42a0-8204-3a218401efba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-befaa6-qp8mv" Oct 30 13:05:19.839676 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:19.839570 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a22eb470-fdda-42a0-8204-3a218401efba-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-befaa6-qp8mv\" (UID: \"a22eb470-fdda-42a0-8204-3a218401efba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-befaa6-qp8mv" Oct 30 13:05:19.849705 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:19.849675 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26z9k\" (UniqueName: \"kubernetes.io/projected/a22eb470-fdda-42a0-8204-3a218401efba-kube-api-access-26z9k\") pod \"maas-api-revoke-keys-e2e-inf-b-befaa6-qp8mv\" (UID: \"a22eb470-fdda-42a0-8204-3a218401efba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-befaa6-qp8mv" Oct 30 13:05:19.904372 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:19.904327 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-befaa6-qp8mv" Oct 30 13:05:20.219490 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:20.219460 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-befaa6-qp8mv"] Oct 30 13:05:20.221149 ip-10-0-141-251 kubenswrapper[4908]: W1030 13:05:20.221121 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda22eb470_fdda_42a0_8204_3a218401efba.slice/crio-b181009f8dbf30f1af146322115b3eeb306e7eddb2999ebf577ef646d25ad7e9 WatchSource:0}: Error finding container b181009f8dbf30f1af146322115b3eeb306e7eddb2999ebf577ef646d25ad7e9: Status 404 returned error can't find the container with id b181009f8dbf30f1af146322115b3eeb306e7eddb2999ebf577ef646d25ad7e9 Oct 30 13:05:20.393378 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:20.393333 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-befaa6-qp8mv" event={"ID":"a22eb470-fdda-42a0-8204-3a218401efba","Type":"ContainerStarted","Data":"b181009f8dbf30f1af146322115b3eeb306e7eddb2999ebf577ef646d25ad7e9"} Oct 30 13:05:21.325520 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:21.325482 4908 patch_prober.go:29] interesting pod/e2e-inf-a-6b67f8-openshift-default-696ff8b6f-6t5q2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 13:05:21.325937 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:21.325570 4908 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-6b67f8-openshift-default-696ff8b6f-6t5q2" podUID="0ab626b9-027f-4799-ba80-e47276070fdb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 13:05:21.406097 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:21.405927 4908 generic.go:356] "Generic (PLEG): container finished" podID="a22eb470-fdda-42a0-8204-3a218401efba" containerID="c975ed37ffbbf8eddddf96c5e6c2e2dfe355b28490eca3d237ea5c1cfb7b4fa2" exitCode=0 Oct 30 13:05:21.406097 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:21.406010 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-befaa6-qp8mv" event={"ID":"a22eb470-fdda-42a0-8204-3a218401efba","Type":"ContainerDied","Data":"c975ed37ffbbf8eddddf96c5e6c2e2dfe355b28490eca3d237ea5c1cfb7b4fa2"} Oct 30 13:05:22.570637 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:22.570604 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-befaa6-qp8mv" Oct 30 13:05:22.664199 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:22.664166 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a22eb470-fdda-42a0-8204-3a218401efba-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a22eb470-fdda-42a0-8204-3a218401efba-maas-api-xks-ca\") pod \"a22eb470-fdda-42a0-8204-3a218401efba\" (UID: \"a22eb470-fdda-42a0-8204-3a218401efba\") " Oct 30 13:05:22.664357 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:22.664266 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a22eb470-fdda-42a0-8204-3a218401efba-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a22eb470-fdda-42a0-8204-3a218401efba-maas-api-openshift-ca\") pod \"a22eb470-fdda-42a0-8204-3a218401efba\" (UID: \"a22eb470-fdda-42a0-8204-3a218401efba\") " Oct 30 13:05:22.664357 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:22.664284 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a22eb470-fdda-42a0-8204-3a218401efba-kube-api-access-26z9k\" (UniqueName: \"kubernetes.io/projected/a22eb470-fdda-42a0-8204-3a218401efba-kube-api-access-26z9k\") pod \"a22eb470-fdda-42a0-8204-3a218401efba\" (UID: \"a22eb470-fdda-42a0-8204-3a218401efba\") " Oct 30 13:05:22.664929 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:22.664672 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a22eb470-fdda-42a0-8204-3a218401efba-maas-api-openshift-ca" pod "a22eb470-fdda-42a0-8204-3a218401efba" (UID: "a22eb470-fdda-42a0-8204-3a218401efba"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:05:22.665066 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:22.665028 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a22eb470-fdda-42a0-8204-3a218401efba-maas-api-openshift-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:05:22.666917 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:22.666896 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a22eb470-fdda-42a0-8204-3a218401efba-maas-api-xks-ca" pod "a22eb470-fdda-42a0-8204-3a218401efba" (UID: "a22eb470-fdda-42a0-8204-3a218401efba"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:05:22.667020 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:22.666986 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a22eb470-fdda-42a0-8204-3a218401efba-kube-api-access-26z9k" pod "a22eb470-fdda-42a0-8204-3a218401efba" (UID: "a22eb470-fdda-42a0-8204-3a218401efba"). InnerVolumeSpecName "kube-api-access-26z9k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:05:22.766267 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:22.766224 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a22eb470-fdda-42a0-8204-3a218401efba-maas-api-xks-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:05:22.766267 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:22.766259 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26z9k\" (UniqueName: \"kubernetes.io/projected/a22eb470-fdda-42a0-8204-3a218401efba-kube-api-access-26z9k\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:05:23.136853 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:23.128491 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c55470c-scdtx"] Oct 30 13:05:23.136853 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:23.129447 4908 cpu_manager.go:414] "containerMap: removing container" podUID="f1ca551b-2beb-407b-9e9c-e410199f6f33" containerName="istio-proxy" Oct 30 13:05:23.136853 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:23.129473 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1ca551b-2beb-407b-9e9c-e410199f6f33" containerName="istio-proxy" Oct 30 13:05:23.136853 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:23.129596 4908 cpu_manager.go:414] "containerMap: removing container" podUID="a22eb470-fdda-42a0-8204-3a218401efba" containerName="revoke-keys" Oct 30 13:05:23.136853 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:23.129610 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a22eb470-fdda-42a0-8204-3a218401efba" containerName="revoke-keys" Oct 30 13:05:23.136853 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:23.135963 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="a22eb470-fdda-42a0-8204-3a218401efba" containerName="revoke-keys" Oct 30 13:05:23.146855 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:23.144660 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c55470c-scdtx" Oct 30 13:05:23.179263 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:23.178127 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff9664e3-b790-4a5e-b96a-5553ef1189d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6c55470c-scdtx\" (UID: \"ff9664e3-b790-4a5e-b96a-5553ef1189d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c55470c-scdtx" Oct 30 13:05:23.179263 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:23.178216 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59gq5\" (UniqueName: \"kubernetes.io/projected/ff9664e3-b790-4a5e-b96a-5553ef1189d8-kube-api-access-59gq5\") pod \"maas-api-revoke-sub-6c55470c-scdtx\" (UID: \"ff9664e3-b790-4a5e-b96a-5553ef1189d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c55470c-scdtx" Oct 30 13:05:23.179263 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:23.178273 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff9664e3-b790-4a5e-b96a-5553ef1189d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6c55470c-scdtx\" (UID: \"ff9664e3-b790-4a5e-b96a-5553ef1189d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c55470c-scdtx" Oct 30 13:05:23.251853 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:23.248169 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c55470c-scdtx"] Oct 30 13:05:23.263925 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:23.253864 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2eac5b06-dd7vs"] Oct 30 13:05:23.284297 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:23.284254 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2eac5b06-dd7vs" Oct 30 13:05:23.312060 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:23.310658 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59gq5\" (UniqueName: \"kubernetes.io/projected/ff9664e3-b790-4a5e-b96a-5553ef1189d8-kube-api-access-59gq5\") pod \"maas-api-revoke-sub-6c55470c-scdtx\" (UID: \"ff9664e3-b790-4a5e-b96a-5553ef1189d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c55470c-scdtx" Oct 30 13:05:23.312060 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:23.310742 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff9664e3-b790-4a5e-b96a-5553ef1189d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6c55470c-scdtx\" (UID: \"ff9664e3-b790-4a5e-b96a-5553ef1189d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c55470c-scdtx" Oct 30 13:05:23.312060 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:23.310875 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4558943a-c9e0-4649-896c-e78d2fe1d25f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2eac5b06-dd7vs\" (UID: \"4558943a-c9e0-4649-896c-e78d2fe1d25f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2eac5b06-dd7vs" Oct 30 13:05:23.312060 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:23.311077 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjq7h\" (UniqueName: \"kubernetes.io/projected/4558943a-c9e0-4649-896c-e78d2fe1d25f-kube-api-access-hjq7h\") pod \"maas-api-revoke-sub-2eac5b06-dd7vs\" (UID: \"4558943a-c9e0-4649-896c-e78d2fe1d25f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2eac5b06-dd7vs" Oct 30 13:05:23.312060 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:23.311145 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff9664e3-b790-4a5e-b96a-5553ef1189d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6c55470c-scdtx\" (UID: \"ff9664e3-b790-4a5e-b96a-5553ef1189d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c55470c-scdtx" Oct 30 13:05:23.312060 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:23.311185 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4558943a-c9e0-4649-896c-e78d2fe1d25f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2eac5b06-dd7vs\" (UID: \"4558943a-c9e0-4649-896c-e78d2fe1d25f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2eac5b06-dd7vs" Oct 30 13:05:23.325945 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:23.325680 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff9664e3-b790-4a5e-b96a-5553ef1189d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6c55470c-scdtx\" (UID: \"ff9664e3-b790-4a5e-b96a-5553ef1189d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c55470c-scdtx" Oct 30 13:05:23.329206 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:23.329170 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff9664e3-b790-4a5e-b96a-5553ef1189d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6c55470c-scdtx\" (UID: \"ff9664e3-b790-4a5e-b96a-5553ef1189d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c55470c-scdtx" Oct 30 13:05:23.341638 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:23.339760 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-59gq5\" (UniqueName: \"kubernetes.io/projected/ff9664e3-b790-4a5e-b96a-5553ef1189d8-kube-api-access-59gq5\") pod \"maas-api-revoke-sub-6c55470c-scdtx\" (UID: \"ff9664e3-b790-4a5e-b96a-5553ef1189d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c55470c-scdtx" Oct 30 13:05:23.342799 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:23.342774 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2eac5b06-dd7vs"] Oct 30 13:05:23.412101 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:23.412010 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4558943a-c9e0-4649-896c-e78d2fe1d25f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2eac5b06-dd7vs\" (UID: \"4558943a-c9e0-4649-896c-e78d2fe1d25f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2eac5b06-dd7vs" Oct 30 13:05:23.412331 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:23.412152 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjq7h\" (UniqueName: \"kubernetes.io/projected/4558943a-c9e0-4649-896c-e78d2fe1d25f-kube-api-access-hjq7h\") pod \"maas-api-revoke-sub-2eac5b06-dd7vs\" (UID: \"4558943a-c9e0-4649-896c-e78d2fe1d25f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2eac5b06-dd7vs" Oct 30 13:05:23.412331 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:23.412209 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4558943a-c9e0-4649-896c-e78d2fe1d25f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2eac5b06-dd7vs\" (UID: \"4558943a-c9e0-4649-896c-e78d2fe1d25f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2eac5b06-dd7vs" Oct 30 13:05:23.412945 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:23.412917 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4558943a-c9e0-4649-896c-e78d2fe1d25f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2eac5b06-dd7vs\" (UID: \"4558943a-c9e0-4649-896c-e78d2fe1d25f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2eac5b06-dd7vs" Oct 30 13:05:23.414821 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:23.414795 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4558943a-c9e0-4649-896c-e78d2fe1d25f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2eac5b06-dd7vs\" (UID: \"4558943a-c9e0-4649-896c-e78d2fe1d25f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2eac5b06-dd7vs" Oct 30 13:05:23.423868 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:23.423813 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-befaa6-qp8mv" Oct 30 13:05:23.423985 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:23.423838 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-befaa6-qp8mv" event={"ID":"a22eb470-fdda-42a0-8204-3a218401efba","Type":"ContainerDied","Data":"b181009f8dbf30f1af146322115b3eeb306e7eddb2999ebf577ef646d25ad7e9"} Oct 30 13:05:23.424049 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:23.423985 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b181009f8dbf30f1af146322115b3eeb306e7eddb2999ebf577ef646d25ad7e9" Oct 30 13:05:23.434889 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:23.434866 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjq7h\" (UniqueName: \"kubernetes.io/projected/4558943a-c9e0-4649-896c-e78d2fe1d25f-kube-api-access-hjq7h\") pod \"maas-api-revoke-sub-2eac5b06-dd7vs\" (UID: \"4558943a-c9e0-4649-896c-e78d2fe1d25f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2eac5b06-dd7vs" Oct 30 13:05:23.525551 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:23.525517 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c55470c-scdtx" Oct 30 13:05:23.639501 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:23.639462 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2eac5b06-dd7vs" Oct 30 13:05:23.691308 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:23.691217 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-479ba8c0-pg9qf"] Oct 30 13:05:23.698349 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:23.698320 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-479ba8c0-pg9qf" Oct 30 13:05:23.771440 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:23.771398 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-479ba8c0-pg9qf"] Oct 30 13:05:23.821530 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:23.821493 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5hfk\" (UniqueName: \"kubernetes.io/projected/e5c2ee3e-6917-4773-8bab-76eaa9f1d9ed-kube-api-access-w5hfk\") pod \"maas-api-revoke-sub-479ba8c0-pg9qf\" (UID: \"e5c2ee3e-6917-4773-8bab-76eaa9f1d9ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-479ba8c0-pg9qf" Oct 30 13:05:23.821714 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:23.821644 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5c2ee3e-6917-4773-8bab-76eaa9f1d9ed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-479ba8c0-pg9qf\" (UID: \"e5c2ee3e-6917-4773-8bab-76eaa9f1d9ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-479ba8c0-pg9qf" Oct 30 13:05:23.821766 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:23.821742 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5c2ee3e-6917-4773-8bab-76eaa9f1d9ed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-479ba8c0-pg9qf\" (UID: \"e5c2ee3e-6917-4773-8bab-76eaa9f1d9ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-479ba8c0-pg9qf" Oct 30 13:05:23.847167 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:23.847139 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c55470c-scdtx"] Oct 30 13:05:23.849068 ip-10-0-141-251 kubenswrapper[4908]: W1030 13:05:23.849035 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff9664e3_b790_4a5e_b96a_5553ef1189d8.slice/crio-8d0938d49d7240ef2fd7583f036c8decd6627b6da95abfe7b9032d5d93efa320 WatchSource:0}: Error finding container 8d0938d49d7240ef2fd7583f036c8decd6627b6da95abfe7b9032d5d93efa320: Status 404 returned error can't find the container with id 8d0938d49d7240ef2fd7583f036c8decd6627b6da95abfe7b9032d5d93efa320 Oct 30 13:05:23.923231 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:23.923195 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5c2ee3e-6917-4773-8bab-76eaa9f1d9ed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-479ba8c0-pg9qf\" (UID: \"e5c2ee3e-6917-4773-8bab-76eaa9f1d9ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-479ba8c0-pg9qf" Oct 30 13:05:23.923424 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:23.923287 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5c2ee3e-6917-4773-8bab-76eaa9f1d9ed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-479ba8c0-pg9qf\" (UID: \"e5c2ee3e-6917-4773-8bab-76eaa9f1d9ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-479ba8c0-pg9qf" Oct 30 13:05:23.923514 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:23.923488 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5hfk\" (UniqueName: \"kubernetes.io/projected/e5c2ee3e-6917-4773-8bab-76eaa9f1d9ed-kube-api-access-w5hfk\") pod \"maas-api-revoke-sub-479ba8c0-pg9qf\" (UID: \"e5c2ee3e-6917-4773-8bab-76eaa9f1d9ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-479ba8c0-pg9qf" Oct 30 13:05:23.923939 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:23.923914 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5c2ee3e-6917-4773-8bab-76eaa9f1d9ed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-479ba8c0-pg9qf\" (UID: \"e5c2ee3e-6917-4773-8bab-76eaa9f1d9ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-479ba8c0-pg9qf" Oct 30 13:05:23.925643 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:23.925615 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5c2ee3e-6917-4773-8bab-76eaa9f1d9ed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-479ba8c0-pg9qf\" (UID: \"e5c2ee3e-6917-4773-8bab-76eaa9f1d9ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-479ba8c0-pg9qf" Oct 30 13:05:23.948969 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:23.948901 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5hfk\" (UniqueName: \"kubernetes.io/projected/e5c2ee3e-6917-4773-8bab-76eaa9f1d9ed-kube-api-access-w5hfk\") pod \"maas-api-revoke-sub-479ba8c0-pg9qf\" (UID: \"e5c2ee3e-6917-4773-8bab-76eaa9f1d9ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-479ba8c0-pg9qf" Oct 30 13:05:24.030435 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:24.030407 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2eac5b06-dd7vs"] Oct 30 13:05:24.060125 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:24.060016 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-479ba8c0-pg9qf" Oct 30 13:05:24.062240 ip-10-0-141-251 kubenswrapper[4908]: W1030 13:05:24.062201 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4558943a_c9e0_4649_896c_e78d2fe1d25f.slice/crio-764cbe88f08e684dc93d6118d4716d515990434e1163eb4fb136eb97ba01987a WatchSource:0}: Error finding container 764cbe88f08e684dc93d6118d4716d515990434e1163eb4fb136eb97ba01987a: Status 404 returned error can't find the container with id 764cbe88f08e684dc93d6118d4716d515990434e1163eb4fb136eb97ba01987a Oct 30 13:05:24.291654 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:24.291629 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6b67f8-openshift-default-696ff8b6f-6t5q2" Oct 30 13:05:24.394168 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:24.394132 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-479ba8c0-pg9qf"] Oct 30 13:05:24.397708 ip-10-0-141-251 kubenswrapper[4908]: W1030 13:05:24.397676 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5c2ee3e_6917_4773_8bab_76eaa9f1d9ed.slice/crio-0adbfa2a85907f1dc6f0c55dd2bd7f5e1b00d9aa6d8d75daeffb58f73d5b5403 WatchSource:0}: Error finding container 0adbfa2a85907f1dc6f0c55dd2bd7f5e1b00d9aa6d8d75daeffb58f73d5b5403: Status 404 returned error can't find the container with id 0adbfa2a85907f1dc6f0c55dd2bd7f5e1b00d9aa6d8d75daeffb58f73d5b5403 Oct 30 13:05:24.429569 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:24.429544 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ab626b9-027f-4799-ba80-e47276070fdb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ab626b9-027f-4799-ba80-e47276070fdb-workload-certs\") pod \"0ab626b9-027f-4799-ba80-e47276070fdb\" (UID: \"0ab626b9-027f-4799-ba80-e47276070fdb\") " Oct 30 13:05:24.429697 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:24.429624 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ab626b9-027f-4799-ba80-e47276070fdb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ab626b9-027f-4799-ba80-e47276070fdb-istio-envoy\") pod \"0ab626b9-027f-4799-ba80-e47276070fdb\" (UID: \"0ab626b9-027f-4799-ba80-e47276070fdb\") " Oct 30 13:05:24.429697 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:24.429673 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ab626b9-027f-4799-ba80-e47276070fdb-kube-api-access-bq5hv\" (UniqueName: \"kubernetes.io/projected/0ab626b9-027f-4799-ba80-e47276070fdb-kube-api-access-bq5hv\") pod \"0ab626b9-027f-4799-ba80-e47276070fdb\" (UID: \"0ab626b9-027f-4799-ba80-e47276070fdb\") " Oct 30 13:05:24.429796 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:24.429732 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ab626b9-027f-4799-ba80-e47276070fdb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ab626b9-027f-4799-ba80-e47276070fdb-workload-socket\") pod \"0ab626b9-027f-4799-ba80-e47276070fdb\" (UID: \"0ab626b9-027f-4799-ba80-e47276070fdb\") " Oct 30 13:05:24.429796 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:24.429769 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ab626b9-027f-4799-ba80-e47276070fdb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ab626b9-027f-4799-ba80-e47276070fdb-istio-data\") pod \"0ab626b9-027f-4799-ba80-e47276070fdb\" (UID: \"0ab626b9-027f-4799-ba80-e47276070fdb\") " Oct 30 13:05:24.429906 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:24.429816 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0ab626b9-027f-4799-ba80-e47276070fdb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ab626b9-027f-4799-ba80-e47276070fdb-istio-podinfo\") pod \"0ab626b9-027f-4799-ba80-e47276070fdb\" (UID: \"0ab626b9-027f-4799-ba80-e47276070fdb\") " Oct 30 13:05:24.429906 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:24.429874 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ab626b9-027f-4799-ba80-e47276070fdb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ab626b9-027f-4799-ba80-e47276070fdb-istiod-ca-cert\") pod \"0ab626b9-027f-4799-ba80-e47276070fdb\" (UID: \"0ab626b9-027f-4799-ba80-e47276070fdb\") " Oct 30 13:05:24.430002 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:24.429936 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ab626b9-027f-4799-ba80-e47276070fdb-workload-certs" pod "0ab626b9-027f-4799-ba80-e47276070fdb" (UID: "0ab626b9-027f-4799-ba80-e47276070fdb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:05:24.430002 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:24.429957 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ab626b9-027f-4799-ba80-e47276070fdb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ab626b9-027f-4799-ba80-e47276070fdb-credential-socket\") pod \"0ab626b9-027f-4799-ba80-e47276070fdb\" (UID: \"0ab626b9-027f-4799-ba80-e47276070fdb\") " Oct 30 13:05:24.430085 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:24.430050 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ab626b9-027f-4799-ba80-e47276070fdb-istio-token\" (UniqueName: \"kubernetes.io/projected/0ab626b9-027f-4799-ba80-e47276070fdb-istio-token\") pod \"0ab626b9-027f-4799-ba80-e47276070fdb\" (UID: \"0ab626b9-027f-4799-ba80-e47276070fdb\") " Oct 30 13:05:24.430205 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:24.430180 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ab626b9-027f-4799-ba80-e47276070fdb-credential-socket" pod "0ab626b9-027f-4799-ba80-e47276070fdb" (UID: "0ab626b9-027f-4799-ba80-e47276070fdb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:05:24.430492 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:24.430473 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ab626b9-027f-4799-ba80-e47276070fdb-istiod-ca-cert" pod "0ab626b9-027f-4799-ba80-e47276070fdb" (UID: "0ab626b9-027f-4799-ba80-e47276070fdb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:05:24.430615 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:24.430485 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ab626b9-027f-4799-ba80-e47276070fdb-istio-data" pod "0ab626b9-027f-4799-ba80-e47276070fdb" (UID: "0ab626b9-027f-4799-ba80-e47276070fdb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:05:24.430727 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:24.430708 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ab626b9-027f-4799-ba80-e47276070fdb-workload-socket" pod "0ab626b9-027f-4799-ba80-e47276070fdb" (UID: "0ab626b9-027f-4799-ba80-e47276070fdb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:05:24.430807 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:24.430783 4908 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ab626b9-027f-4799-ba80-e47276070fdb-workload-certs\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:05:24.430934 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:24.430808 4908 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ab626b9-027f-4799-ba80-e47276070fdb-istio-data\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:05:24.430934 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:24.430821 4908 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ab626b9-027f-4799-ba80-e47276070fdb-istiod-ca-cert\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:05:24.430934 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:24.430850 4908 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ab626b9-027f-4799-ba80-e47276070fdb-credential-socket\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:05:24.432227 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:24.432201 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ab626b9-027f-4799-ba80-e47276070fdb-istio-envoy" pod "0ab626b9-027f-4799-ba80-e47276070fdb" (UID: "0ab626b9-027f-4799-ba80-e47276070fdb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:05:24.432510 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:24.432490 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0ab626b9-027f-4799-ba80-e47276070fdb-istio-podinfo" pod "0ab626b9-027f-4799-ba80-e47276070fdb" (UID: "0ab626b9-027f-4799-ba80-e47276070fdb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:05:24.433119 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:24.433093 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ab626b9-027f-4799-ba80-e47276070fdb-kube-api-access-bq5hv" pod "0ab626b9-027f-4799-ba80-e47276070fdb" (UID: "0ab626b9-027f-4799-ba80-e47276070fdb"). InnerVolumeSpecName "kube-api-access-bq5hv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:05:24.433298 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:24.433277 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ab626b9-027f-4799-ba80-e47276070fdb-istio-token" pod "0ab626b9-027f-4799-ba80-e47276070fdb" (UID: "0ab626b9-027f-4799-ba80-e47276070fdb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:05:24.434646 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:24.434613 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-479ba8c0-pg9qf" event={"ID":"e5c2ee3e-6917-4773-8bab-76eaa9f1d9ed","Type":"ContainerStarted","Data":"0adbfa2a85907f1dc6f0c55dd2bd7f5e1b00d9aa6d8d75daeffb58f73d5b5403"} Oct 30 13:05:24.436041 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:24.436014 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2eac5b06-dd7vs" event={"ID":"4558943a-c9e0-4649-896c-e78d2fe1d25f","Type":"ContainerStarted","Data":"764cbe88f08e684dc93d6118d4716d515990434e1163eb4fb136eb97ba01987a"} Oct 30 13:05:24.437690 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:24.437662 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c55470c-scdtx" event={"ID":"ff9664e3-b790-4a5e-b96a-5553ef1189d8","Type":"ContainerStarted","Data":"43bdc532b8d05302a74859f5c9506b08205208e63930d84d588488739874a80b"} Oct 30 13:05:24.437794 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:24.437710 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c55470c-scdtx" event={"ID":"ff9664e3-b790-4a5e-b96a-5553ef1189d8","Type":"ContainerStarted","Data":"8d0938d49d7240ef2fd7583f036c8decd6627b6da95abfe7b9032d5d93efa320"} Oct 30 13:05:24.439605 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:24.439582 4908 generic.go:356] "Generic (PLEG): container finished" podID="0ab626b9-027f-4799-ba80-e47276070fdb" containerID="f5aca2adf11b10aa6418e97b0c907a75af90a650dd5fe67b5c443a53b26f7af3" exitCode=0 Oct 30 13:05:24.439691 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:24.439626 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6b67f8-openshift-default-696ff8b6f-6t5q2" Oct 30 13:05:24.439741 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:24.439628 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6b67f8-openshift-default-696ff8b6f-6t5q2" event={"ID":"0ab626b9-027f-4799-ba80-e47276070fdb","Type":"ContainerDied","Data":"f5aca2adf11b10aa6418e97b0c907a75af90a650dd5fe67b5c443a53b26f7af3"} Oct 30 13:05:24.439789 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:24.439766 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6b67f8-openshift-default-696ff8b6f-6t5q2" event={"ID":"0ab626b9-027f-4799-ba80-e47276070fdb","Type":"ContainerDied","Data":"d9893206e43269a68e21332fd11dc6e67b2895f2e6c3236422617ec94e8bab60"} Oct 30 13:05:24.439872 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:24.439807 4908 scope.go:122] "RemoveContainer" containerID="f5aca2adf11b10aa6418e97b0c907a75af90a650dd5fe67b5c443a53b26f7af3" Oct 30 13:05:24.451639 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:24.451618 4908 scope.go:122] "RemoveContainer" containerID="3c174bcf714e7781bff416a5de8a3c29a0a190b20f792f857b4e573918540ef7" Oct 30 13:05:24.462933 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:24.462904 4908 scope.go:122] "RemoveContainer" containerID="f5aca2adf11b10aa6418e97b0c907a75af90a650dd5fe67b5c443a53b26f7af3" Oct 30 13:05:24.463318 ip-10-0-141-251 kubenswrapper[4908]: E1030 13:05:24.463290 4908 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5aca2adf11b10aa6418e97b0c907a75af90a650dd5fe67b5c443a53b26f7af3\": container with ID starting with f5aca2adf11b10aa6418e97b0c907a75af90a650dd5fe67b5c443a53b26f7af3 not found: ID does not exist" containerID="f5aca2adf11b10aa6418e97b0c907a75af90a650dd5fe67b5c443a53b26f7af3" Oct 30 13:05:24.463397 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:24.463333 4908 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5aca2adf11b10aa6418e97b0c907a75af90a650dd5fe67b5c443a53b26f7af3"} err="failed to get container status \"f5aca2adf11b10aa6418e97b0c907a75af90a650dd5fe67b5c443a53b26f7af3\": rpc error: code = NotFound desc = could not find container \"f5aca2adf11b10aa6418e97b0c907a75af90a650dd5fe67b5c443a53b26f7af3\": container with ID starting with f5aca2adf11b10aa6418e97b0c907a75af90a650dd5fe67b5c443a53b26f7af3 not found: ID does not exist" Oct 30 13:05:24.463397 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:24.463360 4908 scope.go:122] "RemoveContainer" containerID="3c174bcf714e7781bff416a5de8a3c29a0a190b20f792f857b4e573918540ef7" Oct 30 13:05:24.463674 ip-10-0-141-251 kubenswrapper[4908]: E1030 13:05:24.463653 4908 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c174bcf714e7781bff416a5de8a3c29a0a190b20f792f857b4e573918540ef7\": container with ID starting with 3c174bcf714e7781bff416a5de8a3c29a0a190b20f792f857b4e573918540ef7 not found: ID does not exist" containerID="3c174bcf714e7781bff416a5de8a3c29a0a190b20f792f857b4e573918540ef7" Oct 30 13:05:24.463745 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:24.463681 4908 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c174bcf714e7781bff416a5de8a3c29a0a190b20f792f857b4e573918540ef7"} err="failed to get container status \"3c174bcf714e7781bff416a5de8a3c29a0a190b20f792f857b4e573918540ef7\": rpc error: code = NotFound desc = could not find container \"3c174bcf714e7781bff416a5de8a3c29a0a190b20f792f857b4e573918540ef7\": container with ID starting with 3c174bcf714e7781bff416a5de8a3c29a0a190b20f792f857b4e573918540ef7 not found: ID does not exist" Oct 30 13:05:24.478125 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:24.478073 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c55470c-scdtx" podStartSLOduration=1.478059765 podStartE2EDuration="1.478059765s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:05:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:05:24.475581604 +0000 UTC m=+4280.990010350" watchObservedRunningTime="2026-10-30 13:05:24.478059765 +0000 UTC m=+4280.992488510" Oct 30 13:05:24.514471 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:24.514429 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-6b67f8-openshift-default-696ff8b6f-6t5q2"] Oct 30 13:05:24.518512 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:24.518481 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-6b67f8-openshift-default-696ff8b6f-6t5q2"] Oct 30 13:05:24.532217 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:24.532124 4908 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ab626b9-027f-4799-ba80-e47276070fdb-istio-token\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:05:24.532217 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:24.532159 4908 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ab626b9-027f-4799-ba80-e47276070fdb-istio-envoy\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:05:24.532217 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:24.532189 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bq5hv\" (UniqueName: \"kubernetes.io/projected/0ab626b9-027f-4799-ba80-e47276070fdb-kube-api-access-bq5hv\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:05:24.532217 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:24.532203 4908 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ab626b9-027f-4799-ba80-e47276070fdb-workload-socket\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:05:24.532217 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:24.532218 4908 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ab626b9-027f-4799-ba80-e47276070fdb-istio-podinfo\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:05:25.450766 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:25.450733 4908 generic.go:356] "Generic (PLEG): container finished" podID="ff9664e3-b790-4a5e-b96a-5553ef1189d8" containerID="43bdc532b8d05302a74859f5c9506b08205208e63930d84d588488739874a80b" exitCode=0 Oct 30 13:05:25.451225 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:25.450815 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c55470c-scdtx" event={"ID":"ff9664e3-b790-4a5e-b96a-5553ef1189d8","Type":"ContainerDied","Data":"43bdc532b8d05302a74859f5c9506b08205208e63930d84d588488739874a80b"} Oct 30 13:05:25.453568 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:25.453538 4908 generic.go:356] "Generic (PLEG): container finished" podID="e5c2ee3e-6917-4773-8bab-76eaa9f1d9ed" containerID="d96e4deea0ed29f960d885f6d8f1167db1f49c74aebf5fc38a2ad417d0cf9887" exitCode=0 Oct 30 13:05:25.453703 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:25.453587 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-479ba8c0-pg9qf" event={"ID":"e5c2ee3e-6917-4773-8bab-76eaa9f1d9ed","Type":"ContainerDied","Data":"d96e4deea0ed29f960d885f6d8f1167db1f49c74aebf5fc38a2ad417d0cf9887"} Oct 30 13:05:25.455286 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:25.455264 4908 generic.go:356] "Generic (PLEG): container finished" podID="4558943a-c9e0-4649-896c-e78d2fe1d25f" containerID="b2b93451bc60c58e912083927128fde19a2423190a6188de997613222f166f00" exitCode=0 Oct 30 13:05:25.455410 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:25.455309 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2eac5b06-dd7vs" event={"ID":"4558943a-c9e0-4649-896c-e78d2fe1d25f","Type":"ContainerDied","Data":"b2b93451bc60c58e912083927128fde19a2423190a6188de997613222f166f00"} Oct 30 13:05:25.610248 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:25.610167 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ab626b9-027f-4799-ba80-e47276070fdb" path="/var/lib/kubelet/pods/0ab626b9-027f-4799-ba80-e47276070fdb/volumes" Oct 30 13:05:26.692136 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:26.692104 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c55470c-scdtx" Oct 30 13:05:26.720170 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:26.720099 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2eac5b06-dd7vs" Oct 30 13:05:26.723565 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:26.723537 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-479ba8c0-pg9qf" Oct 30 13:05:26.878220 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:26.878178 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff9664e3-b790-4a5e-b96a-5553ef1189d8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff9664e3-b790-4a5e-b96a-5553ef1189d8-maas-api-openshift-ca\") pod \"ff9664e3-b790-4a5e-b96a-5553ef1189d8\" (UID: \"ff9664e3-b790-4a5e-b96a-5553ef1189d8\") " Oct 30 13:05:26.878220 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:26.878222 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e5c2ee3e-6917-4773-8bab-76eaa9f1d9ed-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5c2ee3e-6917-4773-8bab-76eaa9f1d9ed-maas-api-xks-ca\") pod \"e5c2ee3e-6917-4773-8bab-76eaa9f1d9ed\" (UID: \"e5c2ee3e-6917-4773-8bab-76eaa9f1d9ed\") " Oct 30 13:05:26.878506 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:26.878290 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4558943a-c9e0-4649-896c-e78d2fe1d25f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4558943a-c9e0-4649-896c-e78d2fe1d25f-maas-api-openshift-ca\") pod \"4558943a-c9e0-4649-896c-e78d2fe1d25f\" (UID: \"4558943a-c9e0-4649-896c-e78d2fe1d25f\") " Oct 30 13:05:26.878506 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:26.878315 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5c2ee3e-6917-4773-8bab-76eaa9f1d9ed-kube-api-access-w5hfk\" (UniqueName: \"kubernetes.io/projected/e5c2ee3e-6917-4773-8bab-76eaa9f1d9ed-kube-api-access-w5hfk\") pod \"e5c2ee3e-6917-4773-8bab-76eaa9f1d9ed\" (UID: \"e5c2ee3e-6917-4773-8bab-76eaa9f1d9ed\") " Oct 30 13:05:26.878506 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:26.878354 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff9664e3-b790-4a5e-b96a-5553ef1189d8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff9664e3-b790-4a5e-b96a-5553ef1189d8-maas-api-xks-ca\") pod \"ff9664e3-b790-4a5e-b96a-5553ef1189d8\" (UID: \"ff9664e3-b790-4a5e-b96a-5553ef1189d8\") " Oct 30 13:05:26.878506 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:26.878378 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4558943a-c9e0-4649-896c-e78d2fe1d25f-kube-api-access-hjq7h\" (UniqueName: \"kubernetes.io/projected/4558943a-c9e0-4649-896c-e78d2fe1d25f-kube-api-access-hjq7h\") pod \"4558943a-c9e0-4649-896c-e78d2fe1d25f\" (UID: \"4558943a-c9e0-4649-896c-e78d2fe1d25f\") " Oct 30 13:05:26.878506 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:26.878420 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff9664e3-b790-4a5e-b96a-5553ef1189d8-kube-api-access-59gq5\" (UniqueName: \"kubernetes.io/projected/ff9664e3-b790-4a5e-b96a-5553ef1189d8-kube-api-access-59gq5\") pod \"ff9664e3-b790-4a5e-b96a-5553ef1189d8\" (UID: \"ff9664e3-b790-4a5e-b96a-5553ef1189d8\") " Oct 30 13:05:26.878791 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:26.878522 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4558943a-c9e0-4649-896c-e78d2fe1d25f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4558943a-c9e0-4649-896c-e78d2fe1d25f-maas-api-xks-ca\") pod \"4558943a-c9e0-4649-896c-e78d2fe1d25f\" (UID: \"4558943a-c9e0-4649-896c-e78d2fe1d25f\") " Oct 30 13:05:26.878791 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:26.878552 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5c2ee3e-6917-4773-8bab-76eaa9f1d9ed-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5c2ee3e-6917-4773-8bab-76eaa9f1d9ed-maas-api-openshift-ca\") pod \"e5c2ee3e-6917-4773-8bab-76eaa9f1d9ed\" (UID: \"e5c2ee3e-6917-4773-8bab-76eaa9f1d9ed\") " Oct 30 13:05:26.878791 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:26.878621 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff9664e3-b790-4a5e-b96a-5553ef1189d8-maas-api-openshift-ca" pod "ff9664e3-b790-4a5e-b96a-5553ef1189d8" (UID: "ff9664e3-b790-4a5e-b96a-5553ef1189d8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:05:26.878791 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:26.878705 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4558943a-c9e0-4649-896c-e78d2fe1d25f-maas-api-openshift-ca" pod "4558943a-c9e0-4649-896c-e78d2fe1d25f" (UID: "4558943a-c9e0-4649-896c-e78d2fe1d25f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:05:26.879142 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:26.879106 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5c2ee3e-6917-4773-8bab-76eaa9f1d9ed-maas-api-openshift-ca" pod "e5c2ee3e-6917-4773-8bab-76eaa9f1d9ed" (UID: "e5c2ee3e-6917-4773-8bab-76eaa9f1d9ed"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:05:26.879245 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:26.879172 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff9664e3-b790-4a5e-b96a-5553ef1189d8-maas-api-openshift-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:05:26.879245 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:26.879196 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4558943a-c9e0-4649-896c-e78d2fe1d25f-maas-api-openshift-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:05:26.881193 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:26.881159 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5c2ee3e-6917-4773-8bab-76eaa9f1d9ed-maas-api-xks-ca" pod "e5c2ee3e-6917-4773-8bab-76eaa9f1d9ed" (UID: "e5c2ee3e-6917-4773-8bab-76eaa9f1d9ed"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:05:26.881541 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:26.881514 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff9664e3-b790-4a5e-b96a-5553ef1189d8-kube-api-access-59gq5" pod "ff9664e3-b790-4a5e-b96a-5553ef1189d8" (UID: "ff9664e3-b790-4a5e-b96a-5553ef1189d8"). InnerVolumeSpecName "kube-api-access-59gq5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:05:26.881664 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:26.881550 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5c2ee3e-6917-4773-8bab-76eaa9f1d9ed-kube-api-access-w5hfk" pod "e5c2ee3e-6917-4773-8bab-76eaa9f1d9ed" (UID: "e5c2ee3e-6917-4773-8bab-76eaa9f1d9ed"). InnerVolumeSpecName "kube-api-access-w5hfk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:05:26.882056 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:26.882025 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4558943a-c9e0-4649-896c-e78d2fe1d25f-maas-api-xks-ca" pod "4558943a-c9e0-4649-896c-e78d2fe1d25f" (UID: "4558943a-c9e0-4649-896c-e78d2fe1d25f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:05:26.882212 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:26.882163 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff9664e3-b790-4a5e-b96a-5553ef1189d8-maas-api-xks-ca" pod "ff9664e3-b790-4a5e-b96a-5553ef1189d8" (UID: "ff9664e3-b790-4a5e-b96a-5553ef1189d8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:05:26.882313 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:26.882295 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4558943a-c9e0-4649-896c-e78d2fe1d25f-kube-api-access-hjq7h" pod "4558943a-c9e0-4649-896c-e78d2fe1d25f" (UID: "4558943a-c9e0-4649-896c-e78d2fe1d25f"). InnerVolumeSpecName "kube-api-access-hjq7h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:05:26.980178 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:26.980138 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5c2ee3e-6917-4773-8bab-76eaa9f1d9ed-maas-api-xks-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:05:26.980178 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:26.980174 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w5hfk\" (UniqueName: \"kubernetes.io/projected/e5c2ee3e-6917-4773-8bab-76eaa9f1d9ed-kube-api-access-w5hfk\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:05:26.980178 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:26.980185 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff9664e3-b790-4a5e-b96a-5553ef1189d8-maas-api-xks-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:05:26.980516 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:26.980195 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hjq7h\" (UniqueName: \"kubernetes.io/projected/4558943a-c9e0-4649-896c-e78d2fe1d25f-kube-api-access-hjq7h\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:05:26.980516 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:26.980204 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-59gq5\" (UniqueName: \"kubernetes.io/projected/ff9664e3-b790-4a5e-b96a-5553ef1189d8-kube-api-access-59gq5\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:05:26.980516 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:26.980214 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4558943a-c9e0-4649-896c-e78d2fe1d25f-maas-api-xks-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:05:26.980516 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:26.980224 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5c2ee3e-6917-4773-8bab-76eaa9f1d9ed-maas-api-openshift-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:05:27.476232 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:27.476143 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-479ba8c0-pg9qf" Oct 30 13:05:27.476232 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:27.476172 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-479ba8c0-pg9qf" event={"ID":"e5c2ee3e-6917-4773-8bab-76eaa9f1d9ed","Type":"ContainerDied","Data":"0adbfa2a85907f1dc6f0c55dd2bd7f5e1b00d9aa6d8d75daeffb58f73d5b5403"} Oct 30 13:05:27.476432 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:27.476261 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0adbfa2a85907f1dc6f0c55dd2bd7f5e1b00d9aa6d8d75daeffb58f73d5b5403" Oct 30 13:05:27.477996 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:27.477964 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2eac5b06-dd7vs" Oct 30 13:05:27.478150 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:27.478003 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2eac5b06-dd7vs" event={"ID":"4558943a-c9e0-4649-896c-e78d2fe1d25f","Type":"ContainerDied","Data":"764cbe88f08e684dc93d6118d4716d515990434e1163eb4fb136eb97ba01987a"} Oct 30 13:05:27.478150 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:27.478049 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="764cbe88f08e684dc93d6118d4716d515990434e1163eb4fb136eb97ba01987a" Oct 30 13:05:27.479620 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:27.479598 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c55470c-scdtx" Oct 30 13:05:27.479745 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:27.479602 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c55470c-scdtx" event={"ID":"ff9664e3-b790-4a5e-b96a-5553ef1189d8","Type":"ContainerDied","Data":"8d0938d49d7240ef2fd7583f036c8decd6627b6da95abfe7b9032d5d93efa320"} Oct 30 13:05:27.479745 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:27.479726 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8d0938d49d7240ef2fd7583f036c8decd6627b6da95abfe7b9032d5d93efa320" Oct 30 13:05:28.114822 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:28.114782 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9wwfc"] Oct 30 13:05:31.025571 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:31.025537 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0de4b9-zfpt4"] Oct 30 13:05:31.035633 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:31.035596 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0de4b9-zfpt4"] Oct 30 13:05:31.607421 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:31.607384 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cca21fd0-97fa-42cc-b7d9-a514773ecd97" path="/var/lib/kubelet/pods/cca21fd0-97fa-42cc-b7d9-a514773ecd97/volumes" Oct 30 13:05:39.435936 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:39.435897 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-0de4b9-openshift-default-5d57d77697-g7ct6"] Oct 30 13:05:39.436454 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:39.436154 4908 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-0de4b9-openshift-default-5d57d77697-g7ct6" podUID="cfd62d0e-d1b7-4695-8673-63c1d68f4f40" containerName="istio-proxy" containerID="cri-o://43ef3fa0e82eba6bb1b8aef922b2c49ee38df108ebba18778528befd76f665f5" gracePeriod=30 Oct 30 13:05:40.058208 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:40.058164 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-73bafc-9zb9m"] Oct 30 13:05:40.058729 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:40.058714 4908 cpu_manager.go:414] "containerMap: removing container" podUID="0ab626b9-027f-4799-ba80-e47276070fdb" containerName="istio-proxy" Oct 30 13:05:40.058783 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:40.058730 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ab626b9-027f-4799-ba80-e47276070fdb" containerName="istio-proxy" Oct 30 13:05:40.058783 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:40.058755 4908 cpu_manager.go:414] "containerMap: removing container" podUID="ff9664e3-b790-4a5e-b96a-5553ef1189d8" containerName="revoke-keys" Oct 30 13:05:40.058783 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:40.058760 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff9664e3-b790-4a5e-b96a-5553ef1189d8" containerName="revoke-keys" Oct 30 13:05:40.058783 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:40.058773 4908 cpu_manager.go:414] "containerMap: removing container" podUID="4558943a-c9e0-4649-896c-e78d2fe1d25f" containerName="revoke-keys" Oct 30 13:05:40.058783 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:40.058779 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4558943a-c9e0-4649-896c-e78d2fe1d25f" containerName="revoke-keys" Oct 30 13:05:40.059012 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:40.058793 4908 cpu_manager.go:414] "containerMap: removing container" podUID="e5c2ee3e-6917-4773-8bab-76eaa9f1d9ed" containerName="revoke-keys" Oct 30 13:05:40.059012 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:40.058798 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5c2ee3e-6917-4773-8bab-76eaa9f1d9ed" containerName="revoke-keys" Oct 30 13:05:40.059012 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:40.058915 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ab626b9-027f-4799-ba80-e47276070fdb" containerName="istio-proxy" Oct 30 13:05:40.059012 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:40.058928 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="4558943a-c9e0-4649-896c-e78d2fe1d25f" containerName="revoke-keys" Oct 30 13:05:40.059012 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:40.058938 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5c2ee3e-6917-4773-8bab-76eaa9f1d9ed" containerName="revoke-keys" Oct 30 13:05:40.059012 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:40.058959 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff9664e3-b790-4a5e-b96a-5553ef1189d8" containerName="revoke-keys" Oct 30 13:05:40.064514 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:40.064477 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-73bafc-9zb9m" Oct 30 13:05:40.093509 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:40.093470 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 13:05:40.093509 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:40.093469 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 13:05:40.100311 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:40.100268 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-73bafc-9zb9m"] Oct 30 13:05:40.150028 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:40.149990 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbzk8\" (UniqueName: \"kubernetes.io/projected/4d59db20-5049-4880-9fe4-19156f79dae6-kube-api-access-fbzk8\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-73bafc-9zb9m\" (UID: \"4d59db20-5049-4880-9fe4-19156f79dae6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-73bafc-9zb9m" Oct 30 13:05:40.150235 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:40.150050 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d59db20-5049-4880-9fe4-19156f79dae6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-73bafc-9zb9m\" (UID: \"4d59db20-5049-4880-9fe4-19156f79dae6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-73bafc-9zb9m" Oct 30 13:05:40.150235 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:40.150148 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d59db20-5049-4880-9fe4-19156f79dae6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-73bafc-9zb9m\" (UID: \"4d59db20-5049-4880-9fe4-19156f79dae6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-73bafc-9zb9m" Oct 30 13:05:40.165426 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:40.165376 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:05:40.165673 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:40.165632 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s75hq\"" Oct 30 13:05:40.251895 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:40.251814 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbzk8\" (UniqueName: \"kubernetes.io/projected/4d59db20-5049-4880-9fe4-19156f79dae6-kube-api-access-fbzk8\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-73bafc-9zb9m\" (UID: \"4d59db20-5049-4880-9fe4-19156f79dae6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-73bafc-9zb9m" Oct 30 13:05:40.252063 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:40.251916 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d59db20-5049-4880-9fe4-19156f79dae6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-73bafc-9zb9m\" (UID: \"4d59db20-5049-4880-9fe4-19156f79dae6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-73bafc-9zb9m" Oct 30 13:05:40.252063 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:40.251942 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d59db20-5049-4880-9fe4-19156f79dae6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-73bafc-9zb9m\" (UID: \"4d59db20-5049-4880-9fe4-19156f79dae6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-73bafc-9zb9m" Oct 30 13:05:40.252773 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:40.252742 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d59db20-5049-4880-9fe4-19156f79dae6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-73bafc-9zb9m\" (UID: \"4d59db20-5049-4880-9fe4-19156f79dae6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-73bafc-9zb9m" Oct 30 13:05:40.254474 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:40.254450 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d59db20-5049-4880-9fe4-19156f79dae6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-73bafc-9zb9m\" (UID: \"4d59db20-5049-4880-9fe4-19156f79dae6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-73bafc-9zb9m" Oct 30 13:05:40.275901 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:40.275866 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbzk8\" (UniqueName: \"kubernetes.io/projected/4d59db20-5049-4880-9fe4-19156f79dae6-kube-api-access-fbzk8\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-73bafc-9zb9m\" (UID: \"4d59db20-5049-4880-9fe4-19156f79dae6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-73bafc-9zb9m" Oct 30 13:05:40.383799 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:40.383692 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-73bafc-9zb9m" Oct 30 13:05:40.737315 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:40.734682 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-73bafc-9zb9m"] Oct 30 13:05:41.410711 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:41.410667 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-9c3f12/test-model-9c3f12-kserve-cc6c75df5-ldnbh"] Oct 30 13:05:41.661316 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:41.661276 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1333c8-rf6wc"] Oct 30 13:05:41.661964 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:41.661942 4908 cpu_manager.go:414] "containerMap: removing container" podUID="0ab626b9-027f-4799-ba80-e47276070fdb" containerName="istio-proxy" Oct 30 13:05:41.661964 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:41.661964 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ab626b9-027f-4799-ba80-e47276070fdb" containerName="istio-proxy" Oct 30 13:05:41.662149 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:41.662110 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ab626b9-027f-4799-ba80-e47276070fdb" containerName="istio-proxy" Oct 30 13:05:41.665842 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:41.665803 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1333c8-rf6wc" Oct 30 13:05:41.686119 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:41.686086 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1333c8-rf6wc"] Oct 30 13:05:41.698696 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:41.698660 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-73bafc-9zb9m" event={"ID":"4d59db20-5049-4880-9fe4-19156f79dae6","Type":"ContainerStarted","Data":"86628c5dfdb743d6dcd56548addd635e0d8cd5c22e5d7fd0071a4d152171cf0d"} Oct 30 13:05:41.698973 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:41.698948 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-73bafc-9zb9m" event={"ID":"4d59db20-5049-4880-9fe4-19156f79dae6","Type":"ContainerStarted","Data":"92cf6cdfea3f702f4d7145516ab32d4a785f89a894a36d80e75981f196ecb5f2"} Oct 30 13:05:41.741192 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:41.741070 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-73bafc-9zb9m" podStartSLOduration=1.741054309 podStartE2EDuration="1.741054309s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:05:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:05:41.739812918 +0000 UTC m=+4298.254241686" watchObservedRunningTime="2026-10-30 13:05:41.741054309 +0000 UTC m=+4298.255483056" Oct 30 13:05:41.777008 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:41.776960 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f1942a2-3706-4429-b3a4-c6eba3eb393c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1333c8-rf6wc\" (UID: \"2f1942a2-3706-4429-b3a4-c6eba3eb393c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1333c8-rf6wc" Oct 30 13:05:41.777291 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:41.777263 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jkm4\" (UniqueName: \"kubernetes.io/projected/2f1942a2-3706-4429-b3a4-c6eba3eb393c-kube-api-access-8jkm4\") pod \"maas-api-revoke-keys-e2e-inf-a-1333c8-rf6wc\" (UID: \"2f1942a2-3706-4429-b3a4-c6eba3eb393c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1333c8-rf6wc" Oct 30 13:05:41.777429 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:41.777407 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f1942a2-3706-4429-b3a4-c6eba3eb393c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1333c8-rf6wc\" (UID: \"2f1942a2-3706-4429-b3a4-c6eba3eb393c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1333c8-rf6wc" Oct 30 13:05:41.878915 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:41.878869 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jkm4\" (UniqueName: \"kubernetes.io/projected/2f1942a2-3706-4429-b3a4-c6eba3eb393c-kube-api-access-8jkm4\") pod \"maas-api-revoke-keys-e2e-inf-a-1333c8-rf6wc\" (UID: \"2f1942a2-3706-4429-b3a4-c6eba3eb393c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1333c8-rf6wc" Oct 30 13:05:41.879097 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:41.878988 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f1942a2-3706-4429-b3a4-c6eba3eb393c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1333c8-rf6wc\" (UID: \"2f1942a2-3706-4429-b3a4-c6eba3eb393c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1333c8-rf6wc" Oct 30 13:05:41.879097 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:41.879080 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f1942a2-3706-4429-b3a4-c6eba3eb393c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1333c8-rf6wc\" (UID: \"2f1942a2-3706-4429-b3a4-c6eba3eb393c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1333c8-rf6wc" Oct 30 13:05:41.879753 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:41.879725 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f1942a2-3706-4429-b3a4-c6eba3eb393c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1333c8-rf6wc\" (UID: \"2f1942a2-3706-4429-b3a4-c6eba3eb393c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1333c8-rf6wc" Oct 30 13:05:41.881774 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:41.881745 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f1942a2-3706-4429-b3a4-c6eba3eb393c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1333c8-rf6wc\" (UID: \"2f1942a2-3706-4429-b3a4-c6eba3eb393c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1333c8-rf6wc" Oct 30 13:05:41.902903 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:41.902871 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jkm4\" (UniqueName: \"kubernetes.io/projected/2f1942a2-3706-4429-b3a4-c6eba3eb393c-kube-api-access-8jkm4\") pod \"maas-api-revoke-keys-e2e-inf-a-1333c8-rf6wc\" (UID: \"2f1942a2-3706-4429-b3a4-c6eba3eb393c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1333c8-rf6wc" Oct 30 13:05:41.983117 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:41.983061 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1333c8-rf6wc" Oct 30 13:05:42.315351 ip-10-0-141-251 kubenswrapper[4908]: W1030 13:05:42.315297 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f1942a2_3706_4429_b3a4_c6eba3eb393c.slice/crio-8f7feb7ced7c69f3d4ae138f401ee14b50811caefe2ca76d6f986379660eee48 WatchSource:0}: Error finding container 8f7feb7ced7c69f3d4ae138f401ee14b50811caefe2ca76d6f986379660eee48: Status 404 returned error can't find the container with id 8f7feb7ced7c69f3d4ae138f401ee14b50811caefe2ca76d6f986379660eee48 Oct 30 13:05:42.317263 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:42.317231 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1333c8-rf6wc"] Oct 30 13:05:42.712015 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:42.711977 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1333c8-rf6wc" event={"ID":"2f1942a2-3706-4429-b3a4-c6eba3eb393c","Type":"ContainerStarted","Data":"29d8762478220930ef3dd205658f0a22991049961d2f65cd3ba33c67ac3cfe2f"} Oct 30 13:05:42.712228 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:42.712036 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1333c8-rf6wc" event={"ID":"2f1942a2-3706-4429-b3a4-c6eba3eb393c","Type":"ContainerStarted","Data":"8f7feb7ced7c69f3d4ae138f401ee14b50811caefe2ca76d6f986379660eee48"} Oct 30 13:05:42.713879 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:42.713822 4908 generic.go:356] "Generic (PLEG): container finished" podID="4d59db20-5049-4880-9fe4-19156f79dae6" containerID="86628c5dfdb743d6dcd56548addd635e0d8cd5c22e5d7fd0071a4d152171cf0d" exitCode=0 Oct 30 13:05:42.714018 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:42.713858 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-73bafc-9zb9m" event={"ID":"4d59db20-5049-4880-9fe4-19156f79dae6","Type":"ContainerDied","Data":"86628c5dfdb743d6dcd56548addd635e0d8cd5c22e5d7fd0071a4d152171cf0d"} Oct 30 13:05:42.752138 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:42.752071 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1333c8-rf6wc" podStartSLOduration=1.7520543549999998 podStartE2EDuration="1.752054355s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:05:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:05:42.751234116 +0000 UTC m=+4299.265662862" watchObservedRunningTime="2026-10-30 13:05:42.752054355 +0000 UTC m=+4299.266483101" Oct 30 13:05:43.725176 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:43.725058 4908 generic.go:356] "Generic (PLEG): container finished" podID="2f1942a2-3706-4429-b3a4-c6eba3eb393c" containerID="29d8762478220930ef3dd205658f0a22991049961d2f65cd3ba33c67ac3cfe2f" exitCode=0 Oct 30 13:05:43.725372 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:43.725214 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1333c8-rf6wc" event={"ID":"2f1942a2-3706-4429-b3a4-c6eba3eb393c","Type":"ContainerDied","Data":"29d8762478220930ef3dd205658f0a22991049961d2f65cd3ba33c67ac3cfe2f"} Oct 30 13:05:43.917515 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:43.917486 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-73bafc-9zb9m" Oct 30 13:05:44.037367 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:44.036753 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d59db20-5049-4880-9fe4-19156f79dae6-kube-api-access-fbzk8\" (UniqueName: \"kubernetes.io/projected/4d59db20-5049-4880-9fe4-19156f79dae6-kube-api-access-fbzk8\") pod \"4d59db20-5049-4880-9fe4-19156f79dae6\" (UID: \"4d59db20-5049-4880-9fe4-19156f79dae6\") " Oct 30 13:05:44.037961 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:44.037931 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4d59db20-5049-4880-9fe4-19156f79dae6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d59db20-5049-4880-9fe4-19156f79dae6-maas-api-xks-ca\") pod \"4d59db20-5049-4880-9fe4-19156f79dae6\" (UID: \"4d59db20-5049-4880-9fe4-19156f79dae6\") " Oct 30 13:05:44.038265 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:44.038244 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d59db20-5049-4880-9fe4-19156f79dae6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d59db20-5049-4880-9fe4-19156f79dae6-maas-api-openshift-ca\") pod \"4d59db20-5049-4880-9fe4-19156f79dae6\" (UID: \"4d59db20-5049-4880-9fe4-19156f79dae6\") " Oct 30 13:05:44.039697 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:44.039671 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d59db20-5049-4880-9fe4-19156f79dae6-maas-api-openshift-ca" pod "4d59db20-5049-4880-9fe4-19156f79dae6" (UID: "4d59db20-5049-4880-9fe4-19156f79dae6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:05:44.049448 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:44.049413 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d59db20-5049-4880-9fe4-19156f79dae6-kube-api-access-fbzk8" pod "4d59db20-5049-4880-9fe4-19156f79dae6" (UID: "4d59db20-5049-4880-9fe4-19156f79dae6"). InnerVolumeSpecName "kube-api-access-fbzk8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:05:44.052653 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:44.052625 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d59db20-5049-4880-9fe4-19156f79dae6-maas-api-xks-ca" pod "4d59db20-5049-4880-9fe4-19156f79dae6" (UID: "4d59db20-5049-4880-9fe4-19156f79dae6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:05:44.141412 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:44.141008 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d59db20-5049-4880-9fe4-19156f79dae6-maas-api-xks-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:05:44.141412 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:44.141050 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d59db20-5049-4880-9fe4-19156f79dae6-maas-api-openshift-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:05:44.141412 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:44.141065 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbzk8\" (UniqueName: \"kubernetes.io/projected/4d59db20-5049-4880-9fe4-19156f79dae6-kube-api-access-fbzk8\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:05:44.734479 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:44.734440 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-0de4b9-openshift-default-5d57d77697-g7ct6" Oct 30 13:05:44.736136 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:44.736102 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-73bafc-9zb9m" Oct 30 13:05:44.736282 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:44.736136 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-73bafc-9zb9m" event={"ID":"4d59db20-5049-4880-9fe4-19156f79dae6","Type":"ContainerDied","Data":"92cf6cdfea3f702f4d7145516ab32d4a785f89a894a36d80e75981f196ecb5f2"} Oct 30 13:05:44.736282 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:44.736186 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="92cf6cdfea3f702f4d7145516ab32d4a785f89a894a36d80e75981f196ecb5f2" Oct 30 13:05:44.738160 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:44.738130 4908 generic.go:356] "Generic (PLEG): container finished" podID="cfd62d0e-d1b7-4695-8673-63c1d68f4f40" containerID="43ef3fa0e82eba6bb1b8aef922b2c49ee38df108ebba18778528befd76f665f5" exitCode=0 Oct 30 13:05:44.738312 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:44.738175 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-0de4b9-openshift-default-5d57d77697-g7ct6" Oct 30 13:05:44.738312 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:44.738212 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-0de4b9-openshift-default-5d57d77697-g7ct6" event={"ID":"cfd62d0e-d1b7-4695-8673-63c1d68f4f40","Type":"ContainerDied","Data":"43ef3fa0e82eba6bb1b8aef922b2c49ee38df108ebba18778528befd76f665f5"} Oct 30 13:05:44.738312 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:44.738260 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-0de4b9-openshift-default-5d57d77697-g7ct6" event={"ID":"cfd62d0e-d1b7-4695-8673-63c1d68f4f40","Type":"ContainerDied","Data":"1280ef676eb0df69c00534c047b0fa44cc57b4b3e1d9247ff60476b090098f31"} Oct 30 13:05:44.738312 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:44.738284 4908 scope.go:122] "RemoveContainer" containerID="43ef3fa0e82eba6bb1b8aef922b2c49ee38df108ebba18778528befd76f665f5" Oct 30 13:05:44.753608 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:44.753585 4908 scope.go:122] "RemoveContainer" containerID="1f42a26fc0287f52ffa90881714b4c18f92ff184a9d87eb82271f54bfc0d8b5d" Oct 30 13:05:44.769580 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:44.769551 4908 scope.go:122] "RemoveContainer" containerID="43ef3fa0e82eba6bb1b8aef922b2c49ee38df108ebba18778528befd76f665f5" Oct 30 13:05:44.770055 ip-10-0-141-251 kubenswrapper[4908]: E1030 13:05:44.770022 4908 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43ef3fa0e82eba6bb1b8aef922b2c49ee38df108ebba18778528befd76f665f5\": container with ID starting with 43ef3fa0e82eba6bb1b8aef922b2c49ee38df108ebba18778528befd76f665f5 not found: ID does not exist" containerID="43ef3fa0e82eba6bb1b8aef922b2c49ee38df108ebba18778528befd76f665f5" Oct 30 13:05:44.770160 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:44.770069 4908 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43ef3fa0e82eba6bb1b8aef922b2c49ee38df108ebba18778528befd76f665f5"} err="failed to get container status \"43ef3fa0e82eba6bb1b8aef922b2c49ee38df108ebba18778528befd76f665f5\": rpc error: code = NotFound desc = could not find container \"43ef3fa0e82eba6bb1b8aef922b2c49ee38df108ebba18778528befd76f665f5\": container with ID starting with 43ef3fa0e82eba6bb1b8aef922b2c49ee38df108ebba18778528befd76f665f5 not found: ID does not exist" Oct 30 13:05:44.770160 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:44.770094 4908 scope.go:122] "RemoveContainer" containerID="1f42a26fc0287f52ffa90881714b4c18f92ff184a9d87eb82271f54bfc0d8b5d" Oct 30 13:05:44.770736 ip-10-0-141-251 kubenswrapper[4908]: E1030 13:05:44.770707 4908 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f42a26fc0287f52ffa90881714b4c18f92ff184a9d87eb82271f54bfc0d8b5d\": container with ID starting with 1f42a26fc0287f52ffa90881714b4c18f92ff184a9d87eb82271f54bfc0d8b5d not found: ID does not exist" containerID="1f42a26fc0287f52ffa90881714b4c18f92ff184a9d87eb82271f54bfc0d8b5d" Oct 30 13:05:44.770815 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:44.770746 4908 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f42a26fc0287f52ffa90881714b4c18f92ff184a9d87eb82271f54bfc0d8b5d"} err="failed to get container status \"1f42a26fc0287f52ffa90881714b4c18f92ff184a9d87eb82271f54bfc0d8b5d\": rpc error: code = NotFound desc = could not find container \"1f42a26fc0287f52ffa90881714b4c18f92ff184a9d87eb82271f54bfc0d8b5d\": container with ID starting with 1f42a26fc0287f52ffa90881714b4c18f92ff184a9d87eb82271f54bfc0d8b5d not found: ID does not exist" Oct 30 13:05:44.848783 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:44.848742 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfd62d0e-d1b7-4695-8673-63c1d68f4f40-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfd62d0e-d1b7-4695-8673-63c1d68f4f40-workload-socket\") pod \"cfd62d0e-d1b7-4695-8673-63c1d68f4f40\" (UID: \"cfd62d0e-d1b7-4695-8673-63c1d68f4f40\") " Oct 30 13:05:44.848984 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:44.848855 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfd62d0e-d1b7-4695-8673-63c1d68f4f40-istio-token\" (UniqueName: \"kubernetes.io/projected/cfd62d0e-d1b7-4695-8673-63c1d68f4f40-istio-token\") pod \"cfd62d0e-d1b7-4695-8673-63c1d68f4f40\" (UID: \"cfd62d0e-d1b7-4695-8673-63c1d68f4f40\") " Oct 30 13:05:44.848984 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:44.848888 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfd62d0e-d1b7-4695-8673-63c1d68f4f40-kube-api-access-65qjb\" (UniqueName: \"kubernetes.io/projected/cfd62d0e-d1b7-4695-8673-63c1d68f4f40-kube-api-access-65qjb\") pod \"cfd62d0e-d1b7-4695-8673-63c1d68f4f40\" (UID: \"cfd62d0e-d1b7-4695-8673-63c1d68f4f40\") " Oct 30 13:05:44.848984 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:44.848928 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfd62d0e-d1b7-4695-8673-63c1d68f4f40-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfd62d0e-d1b7-4695-8673-63c1d68f4f40-istio-data\") pod \"cfd62d0e-d1b7-4695-8673-63c1d68f4f40\" (UID: \"cfd62d0e-d1b7-4695-8673-63c1d68f4f40\") " Oct 30 13:05:44.848984 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:44.848979 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cfd62d0e-d1b7-4695-8673-63c1d68f4f40-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfd62d0e-d1b7-4695-8673-63c1d68f4f40-istio-podinfo\") pod \"cfd62d0e-d1b7-4695-8673-63c1d68f4f40\" (UID: \"cfd62d0e-d1b7-4695-8673-63c1d68f4f40\") " Oct 30 13:05:44.849199 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:44.849068 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfd62d0e-d1b7-4695-8673-63c1d68f4f40-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfd62d0e-d1b7-4695-8673-63c1d68f4f40-workload-certs\") pod \"cfd62d0e-d1b7-4695-8673-63c1d68f4f40\" (UID: \"cfd62d0e-d1b7-4695-8673-63c1d68f4f40\") " Oct 30 13:05:44.849199 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:44.849122 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfd62d0e-d1b7-4695-8673-63c1d68f4f40-workload-socket" pod "cfd62d0e-d1b7-4695-8673-63c1d68f4f40" (UID: "cfd62d0e-d1b7-4695-8673-63c1d68f4f40"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:05:44.849378 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:44.849335 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfd62d0e-d1b7-4695-8673-63c1d68f4f40-istio-data" pod "cfd62d0e-d1b7-4695-8673-63c1d68f4f40" (UID: "cfd62d0e-d1b7-4695-8673-63c1d68f4f40"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:05:44.849453 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:44.849385 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfd62d0e-d1b7-4695-8673-63c1d68f4f40-workload-certs" pod "cfd62d0e-d1b7-4695-8673-63c1d68f4f40" (UID: "cfd62d0e-d1b7-4695-8673-63c1d68f4f40"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:05:44.849509 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:44.849485 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfd62d0e-d1b7-4695-8673-63c1d68f4f40-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfd62d0e-d1b7-4695-8673-63c1d68f4f40-credential-socket\") pod \"cfd62d0e-d1b7-4695-8673-63c1d68f4f40\" (UID: \"cfd62d0e-d1b7-4695-8673-63c1d68f4f40\") " Oct 30 13:05:44.849651 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:44.849539 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cfd62d0e-d1b7-4695-8673-63c1d68f4f40-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfd62d0e-d1b7-4695-8673-63c1d68f4f40-istiod-ca-cert\") pod \"cfd62d0e-d1b7-4695-8673-63c1d68f4f40\" (UID: \"cfd62d0e-d1b7-4695-8673-63c1d68f4f40\") " Oct 30 13:05:44.849651 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:44.849597 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfd62d0e-d1b7-4695-8673-63c1d68f4f40-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfd62d0e-d1b7-4695-8673-63c1d68f4f40-istio-envoy\") pod \"cfd62d0e-d1b7-4695-8673-63c1d68f4f40\" (UID: \"cfd62d0e-d1b7-4695-8673-63c1d68f4f40\") " Oct 30 13:05:44.850197 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:44.849992 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfd62d0e-d1b7-4695-8673-63c1d68f4f40-credential-socket" pod "cfd62d0e-d1b7-4695-8673-63c1d68f4f40" (UID: "cfd62d0e-d1b7-4695-8673-63c1d68f4f40"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:05:44.850414 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:44.850391 4908 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfd62d0e-d1b7-4695-8673-63c1d68f4f40-workload-socket\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:05:44.850514 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:44.850416 4908 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfd62d0e-d1b7-4695-8673-63c1d68f4f40-istio-data\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:05:44.850514 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:44.850429 4908 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfd62d0e-d1b7-4695-8673-63c1d68f4f40-workload-certs\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:05:44.850514 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:44.850442 4908 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfd62d0e-d1b7-4695-8673-63c1d68f4f40-credential-socket\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:05:44.850894 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:44.850609 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfd62d0e-d1b7-4695-8673-63c1d68f4f40-istiod-ca-cert" pod "cfd62d0e-d1b7-4695-8673-63c1d68f4f40" (UID: "cfd62d0e-d1b7-4695-8673-63c1d68f4f40"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:05:44.851794 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:44.851772 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfd62d0e-d1b7-4695-8673-63c1d68f4f40-istio-token" pod "cfd62d0e-d1b7-4695-8673-63c1d68f4f40" (UID: "cfd62d0e-d1b7-4695-8673-63c1d68f4f40"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:05:44.859879 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:44.852722 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfd62d0e-d1b7-4695-8673-63c1d68f4f40-istio-envoy" pod "cfd62d0e-d1b7-4695-8673-63c1d68f4f40" (UID: "cfd62d0e-d1b7-4695-8673-63c1d68f4f40"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:05:44.859879 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:44.852854 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfd62d0e-d1b7-4695-8673-63c1d68f4f40-kube-api-access-65qjb" pod "cfd62d0e-d1b7-4695-8673-63c1d68f4f40" (UID: "cfd62d0e-d1b7-4695-8673-63c1d68f4f40"). InnerVolumeSpecName "kube-api-access-65qjb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:05:44.859879 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:44.852813 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cfd62d0e-d1b7-4695-8673-63c1d68f4f40-istio-podinfo" pod "cfd62d0e-d1b7-4695-8673-63c1d68f4f40" (UID: "cfd62d0e-d1b7-4695-8673-63c1d68f4f40"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:05:44.927159 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:44.927125 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1333c8-rf6wc" Oct 30 13:05:44.951957 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:44.951922 4908 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfd62d0e-d1b7-4695-8673-63c1d68f4f40-istio-token\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:05:44.951957 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:44.951952 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65qjb\" (UniqueName: \"kubernetes.io/projected/cfd62d0e-d1b7-4695-8673-63c1d68f4f40-kube-api-access-65qjb\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:05:44.951957 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:44.951962 4908 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfd62d0e-d1b7-4695-8673-63c1d68f4f40-istio-podinfo\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:05:44.952197 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:44.951971 4908 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfd62d0e-d1b7-4695-8673-63c1d68f4f40-istiod-ca-cert\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:05:44.952197 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:44.951981 4908 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfd62d0e-d1b7-4695-8673-63c1d68f4f40-istio-envoy\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:05:45.052850 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:45.052809 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f1942a2-3706-4429-b3a4-c6eba3eb393c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f1942a2-3706-4429-b3a4-c6eba3eb393c-maas-api-openshift-ca\") pod \"2f1942a2-3706-4429-b3a4-c6eba3eb393c\" (UID: \"2f1942a2-3706-4429-b3a4-c6eba3eb393c\") " Oct 30 13:05:45.052984 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:45.052940 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f1942a2-3706-4429-b3a4-c6eba3eb393c-kube-api-access-8jkm4\" (UniqueName: \"kubernetes.io/projected/2f1942a2-3706-4429-b3a4-c6eba3eb393c-kube-api-access-8jkm4\") pod \"2f1942a2-3706-4429-b3a4-c6eba3eb393c\" (UID: \"2f1942a2-3706-4429-b3a4-c6eba3eb393c\") " Oct 30 13:05:45.053079 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:45.053061 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f1942a2-3706-4429-b3a4-c6eba3eb393c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f1942a2-3706-4429-b3a4-c6eba3eb393c-maas-api-xks-ca\") pod \"2f1942a2-3706-4429-b3a4-c6eba3eb393c\" (UID: \"2f1942a2-3706-4429-b3a4-c6eba3eb393c\") " Oct 30 13:05:45.053216 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:45.053195 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f1942a2-3706-4429-b3a4-c6eba3eb393c-maas-api-openshift-ca" pod "2f1942a2-3706-4429-b3a4-c6eba3eb393c" (UID: "2f1942a2-3706-4429-b3a4-c6eba3eb393c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:05:45.053552 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:45.053532 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f1942a2-3706-4429-b3a4-c6eba3eb393c-maas-api-openshift-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:05:45.055163 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:45.055137 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f1942a2-3706-4429-b3a4-c6eba3eb393c-maas-api-xks-ca" pod "2f1942a2-3706-4429-b3a4-c6eba3eb393c" (UID: "2f1942a2-3706-4429-b3a4-c6eba3eb393c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:05:45.055260 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:45.055168 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f1942a2-3706-4429-b3a4-c6eba3eb393c-kube-api-access-8jkm4" pod "2f1942a2-3706-4429-b3a4-c6eba3eb393c" (UID: "2f1942a2-3706-4429-b3a4-c6eba3eb393c"). InnerVolumeSpecName "kube-api-access-8jkm4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:05:45.086894 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:45.086857 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-0de4b9-openshift-default-5d57d77697-g7ct6"] Oct 30 13:05:45.090719 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:45.090690 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-0de4b9-openshift-default-5d57d77697-g7ct6"] Oct 30 13:05:45.154597 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:45.154555 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jkm4\" (UniqueName: \"kubernetes.io/projected/2f1942a2-3706-4429-b3a4-c6eba3eb393c-kube-api-access-8jkm4\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:05:45.154597 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:45.154593 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f1942a2-3706-4429-b3a4-c6eba3eb393c-maas-api-xks-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:05:45.606549 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:45.606508 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cfd62d0e-d1b7-4695-8673-63c1d68f4f40" path="/var/lib/kubelet/pods/cfd62d0e-d1b7-4695-8673-63c1d68f4f40/volumes" Oct 30 13:05:45.748586 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:45.748556 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1333c8-rf6wc" Oct 30 13:05:45.748763 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:45.748598 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1333c8-rf6wc" event={"ID":"2f1942a2-3706-4429-b3a4-c6eba3eb393c","Type":"ContainerDied","Data":"8f7feb7ced7c69f3d4ae138f401ee14b50811caefe2ca76d6f986379660eee48"} Oct 30 13:05:45.748763 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:45.748657 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8f7feb7ced7c69f3d4ae138f401ee14b50811caefe2ca76d6f986379660eee48" Oct 30 13:05:49.686540 ip-10-0-141-251 kubenswrapper[4908]: E1030 13:05:49.686494 4908 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vn6vv" containerName="extractor" Oct 30 13:05:51.069709 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:51.069675 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-734ffe-x6b97"] Oct 30 13:05:51.079879 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:51.079823 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-734ffe-x6b97"] Oct 30 13:05:51.607677 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:51.607640 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01a6f969-4660-4001-b9e7-190f4cfd7633" path="/var/lib/kubelet/pods/01a6f969-4660-4001-b9e7-190f4cfd7633/volumes" Oct 30 13:05:55.146145 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:55.146105 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-befaa6-qp8mv"] Oct 30 13:05:55.156904 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:55.156871 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-befaa6-qp8mv"] Oct 30 13:05:55.606638 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:55.606596 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a22eb470-fdda-42a0-8204-3a218401efba" path="/var/lib/kubelet/pods/a22eb470-fdda-42a0-8204-3a218401efba/volumes" Oct 30 13:05:56.435519 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:56.435469 4908 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-9c3f12/test-model-9c3f12-kserve-cc6c75df5-ldnbh" podUID="b21ead65-1a34-4368-a2ee-b47795620cd3" containerName="main" containerID="cri-o://e4cb88e766fbc3c9abe05612e236a691207719e4222ed4d75d4e35f1eb832736" gracePeriod=45 Oct 30 13:05:56.440319 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:56.440286 4908 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-9c3f12/test-model-9c3f12-kserve-cc6c75df5-ldnbh" podUID="b21ead65-1a34-4368-a2ee-b47795620cd3" containerName="main" probeResult="failure" output="Get \"https://10.132.1.66:8000/ready\": EOF" Oct 30 13:05:56.706678 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:56.706647 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-9c3f12/test-model-9c3f12-kserve-cc6c75df5-ldnbh" Oct 30 13:05:56.756135 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:56.756085 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b21ead65-1a34-4368-a2ee-b47795620cd3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b21ead65-1a34-4368-a2ee-b47795620cd3-model-cache\") pod \"b21ead65-1a34-4368-a2ee-b47795620cd3\" (UID: \"b21ead65-1a34-4368-a2ee-b47795620cd3\") " Oct 30 13:05:56.756322 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:56.756190 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b21ead65-1a34-4368-a2ee-b47795620cd3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b21ead65-1a34-4368-a2ee-b47795620cd3-dshm\") pod \"b21ead65-1a34-4368-a2ee-b47795620cd3\" (UID: \"b21ead65-1a34-4368-a2ee-b47795620cd3\") " Oct 30 13:05:56.756322 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:56.756226 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b21ead65-1a34-4368-a2ee-b47795620cd3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b21ead65-1a34-4368-a2ee-b47795620cd3-tmp-dir\") pod \"b21ead65-1a34-4368-a2ee-b47795620cd3\" (UID: \"b21ead65-1a34-4368-a2ee-b47795620cd3\") " Oct 30 13:05:56.756322 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:56.756243 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b21ead65-1a34-4368-a2ee-b47795620cd3-home\" (UniqueName: \"kubernetes.io/empty-dir/b21ead65-1a34-4368-a2ee-b47795620cd3-home\") pod \"b21ead65-1a34-4368-a2ee-b47795620cd3\" (UID: \"b21ead65-1a34-4368-a2ee-b47795620cd3\") " Oct 30 13:05:56.756510 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:56.756368 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b21ead65-1a34-4368-a2ee-b47795620cd3-tls-certs\" (UniqueName: \"kubernetes.io/secret/b21ead65-1a34-4368-a2ee-b47795620cd3-tls-certs\") pod \"b21ead65-1a34-4368-a2ee-b47795620cd3\" (UID: \"b21ead65-1a34-4368-a2ee-b47795620cd3\") " Oct 30 13:05:56.756510 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:56.756389 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b21ead65-1a34-4368-a2ee-b47795620cd3-model-cache" pod "b21ead65-1a34-4368-a2ee-b47795620cd3" (UID: "b21ead65-1a34-4368-a2ee-b47795620cd3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:05:56.756510 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:56.756409 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b21ead65-1a34-4368-a2ee-b47795620cd3-kube-api-access-gh9cp\" (UniqueName: \"kubernetes.io/projected/b21ead65-1a34-4368-a2ee-b47795620cd3-kube-api-access-gh9cp\") pod \"b21ead65-1a34-4368-a2ee-b47795620cd3\" (UID: \"b21ead65-1a34-4368-a2ee-b47795620cd3\") " Oct 30 13:05:56.756686 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:56.756580 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b21ead65-1a34-4368-a2ee-b47795620cd3-tmp-dir" pod "b21ead65-1a34-4368-a2ee-b47795620cd3" (UID: "b21ead65-1a34-4368-a2ee-b47795620cd3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:05:56.756686 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:56.756618 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b21ead65-1a34-4368-a2ee-b47795620cd3-home" pod "b21ead65-1a34-4368-a2ee-b47795620cd3" (UID: "b21ead65-1a34-4368-a2ee-b47795620cd3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:05:56.757039 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:56.757007 4908 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b21ead65-1a34-4368-a2ee-b47795620cd3-model-cache\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:05:56.757039 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:56.757038 4908 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b21ead65-1a34-4368-a2ee-b47795620cd3-tmp-dir\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:05:56.757211 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:56.757052 4908 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b21ead65-1a34-4368-a2ee-b47795620cd3-home\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:05:56.758572 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:56.758541 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b21ead65-1a34-4368-a2ee-b47795620cd3-dshm" pod "b21ead65-1a34-4368-a2ee-b47795620cd3" (UID: "b21ead65-1a34-4368-a2ee-b47795620cd3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:05:56.758679 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:56.758623 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b21ead65-1a34-4368-a2ee-b47795620cd3-tls-certs" pod "b21ead65-1a34-4368-a2ee-b47795620cd3" (UID: "b21ead65-1a34-4368-a2ee-b47795620cd3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:05:56.758726 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:56.758691 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b21ead65-1a34-4368-a2ee-b47795620cd3-kube-api-access-gh9cp" pod "b21ead65-1a34-4368-a2ee-b47795620cd3" (UID: "b21ead65-1a34-4368-a2ee-b47795620cd3"). InnerVolumeSpecName "kube-api-access-gh9cp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:05:56.857873 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:56.857803 4908 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b21ead65-1a34-4368-a2ee-b47795620cd3-dshm\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:05:56.857873 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:56.857870 4908 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b21ead65-1a34-4368-a2ee-b47795620cd3-tls-certs\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:05:56.857873 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:56.857883 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gh9cp\" (UniqueName: \"kubernetes.io/projected/b21ead65-1a34-4368-a2ee-b47795620cd3-kube-api-access-gh9cp\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:05:56.931991 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:56.931954 4908 generic.go:356] "Generic (PLEG): container finished" podID="b21ead65-1a34-4368-a2ee-b47795620cd3" containerID="e4cb88e766fbc3c9abe05612e236a691207719e4222ed4d75d4e35f1eb832736" exitCode=0 Oct 30 13:05:56.932183 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:56.932004 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-9c3f12/test-model-9c3f12-kserve-cc6c75df5-ldnbh" Oct 30 13:05:56.932183 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:56.932036 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-9c3f12/test-model-9c3f12-kserve-cc6c75df5-ldnbh" event={"ID":"b21ead65-1a34-4368-a2ee-b47795620cd3","Type":"ContainerDied","Data":"e4cb88e766fbc3c9abe05612e236a691207719e4222ed4d75d4e35f1eb832736"} Oct 30 13:05:56.932183 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:56.932083 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-9c3f12/test-model-9c3f12-kserve-cc6c75df5-ldnbh" event={"ID":"b21ead65-1a34-4368-a2ee-b47795620cd3","Type":"ContainerDied","Data":"b1b27702d3ad7ad000f8f3a463a1c24b20fdc04acfcc86d4d4c927ffe8dd53b6"} Oct 30 13:05:56.932183 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:56.932106 4908 scope.go:122] "RemoveContainer" containerID="e4cb88e766fbc3c9abe05612e236a691207719e4222ed4d75d4e35f1eb832736" Oct 30 13:05:56.944307 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:56.944284 4908 scope.go:122] "RemoveContainer" containerID="e4cb88e766fbc3c9abe05612e236a691207719e4222ed4d75d4e35f1eb832736" Oct 30 13:05:56.944590 ip-10-0-141-251 kubenswrapper[4908]: E1030 13:05:56.944570 4908 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4cb88e766fbc3c9abe05612e236a691207719e4222ed4d75d4e35f1eb832736\": container with ID starting with e4cb88e766fbc3c9abe05612e236a691207719e4222ed4d75d4e35f1eb832736 not found: ID does not exist" containerID="e4cb88e766fbc3c9abe05612e236a691207719e4222ed4d75d4e35f1eb832736" Oct 30 13:05:56.944651 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:56.944600 4908 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4cb88e766fbc3c9abe05612e236a691207719e4222ed4d75d4e35f1eb832736"} err="failed to get container status \"e4cb88e766fbc3c9abe05612e236a691207719e4222ed4d75d4e35f1eb832736\": rpc error: code = NotFound desc = could not find container \"e4cb88e766fbc3c9abe05612e236a691207719e4222ed4d75d4e35f1eb832736\": container with ID starting with e4cb88e766fbc3c9abe05612e236a691207719e4222ed4d75d4e35f1eb832736 not found: ID does not exist" Oct 30 13:05:56.989676 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:56.989638 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-9c3f12/test-model-9c3f12-kserve-cc6c75df5-ldnbh"] Oct 30 13:05:56.993483 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:56.993454 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-9c3f12/test-model-9c3f12-kserve-cc6c75df5-ldnbh"] Oct 30 13:05:57.608355 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:57.608319 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b21ead65-1a34-4368-a2ee-b47795620cd3" path="/var/lib/kubelet/pods/b21ead65-1a34-4368-a2ee-b47795620cd3/volumes" Oct 30 13:05:58.719885 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:58.715965 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-734ffe-openshift-default-5db45fbc99-5vzmr"] Oct 30 13:05:58.719885 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:58.716213 4908 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-734ffe-openshift-default-5db45fbc99-5vzmr" podUID="7e96c960-abe0-4b56-b3f9-ce46f2ce307a" containerName="istio-proxy" containerID="cri-o://e44f7f042c249b44a8aeea37b5f6c38de531e2e11fbaac541006930b57b32133" gracePeriod=30 Oct 30 13:05:59.365023 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:59.364983 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2545c7-mctbc"] Oct 30 13:05:59.365840 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:59.365803 4908 cpu_manager.go:414] "containerMap: removing container" podUID="cfd62d0e-d1b7-4695-8673-63c1d68f4f40" containerName="istio-proxy" Oct 30 13:05:59.365979 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:59.365842 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfd62d0e-d1b7-4695-8673-63c1d68f4f40" containerName="istio-proxy" Oct 30 13:05:59.365979 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:59.365872 4908 cpu_manager.go:414] "containerMap: removing container" podUID="b21ead65-1a34-4368-a2ee-b47795620cd3" containerName="main" Oct 30 13:05:59.365979 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:59.365880 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b21ead65-1a34-4368-a2ee-b47795620cd3" containerName="main" Oct 30 13:05:59.365979 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:59.365925 4908 cpu_manager.go:414] "containerMap: removing container" podUID="4d59db20-5049-4880-9fe4-19156f79dae6" containerName="revoke-keys" Oct 30 13:05:59.365979 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:59.365934 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d59db20-5049-4880-9fe4-19156f79dae6" containerName="revoke-keys" Oct 30 13:05:59.365979 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:59.365964 4908 cpu_manager.go:414] "containerMap: removing container" podUID="2f1942a2-3706-4429-b3a4-c6eba3eb393c" containerName="revoke-keys" Oct 30 13:05:59.365979 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:59.365971 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f1942a2-3706-4429-b3a4-c6eba3eb393c" containerName="revoke-keys" Oct 30 13:05:59.366225 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:59.366086 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfd62d0e-d1b7-4695-8673-63c1d68f4f40" containerName="istio-proxy" Oct 30 13:05:59.366225 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:59.366101 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="b21ead65-1a34-4368-a2ee-b47795620cd3" containerName="main" Oct 30 13:05:59.366225 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:59.366112 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfd62d0e-d1b7-4695-8673-63c1d68f4f40" containerName="istio-proxy" Oct 30 13:05:59.366225 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:59.366126 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d59db20-5049-4880-9fe4-19156f79dae6" containerName="revoke-keys" Oct 30 13:05:59.366225 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:59.366138 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f1942a2-3706-4429-b3a4-c6eba3eb393c" containerName="revoke-keys" Oct 30 13:05:59.371574 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:59.371554 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2545c7-mctbc" Oct 30 13:05:59.388923 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:59.388884 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2545c7-mctbc"] Oct 30 13:05:59.389889 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:59.389801 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 13:05:59.390346 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:59.390319 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 13:05:59.405459 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:59.405424 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:05:59.405691 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:59.405665 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s75hq\"" Oct 30 13:05:59.410537 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:59.410496 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pq7m\" (UniqueName: \"kubernetes.io/projected/8ebf1f1c-f8ed-4960-8f9b-2b750fcf4885-kube-api-access-5pq7m\") pod \"maas-api-revoke-keys-e2e-extmod-a-2545c7-mctbc\" (UID: \"8ebf1f1c-f8ed-4960-8f9b-2b750fcf4885\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2545c7-mctbc" Oct 30 13:05:59.410689 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:59.410640 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ebf1f1c-f8ed-4960-8f9b-2b750fcf4885-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-2545c7-mctbc\" (UID: \"8ebf1f1c-f8ed-4960-8f9b-2b750fcf4885\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2545c7-mctbc" Oct 30 13:05:59.410933 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:59.410907 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ebf1f1c-f8ed-4960-8f9b-2b750fcf4885-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-2545c7-mctbc\" (UID: \"8ebf1f1c-f8ed-4960-8f9b-2b750fcf4885\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2545c7-mctbc" Oct 30 13:05:59.511949 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:59.511907 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ebf1f1c-f8ed-4960-8f9b-2b750fcf4885-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-2545c7-mctbc\" (UID: \"8ebf1f1c-f8ed-4960-8f9b-2b750fcf4885\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2545c7-mctbc" Oct 30 13:05:59.512179 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:59.512021 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pq7m\" (UniqueName: \"kubernetes.io/projected/8ebf1f1c-f8ed-4960-8f9b-2b750fcf4885-kube-api-access-5pq7m\") pod \"maas-api-revoke-keys-e2e-extmod-a-2545c7-mctbc\" (UID: \"8ebf1f1c-f8ed-4960-8f9b-2b750fcf4885\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2545c7-mctbc" Oct 30 13:05:59.512179 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:59.512079 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ebf1f1c-f8ed-4960-8f9b-2b750fcf4885-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-2545c7-mctbc\" (UID: \"8ebf1f1c-f8ed-4960-8f9b-2b750fcf4885\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2545c7-mctbc" Oct 30 13:05:59.512734 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:59.512708 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ebf1f1c-f8ed-4960-8f9b-2b750fcf4885-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-2545c7-mctbc\" (UID: \"8ebf1f1c-f8ed-4960-8f9b-2b750fcf4885\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2545c7-mctbc" Oct 30 13:05:59.514389 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:59.514366 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ebf1f1c-f8ed-4960-8f9b-2b750fcf4885-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-2545c7-mctbc\" (UID: \"8ebf1f1c-f8ed-4960-8f9b-2b750fcf4885\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2545c7-mctbc" Oct 30 13:05:59.534385 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:59.534354 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pq7m\" (UniqueName: \"kubernetes.io/projected/8ebf1f1c-f8ed-4960-8f9b-2b750fcf4885-kube-api-access-5pq7m\") pod \"maas-api-revoke-keys-e2e-extmod-a-2545c7-mctbc\" (UID: \"8ebf1f1c-f8ed-4960-8f9b-2b750fcf4885\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2545c7-mctbc" Oct 30 13:05:59.687335 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:05:59.687228 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2545c7-mctbc" Oct 30 13:06:00.026531 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:00.026496 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2545c7-mctbc"] Oct 30 13:06:00.028582 ip-10-0-141-251 kubenswrapper[4908]: W1030 13:06:00.028541 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ebf1f1c_f8ed_4960_8f9b_2b750fcf4885.slice/crio-538e9871cc4e06181597291c69216bc176f6fea4344a5b6c09b07bdbd8cee1ce WatchSource:0}: Error finding container 538e9871cc4e06181597291c69216bc176f6fea4344a5b6c09b07bdbd8cee1ce: Status 404 returned error can't find the container with id 538e9871cc4e06181597291c69216bc176f6fea4344a5b6c09b07bdbd8cee1ce Oct 30 13:06:01.016643 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:01.016607 4908 generic.go:356] "Generic (PLEG): container finished" podID="8ebf1f1c-f8ed-4960-8f9b-2b750fcf4885" containerID="99e3be6bef768b61b57cd5e42d08321634407c6ea6f63520470b7ccf184a5260" exitCode=0 Oct 30 13:06:01.016858 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:01.016693 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2545c7-mctbc" event={"ID":"8ebf1f1c-f8ed-4960-8f9b-2b750fcf4885","Type":"ContainerDied","Data":"99e3be6bef768b61b57cd5e42d08321634407c6ea6f63520470b7ccf184a5260"} Oct 30 13:06:01.016858 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:01.016757 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2545c7-mctbc" event={"ID":"8ebf1f1c-f8ed-4960-8f9b-2b750fcf4885","Type":"ContainerStarted","Data":"538e9871cc4e06181597291c69216bc176f6fea4344a5b6c09b07bdbd8cee1ce"} Oct 30 13:06:02.212876 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:02.212818 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2545c7-mctbc" Oct 30 13:06:02.244331 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:02.244281 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ebf1f1c-f8ed-4960-8f9b-2b750fcf4885-kube-api-access-5pq7m\" (UniqueName: \"kubernetes.io/projected/8ebf1f1c-f8ed-4960-8f9b-2b750fcf4885-kube-api-access-5pq7m\") pod \"8ebf1f1c-f8ed-4960-8f9b-2b750fcf4885\" (UID: \"8ebf1f1c-f8ed-4960-8f9b-2b750fcf4885\") " Oct 30 13:06:02.244522 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:02.244466 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ebf1f1c-f8ed-4960-8f9b-2b750fcf4885-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ebf1f1c-f8ed-4960-8f9b-2b750fcf4885-maas-api-openshift-ca\") pod \"8ebf1f1c-f8ed-4960-8f9b-2b750fcf4885\" (UID: \"8ebf1f1c-f8ed-4960-8f9b-2b750fcf4885\") " Oct 30 13:06:02.244588 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:02.244564 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8ebf1f1c-f8ed-4960-8f9b-2b750fcf4885-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ebf1f1c-f8ed-4960-8f9b-2b750fcf4885-maas-api-xks-ca\") pod \"8ebf1f1c-f8ed-4960-8f9b-2b750fcf4885\" (UID: \"8ebf1f1c-f8ed-4960-8f9b-2b750fcf4885\") " Oct 30 13:06:02.244890 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:02.244862 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ebf1f1c-f8ed-4960-8f9b-2b750fcf4885-maas-api-openshift-ca" pod "8ebf1f1c-f8ed-4960-8f9b-2b750fcf4885" (UID: "8ebf1f1c-f8ed-4960-8f9b-2b750fcf4885"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:06:02.246672 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:02.246650 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ebf1f1c-f8ed-4960-8f9b-2b750fcf4885-maas-api-xks-ca" pod "8ebf1f1c-f8ed-4960-8f9b-2b750fcf4885" (UID: "8ebf1f1c-f8ed-4960-8f9b-2b750fcf4885"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:06:02.246756 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:02.246682 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ebf1f1c-f8ed-4960-8f9b-2b750fcf4885-kube-api-access-5pq7m" pod "8ebf1f1c-f8ed-4960-8f9b-2b750fcf4885" (UID: "8ebf1f1c-f8ed-4960-8f9b-2b750fcf4885"). InnerVolumeSpecName "kube-api-access-5pq7m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:06:02.346361 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:02.346270 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ebf1f1c-f8ed-4960-8f9b-2b750fcf4885-maas-api-openshift-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:06:02.346361 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:02.346316 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ebf1f1c-f8ed-4960-8f9b-2b750fcf4885-maas-api-xks-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:06:02.346361 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:02.346332 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5pq7m\" (UniqueName: \"kubernetes.io/projected/8ebf1f1c-f8ed-4960-8f9b-2b750fcf4885-kube-api-access-5pq7m\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:06:02.362506 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:02.362464 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-befaa6-openshift-default-767bb98587-6bvqz"] Oct 30 13:06:02.362887 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:02.362712 4908 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-befaa6-openshift-default-767bb98587-6bvqz" podUID="93f3f694-3b06-430c-b4f0-ed5afbea43d1" containerName="istio-proxy" containerID="cri-o://8894ef6754187ee1bbbcb40065dbe64af9a2a5d915d0602b37488466a274eda0" gracePeriod=30 Oct 30 13:06:02.968006 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:02.967968 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a786af-whx6v"] Oct 30 13:06:02.968537 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:02.968516 4908 cpu_manager.go:414] "containerMap: removing container" podUID="8ebf1f1c-f8ed-4960-8f9b-2b750fcf4885" containerName="revoke-keys" Oct 30 13:06:02.968537 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:02.968535 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ebf1f1c-f8ed-4960-8f9b-2b750fcf4885" containerName="revoke-keys" Oct 30 13:06:02.968741 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:02.968556 4908 cpu_manager.go:414] "containerMap: removing container" podUID="cfd62d0e-d1b7-4695-8673-63c1d68f4f40" containerName="istio-proxy" Oct 30 13:06:02.968741 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:02.968562 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfd62d0e-d1b7-4695-8673-63c1d68f4f40" containerName="istio-proxy" Oct 30 13:06:02.968741 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:02.968641 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ebf1f1c-f8ed-4960-8f9b-2b750fcf4885" containerName="revoke-keys" Oct 30 13:06:02.972205 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:02.972180 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a786af-whx6v" Oct 30 13:06:02.992553 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:02.992468 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a786af-whx6v"] Oct 30 13:06:03.039152 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:03.039121 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2545c7-mctbc" Oct 30 13:06:03.039152 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:03.039144 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2545c7-mctbc" event={"ID":"8ebf1f1c-f8ed-4960-8f9b-2b750fcf4885","Type":"ContainerDied","Data":"538e9871cc4e06181597291c69216bc176f6fea4344a5b6c09b07bdbd8cee1ce"} Oct 30 13:06:03.039389 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:03.039210 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="538e9871cc4e06181597291c69216bc176f6fea4344a5b6c09b07bdbd8cee1ce" Oct 30 13:06:03.076117 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:03.076059 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05338db8-262a-4d4b-8edb-8c06daa80d30-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-a786af-whx6v\" (UID: \"05338db8-262a-4d4b-8edb-8c06daa80d30\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a786af-whx6v" Oct 30 13:06:03.076323 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:03.076305 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05338db8-262a-4d4b-8edb-8c06daa80d30-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-a786af-whx6v\" (UID: \"05338db8-262a-4d4b-8edb-8c06daa80d30\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a786af-whx6v" Oct 30 13:06:03.076390 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:03.076349 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxqht\" (UniqueName: \"kubernetes.io/projected/05338db8-262a-4d4b-8edb-8c06daa80d30-kube-api-access-gxqht\") pod \"maas-api-revoke-keys-e2e-extmod-a-a786af-whx6v\" (UID: \"05338db8-262a-4d4b-8edb-8c06daa80d30\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a786af-whx6v" Oct 30 13:06:03.178280 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:03.178235 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05338db8-262a-4d4b-8edb-8c06daa80d30-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-a786af-whx6v\" (UID: \"05338db8-262a-4d4b-8edb-8c06daa80d30\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a786af-whx6v" Oct 30 13:06:03.178508 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:03.178310 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxqht\" (UniqueName: \"kubernetes.io/projected/05338db8-262a-4d4b-8edb-8c06daa80d30-kube-api-access-gxqht\") pod \"maas-api-revoke-keys-e2e-extmod-a-a786af-whx6v\" (UID: \"05338db8-262a-4d4b-8edb-8c06daa80d30\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a786af-whx6v" Oct 30 13:06:03.178508 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:03.178452 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05338db8-262a-4d4b-8edb-8c06daa80d30-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-a786af-whx6v\" (UID: \"05338db8-262a-4d4b-8edb-8c06daa80d30\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a786af-whx6v" Oct 30 13:06:03.179360 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:03.179331 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05338db8-262a-4d4b-8edb-8c06daa80d30-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-a786af-whx6v\" (UID: \"05338db8-262a-4d4b-8edb-8c06daa80d30\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a786af-whx6v" Oct 30 13:06:03.181221 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:03.181195 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05338db8-262a-4d4b-8edb-8c06daa80d30-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-a786af-whx6v\" (UID: \"05338db8-262a-4d4b-8edb-8c06daa80d30\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a786af-whx6v" Oct 30 13:06:03.201705 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:03.201672 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxqht\" (UniqueName: \"kubernetes.io/projected/05338db8-262a-4d4b-8edb-8c06daa80d30-kube-api-access-gxqht\") pod \"maas-api-revoke-keys-e2e-extmod-a-a786af-whx6v\" (UID: \"05338db8-262a-4d4b-8edb-8c06daa80d30\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a786af-whx6v" Oct 30 13:06:03.290272 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:03.290231 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a786af-whx6v" Oct 30 13:06:03.596191 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:03.596157 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a786af-whx6v"] Oct 30 13:06:03.994327 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:03.994286 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-734ffe-openshift-default-5db45fbc99-5vzmr" Oct 30 13:06:04.048535 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:04.048475 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a786af-whx6v" event={"ID":"05338db8-262a-4d4b-8edb-8c06daa80d30","Type":"ContainerStarted","Data":"9ab61f5b4a3be6fa681775e115fdfe5acad5e5eb319950739dce4b016b0f3e50"} Oct 30 13:06:04.050923 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:04.050880 4908 generic.go:356] "Generic (PLEG): container finished" podID="7e96c960-abe0-4b56-b3f9-ce46f2ce307a" containerID="e44f7f042c249b44a8aeea37b5f6c38de531e2e11fbaac541006930b57b32133" exitCode=0 Oct 30 13:06:04.051088 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:04.051021 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-734ffe-openshift-default-5db45fbc99-5vzmr" event={"ID":"7e96c960-abe0-4b56-b3f9-ce46f2ce307a","Type":"ContainerDied","Data":"e44f7f042c249b44a8aeea37b5f6c38de531e2e11fbaac541006930b57b32133"} Oct 30 13:06:04.051088 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:04.051082 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-734ffe-openshift-default-5db45fbc99-5vzmr" event={"ID":"7e96c960-abe0-4b56-b3f9-ce46f2ce307a","Type":"ContainerDied","Data":"2417315f235ea85756bf0cff4590cf2ed76756a72eaeac8e80e3e7c0c2806024"} Oct 30 13:06:04.051253 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:04.051137 4908 scope.go:122] "RemoveContainer" containerID="e44f7f042c249b44a8aeea37b5f6c38de531e2e11fbaac541006930b57b32133" Oct 30 13:06:04.051378 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:04.051354 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-734ffe-openshift-default-5db45fbc99-5vzmr" Oct 30 13:06:04.064581 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:04.064509 4908 scope.go:122] "RemoveContainer" containerID="0f32041b6b6e591be1af63761a9a5f99db3675b682c4ee06b9fa1e362c1dbf5d" Oct 30 13:06:04.077899 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:04.077519 4908 scope.go:122] "RemoveContainer" containerID="e44f7f042c249b44a8aeea37b5f6c38de531e2e11fbaac541006930b57b32133" Oct 30 13:06:04.078446 ip-10-0-141-251 kubenswrapper[4908]: E1030 13:06:04.078291 4908 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e44f7f042c249b44a8aeea37b5f6c38de531e2e11fbaac541006930b57b32133\": container with ID starting with e44f7f042c249b44a8aeea37b5f6c38de531e2e11fbaac541006930b57b32133 not found: ID does not exist" containerID="e44f7f042c249b44a8aeea37b5f6c38de531e2e11fbaac541006930b57b32133" Oct 30 13:06:04.078446 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:04.078336 4908 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e44f7f042c249b44a8aeea37b5f6c38de531e2e11fbaac541006930b57b32133"} err="failed to get container status \"e44f7f042c249b44a8aeea37b5f6c38de531e2e11fbaac541006930b57b32133\": rpc error: code = NotFound desc = could not find container \"e44f7f042c249b44a8aeea37b5f6c38de531e2e11fbaac541006930b57b32133\": container with ID starting with e44f7f042c249b44a8aeea37b5f6c38de531e2e11fbaac541006930b57b32133 not found: ID does not exist" Oct 30 13:06:04.078446 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:04.078363 4908 scope.go:122] "RemoveContainer" containerID="0f32041b6b6e591be1af63761a9a5f99db3675b682c4ee06b9fa1e362c1dbf5d" Oct 30 13:06:04.078795 ip-10-0-141-251 kubenswrapper[4908]: E1030 13:06:04.078765 4908 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f32041b6b6e591be1af63761a9a5f99db3675b682c4ee06b9fa1e362c1dbf5d\": container with ID starting with 0f32041b6b6e591be1af63761a9a5f99db3675b682c4ee06b9fa1e362c1dbf5d not found: ID does not exist" containerID="0f32041b6b6e591be1af63761a9a5f99db3675b682c4ee06b9fa1e362c1dbf5d" Oct 30 13:06:04.078931 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:04.078801 4908 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f32041b6b6e591be1af63761a9a5f99db3675b682c4ee06b9fa1e362c1dbf5d"} err="failed to get container status \"0f32041b6b6e591be1af63761a9a5f99db3675b682c4ee06b9fa1e362c1dbf5d\": rpc error: code = NotFound desc = could not find container \"0f32041b6b6e591be1af63761a9a5f99db3675b682c4ee06b9fa1e362c1dbf5d\": container with ID starting with 0f32041b6b6e591be1af63761a9a5f99db3675b682c4ee06b9fa1e362c1dbf5d not found: ID does not exist" Oct 30 13:06:04.091429 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:04.091403 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e96c960-abe0-4b56-b3f9-ce46f2ce307a-istio-token\" (UniqueName: \"kubernetes.io/projected/7e96c960-abe0-4b56-b3f9-ce46f2ce307a-istio-token\") pod \"7e96c960-abe0-4b56-b3f9-ce46f2ce307a\" (UID: \"7e96c960-abe0-4b56-b3f9-ce46f2ce307a\") " Oct 30 13:06:04.091556 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:04.091486 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e96c960-abe0-4b56-b3f9-ce46f2ce307a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e96c960-abe0-4b56-b3f9-ce46f2ce307a-istio-envoy\") pod \"7e96c960-abe0-4b56-b3f9-ce46f2ce307a\" (UID: \"7e96c960-abe0-4b56-b3f9-ce46f2ce307a\") " Oct 30 13:06:04.091556 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:04.091533 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e96c960-abe0-4b56-b3f9-ce46f2ce307a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e96c960-abe0-4b56-b3f9-ce46f2ce307a-workload-socket\") pod \"7e96c960-abe0-4b56-b3f9-ce46f2ce307a\" (UID: \"7e96c960-abe0-4b56-b3f9-ce46f2ce307a\") " Oct 30 13:06:04.091692 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:04.091571 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e96c960-abe0-4b56-b3f9-ce46f2ce307a-kube-api-access-q9pzp\" (UniqueName: \"kubernetes.io/projected/7e96c960-abe0-4b56-b3f9-ce46f2ce307a-kube-api-access-q9pzp\") pod \"7e96c960-abe0-4b56-b3f9-ce46f2ce307a\" (UID: \"7e96c960-abe0-4b56-b3f9-ce46f2ce307a\") " Oct 30 13:06:04.091692 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:04.091663 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e96c960-abe0-4b56-b3f9-ce46f2ce307a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e96c960-abe0-4b56-b3f9-ce46f2ce307a-workload-certs\") pod \"7e96c960-abe0-4b56-b3f9-ce46f2ce307a\" (UID: \"7e96c960-abe0-4b56-b3f9-ce46f2ce307a\") " Oct 30 13:06:04.091807 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:04.091720 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e96c960-abe0-4b56-b3f9-ce46f2ce307a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e96c960-abe0-4b56-b3f9-ce46f2ce307a-istiod-ca-cert\") pod \"7e96c960-abe0-4b56-b3f9-ce46f2ce307a\" (UID: \"7e96c960-abe0-4b56-b3f9-ce46f2ce307a\") " Oct 30 13:06:04.091807 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:04.091768 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7e96c960-abe0-4b56-b3f9-ce46f2ce307a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e96c960-abe0-4b56-b3f9-ce46f2ce307a-istio-podinfo\") pod \"7e96c960-abe0-4b56-b3f9-ce46f2ce307a\" (UID: \"7e96c960-abe0-4b56-b3f9-ce46f2ce307a\") " Oct 30 13:06:04.091933 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:04.091879 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e96c960-abe0-4b56-b3f9-ce46f2ce307a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e96c960-abe0-4b56-b3f9-ce46f2ce307a-credential-socket\") pod \"7e96c960-abe0-4b56-b3f9-ce46f2ce307a\" (UID: \"7e96c960-abe0-4b56-b3f9-ce46f2ce307a\") " Oct 30 13:06:04.092009 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:04.091958 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e96c960-abe0-4b56-b3f9-ce46f2ce307a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e96c960-abe0-4b56-b3f9-ce46f2ce307a-istio-data\") pod \"7e96c960-abe0-4b56-b3f9-ce46f2ce307a\" (UID: \"7e96c960-abe0-4b56-b3f9-ce46f2ce307a\") " Oct 30 13:06:04.092066 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:04.092044 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e96c960-abe0-4b56-b3f9-ce46f2ce307a-workload-certs" pod "7e96c960-abe0-4b56-b3f9-ce46f2ce307a" (UID: "7e96c960-abe0-4b56-b3f9-ce46f2ce307a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:06:04.092124 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:04.092090 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e96c960-abe0-4b56-b3f9-ce46f2ce307a-workload-socket" pod "7e96c960-abe0-4b56-b3f9-ce46f2ce307a" (UID: "7e96c960-abe0-4b56-b3f9-ce46f2ce307a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:06:04.092571 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:04.092525 4908 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e96c960-abe0-4b56-b3f9-ce46f2ce307a-workload-socket\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:06:04.092571 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:04.092550 4908 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e96c960-abe0-4b56-b3f9-ce46f2ce307a-workload-certs\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:06:04.092773 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:04.092749 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e96c960-abe0-4b56-b3f9-ce46f2ce307a-credential-socket" pod "7e96c960-abe0-4b56-b3f9-ce46f2ce307a" (UID: "7e96c960-abe0-4b56-b3f9-ce46f2ce307a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:06:04.092862 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:04.092771 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e96c960-abe0-4b56-b3f9-ce46f2ce307a-istio-data" pod "7e96c960-abe0-4b56-b3f9-ce46f2ce307a" (UID: "7e96c960-abe0-4b56-b3f9-ce46f2ce307a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:06:04.092924 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:04.092874 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e96c960-abe0-4b56-b3f9-ce46f2ce307a-istiod-ca-cert" pod "7e96c960-abe0-4b56-b3f9-ce46f2ce307a" (UID: "7e96c960-abe0-4b56-b3f9-ce46f2ce307a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:06:04.094369 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:04.094337 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e96c960-abe0-4b56-b3f9-ce46f2ce307a-kube-api-access-q9pzp" pod "7e96c960-abe0-4b56-b3f9-ce46f2ce307a" (UID: "7e96c960-abe0-4b56-b3f9-ce46f2ce307a"). InnerVolumeSpecName "kube-api-access-q9pzp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:06:04.094484 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:04.094443 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e96c960-abe0-4b56-b3f9-ce46f2ce307a-istio-envoy" pod "7e96c960-abe0-4b56-b3f9-ce46f2ce307a" (UID: "7e96c960-abe0-4b56-b3f9-ce46f2ce307a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:06:04.094756 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:04.094721 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7e96c960-abe0-4b56-b3f9-ce46f2ce307a-istio-podinfo" pod "7e96c960-abe0-4b56-b3f9-ce46f2ce307a" (UID: "7e96c960-abe0-4b56-b3f9-ce46f2ce307a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:06:04.094897 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:04.094845 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e96c960-abe0-4b56-b3f9-ce46f2ce307a-istio-token" pod "7e96c960-abe0-4b56-b3f9-ce46f2ce307a" (UID: "7e96c960-abe0-4b56-b3f9-ce46f2ce307a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:06:04.194139 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:04.194047 4908 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e96c960-abe0-4b56-b3f9-ce46f2ce307a-credential-socket\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:06:04.194139 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:04.194086 4908 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e96c960-abe0-4b56-b3f9-ce46f2ce307a-istio-data\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:06:04.194139 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:04.194101 4908 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e96c960-abe0-4b56-b3f9-ce46f2ce307a-istio-token\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:06:04.194139 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:04.194114 4908 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e96c960-abe0-4b56-b3f9-ce46f2ce307a-istio-envoy\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:06:04.194139 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:04.194127 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9pzp\" (UniqueName: \"kubernetes.io/projected/7e96c960-abe0-4b56-b3f9-ce46f2ce307a-kube-api-access-q9pzp\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:06:04.194139 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:04.194140 4908 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e96c960-abe0-4b56-b3f9-ce46f2ce307a-istiod-ca-cert\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:06:04.194457 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:04.194153 4908 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e96c960-abe0-4b56-b3f9-ce46f2ce307a-istio-podinfo\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:06:04.398550 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:04.398511 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-734ffe-openshift-default-5db45fbc99-5vzmr"] Oct 30 13:06:04.408792 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:04.408754 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-734ffe-openshift-default-5db45fbc99-5vzmr"] Oct 30 13:06:04.786453 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:04.786417 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9wwfc"] Oct 30 13:06:05.062987 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:05.062947 4908 generic.go:356] "Generic (PLEG): container finished" podID="05338db8-262a-4d4b-8edb-8c06daa80d30" containerID="6d04a4d751f0fb9c4b41ce740ae18a2305b78c3a80797d56806b8afaa4e0e65a" exitCode=0 Oct 30 13:06:05.063182 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:05.063002 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a786af-whx6v" event={"ID":"05338db8-262a-4d4b-8edb-8c06daa80d30","Type":"ContainerDied","Data":"6d04a4d751f0fb9c4b41ce740ae18a2305b78c3a80797d56806b8afaa4e0e65a"} Oct 30 13:06:05.609194 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:05.609154 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e96c960-abe0-4b56-b3f9-ce46f2ce307a" path="/var/lib/kubelet/pods/7e96c960-abe0-4b56-b3f9-ce46f2ce307a/volumes" Oct 30 13:06:06.295632 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:06.295598 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a786af-whx6v" Oct 30 13:06:06.336859 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:06.336758 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05338db8-262a-4d4b-8edb-8c06daa80d30-kube-api-access-gxqht\" (UniqueName: \"kubernetes.io/projected/05338db8-262a-4d4b-8edb-8c06daa80d30-kube-api-access-gxqht\") pod \"05338db8-262a-4d4b-8edb-8c06daa80d30\" (UID: \"05338db8-262a-4d4b-8edb-8c06daa80d30\") " Oct 30 13:06:06.337032 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:06.336948 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/05338db8-262a-4d4b-8edb-8c06daa80d30-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05338db8-262a-4d4b-8edb-8c06daa80d30-maas-api-xks-ca\") pod \"05338db8-262a-4d4b-8edb-8c06daa80d30\" (UID: \"05338db8-262a-4d4b-8edb-8c06daa80d30\") " Oct 30 13:06:06.337119 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:06.337103 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05338db8-262a-4d4b-8edb-8c06daa80d30-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05338db8-262a-4d4b-8edb-8c06daa80d30-maas-api-openshift-ca\") pod \"05338db8-262a-4d4b-8edb-8c06daa80d30\" (UID: \"05338db8-262a-4d4b-8edb-8c06daa80d30\") " Oct 30 13:06:06.337845 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:06.337800 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05338db8-262a-4d4b-8edb-8c06daa80d30-maas-api-openshift-ca" pod "05338db8-262a-4d4b-8edb-8c06daa80d30" (UID: "05338db8-262a-4d4b-8edb-8c06daa80d30"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:06:06.342602 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:06.341964 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05338db8-262a-4d4b-8edb-8c06daa80d30-maas-api-xks-ca" pod "05338db8-262a-4d4b-8edb-8c06daa80d30" (UID: "05338db8-262a-4d4b-8edb-8c06daa80d30"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:06:06.342602 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:06.342011 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05338db8-262a-4d4b-8edb-8c06daa80d30-kube-api-access-gxqht" pod "05338db8-262a-4d4b-8edb-8c06daa80d30" (UID: "05338db8-262a-4d4b-8edb-8c06daa80d30"). InnerVolumeSpecName "kube-api-access-gxqht". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:06:06.438468 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:06.438433 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05338db8-262a-4d4b-8edb-8c06daa80d30-maas-api-openshift-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:06:06.438468 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:06.438464 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gxqht\" (UniqueName: \"kubernetes.io/projected/05338db8-262a-4d4b-8edb-8c06daa80d30-kube-api-access-gxqht\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:06:06.438468 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:06.438474 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05338db8-262a-4d4b-8edb-8c06daa80d30-maas-api-xks-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:06:07.091754 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:07.091718 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a786af-whx6v" Oct 30 13:06:07.092189 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:07.091749 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a786af-whx6v" event={"ID":"05338db8-262a-4d4b-8edb-8c06daa80d30","Type":"ContainerDied","Data":"9ab61f5b4a3be6fa681775e115fdfe5acad5e5eb319950739dce4b016b0f3e50"} Oct 30 13:06:07.092189 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:07.091802 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9ab61f5b4a3be6fa681775e115fdfe5acad5e5eb319950739dce4b016b0f3e50" Oct 30 13:06:07.748388 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:07.748351 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-befaa6-openshift-default-767bb98587-6bvqz" Oct 30 13:06:07.835667 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:07.835630 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93f3f694-3b06-430c-b4f0-ed5afbea43d1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93f3f694-3b06-430c-b4f0-ed5afbea43d1-workload-socket\") pod \"93f3f694-3b06-430c-b4f0-ed5afbea43d1\" (UID: \"93f3f694-3b06-430c-b4f0-ed5afbea43d1\") " Oct 30 13:06:07.835667 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:07.835673 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/93f3f694-3b06-430c-b4f0-ed5afbea43d1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93f3f694-3b06-430c-b4f0-ed5afbea43d1-istio-podinfo\") pod \"93f3f694-3b06-430c-b4f0-ed5afbea43d1\" (UID: \"93f3f694-3b06-430c-b4f0-ed5afbea43d1\") " Oct 30 13:06:07.835976 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:07.835704 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93f3f694-3b06-430c-b4f0-ed5afbea43d1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93f3f694-3b06-430c-b4f0-ed5afbea43d1-workload-certs\") pod \"93f3f694-3b06-430c-b4f0-ed5afbea43d1\" (UID: \"93f3f694-3b06-430c-b4f0-ed5afbea43d1\") " Oct 30 13:06:07.835976 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:07.835900 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93f3f694-3b06-430c-b4f0-ed5afbea43d1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93f3f694-3b06-430c-b4f0-ed5afbea43d1-istio-data\") pod \"93f3f694-3b06-430c-b4f0-ed5afbea43d1\" (UID: \"93f3f694-3b06-430c-b4f0-ed5afbea43d1\") " Oct 30 13:06:07.835976 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:07.835961 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93f3f694-3b06-430c-b4f0-ed5afbea43d1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93f3f694-3b06-430c-b4f0-ed5afbea43d1-istio-envoy\") pod \"93f3f694-3b06-430c-b4f0-ed5afbea43d1\" (UID: \"93f3f694-3b06-430c-b4f0-ed5afbea43d1\") " Oct 30 13:06:07.836145 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:07.835968 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93f3f694-3b06-430c-b4f0-ed5afbea43d1-workload-socket" pod "93f3f694-3b06-430c-b4f0-ed5afbea43d1" (UID: "93f3f694-3b06-430c-b4f0-ed5afbea43d1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:06:07.836145 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:07.835999 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93f3f694-3b06-430c-b4f0-ed5afbea43d1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93f3f694-3b06-430c-b4f0-ed5afbea43d1-credential-socket\") pod \"93f3f694-3b06-430c-b4f0-ed5afbea43d1\" (UID: \"93f3f694-3b06-430c-b4f0-ed5afbea43d1\") " Oct 30 13:06:07.836145 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:07.836026 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93f3f694-3b06-430c-b4f0-ed5afbea43d1-workload-certs" pod "93f3f694-3b06-430c-b4f0-ed5afbea43d1" (UID: "93f3f694-3b06-430c-b4f0-ed5afbea43d1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:06:07.836145 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:07.836045 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93f3f694-3b06-430c-b4f0-ed5afbea43d1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93f3f694-3b06-430c-b4f0-ed5afbea43d1-istiod-ca-cert\") pod \"93f3f694-3b06-430c-b4f0-ed5afbea43d1\" (UID: \"93f3f694-3b06-430c-b4f0-ed5afbea43d1\") " Oct 30 13:06:07.836363 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:07.836156 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93f3f694-3b06-430c-b4f0-ed5afbea43d1-istio-token\" (UniqueName: \"kubernetes.io/projected/93f3f694-3b06-430c-b4f0-ed5afbea43d1-istio-token\") pod \"93f3f694-3b06-430c-b4f0-ed5afbea43d1\" (UID: \"93f3f694-3b06-430c-b4f0-ed5afbea43d1\") " Oct 30 13:06:07.836363 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:07.836173 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93f3f694-3b06-430c-b4f0-ed5afbea43d1-istio-data" pod "93f3f694-3b06-430c-b4f0-ed5afbea43d1" (UID: "93f3f694-3b06-430c-b4f0-ed5afbea43d1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:06:07.836363 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:07.836217 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93f3f694-3b06-430c-b4f0-ed5afbea43d1-credential-socket" pod "93f3f694-3b06-430c-b4f0-ed5afbea43d1" (UID: "93f3f694-3b06-430c-b4f0-ed5afbea43d1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:06:07.836363 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:07.836214 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93f3f694-3b06-430c-b4f0-ed5afbea43d1-kube-api-access-6qjkl\" (UniqueName: \"kubernetes.io/projected/93f3f694-3b06-430c-b4f0-ed5afbea43d1-kube-api-access-6qjkl\") pod \"93f3f694-3b06-430c-b4f0-ed5afbea43d1\" (UID: \"93f3f694-3b06-430c-b4f0-ed5afbea43d1\") " Oct 30 13:06:07.836608 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:07.836409 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93f3f694-3b06-430c-b4f0-ed5afbea43d1-istiod-ca-cert" pod "93f3f694-3b06-430c-b4f0-ed5afbea43d1" (UID: "93f3f694-3b06-430c-b4f0-ed5afbea43d1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:06:07.836813 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:07.836787 4908 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93f3f694-3b06-430c-b4f0-ed5afbea43d1-workload-certs\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:06:07.836930 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:07.836816 4908 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93f3f694-3b06-430c-b4f0-ed5afbea43d1-istio-data\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:06:07.836930 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:07.836851 4908 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93f3f694-3b06-430c-b4f0-ed5afbea43d1-credential-socket\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:06:07.836930 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:07.836874 4908 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93f3f694-3b06-430c-b4f0-ed5afbea43d1-istiod-ca-cert\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:06:07.836930 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:07.836886 4908 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93f3f694-3b06-430c-b4f0-ed5afbea43d1-workload-socket\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:06:07.842364 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:07.842333 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/93f3f694-3b06-430c-b4f0-ed5afbea43d1-istio-podinfo" pod "93f3f694-3b06-430c-b4f0-ed5afbea43d1" (UID: "93f3f694-3b06-430c-b4f0-ed5afbea43d1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:06:07.847361 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:07.847328 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93f3f694-3b06-430c-b4f0-ed5afbea43d1-kube-api-access-6qjkl" pod "93f3f694-3b06-430c-b4f0-ed5afbea43d1" (UID: "93f3f694-3b06-430c-b4f0-ed5afbea43d1"). InnerVolumeSpecName "kube-api-access-6qjkl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:06:07.847977 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:07.847949 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93f3f694-3b06-430c-b4f0-ed5afbea43d1-istio-envoy" pod "93f3f694-3b06-430c-b4f0-ed5afbea43d1" (UID: "93f3f694-3b06-430c-b4f0-ed5afbea43d1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:06:07.850981 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:07.850953 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93f3f694-3b06-430c-b4f0-ed5afbea43d1-istio-token" pod "93f3f694-3b06-430c-b4f0-ed5afbea43d1" (UID: "93f3f694-3b06-430c-b4f0-ed5afbea43d1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:06:07.945938 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:07.942434 4908 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93f3f694-3b06-430c-b4f0-ed5afbea43d1-istio-podinfo\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:06:07.947390 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:07.946171 4908 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93f3f694-3b06-430c-b4f0-ed5afbea43d1-istio-envoy\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:06:07.951229 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:07.951196 4908 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93f3f694-3b06-430c-b4f0-ed5afbea43d1-istio-token\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:06:07.951383 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:07.951232 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6qjkl\" (UniqueName: \"kubernetes.io/projected/93f3f694-3b06-430c-b4f0-ed5afbea43d1-kube-api-access-6qjkl\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:06:08.102218 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:08.102178 4908 generic.go:356] "Generic (PLEG): container finished" podID="93f3f694-3b06-430c-b4f0-ed5afbea43d1" containerID="8894ef6754187ee1bbbcb40065dbe64af9a2a5d915d0602b37488466a274eda0" exitCode=0 Oct 30 13:06:08.102669 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:08.102235 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-befaa6-openshift-default-767bb98587-6bvqz" Oct 30 13:06:08.102669 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:08.102265 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-befaa6-openshift-default-767bb98587-6bvqz" event={"ID":"93f3f694-3b06-430c-b4f0-ed5afbea43d1","Type":"ContainerDied","Data":"8894ef6754187ee1bbbcb40065dbe64af9a2a5d915d0602b37488466a274eda0"} Oct 30 13:06:08.102669 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:08.102355 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-befaa6-openshift-default-767bb98587-6bvqz" event={"ID":"93f3f694-3b06-430c-b4f0-ed5afbea43d1","Type":"ContainerDied","Data":"16328ff68490e6046260f446763ea1165bdd57c4280c4f808674ce62cebdef44"} Oct 30 13:06:08.102669 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:08.102380 4908 scope.go:122] "RemoveContainer" containerID="8894ef6754187ee1bbbcb40065dbe64af9a2a5d915d0602b37488466a274eda0" Oct 30 13:06:08.114395 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:08.114375 4908 scope.go:122] "RemoveContainer" containerID="fd793e758613ea073329301aa04b1d67dc621e22ffb6618363f63290a4ffbe51" Oct 30 13:06:08.124706 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:08.124682 4908 scope.go:122] "RemoveContainer" containerID="8894ef6754187ee1bbbcb40065dbe64af9a2a5d915d0602b37488466a274eda0" Oct 30 13:06:08.125001 ip-10-0-141-251 kubenswrapper[4908]: E1030 13:06:08.124981 4908 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8894ef6754187ee1bbbcb40065dbe64af9a2a5d915d0602b37488466a274eda0\": container with ID starting with 8894ef6754187ee1bbbcb40065dbe64af9a2a5d915d0602b37488466a274eda0 not found: ID does not exist" containerID="8894ef6754187ee1bbbcb40065dbe64af9a2a5d915d0602b37488466a274eda0" Oct 30 13:06:08.125050 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:08.125012 4908 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8894ef6754187ee1bbbcb40065dbe64af9a2a5d915d0602b37488466a274eda0"} err="failed to get container status \"8894ef6754187ee1bbbcb40065dbe64af9a2a5d915d0602b37488466a274eda0\": rpc error: code = NotFound desc = could not find container \"8894ef6754187ee1bbbcb40065dbe64af9a2a5d915d0602b37488466a274eda0\": container with ID starting with 8894ef6754187ee1bbbcb40065dbe64af9a2a5d915d0602b37488466a274eda0 not found: ID does not exist" Oct 30 13:06:08.125050 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:08.125031 4908 scope.go:122] "RemoveContainer" containerID="fd793e758613ea073329301aa04b1d67dc621e22ffb6618363f63290a4ffbe51" Oct 30 13:06:08.125263 ip-10-0-141-251 kubenswrapper[4908]: E1030 13:06:08.125248 4908 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd793e758613ea073329301aa04b1d67dc621e22ffb6618363f63290a4ffbe51\": container with ID starting with fd793e758613ea073329301aa04b1d67dc621e22ffb6618363f63290a4ffbe51 not found: ID does not exist" containerID="fd793e758613ea073329301aa04b1d67dc621e22ffb6618363f63290a4ffbe51" Oct 30 13:06:08.125303 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:08.125266 4908 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd793e758613ea073329301aa04b1d67dc621e22ffb6618363f63290a4ffbe51"} err="failed to get container status \"fd793e758613ea073329301aa04b1d67dc621e22ffb6618363f63290a4ffbe51\": rpc error: code = NotFound desc = could not find container \"fd793e758613ea073329301aa04b1d67dc621e22ffb6618363f63290a4ffbe51\": container with ID starting with fd793e758613ea073329301aa04b1d67dc621e22ffb6618363f63290a4ffbe51 not found: ID does not exist" Oct 30 13:06:08.153277 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:08.153247 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-befaa6-openshift-default-767bb98587-6bvqz"] Oct 30 13:06:08.160459 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:08.160434 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-befaa6-openshift-default-767bb98587-6bvqz"] Oct 30 13:06:08.199909 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:08.188101 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c1b1e8-tdxb5"] Oct 30 13:06:08.212082 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:08.207942 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c1b1e8-tdxb5"] Oct 30 13:06:09.609108 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:09.609019 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92dbbda1-816b-4692-8b8f-85441a6609d0" path="/var/lib/kubelet/pods/92dbbda1-816b-4692-8b8f-85441a6609d0/volumes" Oct 30 13:06:09.609621 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:09.609602 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93f3f694-3b06-430c-b4f0-ed5afbea43d1" path="/var/lib/kubelet/pods/93f3f694-3b06-430c-b4f0-ed5afbea43d1/volumes" Oct 30 13:06:10.415681 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:10.415639 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-73bafc-9zb9m"] Oct 30 13:06:10.426530 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:10.426497 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-73bafc-9zb9m"] Oct 30 13:06:11.606290 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:11.606233 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d59db20-5049-4880-9fe4-19156f79dae6" path="/var/lib/kubelet/pods/4d59db20-5049-4880-9fe4-19156f79dae6/volumes" Oct 30 13:06:16.199853 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:16.199152 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1333c8-rf6wc"] Oct 30 13:06:16.213698 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:16.212135 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1333c8-rf6wc"] Oct 30 13:06:17.609452 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:17.609412 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f1942a2-3706-4429-b3a4-c6eba3eb393c" path="/var/lib/kubelet/pods/2f1942a2-3706-4429-b3a4-c6eba3eb393c/volumes" Oct 30 13:06:18.404739 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:18.404698 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-73bafc-openshift-default-64bfdb8459-rfkqt"] Oct 30 13:06:18.405279 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:18.405229 4908 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-73bafc-openshift-default-64bfdb8459-rfkqt" podUID="c9c9d1f3-e29c-42d4-96e6-765da8de6ab9" containerName="istio-proxy" containerID="cri-o://19cf3f8257259bb14b18c415fd3fceb8beec31070c4c2a6fc2f61f373e2737a6" gracePeriod=30 Oct 30 13:06:18.952903 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:18.952865 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-dd0628-xdwfv"] Oct 30 13:06:18.953480 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:18.953464 4908 cpu_manager.go:414] "containerMap: removing container" podUID="93f3f694-3b06-430c-b4f0-ed5afbea43d1" containerName="istio-proxy" Oct 30 13:06:18.953530 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:18.953482 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93f3f694-3b06-430c-b4f0-ed5afbea43d1" containerName="istio-proxy" Oct 30 13:06:18.953530 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:18.953500 4908 cpu_manager.go:414] "containerMap: removing container" podUID="7e96c960-abe0-4b56-b3f9-ce46f2ce307a" containerName="istio-proxy" Oct 30 13:06:18.953530 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:18.953506 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e96c960-abe0-4b56-b3f9-ce46f2ce307a" containerName="istio-proxy" Oct 30 13:06:18.953530 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:18.953514 4908 cpu_manager.go:414] "containerMap: removing container" podUID="93f3f694-3b06-430c-b4f0-ed5afbea43d1" containerName="istio-proxy" Oct 30 13:06:18.953530 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:18.953519 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93f3f694-3b06-430c-b4f0-ed5afbea43d1" containerName="istio-proxy" Oct 30 13:06:18.953530 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:18.953528 4908 cpu_manager.go:414] "containerMap: removing container" podUID="05338db8-262a-4d4b-8edb-8c06daa80d30" containerName="revoke-keys" Oct 30 13:06:18.953530 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:18.953532 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05338db8-262a-4d4b-8edb-8c06daa80d30" containerName="revoke-keys" Oct 30 13:06:18.953766 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:18.953541 4908 cpu_manager.go:414] "containerMap: removing container" podUID="7e96c960-abe0-4b56-b3f9-ce46f2ce307a" containerName="istio-proxy" Oct 30 13:06:18.953766 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:18.953546 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e96c960-abe0-4b56-b3f9-ce46f2ce307a" containerName="istio-proxy" Oct 30 13:06:18.953766 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:18.953652 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="93f3f694-3b06-430c-b4f0-ed5afbea43d1" containerName="istio-proxy" Oct 30 13:06:18.953766 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:18.953662 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e96c960-abe0-4b56-b3f9-ce46f2ce307a" containerName="istio-proxy" Oct 30 13:06:18.953766 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:18.953668 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="05338db8-262a-4d4b-8edb-8c06daa80d30" containerName="revoke-keys" Oct 30 13:06:18.953766 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:18.953675 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e96c960-abe0-4b56-b3f9-ce46f2ce307a" containerName="istio-proxy" Oct 30 13:06:18.957425 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:18.957405 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-dd0628-xdwfv" Oct 30 13:06:18.982882 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:18.982819 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 13:06:18.983062 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:18.982941 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:06:18.983062 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:18.982955 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 13:06:18.983062 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:18.983024 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s75hq\"" Oct 30 13:06:18.987863 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:18.987810 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-dd0628-xdwfv"] Oct 30 13:06:19.080144 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:19.080107 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdkqw\" (UniqueName: \"kubernetes.io/projected/ef849d43-7bb0-4096-9736-40ae518c6526-kube-api-access-rdkqw\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-dd0628-xdwfv\" (UID: \"ef849d43-7bb0-4096-9736-40ae518c6526\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-dd0628-xdwfv" Oct 30 13:06:19.080321 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:19.080148 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef849d43-7bb0-4096-9736-40ae518c6526-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-dd0628-xdwfv\" (UID: \"ef849d43-7bb0-4096-9736-40ae518c6526\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-dd0628-xdwfv" Oct 30 13:06:19.080391 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:19.080364 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef849d43-7bb0-4096-9736-40ae518c6526-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-dd0628-xdwfv\" (UID: \"ef849d43-7bb0-4096-9736-40ae518c6526\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-dd0628-xdwfv" Oct 30 13:06:19.181583 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:19.181543 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdkqw\" (UniqueName: \"kubernetes.io/projected/ef849d43-7bb0-4096-9736-40ae518c6526-kube-api-access-rdkqw\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-dd0628-xdwfv\" (UID: \"ef849d43-7bb0-4096-9736-40ae518c6526\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-dd0628-xdwfv" Oct 30 13:06:19.181583 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:19.181582 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef849d43-7bb0-4096-9736-40ae518c6526-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-dd0628-xdwfv\" (UID: \"ef849d43-7bb0-4096-9736-40ae518c6526\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-dd0628-xdwfv" Oct 30 13:06:19.181871 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:19.181660 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef849d43-7bb0-4096-9736-40ae518c6526-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-dd0628-xdwfv\" (UID: \"ef849d43-7bb0-4096-9736-40ae518c6526\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-dd0628-xdwfv" Oct 30 13:06:19.182521 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:19.182491 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef849d43-7bb0-4096-9736-40ae518c6526-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-dd0628-xdwfv\" (UID: \"ef849d43-7bb0-4096-9736-40ae518c6526\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-dd0628-xdwfv" Oct 30 13:06:19.184198 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:19.184175 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef849d43-7bb0-4096-9736-40ae518c6526-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-dd0628-xdwfv\" (UID: \"ef849d43-7bb0-4096-9736-40ae518c6526\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-dd0628-xdwfv" Oct 30 13:06:19.204375 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:19.204305 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdkqw\" (UniqueName: \"kubernetes.io/projected/ef849d43-7bb0-4096-9736-40ae518c6526-kube-api-access-rdkqw\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-dd0628-xdwfv\" (UID: \"ef849d43-7bb0-4096-9736-40ae518c6526\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-dd0628-xdwfv" Oct 30 13:06:19.271162 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:19.271130 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-dd0628-xdwfv" Oct 30 13:06:19.576278 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:19.576232 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-dd0628-xdwfv"] Oct 30 13:06:19.579076 ip-10-0-141-251 kubenswrapper[4908]: W1030 13:06:19.579032 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef849d43_7bb0_4096_9736_40ae518c6526.slice/crio-cad88d84afd362738dfba4660fb21c4ec75961b6f33b1ceeb28f11a5f42f1e8c WatchSource:0}: Error finding container cad88d84afd362738dfba4660fb21c4ec75961b6f33b1ceeb28f11a5f42f1e8c: Status 404 returned error can't find the container with id cad88d84afd362738dfba4660fb21c4ec75961b6f33b1ceeb28f11a5f42f1e8c Oct 30 13:06:19.685369 ip-10-0-141-251 kubenswrapper[4908]: E1030 13:06:19.685327 4908 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vn6vv" containerName="extractor" Oct 30 13:06:20.226264 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:20.226163 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-dd0628-xdwfv" event={"ID":"ef849d43-7bb0-4096-9736-40ae518c6526","Type":"ContainerStarted","Data":"1e0ce98464dea8d08dfdb703655d3540f4ef4a5189fc954cff7be7bb58877470"} Oct 30 13:06:20.226264 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:20.226229 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-dd0628-xdwfv" event={"ID":"ef849d43-7bb0-4096-9736-40ae518c6526","Type":"ContainerStarted","Data":"cad88d84afd362738dfba4660fb21c4ec75961b6f33b1ceeb28f11a5f42f1e8c"} Oct 30 13:06:20.270960 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:20.270905 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-dd0628-xdwfv" podStartSLOduration=2.270890623 podStartE2EDuration="2.270890623s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:06:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:06:20.268983983 +0000 UTC m=+4336.783412731" watchObservedRunningTime="2026-10-30 13:06:20.270890623 +0000 UTC m=+4336.785319427" Oct 30 13:06:21.238683 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:21.238646 4908 generic.go:356] "Generic (PLEG): container finished" podID="ef849d43-7bb0-4096-9736-40ae518c6526" containerID="1e0ce98464dea8d08dfdb703655d3540f4ef4a5189fc954cff7be7bb58877470" exitCode=0 Oct 30 13:06:21.239107 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:21.238744 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-dd0628-xdwfv" event={"ID":"ef849d43-7bb0-4096-9736-40ae518c6526","Type":"ContainerDied","Data":"1e0ce98464dea8d08dfdb703655d3540f4ef4a5189fc954cff7be7bb58877470"} Oct 30 13:06:22.407769 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:22.407742 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-dd0628-xdwfv" Oct 30 13:06:22.420767 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:22.420733 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef849d43-7bb0-4096-9736-40ae518c6526-kube-api-access-rdkqw\" (UniqueName: \"kubernetes.io/projected/ef849d43-7bb0-4096-9736-40ae518c6526-kube-api-access-rdkqw\") pod \"ef849d43-7bb0-4096-9736-40ae518c6526\" (UID: \"ef849d43-7bb0-4096-9736-40ae518c6526\") " Oct 30 13:06:22.420952 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:22.420937 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef849d43-7bb0-4096-9736-40ae518c6526-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef849d43-7bb0-4096-9736-40ae518c6526-maas-api-xks-ca\") pod \"ef849d43-7bb0-4096-9736-40ae518c6526\" (UID: \"ef849d43-7bb0-4096-9736-40ae518c6526\") " Oct 30 13:06:22.421023 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:22.421009 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef849d43-7bb0-4096-9736-40ae518c6526-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef849d43-7bb0-4096-9736-40ae518c6526-maas-api-openshift-ca\") pod \"ef849d43-7bb0-4096-9736-40ae518c6526\" (UID: \"ef849d43-7bb0-4096-9736-40ae518c6526\") " Oct 30 13:06:22.421514 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:22.421474 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef849d43-7bb0-4096-9736-40ae518c6526-maas-api-openshift-ca" pod "ef849d43-7bb0-4096-9736-40ae518c6526" (UID: "ef849d43-7bb0-4096-9736-40ae518c6526"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:06:22.423236 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:22.423211 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef849d43-7bb0-4096-9736-40ae518c6526-kube-api-access-rdkqw" pod "ef849d43-7bb0-4096-9736-40ae518c6526" (UID: "ef849d43-7bb0-4096-9736-40ae518c6526"). InnerVolumeSpecName "kube-api-access-rdkqw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:06:22.423512 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:22.423482 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef849d43-7bb0-4096-9736-40ae518c6526-maas-api-xks-ca" pod "ef849d43-7bb0-4096-9736-40ae518c6526" (UID: "ef849d43-7bb0-4096-9736-40ae518c6526"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:06:22.522636 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:22.522555 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef849d43-7bb0-4096-9736-40ae518c6526-maas-api-xks-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:06:22.522636 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:22.522585 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef849d43-7bb0-4096-9736-40ae518c6526-maas-api-openshift-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:06:22.522636 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:22.522596 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdkqw\" (UniqueName: \"kubernetes.io/projected/ef849d43-7bb0-4096-9736-40ae518c6526-kube-api-access-rdkqw\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:06:23.256122 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:23.256087 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-dd0628-xdwfv" Oct 30 13:06:23.256315 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:23.256085 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-dd0628-xdwfv" event={"ID":"ef849d43-7bb0-4096-9736-40ae518c6526","Type":"ContainerDied","Data":"cad88d84afd362738dfba4660fb21c4ec75961b6f33b1ceeb28f11a5f42f1e8c"} Oct 30 13:06:23.256315 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:23.256261 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cad88d84afd362738dfba4660fb21c4ec75961b6f33b1ceeb28f11a5f42f1e8c" Oct 30 13:06:23.690951 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:23.690907 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-73bafc-openshift-default-64bfdb8459-rfkqt" Oct 30 13:06:23.736446 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:23.736271 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9c9d1f3-e29c-42d4-96e6-765da8de6ab9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9c9d1f3-e29c-42d4-96e6-765da8de6ab9-workload-socket\") pod \"c9c9d1f3-e29c-42d4-96e6-765da8de6ab9\" (UID: \"c9c9d1f3-e29c-42d4-96e6-765da8de6ab9\") " Oct 30 13:06:23.736446 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:23.736365 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9c9d1f3-e29c-42d4-96e6-765da8de6ab9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9c9d1f3-e29c-42d4-96e6-765da8de6ab9-istio-data\") pod \"c9c9d1f3-e29c-42d4-96e6-765da8de6ab9\" (UID: \"c9c9d1f3-e29c-42d4-96e6-765da8de6ab9\") " Oct 30 13:06:23.736711 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:23.736518 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9c9d1f3-e29c-42d4-96e6-765da8de6ab9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9c9d1f3-e29c-42d4-96e6-765da8de6ab9-credential-socket\") pod \"c9c9d1f3-e29c-42d4-96e6-765da8de6ab9\" (UID: \"c9c9d1f3-e29c-42d4-96e6-765da8de6ab9\") " Oct 30 13:06:23.736711 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:23.736553 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9c9d1f3-e29c-42d4-96e6-765da8de6ab9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9c9d1f3-e29c-42d4-96e6-765da8de6ab9-workload-certs\") pod \"c9c9d1f3-e29c-42d4-96e6-765da8de6ab9\" (UID: \"c9c9d1f3-e29c-42d4-96e6-765da8de6ab9\") " Oct 30 13:06:23.736711 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:23.736612 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9c9d1f3-e29c-42d4-96e6-765da8de6ab9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9c9d1f3-e29c-42d4-96e6-765da8de6ab9-istio-envoy\") pod \"c9c9d1f3-e29c-42d4-96e6-765da8de6ab9\" (UID: \"c9c9d1f3-e29c-42d4-96e6-765da8de6ab9\") " Oct 30 13:06:23.737032 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:23.736729 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9c9d1f3-e29c-42d4-96e6-765da8de6ab9-istio-token\" (UniqueName: \"kubernetes.io/projected/c9c9d1f3-e29c-42d4-96e6-765da8de6ab9-istio-token\") pod \"c9c9d1f3-e29c-42d4-96e6-765da8de6ab9\" (UID: \"c9c9d1f3-e29c-42d4-96e6-765da8de6ab9\") " Oct 30 13:06:23.737032 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:23.736789 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9c9d1f3-e29c-42d4-96e6-765da8de6ab9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9c9d1f3-e29c-42d4-96e6-765da8de6ab9-istiod-ca-cert\") pod \"c9c9d1f3-e29c-42d4-96e6-765da8de6ab9\" (UID: \"c9c9d1f3-e29c-42d4-96e6-765da8de6ab9\") " Oct 30 13:06:23.737032 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:23.736868 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c9c9d1f3-e29c-42d4-96e6-765da8de6ab9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9c9d1f3-e29c-42d4-96e6-765da8de6ab9-istio-podinfo\") pod \"c9c9d1f3-e29c-42d4-96e6-765da8de6ab9\" (UID: \"c9c9d1f3-e29c-42d4-96e6-765da8de6ab9\") " Oct 30 13:06:23.737032 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:23.736894 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9c9d1f3-e29c-42d4-96e6-765da8de6ab9-kube-api-access-f47fz\" (UniqueName: \"kubernetes.io/projected/c9c9d1f3-e29c-42d4-96e6-765da8de6ab9-kube-api-access-f47fz\") pod \"c9c9d1f3-e29c-42d4-96e6-765da8de6ab9\" (UID: \"c9c9d1f3-e29c-42d4-96e6-765da8de6ab9\") " Oct 30 13:06:23.737818 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:23.737613 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9c9d1f3-e29c-42d4-96e6-765da8de6ab9-workload-certs" pod "c9c9d1f3-e29c-42d4-96e6-765da8de6ab9" (UID: "c9c9d1f3-e29c-42d4-96e6-765da8de6ab9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:06:23.738175 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:23.737989 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9c9d1f3-e29c-42d4-96e6-765da8de6ab9-workload-socket" pod "c9c9d1f3-e29c-42d4-96e6-765da8de6ab9" (UID: "c9c9d1f3-e29c-42d4-96e6-765da8de6ab9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:06:23.738342 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:23.738314 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9c9d1f3-e29c-42d4-96e6-765da8de6ab9-istio-data" pod "c9c9d1f3-e29c-42d4-96e6-765da8de6ab9" (UID: "c9c9d1f3-e29c-42d4-96e6-765da8de6ab9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:06:23.738687 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:23.738666 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9c9d1f3-e29c-42d4-96e6-765da8de6ab9-credential-socket" pod "c9c9d1f3-e29c-42d4-96e6-765da8de6ab9" (UID: "c9c9d1f3-e29c-42d4-96e6-765da8de6ab9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:06:23.739639 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:23.739614 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9c9d1f3-e29c-42d4-96e6-765da8de6ab9-istiod-ca-cert" pod "c9c9d1f3-e29c-42d4-96e6-765da8de6ab9" (UID: "c9c9d1f3-e29c-42d4-96e6-765da8de6ab9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:06:23.744781 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:23.743276 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9c9d1f3-e29c-42d4-96e6-765da8de6ab9-kube-api-access-f47fz" pod "c9c9d1f3-e29c-42d4-96e6-765da8de6ab9" (UID: "c9c9d1f3-e29c-42d4-96e6-765da8de6ab9"). InnerVolumeSpecName "kube-api-access-f47fz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:06:23.745204 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:23.745158 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9c9d1f3-e29c-42d4-96e6-765da8de6ab9-istio-token" pod "c9c9d1f3-e29c-42d4-96e6-765da8de6ab9" (UID: "c9c9d1f3-e29c-42d4-96e6-765da8de6ab9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:06:23.745450 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:23.745424 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9c9d1f3-e29c-42d4-96e6-765da8de6ab9-istio-envoy" pod "c9c9d1f3-e29c-42d4-96e6-765da8de6ab9" (UID: "c9c9d1f3-e29c-42d4-96e6-765da8de6ab9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:06:23.745637 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:23.745611 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c9c9d1f3-e29c-42d4-96e6-765da8de6ab9-istio-podinfo" pod "c9c9d1f3-e29c-42d4-96e6-765da8de6ab9" (UID: "c9c9d1f3-e29c-42d4-96e6-765da8de6ab9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:06:23.838146 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:23.838055 4908 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9c9d1f3-e29c-42d4-96e6-765da8de6ab9-workload-socket\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:06:23.838146 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:23.838090 4908 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9c9d1f3-e29c-42d4-96e6-765da8de6ab9-istio-data\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:06:23.838146 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:23.838099 4908 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9c9d1f3-e29c-42d4-96e6-765da8de6ab9-credential-socket\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:06:23.838146 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:23.838108 4908 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9c9d1f3-e29c-42d4-96e6-765da8de6ab9-workload-certs\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:06:23.838146 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:23.838117 4908 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9c9d1f3-e29c-42d4-96e6-765da8de6ab9-istio-envoy\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:06:23.838146 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:23.838126 4908 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9c9d1f3-e29c-42d4-96e6-765da8de6ab9-istio-token\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:06:23.838146 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:23.838133 4908 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9c9d1f3-e29c-42d4-96e6-765da8de6ab9-istiod-ca-cert\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:06:23.838146 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:23.838141 4908 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9c9d1f3-e29c-42d4-96e6-765da8de6ab9-istio-podinfo\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:06:23.838146 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:23.838150 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f47fz\" (UniqueName: \"kubernetes.io/projected/c9c9d1f3-e29c-42d4-96e6-765da8de6ab9-kube-api-access-f47fz\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:06:24.039352 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:24.039316 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-1333c8-openshift-default-847ccc6485-tp95k"] Oct 30 13:06:24.039563 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:24.039537 4908 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-1333c8-openshift-default-847ccc6485-tp95k" podUID="75b90e54-7d8b-44cf-9e8d-06fe342acc3b" containerName="istio-proxy" containerID="cri-o://dc3a66352a2581e2b79c62f2c986f51e7f7feeccee51d3d0b74097990e1bc134" gracePeriod=30 Oct 30 13:06:24.268132 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:24.268097 4908 generic.go:356] "Generic (PLEG): container finished" podID="c9c9d1f3-e29c-42d4-96e6-765da8de6ab9" containerID="19cf3f8257259bb14b18c415fd3fceb8beec31070c4c2a6fc2f61f373e2737a6" exitCode=0 Oct 30 13:06:24.268358 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:24.268194 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-73bafc-openshift-default-64bfdb8459-rfkqt" Oct 30 13:06:24.268358 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:24.268190 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-73bafc-openshift-default-64bfdb8459-rfkqt" event={"ID":"c9c9d1f3-e29c-42d4-96e6-765da8de6ab9","Type":"ContainerDied","Data":"19cf3f8257259bb14b18c415fd3fceb8beec31070c4c2a6fc2f61f373e2737a6"} Oct 30 13:06:24.268358 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:24.268326 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-73bafc-openshift-default-64bfdb8459-rfkqt" event={"ID":"c9c9d1f3-e29c-42d4-96e6-765da8de6ab9","Type":"ContainerDied","Data":"b31c36a17ea35a625e9e6a669a21289b6baea6c208eb31a106384f0c541a1d69"} Oct 30 13:06:24.268358 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:24.268357 4908 scope.go:122] "RemoveContainer" containerID="19cf3f8257259bb14b18c415fd3fceb8beec31070c4c2a6fc2f61f373e2737a6" Oct 30 13:06:24.283106 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:24.283085 4908 scope.go:122] "RemoveContainer" containerID="b7fd1b50b654a93ce527f34005feaed728e662acddbe2b46b84e874364693d3a" Oct 30 13:06:24.292809 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:24.292783 4908 scope.go:122] "RemoveContainer" containerID="19cf3f8257259bb14b18c415fd3fceb8beec31070c4c2a6fc2f61f373e2737a6" Oct 30 13:06:24.293107 ip-10-0-141-251 kubenswrapper[4908]: E1030 13:06:24.293084 4908 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19cf3f8257259bb14b18c415fd3fceb8beec31070c4c2a6fc2f61f373e2737a6\": container with ID starting with 19cf3f8257259bb14b18c415fd3fceb8beec31070c4c2a6fc2f61f373e2737a6 not found: ID does not exist" containerID="19cf3f8257259bb14b18c415fd3fceb8beec31070c4c2a6fc2f61f373e2737a6" Oct 30 13:06:24.293170 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:24.293117 4908 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19cf3f8257259bb14b18c415fd3fceb8beec31070c4c2a6fc2f61f373e2737a6"} err="failed to get container status \"19cf3f8257259bb14b18c415fd3fceb8beec31070c4c2a6fc2f61f373e2737a6\": rpc error: code = NotFound desc = could not find container \"19cf3f8257259bb14b18c415fd3fceb8beec31070c4c2a6fc2f61f373e2737a6\": container with ID starting with 19cf3f8257259bb14b18c415fd3fceb8beec31070c4c2a6fc2f61f373e2737a6 not found: ID does not exist" Oct 30 13:06:24.293170 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:24.293137 4908 scope.go:122] "RemoveContainer" containerID="b7fd1b50b654a93ce527f34005feaed728e662acddbe2b46b84e874364693d3a" Oct 30 13:06:24.293389 ip-10-0-141-251 kubenswrapper[4908]: E1030 13:06:24.293368 4908 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7fd1b50b654a93ce527f34005feaed728e662acddbe2b46b84e874364693d3a\": container with ID starting with b7fd1b50b654a93ce527f34005feaed728e662acddbe2b46b84e874364693d3a not found: ID does not exist" containerID="b7fd1b50b654a93ce527f34005feaed728e662acddbe2b46b84e874364693d3a" Oct 30 13:06:24.293449 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:24.293401 4908 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7fd1b50b654a93ce527f34005feaed728e662acddbe2b46b84e874364693d3a"} err="failed to get container status \"b7fd1b50b654a93ce527f34005feaed728e662acddbe2b46b84e874364693d3a\": rpc error: code = NotFound desc = could not find container \"b7fd1b50b654a93ce527f34005feaed728e662acddbe2b46b84e874364693d3a\": container with ID starting with b7fd1b50b654a93ce527f34005feaed728e662acddbe2b46b84e874364693d3a not found: ID does not exist" Oct 30 13:06:24.318588 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:24.318557 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-73bafc-openshift-default-64bfdb8459-rfkqt"] Oct 30 13:06:24.322776 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:24.322746 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-73bafc-openshift-default-64bfdb8459-rfkqt"] Oct 30 13:06:24.558203 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:24.558118 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9c3f12-zrzk7"] Oct 30 13:06:24.558645 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:24.558625 4908 cpu_manager.go:414] "containerMap: removing container" podUID="ef849d43-7bb0-4096-9736-40ae518c6526" containerName="revoke-keys" Oct 30 13:06:24.558645 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:24.558642 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef849d43-7bb0-4096-9736-40ae518c6526" containerName="revoke-keys" Oct 30 13:06:24.558851 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:24.558663 4908 cpu_manager.go:414] "containerMap: removing container" podUID="c9c9d1f3-e29c-42d4-96e6-765da8de6ab9" containerName="istio-proxy" Oct 30 13:06:24.558851 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:24.558670 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9c9d1f3-e29c-42d4-96e6-765da8de6ab9" containerName="istio-proxy" Oct 30 13:06:24.558851 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:24.558780 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef849d43-7bb0-4096-9736-40ae518c6526" containerName="revoke-keys" Oct 30 13:06:24.558851 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:24.558794 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9c9d1f3-e29c-42d4-96e6-765da8de6ab9" containerName="istio-proxy" Oct 30 13:06:24.558851 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:24.558802 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="93f3f694-3b06-430c-b4f0-ed5afbea43d1" containerName="istio-proxy" Oct 30 13:06:24.558851 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:24.558813 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9c9d1f3-e29c-42d4-96e6-765da8de6ab9" containerName="istio-proxy" Oct 30 13:06:24.563701 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:24.563679 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9c3f12-zrzk7" Oct 30 13:06:24.578253 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:24.578220 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 13:06:24.578402 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:24.578255 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 13:06:24.580994 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:24.580966 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9c3f12-zrzk7"] Oct 30 13:06:24.599630 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:24.599589 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:06:24.599891 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:24.599740 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s75hq\"" Oct 30 13:06:24.647651 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:24.647611 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8d76aa5-3fbc-4311-a104-242d252c5be3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-9c3f12-zrzk7\" (UID: \"c8d76aa5-3fbc-4311-a104-242d252c5be3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9c3f12-zrzk7" Oct 30 13:06:24.647651 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:24.647650 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64p9b\" (UniqueName: \"kubernetes.io/projected/c8d76aa5-3fbc-4311-a104-242d252c5be3-kube-api-access-64p9b\") pod \"maas-api-revoke-keys-e2e-inf-b-9c3f12-zrzk7\" (UID: \"c8d76aa5-3fbc-4311-a104-242d252c5be3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9c3f12-zrzk7" Oct 30 13:06:24.647895 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:24.647805 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8d76aa5-3fbc-4311-a104-242d252c5be3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-9c3f12-zrzk7\" (UID: \"c8d76aa5-3fbc-4311-a104-242d252c5be3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9c3f12-zrzk7" Oct 30 13:06:24.749037 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:24.748992 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8d76aa5-3fbc-4311-a104-242d252c5be3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-9c3f12-zrzk7\" (UID: \"c8d76aa5-3fbc-4311-a104-242d252c5be3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9c3f12-zrzk7" Oct 30 13:06:24.749534 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:24.749136 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8d76aa5-3fbc-4311-a104-242d252c5be3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-9c3f12-zrzk7\" (UID: \"c8d76aa5-3fbc-4311-a104-242d252c5be3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9c3f12-zrzk7" Oct 30 13:06:24.749534 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:24.749159 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64p9b\" (UniqueName: \"kubernetes.io/projected/c8d76aa5-3fbc-4311-a104-242d252c5be3-kube-api-access-64p9b\") pod \"maas-api-revoke-keys-e2e-inf-b-9c3f12-zrzk7\" (UID: \"c8d76aa5-3fbc-4311-a104-242d252c5be3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9c3f12-zrzk7" Oct 30 13:06:24.749896 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:24.749872 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8d76aa5-3fbc-4311-a104-242d252c5be3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-9c3f12-zrzk7\" (UID: \"c8d76aa5-3fbc-4311-a104-242d252c5be3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9c3f12-zrzk7" Oct 30 13:06:24.751670 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:24.751641 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8d76aa5-3fbc-4311-a104-242d252c5be3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-9c3f12-zrzk7\" (UID: \"c8d76aa5-3fbc-4311-a104-242d252c5be3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9c3f12-zrzk7" Oct 30 13:06:24.771755 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:24.771719 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64p9b\" (UniqueName: \"kubernetes.io/projected/c8d76aa5-3fbc-4311-a104-242d252c5be3-kube-api-access-64p9b\") pod \"maas-api-revoke-keys-e2e-inf-b-9c3f12-zrzk7\" (UID: \"c8d76aa5-3fbc-4311-a104-242d252c5be3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9c3f12-zrzk7" Oct 30 13:06:24.877850 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:24.877745 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9c3f12-zrzk7" Oct 30 13:06:24.920198 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:24.920158 4908 patch_prober.go:29] interesting pod/e2e-inf-a-1333c8-openshift-default-847ccc6485-tp95k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 13:06:24.920365 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:24.920218 4908 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-1333c8-openshift-default-847ccc6485-tp95k" podUID="75b90e54-7d8b-44cf-9e8d-06fe342acc3b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 13:06:25.190756 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:25.190721 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9c3f12-zrzk7"] Oct 30 13:06:25.193180 ip-10-0-141-251 kubenswrapper[4908]: W1030 13:06:25.193149 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8d76aa5_3fbc_4311_a104_242d252c5be3.slice/crio-b505fc59a60b7a3a5c92787ee1eeaec284ca6b94f9a7bcf1e0eb1688c2f714c0 WatchSource:0}: Error finding container b505fc59a60b7a3a5c92787ee1eeaec284ca6b94f9a7bcf1e0eb1688c2f714c0: Status 404 returned error can't find the container with id b505fc59a60b7a3a5c92787ee1eeaec284ca6b94f9a7bcf1e0eb1688c2f714c0 Oct 30 13:06:25.277440 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:25.277398 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9c3f12-zrzk7" event={"ID":"c8d76aa5-3fbc-4311-a104-242d252c5be3","Type":"ContainerStarted","Data":"b505fc59a60b7a3a5c92787ee1eeaec284ca6b94f9a7bcf1e0eb1688c2f714c0"} Oct 30 13:06:25.606314 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:25.606276 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9c9d1f3-e29c-42d4-96e6-765da8de6ab9" path="/var/lib/kubelet/pods/c9c9d1f3-e29c-42d4-96e6-765da8de6ab9/volumes" Oct 30 13:06:26.288174 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:26.288131 4908 generic.go:356] "Generic (PLEG): container finished" podID="c8d76aa5-3fbc-4311-a104-242d252c5be3" containerID="798ffa9205f0a684ce06393712a0b312c28d4632f374786208f98f84a9d48205" exitCode=0 Oct 30 13:06:26.288718 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:26.288208 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9c3f12-zrzk7" event={"ID":"c8d76aa5-3fbc-4311-a104-242d252c5be3","Type":"ContainerDied","Data":"798ffa9205f0a684ce06393712a0b312c28d4632f374786208f98f84a9d48205"} Oct 30 13:06:27.450882 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:27.450851 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a786af-whx6v"] Oct 30 13:06:27.490880 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:27.490800 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a786af-whx6v"] Oct 30 13:06:27.495487 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:27.495461 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9c3f12-zrzk7" Oct 30 13:06:27.589939 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:27.589550 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c8d76aa5-3fbc-4311-a104-242d252c5be3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8d76aa5-3fbc-4311-a104-242d252c5be3-maas-api-xks-ca\") pod \"c8d76aa5-3fbc-4311-a104-242d252c5be3\" (UID: \"c8d76aa5-3fbc-4311-a104-242d252c5be3\") " Oct 30 13:06:27.592881 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:27.590115 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8d76aa5-3fbc-4311-a104-242d252c5be3-kube-api-access-64p9b\" (UniqueName: \"kubernetes.io/projected/c8d76aa5-3fbc-4311-a104-242d252c5be3-kube-api-access-64p9b\") pod \"c8d76aa5-3fbc-4311-a104-242d252c5be3\" (UID: \"c8d76aa5-3fbc-4311-a104-242d252c5be3\") " Oct 30 13:06:27.592881 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:27.590638 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8d76aa5-3fbc-4311-a104-242d252c5be3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8d76aa5-3fbc-4311-a104-242d252c5be3-maas-api-openshift-ca\") pod \"c8d76aa5-3fbc-4311-a104-242d252c5be3\" (UID: \"c8d76aa5-3fbc-4311-a104-242d252c5be3\") " Oct 30 13:06:27.592881 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:27.591062 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8d76aa5-3fbc-4311-a104-242d252c5be3-maas-api-openshift-ca" pod "c8d76aa5-3fbc-4311-a104-242d252c5be3" (UID: "c8d76aa5-3fbc-4311-a104-242d252c5be3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:06:27.592881 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:27.592363 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8d76aa5-3fbc-4311-a104-242d252c5be3-maas-api-xks-ca" pod "c8d76aa5-3fbc-4311-a104-242d252c5be3" (UID: "c8d76aa5-3fbc-4311-a104-242d252c5be3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:06:27.592881 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:27.592395 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8d76aa5-3fbc-4311-a104-242d252c5be3-maas-api-openshift-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:06:27.612923 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:27.612809 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05338db8-262a-4d4b-8edb-8c06daa80d30" path="/var/lib/kubelet/pods/05338db8-262a-4d4b-8edb-8c06daa80d30/volumes" Oct 30 13:06:27.616014 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:27.615982 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8d76aa5-3fbc-4311-a104-242d252c5be3-kube-api-access-64p9b" pod "c8d76aa5-3fbc-4311-a104-242d252c5be3" (UID: "c8d76aa5-3fbc-4311-a104-242d252c5be3"). InnerVolumeSpecName "kube-api-access-64p9b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:06:27.694113 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:27.694073 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8d76aa5-3fbc-4311-a104-242d252c5be3-maas-api-xks-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:06:27.694113 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:27.694113 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64p9b\" (UniqueName: \"kubernetes.io/projected/c8d76aa5-3fbc-4311-a104-242d252c5be3-kube-api-access-64p9b\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:06:28.307886 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:28.307841 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9c3f12-zrzk7" Oct 30 13:06:28.307886 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:28.307863 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9c3f12-zrzk7" event={"ID":"c8d76aa5-3fbc-4311-a104-242d252c5be3","Type":"ContainerDied","Data":"b505fc59a60b7a3a5c92787ee1eeaec284ca6b94f9a7bcf1e0eb1688c2f714c0"} Oct 30 13:06:28.308129 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:28.307924 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b505fc59a60b7a3a5c92787ee1eeaec284ca6b94f9a7bcf1e0eb1688c2f714c0" Oct 30 13:06:29.324221 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:29.324118 4908 generic.go:356] "Generic (PLEG): container finished" podID="75b90e54-7d8b-44cf-9e8d-06fe342acc3b" containerID="dc3a66352a2581e2b79c62f2c986f51e7f7feeccee51d3d0b74097990e1bc134" exitCode=0 Oct 30 13:06:29.324221 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:29.324166 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1333c8-openshift-default-847ccc6485-tp95k" event={"ID":"75b90e54-7d8b-44cf-9e8d-06fe342acc3b","Type":"ContainerDied","Data":"dc3a66352a2581e2b79c62f2c986f51e7f7feeccee51d3d0b74097990e1bc134"} Oct 30 13:06:29.324221 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:29.324212 4908 scope.go:122] "RemoveContainer" containerID="aeb670d117bd2951334cae801bb431a0a884adf89d173a4d11efdaa3c2610696" Oct 30 13:06:29.367868 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:29.367801 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1333c8-openshift-default-847ccc6485-tp95k" Oct 30 13:06:29.416185 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:29.416142 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75b90e54-7d8b-44cf-9e8d-06fe342acc3b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75b90e54-7d8b-44cf-9e8d-06fe342acc3b-credential-socket\") pod \"75b90e54-7d8b-44cf-9e8d-06fe342acc3b\" (UID: \"75b90e54-7d8b-44cf-9e8d-06fe342acc3b\") " Oct 30 13:06:29.416185 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:29.416189 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/75b90e54-7d8b-44cf-9e8d-06fe342acc3b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75b90e54-7d8b-44cf-9e8d-06fe342acc3b-istio-podinfo\") pod \"75b90e54-7d8b-44cf-9e8d-06fe342acc3b\" (UID: \"75b90e54-7d8b-44cf-9e8d-06fe342acc3b\") " Oct 30 13:06:29.416445 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:29.416210 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75b90e54-7d8b-44cf-9e8d-06fe342acc3b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75b90e54-7d8b-44cf-9e8d-06fe342acc3b-workload-certs\") pod \"75b90e54-7d8b-44cf-9e8d-06fe342acc3b\" (UID: \"75b90e54-7d8b-44cf-9e8d-06fe342acc3b\") " Oct 30 13:06:29.416445 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:29.416258 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75b90e54-7d8b-44cf-9e8d-06fe342acc3b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75b90e54-7d8b-44cf-9e8d-06fe342acc3b-istio-envoy\") pod \"75b90e54-7d8b-44cf-9e8d-06fe342acc3b\" (UID: \"75b90e54-7d8b-44cf-9e8d-06fe342acc3b\") " Oct 30 13:06:29.416445 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:29.416323 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75b90e54-7d8b-44cf-9e8d-06fe342acc3b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75b90e54-7d8b-44cf-9e8d-06fe342acc3b-workload-socket\") pod \"75b90e54-7d8b-44cf-9e8d-06fe342acc3b\" (UID: \"75b90e54-7d8b-44cf-9e8d-06fe342acc3b\") " Oct 30 13:06:29.416445 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:29.416361 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75b90e54-7d8b-44cf-9e8d-06fe342acc3b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75b90e54-7d8b-44cf-9e8d-06fe342acc3b-istiod-ca-cert\") pod \"75b90e54-7d8b-44cf-9e8d-06fe342acc3b\" (UID: \"75b90e54-7d8b-44cf-9e8d-06fe342acc3b\") " Oct 30 13:06:29.416685 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:29.416465 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75b90e54-7d8b-44cf-9e8d-06fe342acc3b-istio-token\" (UniqueName: \"kubernetes.io/projected/75b90e54-7d8b-44cf-9e8d-06fe342acc3b-istio-token\") pod \"75b90e54-7d8b-44cf-9e8d-06fe342acc3b\" (UID: \"75b90e54-7d8b-44cf-9e8d-06fe342acc3b\") " Oct 30 13:06:29.416685 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:29.416577 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75b90e54-7d8b-44cf-9e8d-06fe342acc3b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75b90e54-7d8b-44cf-9e8d-06fe342acc3b-istio-data\") pod \"75b90e54-7d8b-44cf-9e8d-06fe342acc3b\" (UID: \"75b90e54-7d8b-44cf-9e8d-06fe342acc3b\") " Oct 30 13:06:29.416685 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:29.416577 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75b90e54-7d8b-44cf-9e8d-06fe342acc3b-credential-socket" pod "75b90e54-7d8b-44cf-9e8d-06fe342acc3b" (UID: "75b90e54-7d8b-44cf-9e8d-06fe342acc3b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:06:29.416685 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:29.416621 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75b90e54-7d8b-44cf-9e8d-06fe342acc3b-kube-api-access-4gx9n\" (UniqueName: \"kubernetes.io/projected/75b90e54-7d8b-44cf-9e8d-06fe342acc3b-kube-api-access-4gx9n\") pod \"75b90e54-7d8b-44cf-9e8d-06fe342acc3b\" (UID: \"75b90e54-7d8b-44cf-9e8d-06fe342acc3b\") " Oct 30 13:06:29.416685 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:29.416626 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75b90e54-7d8b-44cf-9e8d-06fe342acc3b-workload-certs" pod "75b90e54-7d8b-44cf-9e8d-06fe342acc3b" (UID: "75b90e54-7d8b-44cf-9e8d-06fe342acc3b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:06:29.417031 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:29.416686 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75b90e54-7d8b-44cf-9e8d-06fe342acc3b-workload-socket" pod "75b90e54-7d8b-44cf-9e8d-06fe342acc3b" (UID: "75b90e54-7d8b-44cf-9e8d-06fe342acc3b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:06:29.417031 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:29.416856 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75b90e54-7d8b-44cf-9e8d-06fe342acc3b-istio-data" pod "75b90e54-7d8b-44cf-9e8d-06fe342acc3b" (UID: "75b90e54-7d8b-44cf-9e8d-06fe342acc3b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:06:29.417031 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:29.416941 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75b90e54-7d8b-44cf-9e8d-06fe342acc3b-istiod-ca-cert" pod "75b90e54-7d8b-44cf-9e8d-06fe342acc3b" (UID: "75b90e54-7d8b-44cf-9e8d-06fe342acc3b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:06:29.417223 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:29.417135 4908 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75b90e54-7d8b-44cf-9e8d-06fe342acc3b-workload-certs\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:06:29.417223 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:29.417159 4908 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75b90e54-7d8b-44cf-9e8d-06fe342acc3b-workload-socket\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:06:29.417223 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:29.417175 4908 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75b90e54-7d8b-44cf-9e8d-06fe342acc3b-istiod-ca-cert\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:06:29.417223 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:29.417187 4908 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75b90e54-7d8b-44cf-9e8d-06fe342acc3b-istio-data\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:06:29.417223 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:29.417199 4908 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75b90e54-7d8b-44cf-9e8d-06fe342acc3b-credential-socket\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:06:29.418596 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:29.418569 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/75b90e54-7d8b-44cf-9e8d-06fe342acc3b-istio-podinfo" pod "75b90e54-7d8b-44cf-9e8d-06fe342acc3b" (UID: "75b90e54-7d8b-44cf-9e8d-06fe342acc3b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:06:29.418905 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:29.418879 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75b90e54-7d8b-44cf-9e8d-06fe342acc3b-kube-api-access-4gx9n" pod "75b90e54-7d8b-44cf-9e8d-06fe342acc3b" (UID: "75b90e54-7d8b-44cf-9e8d-06fe342acc3b"). InnerVolumeSpecName "kube-api-access-4gx9n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:06:29.419141 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:29.419118 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75b90e54-7d8b-44cf-9e8d-06fe342acc3b-istio-envoy" pod "75b90e54-7d8b-44cf-9e8d-06fe342acc3b" (UID: "75b90e54-7d8b-44cf-9e8d-06fe342acc3b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:06:29.419223 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:29.419206 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75b90e54-7d8b-44cf-9e8d-06fe342acc3b-istio-token" pod "75b90e54-7d8b-44cf-9e8d-06fe342acc3b" (UID: "75b90e54-7d8b-44cf-9e8d-06fe342acc3b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:06:29.518170 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:29.518134 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4gx9n\" (UniqueName: \"kubernetes.io/projected/75b90e54-7d8b-44cf-9e8d-06fe342acc3b-kube-api-access-4gx9n\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:06:29.518170 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:29.518168 4908 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75b90e54-7d8b-44cf-9e8d-06fe342acc3b-istio-podinfo\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:06:29.518170 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:29.518180 4908 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75b90e54-7d8b-44cf-9e8d-06fe342acc3b-istio-envoy\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:06:29.518412 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:29.518188 4908 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75b90e54-7d8b-44cf-9e8d-06fe342acc3b-istio-token\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:06:30.333908 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:30.333867 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1333c8-openshift-default-847ccc6485-tp95k" Oct 30 13:06:30.334455 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:30.333873 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1333c8-openshift-default-847ccc6485-tp95k" event={"ID":"75b90e54-7d8b-44cf-9e8d-06fe342acc3b","Type":"ContainerDied","Data":"f18e2945f6c50788f0e357058220fdcc1b3f3cd11c762b40a9b275c169e0c5b9"} Oct 30 13:06:30.334455 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:30.334006 4908 scope.go:122] "RemoveContainer" containerID="dc3a66352a2581e2b79c62f2c986f51e7f7feeccee51d3d0b74097990e1bc134" Oct 30 13:06:30.405495 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:30.405452 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-1333c8-openshift-default-847ccc6485-tp95k"] Oct 30 13:06:30.418677 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:30.418642 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-1333c8-openshift-default-847ccc6485-tp95k"] Oct 30 13:06:31.608254 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:31.608210 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75b90e54-7d8b-44cf-9e8d-06fe342acc3b" path="/var/lib/kubelet/pods/75b90e54-7d8b-44cf-9e8d-06fe342acc3b/volumes" Oct 30 13:06:34.680886 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:34.678127 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-a786af-openshift-default-f89f5ffc7-b62z2"] Oct 30 13:06:34.680886 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:34.678408 4908 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-a786af-openshift-default-f89f5ffc7-b62z2" podUID="20af951f-19a7-49e6-ab1d-3bdee930084d" containerName="istio-proxy" containerID="cri-o://1944c701fba206b3f327f0931743a5746733758cced662ffe3dc1b778b05a403" gracePeriod=30 Oct 30 13:06:34.863852 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:34.862883 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2545c7-mctbc"] Oct 30 13:06:34.887873 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:34.873435 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2545c7-mctbc"] Oct 30 13:06:35.038542 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:35.038501 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd707416-8fx5n"] Oct 30 13:06:35.044071 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:35.044039 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd707416-8fx5n"] Oct 30 13:06:35.278872 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:35.278820 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0f7d14-fdln9"] Oct 30 13:06:35.279315 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:35.279300 4908 cpu_manager.go:414] "containerMap: removing container" podUID="75b90e54-7d8b-44cf-9e8d-06fe342acc3b" containerName="istio-proxy" Oct 30 13:06:35.279389 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:35.279315 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75b90e54-7d8b-44cf-9e8d-06fe342acc3b" containerName="istio-proxy" Oct 30 13:06:35.279389 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:35.279347 4908 cpu_manager.go:414] "containerMap: removing container" podUID="c9c9d1f3-e29c-42d4-96e6-765da8de6ab9" containerName="istio-proxy" Oct 30 13:06:35.279389 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:35.279353 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9c9d1f3-e29c-42d4-96e6-765da8de6ab9" containerName="istio-proxy" Oct 30 13:06:35.279389 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:35.279365 4908 cpu_manager.go:414] "containerMap: removing container" podUID="c8d76aa5-3fbc-4311-a104-242d252c5be3" containerName="revoke-keys" Oct 30 13:06:35.279389 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:35.279370 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8d76aa5-3fbc-4311-a104-242d252c5be3" containerName="revoke-keys" Oct 30 13:06:35.279673 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:35.279451 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8d76aa5-3fbc-4311-a104-242d252c5be3" containerName="revoke-keys" Oct 30 13:06:35.279673 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:35.279459 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="75b90e54-7d8b-44cf-9e8d-06fe342acc3b" containerName="istio-proxy" Oct 30 13:06:35.279673 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:35.279467 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="75b90e54-7d8b-44cf-9e8d-06fe342acc3b" containerName="istio-proxy" Oct 30 13:06:35.284511 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:35.284485 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0f7d14-fdln9" Oct 30 13:06:35.301277 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:35.301249 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 13:06:35.301448 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:35.301346 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 13:06:35.303132 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:35.303105 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0f7d14-fdln9"] Oct 30 13:06:35.356149 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:35.356118 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:06:35.356365 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:35.356341 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s75hq\"" Oct 30 13:06:35.413013 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:35.412974 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2335eb21-f2c9-49ab-b060-932461ba8b38-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-0f7d14-fdln9\" (UID: \"2335eb21-f2c9-49ab-b060-932461ba8b38\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0f7d14-fdln9" Oct 30 13:06:35.413197 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:35.413034 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94cbz\" (UniqueName: \"kubernetes.io/projected/2335eb21-f2c9-49ab-b060-932461ba8b38-kube-api-access-94cbz\") pod \"maas-api-revoke-keys-e2e-extmod-b-0f7d14-fdln9\" (UID: \"2335eb21-f2c9-49ab-b060-932461ba8b38\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0f7d14-fdln9" Oct 30 13:06:35.413197 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:35.413134 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2335eb21-f2c9-49ab-b060-932461ba8b38-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-0f7d14-fdln9\" (UID: \"2335eb21-f2c9-49ab-b060-932461ba8b38\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0f7d14-fdln9" Oct 30 13:06:35.514476 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:35.514437 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2335eb21-f2c9-49ab-b060-932461ba8b38-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-0f7d14-fdln9\" (UID: \"2335eb21-f2c9-49ab-b060-932461ba8b38\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0f7d14-fdln9" Oct 30 13:06:35.514679 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:35.514489 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94cbz\" (UniqueName: \"kubernetes.io/projected/2335eb21-f2c9-49ab-b060-932461ba8b38-kube-api-access-94cbz\") pod \"maas-api-revoke-keys-e2e-extmod-b-0f7d14-fdln9\" (UID: \"2335eb21-f2c9-49ab-b060-932461ba8b38\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0f7d14-fdln9" Oct 30 13:06:35.514679 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:35.514546 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2335eb21-f2c9-49ab-b060-932461ba8b38-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-0f7d14-fdln9\" (UID: \"2335eb21-f2c9-49ab-b060-932461ba8b38\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0f7d14-fdln9" Oct 30 13:06:35.526250 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:35.516253 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2335eb21-f2c9-49ab-b060-932461ba8b38-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-0f7d14-fdln9\" (UID: \"2335eb21-f2c9-49ab-b060-932461ba8b38\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0f7d14-fdln9" Oct 30 13:06:35.526971 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:35.526941 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2335eb21-f2c9-49ab-b060-932461ba8b38-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-0f7d14-fdln9\" (UID: \"2335eb21-f2c9-49ab-b060-932461ba8b38\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0f7d14-fdln9" Oct 30 13:06:35.539760 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:35.539731 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-94cbz\" (UniqueName: \"kubernetes.io/projected/2335eb21-f2c9-49ab-b060-932461ba8b38-kube-api-access-94cbz\") pod \"maas-api-revoke-keys-e2e-extmod-b-0f7d14-fdln9\" (UID: \"2335eb21-f2c9-49ab-b060-932461ba8b38\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0f7d14-fdln9" Oct 30 13:06:35.602090 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:35.601973 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0f7d14-fdln9" Oct 30 13:06:35.606257 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:35.606224 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ebf1f1c-f8ed-4960-8f9b-2b750fcf4885" path="/var/lib/kubelet/pods/8ebf1f1c-f8ed-4960-8f9b-2b750fcf4885/volumes" Oct 30 13:06:35.606909 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:35.606887 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b64ae455-69c2-4294-b6e3-f892be94b6bb" path="/var/lib/kubelet/pods/b64ae455-69c2-4294-b6e3-f892be94b6bb/volumes" Oct 30 13:06:35.971897 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:35.971862 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0f7d14-fdln9"] Oct 30 13:06:35.973441 ip-10-0-141-251 kubenswrapper[4908]: W1030 13:06:35.973408 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2335eb21_f2c9_49ab_b060_932461ba8b38.slice/crio-bd1a23fd4605ca72cd8202de93d572f6c8320c3917e4c69b4f5a7400442217d7 WatchSource:0}: Error finding container bd1a23fd4605ca72cd8202de93d572f6c8320c3917e4c69b4f5a7400442217d7: Status 404 returned error can't find the container with id bd1a23fd4605ca72cd8202de93d572f6c8320c3917e4c69b4f5a7400442217d7 Oct 30 13:06:36.387875 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:36.387821 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0f7d14-fdln9" event={"ID":"2335eb21-f2c9-49ab-b060-932461ba8b38","Type":"ContainerStarted","Data":"ea4d135a3532067a312d946c75fad3aa80a98f81d548fb5cdd6e70e5c962078b"} Oct 30 13:06:36.388033 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:36.387907 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0f7d14-fdln9" event={"ID":"2335eb21-f2c9-49ab-b060-932461ba8b38","Type":"ContainerStarted","Data":"bd1a23fd4605ca72cd8202de93d572f6c8320c3917e4c69b4f5a7400442217d7"} Oct 30 13:06:36.432635 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:36.432575 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0f7d14-fdln9" podStartSLOduration=1.4325622980000001 podStartE2EDuration="1.432562298s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:06:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:06:36.430646766 +0000 UTC m=+4352.945075516" watchObservedRunningTime="2026-10-30 13:06:36.432562298 +0000 UTC m=+4352.946991044" Oct 30 13:06:36.470965 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:36.470932 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9wwfc"] Oct 30 13:06:37.396806 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:37.396715 4908 generic.go:356] "Generic (PLEG): container finished" podID="2335eb21-f2c9-49ab-b060-932461ba8b38" containerID="ea4d135a3532067a312d946c75fad3aa80a98f81d548fb5cdd6e70e5c962078b" exitCode=0 Oct 30 13:06:37.397222 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:37.396798 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0f7d14-fdln9" event={"ID":"2335eb21-f2c9-49ab-b060-932461ba8b38","Type":"ContainerDied","Data":"ea4d135a3532067a312d946c75fad3aa80a98f81d548fb5cdd6e70e5c962078b"} Oct 30 13:06:38.557192 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:38.557167 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0f7d14-fdln9" Oct 30 13:06:38.562511 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:38.562485 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2335eb21-f2c9-49ab-b060-932461ba8b38-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2335eb21-f2c9-49ab-b060-932461ba8b38-maas-api-xks-ca\") pod \"2335eb21-f2c9-49ab-b060-932461ba8b38\" (UID: \"2335eb21-f2c9-49ab-b060-932461ba8b38\") " Oct 30 13:06:38.562629 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:38.562568 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2335eb21-f2c9-49ab-b060-932461ba8b38-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2335eb21-f2c9-49ab-b060-932461ba8b38-maas-api-openshift-ca\") pod \"2335eb21-f2c9-49ab-b060-932461ba8b38\" (UID: \"2335eb21-f2c9-49ab-b060-932461ba8b38\") " Oct 30 13:06:38.562678 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:38.562631 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2335eb21-f2c9-49ab-b060-932461ba8b38-kube-api-access-94cbz\" (UniqueName: \"kubernetes.io/projected/2335eb21-f2c9-49ab-b060-932461ba8b38-kube-api-access-94cbz\") pod \"2335eb21-f2c9-49ab-b060-932461ba8b38\" (UID: \"2335eb21-f2c9-49ab-b060-932461ba8b38\") " Oct 30 13:06:38.562952 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:38.562927 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2335eb21-f2c9-49ab-b060-932461ba8b38-maas-api-openshift-ca" pod "2335eb21-f2c9-49ab-b060-932461ba8b38" (UID: "2335eb21-f2c9-49ab-b060-932461ba8b38"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:06:38.564569 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:38.564536 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2335eb21-f2c9-49ab-b060-932461ba8b38-maas-api-xks-ca" pod "2335eb21-f2c9-49ab-b060-932461ba8b38" (UID: "2335eb21-f2c9-49ab-b060-932461ba8b38"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:06:38.564685 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:38.564661 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2335eb21-f2c9-49ab-b060-932461ba8b38-kube-api-access-94cbz" pod "2335eb21-f2c9-49ab-b060-932461ba8b38" (UID: "2335eb21-f2c9-49ab-b060-932461ba8b38"). InnerVolumeSpecName "kube-api-access-94cbz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:06:38.663974 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:38.663875 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2335eb21-f2c9-49ab-b060-932461ba8b38-maas-api-xks-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:06:38.663974 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:38.663910 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2335eb21-f2c9-49ab-b060-932461ba8b38-maas-api-openshift-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:06:38.663974 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:38.663924 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-94cbz\" (UniqueName: \"kubernetes.io/projected/2335eb21-f2c9-49ab-b060-932461ba8b38-kube-api-access-94cbz\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:06:39.415609 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:39.415576 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0f7d14-fdln9" Oct 30 13:06:39.415794 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:39.415574 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0f7d14-fdln9" event={"ID":"2335eb21-f2c9-49ab-b060-932461ba8b38","Type":"ContainerDied","Data":"bd1a23fd4605ca72cd8202de93d572f6c8320c3917e4c69b4f5a7400442217d7"} Oct 30 13:06:39.415794 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:39.415708 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bd1a23fd4605ca72cd8202de93d572f6c8320c3917e4c69b4f5a7400442217d7" Oct 30 13:06:39.978215 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:39.978182 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-a786af-openshift-default-f89f5ffc7-b62z2" Oct 30 13:06:40.079042 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:40.079006 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20af951f-19a7-49e6-ab1d-3bdee930084d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20af951f-19a7-49e6-ab1d-3bdee930084d-credential-socket\") pod \"20af951f-19a7-49e6-ab1d-3bdee930084d\" (UID: \"20af951f-19a7-49e6-ab1d-3bdee930084d\") " Oct 30 13:06:40.079042 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:40.079056 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/20af951f-19a7-49e6-ab1d-3bdee930084d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20af951f-19a7-49e6-ab1d-3bdee930084d-istio-podinfo\") pod \"20af951f-19a7-49e6-ab1d-3bdee930084d\" (UID: \"20af951f-19a7-49e6-ab1d-3bdee930084d\") " Oct 30 13:06:40.079308 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:40.079092 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20af951f-19a7-49e6-ab1d-3bdee930084d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20af951f-19a7-49e6-ab1d-3bdee930084d-workload-certs\") pod \"20af951f-19a7-49e6-ab1d-3bdee930084d\" (UID: \"20af951f-19a7-49e6-ab1d-3bdee930084d\") " Oct 30 13:06:40.079308 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:40.079254 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20af951f-19a7-49e6-ab1d-3bdee930084d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20af951f-19a7-49e6-ab1d-3bdee930084d-istiod-ca-cert\") pod \"20af951f-19a7-49e6-ab1d-3bdee930084d\" (UID: \"20af951f-19a7-49e6-ab1d-3bdee930084d\") " Oct 30 13:06:40.079437 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:40.079312 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20af951f-19a7-49e6-ab1d-3bdee930084d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20af951f-19a7-49e6-ab1d-3bdee930084d-istio-envoy\") pod \"20af951f-19a7-49e6-ab1d-3bdee930084d\" (UID: \"20af951f-19a7-49e6-ab1d-3bdee930084d\") " Oct 30 13:06:40.079437 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:40.079346 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20af951f-19a7-49e6-ab1d-3bdee930084d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20af951f-19a7-49e6-ab1d-3bdee930084d-workload-socket\") pod \"20af951f-19a7-49e6-ab1d-3bdee930084d\" (UID: \"20af951f-19a7-49e6-ab1d-3bdee930084d\") " Oct 30 13:06:40.079437 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:40.079380 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20af951f-19a7-49e6-ab1d-3bdee930084d-istio-token\" (UniqueName: \"kubernetes.io/projected/20af951f-19a7-49e6-ab1d-3bdee930084d-istio-token\") pod \"20af951f-19a7-49e6-ab1d-3bdee930084d\" (UID: \"20af951f-19a7-49e6-ab1d-3bdee930084d\") " Oct 30 13:06:40.079437 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:40.079413 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20af951f-19a7-49e6-ab1d-3bdee930084d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20af951f-19a7-49e6-ab1d-3bdee930084d-istio-data\") pod \"20af951f-19a7-49e6-ab1d-3bdee930084d\" (UID: \"20af951f-19a7-49e6-ab1d-3bdee930084d\") " Oct 30 13:06:40.079660 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:40.079486 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20af951f-19a7-49e6-ab1d-3bdee930084d-kube-api-access-zq55s\" (UniqueName: \"kubernetes.io/projected/20af951f-19a7-49e6-ab1d-3bdee930084d-kube-api-access-zq55s\") pod \"20af951f-19a7-49e6-ab1d-3bdee930084d\" (UID: \"20af951f-19a7-49e6-ab1d-3bdee930084d\") " Oct 30 13:06:40.080633 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:40.079748 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20af951f-19a7-49e6-ab1d-3bdee930084d-workload-socket" pod "20af951f-19a7-49e6-ab1d-3bdee930084d" (UID: "20af951f-19a7-49e6-ab1d-3bdee930084d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:06:40.080633 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:40.079995 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20af951f-19a7-49e6-ab1d-3bdee930084d-workload-certs" pod "20af951f-19a7-49e6-ab1d-3bdee930084d" (UID: "20af951f-19a7-49e6-ab1d-3bdee930084d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:06:40.080633 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:40.080474 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20af951f-19a7-49e6-ab1d-3bdee930084d-istiod-ca-cert" pod "20af951f-19a7-49e6-ab1d-3bdee930084d" (UID: "20af951f-19a7-49e6-ab1d-3bdee930084d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:06:40.081884 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:40.080466 4908 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20af951f-19a7-49e6-ab1d-3bdee930084d-workload-certs\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:06:40.081884 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:40.081459 4908 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20af951f-19a7-49e6-ab1d-3bdee930084d-workload-socket\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:06:40.082326 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:40.082294 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20af951f-19a7-49e6-ab1d-3bdee930084d-istio-data" pod "20af951f-19a7-49e6-ab1d-3bdee930084d" (UID: "20af951f-19a7-49e6-ab1d-3bdee930084d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:06:40.083588 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:40.083278 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20af951f-19a7-49e6-ab1d-3bdee930084d-credential-socket" pod "20af951f-19a7-49e6-ab1d-3bdee930084d" (UID: "20af951f-19a7-49e6-ab1d-3bdee930084d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:06:40.087984 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:40.087950 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20af951f-19a7-49e6-ab1d-3bdee930084d-istio-envoy" pod "20af951f-19a7-49e6-ab1d-3bdee930084d" (UID: "20af951f-19a7-49e6-ab1d-3bdee930084d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:06:40.088129 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:40.088055 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/20af951f-19a7-49e6-ab1d-3bdee930084d-istio-podinfo" pod "20af951f-19a7-49e6-ab1d-3bdee930084d" (UID: "20af951f-19a7-49e6-ab1d-3bdee930084d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:06:40.094688 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:40.093448 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20af951f-19a7-49e6-ab1d-3bdee930084d-kube-api-access-zq55s" pod "20af951f-19a7-49e6-ab1d-3bdee930084d" (UID: "20af951f-19a7-49e6-ab1d-3bdee930084d"). InnerVolumeSpecName "kube-api-access-zq55s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:06:40.094688 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:40.093512 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20af951f-19a7-49e6-ab1d-3bdee930084d-istio-token" pod "20af951f-19a7-49e6-ab1d-3bdee930084d" (UID: "20af951f-19a7-49e6-ab1d-3bdee930084d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:06:40.182189 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:40.182144 4908 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20af951f-19a7-49e6-ab1d-3bdee930084d-istiod-ca-cert\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:06:40.182189 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:40.182188 4908 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20af951f-19a7-49e6-ab1d-3bdee930084d-istio-envoy\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:06:40.182434 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:40.182200 4908 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20af951f-19a7-49e6-ab1d-3bdee930084d-istio-token\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:06:40.182434 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:40.182211 4908 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20af951f-19a7-49e6-ab1d-3bdee930084d-istio-data\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:06:40.182434 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:40.182223 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zq55s\" (UniqueName: \"kubernetes.io/projected/20af951f-19a7-49e6-ab1d-3bdee930084d-kube-api-access-zq55s\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:06:40.182434 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:40.182238 4908 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20af951f-19a7-49e6-ab1d-3bdee930084d-credential-socket\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:06:40.182434 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:40.182250 4908 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20af951f-19a7-49e6-ab1d-3bdee930084d-istio-podinfo\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:06:40.424958 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:40.424916 4908 generic.go:356] "Generic (PLEG): container finished" podID="20af951f-19a7-49e6-ab1d-3bdee930084d" containerID="1944c701fba206b3f327f0931743a5746733758cced662ffe3dc1b778b05a403" exitCode=0 Oct 30 13:06:40.425134 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:40.424958 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a786af-openshift-default-f89f5ffc7-b62z2" event={"ID":"20af951f-19a7-49e6-ab1d-3bdee930084d","Type":"ContainerDied","Data":"1944c701fba206b3f327f0931743a5746733758cced662ffe3dc1b778b05a403"} Oct 30 13:06:40.425134 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:40.424999 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-a786af-openshift-default-f89f5ffc7-b62z2" Oct 30 13:06:40.425134 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:40.425066 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a786af-openshift-default-f89f5ffc7-b62z2" event={"ID":"20af951f-19a7-49e6-ab1d-3bdee930084d","Type":"ContainerDied","Data":"15c8456823cdf8e59d0b9b3a7c9a2aa85b704de19d4224e9dd6ee8507df047e1"} Oct 30 13:06:40.425134 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:40.425098 4908 scope.go:122] "RemoveContainer" containerID="1944c701fba206b3f327f0931743a5746733758cced662ffe3dc1b778b05a403" Oct 30 13:06:40.435994 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:40.435974 4908 scope.go:122] "RemoveContainer" containerID="134480ee395452bba7c2b489dee184e5ec1afcfd6618812b3430bd3479a010d8" Oct 30 13:06:40.446141 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:40.446117 4908 scope.go:122] "RemoveContainer" containerID="1944c701fba206b3f327f0931743a5746733758cced662ffe3dc1b778b05a403" Oct 30 13:06:40.446413 ip-10-0-141-251 kubenswrapper[4908]: E1030 13:06:40.446396 4908 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1944c701fba206b3f327f0931743a5746733758cced662ffe3dc1b778b05a403\": container with ID starting with 1944c701fba206b3f327f0931743a5746733758cced662ffe3dc1b778b05a403 not found: ID does not exist" containerID="1944c701fba206b3f327f0931743a5746733758cced662ffe3dc1b778b05a403" Oct 30 13:06:40.446465 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:40.446421 4908 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1944c701fba206b3f327f0931743a5746733758cced662ffe3dc1b778b05a403"} err="failed to get container status \"1944c701fba206b3f327f0931743a5746733758cced662ffe3dc1b778b05a403\": rpc error: code = NotFound desc = could not find container \"1944c701fba206b3f327f0931743a5746733758cced662ffe3dc1b778b05a403\": container with ID starting with 1944c701fba206b3f327f0931743a5746733758cced662ffe3dc1b778b05a403 not found: ID does not exist" Oct 30 13:06:40.446465 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:40.446438 4908 scope.go:122] "RemoveContainer" containerID="134480ee395452bba7c2b489dee184e5ec1afcfd6618812b3430bd3479a010d8" Oct 30 13:06:40.446681 ip-10-0-141-251 kubenswrapper[4908]: E1030 13:06:40.446661 4908 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"134480ee395452bba7c2b489dee184e5ec1afcfd6618812b3430bd3479a010d8\": container with ID starting with 134480ee395452bba7c2b489dee184e5ec1afcfd6618812b3430bd3479a010d8 not found: ID does not exist" containerID="134480ee395452bba7c2b489dee184e5ec1afcfd6618812b3430bd3479a010d8" Oct 30 13:06:40.446719 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:40.446688 4908 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"134480ee395452bba7c2b489dee184e5ec1afcfd6618812b3430bd3479a010d8"} err="failed to get container status \"134480ee395452bba7c2b489dee184e5ec1afcfd6618812b3430bd3479a010d8\": rpc error: code = NotFound desc = could not find container \"134480ee395452bba7c2b489dee184e5ec1afcfd6618812b3430bd3479a010d8\": container with ID starting with 134480ee395452bba7c2b489dee184e5ec1afcfd6618812b3430bd3479a010d8 not found: ID does not exist" Oct 30 13:06:40.482956 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:40.482917 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-a786af-openshift-default-f89f5ffc7-b62z2"] Oct 30 13:06:40.495860 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:40.495811 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-a786af-openshift-default-f89f5ffc7-b62z2"] Oct 30 13:06:41.615049 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:41.615003 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20af951f-19a7-49e6-ab1d-3bdee930084d" path="/var/lib/kubelet/pods/20af951f-19a7-49e6-ab1d-3bdee930084d/volumes" Oct 30 13:06:41.861775 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:41.861737 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-2545c7-openshift-default-69cf958465-vb4sg"] Oct 30 13:06:41.862048 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:41.862020 4908 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-2545c7-openshift-default-69cf958465-vb4sg" podUID="136fdf42-e3f2-4e4a-83de-a2482e084e5b" containerName="istio-proxy" containerID="cri-o://6f0bc4f496d801fc0d5151f7869ad45663818f6e21061d36165dfe95e4a3636e" gracePeriod=30 Oct 30 13:06:42.399220 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:42.399178 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-12fd33-tn7wc"] Oct 30 13:06:42.399954 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:42.399932 4908 cpu_manager.go:414] "containerMap: removing container" podUID="75b90e54-7d8b-44cf-9e8d-06fe342acc3b" containerName="istio-proxy" Oct 30 13:06:42.400001 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:42.399957 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75b90e54-7d8b-44cf-9e8d-06fe342acc3b" containerName="istio-proxy" Oct 30 13:06:42.400044 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:42.400017 4908 cpu_manager.go:414] "containerMap: removing container" podUID="20af951f-19a7-49e6-ab1d-3bdee930084d" containerName="istio-proxy" Oct 30 13:06:42.400044 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:42.400026 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20af951f-19a7-49e6-ab1d-3bdee930084d" containerName="istio-proxy" Oct 30 13:06:42.400044 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:42.400039 4908 cpu_manager.go:414] "containerMap: removing container" podUID="20af951f-19a7-49e6-ab1d-3bdee930084d" containerName="istio-proxy" Oct 30 13:06:42.400142 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:42.400047 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20af951f-19a7-49e6-ab1d-3bdee930084d" containerName="istio-proxy" Oct 30 13:06:42.400142 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:42.400063 4908 cpu_manager.go:414] "containerMap: removing container" podUID="2335eb21-f2c9-49ab-b060-932461ba8b38" containerName="revoke-keys" Oct 30 13:06:42.400142 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:42.400070 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2335eb21-f2c9-49ab-b060-932461ba8b38" containerName="revoke-keys" Oct 30 13:06:42.400247 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:42.400181 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="20af951f-19a7-49e6-ab1d-3bdee930084d" containerName="istio-proxy" Oct 30 13:06:42.400247 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:42.400192 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="20af951f-19a7-49e6-ab1d-3bdee930084d" containerName="istio-proxy" Oct 30 13:06:42.400247 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:42.400203 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="2335eb21-f2c9-49ab-b060-932461ba8b38" containerName="revoke-keys" Oct 30 13:06:42.405761 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:42.405736 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-12fd33-tn7wc" Oct 30 13:06:42.418123 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:42.418093 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvblq\" (UniqueName: \"kubernetes.io/projected/978bb9d4-019d-4cf5-90de-d90a6abc4c84-kube-api-access-xvblq\") pod \"maas-api-revoke-keys-e2e-extmod-b-12fd33-tn7wc\" (UID: \"978bb9d4-019d-4cf5-90de-d90a6abc4c84\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-12fd33-tn7wc" Oct 30 13:06:42.418265 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:42.418154 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/978bb9d4-019d-4cf5-90de-d90a6abc4c84-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-12fd33-tn7wc\" (UID: \"978bb9d4-019d-4cf5-90de-d90a6abc4c84\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-12fd33-tn7wc" Oct 30 13:06:42.418312 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:42.418269 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/978bb9d4-019d-4cf5-90de-d90a6abc4c84-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-12fd33-tn7wc\" (UID: \"978bb9d4-019d-4cf5-90de-d90a6abc4c84\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-12fd33-tn7wc" Oct 30 13:06:42.420094 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:42.420071 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-12fd33-tn7wc"] Oct 30 13:06:42.428988 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:42.428965 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 13:06:42.428988 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:42.428971 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 13:06:42.457666 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:42.457626 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:06:42.457666 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:42.457644 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s75hq\"" Oct 30 13:06:42.519999 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:42.519964 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvblq\" (UniqueName: \"kubernetes.io/projected/978bb9d4-019d-4cf5-90de-d90a6abc4c84-kube-api-access-xvblq\") pod \"maas-api-revoke-keys-e2e-extmod-b-12fd33-tn7wc\" (UID: \"978bb9d4-019d-4cf5-90de-d90a6abc4c84\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-12fd33-tn7wc" Oct 30 13:06:42.520196 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:42.520011 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/978bb9d4-019d-4cf5-90de-d90a6abc4c84-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-12fd33-tn7wc\" (UID: \"978bb9d4-019d-4cf5-90de-d90a6abc4c84\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-12fd33-tn7wc" Oct 30 13:06:42.520196 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:42.520039 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/978bb9d4-019d-4cf5-90de-d90a6abc4c84-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-12fd33-tn7wc\" (UID: \"978bb9d4-019d-4cf5-90de-d90a6abc4c84\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-12fd33-tn7wc" Oct 30 13:06:42.520761 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:42.520740 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/978bb9d4-019d-4cf5-90de-d90a6abc4c84-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-12fd33-tn7wc\" (UID: \"978bb9d4-019d-4cf5-90de-d90a6abc4c84\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-12fd33-tn7wc" Oct 30 13:06:42.522510 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:42.522485 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/978bb9d4-019d-4cf5-90de-d90a6abc4c84-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-12fd33-tn7wc\" (UID: \"978bb9d4-019d-4cf5-90de-d90a6abc4c84\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-12fd33-tn7wc" Oct 30 13:06:42.546089 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:42.546066 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvblq\" (UniqueName: \"kubernetes.io/projected/978bb9d4-019d-4cf5-90de-d90a6abc4c84-kube-api-access-xvblq\") pod \"maas-api-revoke-keys-e2e-extmod-b-12fd33-tn7wc\" (UID: \"978bb9d4-019d-4cf5-90de-d90a6abc4c84\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-12fd33-tn7wc" Oct 30 13:06:42.722452 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:42.722373 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-12fd33-tn7wc" Oct 30 13:06:43.034628 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:43.034599 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-12fd33-tn7wc"] Oct 30 13:06:43.035128 ip-10-0-141-251 kubenswrapper[4908]: W1030 13:06:43.035102 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod978bb9d4_019d_4cf5_90de_d90a6abc4c84.slice/crio-748d8a238a7a639f5fa0b78551177e9f4b2f21e821a69ace08628e093a552805 WatchSource:0}: Error finding container 748d8a238a7a639f5fa0b78551177e9f4b2f21e821a69ace08628e093a552805: Status 404 returned error can't find the container with id 748d8a238a7a639f5fa0b78551177e9f4b2f21e821a69ace08628e093a552805 Oct 30 13:06:43.452534 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:43.452492 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-12fd33-tn7wc" event={"ID":"978bb9d4-019d-4cf5-90de-d90a6abc4c84","Type":"ContainerStarted","Data":"54b2830545aad83d8792c41d4093dd1c86e5724f26b936323378a617bbcdf0fe"} Oct 30 13:06:43.452722 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:43.452552 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-12fd33-tn7wc" event={"ID":"978bb9d4-019d-4cf5-90de-d90a6abc4c84","Type":"ContainerStarted","Data":"748d8a238a7a639f5fa0b78551177e9f4b2f21e821a69ace08628e093a552805"} Oct 30 13:06:43.499703 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:43.499640 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-12fd33-tn7wc" podStartSLOduration=1.499623946 podStartE2EDuration="1.499623946s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:06:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:06:43.498475496 +0000 UTC m=+4360.012904268" watchObservedRunningTime="2026-10-30 13:06:43.499623946 +0000 UTC m=+4360.014052691" Oct 30 13:06:44.462421 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:44.462371 4908 generic.go:356] "Generic (PLEG): container finished" podID="978bb9d4-019d-4cf5-90de-d90a6abc4c84" containerID="54b2830545aad83d8792c41d4093dd1c86e5724f26b936323378a617bbcdf0fe" exitCode=0 Oct 30 13:06:44.462873 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:44.462443 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-12fd33-tn7wc" event={"ID":"978bb9d4-019d-4cf5-90de-d90a6abc4c84","Type":"ContainerDied","Data":"54b2830545aad83d8792c41d4093dd1c86e5724f26b936323378a617bbcdf0fe"} Oct 30 13:06:45.641555 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:45.641526 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-12fd33-tn7wc" Oct 30 13:06:45.655240 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:45.655209 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/978bb9d4-019d-4cf5-90de-d90a6abc4c84-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/978bb9d4-019d-4cf5-90de-d90a6abc4c84-maas-api-xks-ca\") pod \"978bb9d4-019d-4cf5-90de-d90a6abc4c84\" (UID: \"978bb9d4-019d-4cf5-90de-d90a6abc4c84\") " Oct 30 13:06:45.655336 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:45.655276 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/978bb9d4-019d-4cf5-90de-d90a6abc4c84-kube-api-access-xvblq\" (UniqueName: \"kubernetes.io/projected/978bb9d4-019d-4cf5-90de-d90a6abc4c84-kube-api-access-xvblq\") pod \"978bb9d4-019d-4cf5-90de-d90a6abc4c84\" (UID: \"978bb9d4-019d-4cf5-90de-d90a6abc4c84\") " Oct 30 13:06:45.655336 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:45.655307 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/978bb9d4-019d-4cf5-90de-d90a6abc4c84-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/978bb9d4-019d-4cf5-90de-d90a6abc4c84-maas-api-openshift-ca\") pod \"978bb9d4-019d-4cf5-90de-d90a6abc4c84\" (UID: \"978bb9d4-019d-4cf5-90de-d90a6abc4c84\") " Oct 30 13:06:45.655741 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:45.655713 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/978bb9d4-019d-4cf5-90de-d90a6abc4c84-maas-api-openshift-ca" pod "978bb9d4-019d-4cf5-90de-d90a6abc4c84" (UID: "978bb9d4-019d-4cf5-90de-d90a6abc4c84"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:06:45.657405 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:45.657375 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/978bb9d4-019d-4cf5-90de-d90a6abc4c84-maas-api-xks-ca" pod "978bb9d4-019d-4cf5-90de-d90a6abc4c84" (UID: "978bb9d4-019d-4cf5-90de-d90a6abc4c84"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:06:45.657405 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:45.657382 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/978bb9d4-019d-4cf5-90de-d90a6abc4c84-kube-api-access-xvblq" pod "978bb9d4-019d-4cf5-90de-d90a6abc4c84" (UID: "978bb9d4-019d-4cf5-90de-d90a6abc4c84"). InnerVolumeSpecName "kube-api-access-xvblq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:06:45.756753 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:45.756715 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/978bb9d4-019d-4cf5-90de-d90a6abc4c84-maas-api-xks-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:06:45.756753 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:45.756753 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvblq\" (UniqueName: \"kubernetes.io/projected/978bb9d4-019d-4cf5-90de-d90a6abc4c84-kube-api-access-xvblq\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:06:45.757019 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:45.756766 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/978bb9d4-019d-4cf5-90de-d90a6abc4c84-maas-api-openshift-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:06:46.522566 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:46.522534 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-12fd33-tn7wc" Oct 30 13:06:46.522791 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:46.522533 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-12fd33-tn7wc" event={"ID":"978bb9d4-019d-4cf5-90de-d90a6abc4c84","Type":"ContainerDied","Data":"748d8a238a7a639f5fa0b78551177e9f4b2f21e821a69ace08628e093a552805"} Oct 30 13:06:46.522791 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:46.522664 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="748d8a238a7a639f5fa0b78551177e9f4b2f21e821a69ace08628e093a552805" Oct 30 13:06:47.087838 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:47.087790 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7f73653-99bl7"] Oct 30 13:06:47.101164 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:47.101134 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7f73653-99bl7"] Oct 30 13:06:47.148779 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:47.148694 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-2545c7-openshift-default-69cf958465-vb4sg" Oct 30 13:06:47.172126 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:47.172076 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/136fdf42-e3f2-4e4a-83de-a2482e084e5b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/136fdf42-e3f2-4e4a-83de-a2482e084e5b-workload-certs\") pod \"136fdf42-e3f2-4e4a-83de-a2482e084e5b\" (UID: \"136fdf42-e3f2-4e4a-83de-a2482e084e5b\") " Oct 30 13:06:47.172325 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:47.172136 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/136fdf42-e3f2-4e4a-83de-a2482e084e5b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/136fdf42-e3f2-4e4a-83de-a2482e084e5b-workload-socket\") pod \"136fdf42-e3f2-4e4a-83de-a2482e084e5b\" (UID: \"136fdf42-e3f2-4e4a-83de-a2482e084e5b\") " Oct 30 13:06:47.172325 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:47.172178 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/136fdf42-e3f2-4e4a-83de-a2482e084e5b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/136fdf42-e3f2-4e4a-83de-a2482e084e5b-credential-socket\") pod \"136fdf42-e3f2-4e4a-83de-a2482e084e5b\" (UID: \"136fdf42-e3f2-4e4a-83de-a2482e084e5b\") " Oct 30 13:06:47.172325 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:47.172221 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/136fdf42-e3f2-4e4a-83de-a2482e084e5b-kube-api-access-qqftf\" (UniqueName: \"kubernetes.io/projected/136fdf42-e3f2-4e4a-83de-a2482e084e5b-kube-api-access-qqftf\") pod \"136fdf42-e3f2-4e4a-83de-a2482e084e5b\" (UID: \"136fdf42-e3f2-4e4a-83de-a2482e084e5b\") " Oct 30 13:06:47.172325 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:47.172247 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/136fdf42-e3f2-4e4a-83de-a2482e084e5b-istio-token\" (UniqueName: \"kubernetes.io/projected/136fdf42-e3f2-4e4a-83de-a2482e084e5b-istio-token\") pod \"136fdf42-e3f2-4e4a-83de-a2482e084e5b\" (UID: \"136fdf42-e3f2-4e4a-83de-a2482e084e5b\") " Oct 30 13:06:47.172582 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:47.172334 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/136fdf42-e3f2-4e4a-83de-a2482e084e5b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/136fdf42-e3f2-4e4a-83de-a2482e084e5b-istio-envoy\") pod \"136fdf42-e3f2-4e4a-83de-a2482e084e5b\" (UID: \"136fdf42-e3f2-4e4a-83de-a2482e084e5b\") " Oct 30 13:06:47.172582 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:47.172426 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/136fdf42-e3f2-4e4a-83de-a2482e084e5b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/136fdf42-e3f2-4e4a-83de-a2482e084e5b-istio-data\") pod \"136fdf42-e3f2-4e4a-83de-a2482e084e5b\" (UID: \"136fdf42-e3f2-4e4a-83de-a2482e084e5b\") " Oct 30 13:06:47.172582 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:47.172465 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/136fdf42-e3f2-4e4a-83de-a2482e084e5b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/136fdf42-e3f2-4e4a-83de-a2482e084e5b-istio-podinfo\") pod \"136fdf42-e3f2-4e4a-83de-a2482e084e5b\" (UID: \"136fdf42-e3f2-4e4a-83de-a2482e084e5b\") " Oct 30 13:06:47.172582 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:47.172496 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/136fdf42-e3f2-4e4a-83de-a2482e084e5b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/136fdf42-e3f2-4e4a-83de-a2482e084e5b-istiod-ca-cert\") pod \"136fdf42-e3f2-4e4a-83de-a2482e084e5b\" (UID: \"136fdf42-e3f2-4e4a-83de-a2482e084e5b\") " Oct 30 13:06:47.172582 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:47.172495 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/136fdf42-e3f2-4e4a-83de-a2482e084e5b-workload-certs" pod "136fdf42-e3f2-4e4a-83de-a2482e084e5b" (UID: "136fdf42-e3f2-4e4a-83de-a2482e084e5b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:06:47.172582 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:47.172559 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/136fdf42-e3f2-4e4a-83de-a2482e084e5b-credential-socket" pod "136fdf42-e3f2-4e4a-83de-a2482e084e5b" (UID: "136fdf42-e3f2-4e4a-83de-a2482e084e5b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:06:47.172934 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:47.172817 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/136fdf42-e3f2-4e4a-83de-a2482e084e5b-istio-data" pod "136fdf42-e3f2-4e4a-83de-a2482e084e5b" (UID: "136fdf42-e3f2-4e4a-83de-a2482e084e5b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:06:47.173120 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:47.173025 4908 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/136fdf42-e3f2-4e4a-83de-a2482e084e5b-credential-socket\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:06:47.173120 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:47.173019 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/136fdf42-e3f2-4e4a-83de-a2482e084e5b-workload-socket" pod "136fdf42-e3f2-4e4a-83de-a2482e084e5b" (UID: "136fdf42-e3f2-4e4a-83de-a2482e084e5b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:06:47.173120 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:47.173051 4908 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/136fdf42-e3f2-4e4a-83de-a2482e084e5b-istio-data\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:06:47.173120 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:47.173066 4908 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/136fdf42-e3f2-4e4a-83de-a2482e084e5b-workload-certs\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:06:47.173406 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:47.173229 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/136fdf42-e3f2-4e4a-83de-a2482e084e5b-istiod-ca-cert" pod "136fdf42-e3f2-4e4a-83de-a2482e084e5b" (UID: "136fdf42-e3f2-4e4a-83de-a2482e084e5b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:06:47.175742 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:47.175702 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/136fdf42-e3f2-4e4a-83de-a2482e084e5b-istio-token" pod "136fdf42-e3f2-4e4a-83de-a2482e084e5b" (UID: "136fdf42-e3f2-4e4a-83de-a2482e084e5b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:06:47.175742 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:47.175707 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/136fdf42-e3f2-4e4a-83de-a2482e084e5b-istio-envoy" pod "136fdf42-e3f2-4e4a-83de-a2482e084e5b" (UID: "136fdf42-e3f2-4e4a-83de-a2482e084e5b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:06:47.176076 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:47.175745 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/136fdf42-e3f2-4e4a-83de-a2482e084e5b-kube-api-access-qqftf" pod "136fdf42-e3f2-4e4a-83de-a2482e084e5b" (UID: "136fdf42-e3f2-4e4a-83de-a2482e084e5b"). InnerVolumeSpecName "kube-api-access-qqftf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:06:47.176076 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:47.175979 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/136fdf42-e3f2-4e4a-83de-a2482e084e5b-istio-podinfo" pod "136fdf42-e3f2-4e4a-83de-a2482e084e5b" (UID: "136fdf42-e3f2-4e4a-83de-a2482e084e5b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:06:47.274266 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:47.274229 4908 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/136fdf42-e3f2-4e4a-83de-a2482e084e5b-istio-podinfo\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:06:47.274266 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:47.274260 4908 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/136fdf42-e3f2-4e4a-83de-a2482e084e5b-istiod-ca-cert\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:06:47.274266 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:47.274270 4908 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/136fdf42-e3f2-4e4a-83de-a2482e084e5b-workload-socket\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:06:47.274528 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:47.274279 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqftf\" (UniqueName: \"kubernetes.io/projected/136fdf42-e3f2-4e4a-83de-a2482e084e5b-kube-api-access-qqftf\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:06:47.274528 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:47.274289 4908 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/136fdf42-e3f2-4e4a-83de-a2482e084e5b-istio-token\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:06:47.274528 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:47.274298 4908 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/136fdf42-e3f2-4e4a-83de-a2482e084e5b-istio-envoy\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:06:47.533472 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:47.533431 4908 generic.go:356] "Generic (PLEG): container finished" podID="136fdf42-e3f2-4e4a-83de-a2482e084e5b" containerID="6f0bc4f496d801fc0d5151f7869ad45663818f6e21061d36165dfe95e4a3636e" exitCode=0 Oct 30 13:06:47.533638 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:47.533480 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-2545c7-openshift-default-69cf958465-vb4sg" Oct 30 13:06:47.533638 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:47.533513 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-2545c7-openshift-default-69cf958465-vb4sg" event={"ID":"136fdf42-e3f2-4e4a-83de-a2482e084e5b","Type":"ContainerDied","Data":"6f0bc4f496d801fc0d5151f7869ad45663818f6e21061d36165dfe95e4a3636e"} Oct 30 13:06:47.533638 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:47.533572 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-2545c7-openshift-default-69cf958465-vb4sg" event={"ID":"136fdf42-e3f2-4e4a-83de-a2482e084e5b","Type":"ContainerDied","Data":"7c91310107e7843b1ad04b84f339e4e434ae7e717b36d675f064b0a218f8705b"} Oct 30 13:06:47.533638 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:47.533601 4908 scope.go:122] "RemoveContainer" containerID="6f0bc4f496d801fc0d5151f7869ad45663818f6e21061d36165dfe95e4a3636e" Oct 30 13:06:47.545691 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:47.545669 4908 scope.go:122] "RemoveContainer" containerID="0885278d558403846e921e2da33c6a94cc6ac2b4ae2d0710882eaa03c1a72b2c" Oct 30 13:06:47.555631 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:47.555608 4908 scope.go:122] "RemoveContainer" containerID="6f0bc4f496d801fc0d5151f7869ad45663818f6e21061d36165dfe95e4a3636e" Oct 30 13:06:47.555981 ip-10-0-141-251 kubenswrapper[4908]: E1030 13:06:47.555956 4908 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f0bc4f496d801fc0d5151f7869ad45663818f6e21061d36165dfe95e4a3636e\": container with ID starting with 6f0bc4f496d801fc0d5151f7869ad45663818f6e21061d36165dfe95e4a3636e not found: ID does not exist" containerID="6f0bc4f496d801fc0d5151f7869ad45663818f6e21061d36165dfe95e4a3636e" Oct 30 13:06:47.556075 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:47.555993 4908 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f0bc4f496d801fc0d5151f7869ad45663818f6e21061d36165dfe95e4a3636e"} err="failed to get container status \"6f0bc4f496d801fc0d5151f7869ad45663818f6e21061d36165dfe95e4a3636e\": rpc error: code = NotFound desc = could not find container \"6f0bc4f496d801fc0d5151f7869ad45663818f6e21061d36165dfe95e4a3636e\": container with ID starting with 6f0bc4f496d801fc0d5151f7869ad45663818f6e21061d36165dfe95e4a3636e not found: ID does not exist" Oct 30 13:06:47.556075 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:47.556017 4908 scope.go:122] "RemoveContainer" containerID="0885278d558403846e921e2da33c6a94cc6ac2b4ae2d0710882eaa03c1a72b2c" Oct 30 13:06:47.556265 ip-10-0-141-251 kubenswrapper[4908]: E1030 13:06:47.556245 4908 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0885278d558403846e921e2da33c6a94cc6ac2b4ae2d0710882eaa03c1a72b2c\": container with ID starting with 0885278d558403846e921e2da33c6a94cc6ac2b4ae2d0710882eaa03c1a72b2c not found: ID does not exist" containerID="0885278d558403846e921e2da33c6a94cc6ac2b4ae2d0710882eaa03c1a72b2c" Oct 30 13:06:47.556329 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:47.556269 4908 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0885278d558403846e921e2da33c6a94cc6ac2b4ae2d0710882eaa03c1a72b2c"} err="failed to get container status \"0885278d558403846e921e2da33c6a94cc6ac2b4ae2d0710882eaa03c1a72b2c\": rpc error: code = NotFound desc = could not find container \"0885278d558403846e921e2da33c6a94cc6ac2b4ae2d0710882eaa03c1a72b2c\": container with ID starting with 0885278d558403846e921e2da33c6a94cc6ac2b4ae2d0710882eaa03c1a72b2c not found: ID does not exist" Oct 30 13:06:47.606158 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:47.606122 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eec2d564-368e-4413-9ccb-9010f03eb27c" path="/var/lib/kubelet/pods/eec2d564-368e-4413-9ccb-9010f03eb27c/volumes" Oct 30 13:06:47.611208 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:47.611181 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-2545c7-openshift-default-69cf958465-vb4sg"] Oct 30 13:06:47.625948 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:47.625918 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-2545c7-openshift-default-69cf958465-vb4sg"] Oct 30 13:06:49.605988 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:49.605951 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="136fdf42-e3f2-4e4a-83de-a2482e084e5b" path="/var/lib/kubelet/pods/136fdf42-e3f2-4e4a-83de-a2482e084e5b/volumes" Oct 30 13:06:49.686309 ip-10-0-141-251 kubenswrapper[4908]: E1030 13:06:49.686269 4908 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vn6vv" containerName="extractor" Oct 30 13:06:51.129303 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:51.129263 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-dd0628-xdwfv"] Oct 30 13:06:51.142350 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:51.142313 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-dd0628-xdwfv"] Oct 30 13:06:51.606980 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:51.606944 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef849d43-7bb0-4096-9736-40ae518c6526" path="/var/lib/kubelet/pods/ef849d43-7bb0-4096-9736-40ae518c6526/volumes" Oct 30 13:06:51.756810 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:51.756767 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9c3f12-zrzk7"] Oct 30 13:06:51.783029 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:51.779716 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9c3f12-zrzk7"] Oct 30 13:06:53.607597 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:53.607560 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8d76aa5-3fbc-4311-a104-242d252c5be3" path="/var/lib/kubelet/pods/c8d76aa5-3fbc-4311-a104-242d252c5be3/volumes" Oct 30 13:06:58.902883 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:58.902685 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-9c3f12-openshift-default-78b944cc9c-fdpqh"] Oct 30 13:06:58.903549 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:58.903351 4908 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-9c3f12-openshift-default-78b944cc9c-fdpqh" podUID="a2029481-e8ad-40ef-97b7-81140abedb5c" containerName="istio-proxy" containerID="cri-o://8c2d59e692b8a4c74147a8d0aae249a6b34db995d531a52e86abb25c5ce863e1" gracePeriod=30 Oct 30 13:06:58.927656 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:58.927609 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-dd0628-openshift-default-54686d76fc-x98kl"] Oct 30 13:06:58.927880 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:58.927811 4908 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-dd0628-openshift-default-54686d76fc-x98kl" podUID="2d070106-c79c-44d5-a7a9-b766ee82b439" containerName="istio-proxy" containerID="cri-o://5c86db12fa0656d65f0487fd1defd3ebacc6d436d4eab07243d282cfcbb509b4" gracePeriod=30 Oct 30 13:06:59.040988 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:59.040932 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09c34f4e-92lmw"] Oct 30 13:06:59.043518 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:59.043474 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09c34f4e-92lmw"] Oct 30 13:06:59.511450 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:59.511410 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-10f364-nmjt8"] Oct 30 13:06:59.511928 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:59.511913 4908 cpu_manager.go:414] "containerMap: removing container" podUID="136fdf42-e3f2-4e4a-83de-a2482e084e5b" containerName="istio-proxy" Oct 30 13:06:59.511999 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:59.511929 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="136fdf42-e3f2-4e4a-83de-a2482e084e5b" containerName="istio-proxy" Oct 30 13:06:59.511999 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:59.511955 4908 cpu_manager.go:414] "containerMap: removing container" podUID="978bb9d4-019d-4cf5-90de-d90a6abc4c84" containerName="revoke-keys" Oct 30 13:06:59.511999 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:59.511960 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="978bb9d4-019d-4cf5-90de-d90a6abc4c84" containerName="revoke-keys" Oct 30 13:06:59.512126 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:59.512027 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="978bb9d4-019d-4cf5-90de-d90a6abc4c84" containerName="revoke-keys" Oct 30 13:06:59.512126 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:59.512036 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="136fdf42-e3f2-4e4a-83de-a2482e084e5b" containerName="istio-proxy" Oct 30 13:06:59.517101 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:59.517077 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-10f364-nmjt8" Oct 30 13:06:59.533183 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:59.533129 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 13:06:59.533351 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:59.533288 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 13:06:59.535020 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:59.534992 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-10f364-nmjt8"] Oct 30 13:06:59.567676 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:59.567633 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:06:59.567905 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:59.567745 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s75hq\"" Oct 30 13:06:59.608343 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:59.608309 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8888a20-3251-4d90-b5ac-eb8ea3b81245" path="/var/lib/kubelet/pods/f8888a20-3251-4d90-b5ac-eb8ea3b81245/volumes" Oct 30 13:06:59.631317 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:59.631271 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39a5d567-2188-4af8-9171-9879694ecc67-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-10f364-nmjt8\" (UID: \"39a5d567-2188-4af8-9171-9879694ecc67\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-10f364-nmjt8" Oct 30 13:06:59.631317 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:59.631320 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hk4w\" (UniqueName: \"kubernetes.io/projected/39a5d567-2188-4af8-9171-9879694ecc67-kube-api-access-6hk4w\") pod \"maas-api-revoke-keys-e2e-extmod-a-10f364-nmjt8\" (UID: \"39a5d567-2188-4af8-9171-9879694ecc67\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-10f364-nmjt8" Oct 30 13:06:59.631547 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:59.631522 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39a5d567-2188-4af8-9171-9879694ecc67-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-10f364-nmjt8\" (UID: \"39a5d567-2188-4af8-9171-9879694ecc67\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-10f364-nmjt8" Oct 30 13:06:59.669147 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:59.669106 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f920f7-kfk24"] Oct 30 13:06:59.669614 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:59.669598 4908 cpu_manager.go:414] "containerMap: removing container" podUID="136fdf42-e3f2-4e4a-83de-a2482e084e5b" containerName="istio-proxy" Oct 30 13:06:59.669614 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:59.669613 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="136fdf42-e3f2-4e4a-83de-a2482e084e5b" containerName="istio-proxy" Oct 30 13:06:59.669750 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:59.669720 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="136fdf42-e3f2-4e4a-83de-a2482e084e5b" containerName="istio-proxy" Oct 30 13:06:59.680571 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:59.680523 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f920f7-kfk24" Oct 30 13:06:59.695258 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:59.695221 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f920f7-kfk24"] Oct 30 13:06:59.733877 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:59.732946 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39a5d567-2188-4af8-9171-9879694ecc67-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-10f364-nmjt8\" (UID: \"39a5d567-2188-4af8-9171-9879694ecc67\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-10f364-nmjt8" Oct 30 13:06:59.733877 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:59.733061 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d4f38b5-6492-406c-80e1-cbd779f9fd73-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-f920f7-kfk24\" (UID: \"1d4f38b5-6492-406c-80e1-cbd779f9fd73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f920f7-kfk24" Oct 30 13:06:59.733877 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:59.733208 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4x4cb\" (UniqueName: \"kubernetes.io/projected/1d4f38b5-6492-406c-80e1-cbd779f9fd73-kube-api-access-4x4cb\") pod \"maas-api-revoke-keys-e2e-worker-w0-f920f7-kfk24\" (UID: \"1d4f38b5-6492-406c-80e1-cbd779f9fd73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f920f7-kfk24" Oct 30 13:06:59.733877 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:59.733260 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39a5d567-2188-4af8-9171-9879694ecc67-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-10f364-nmjt8\" (UID: \"39a5d567-2188-4af8-9171-9879694ecc67\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-10f364-nmjt8" Oct 30 13:06:59.733877 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:59.733278 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d4f38b5-6492-406c-80e1-cbd779f9fd73-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-f920f7-kfk24\" (UID: \"1d4f38b5-6492-406c-80e1-cbd779f9fd73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f920f7-kfk24" Oct 30 13:06:59.733877 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:59.733398 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hk4w\" (UniqueName: \"kubernetes.io/projected/39a5d567-2188-4af8-9171-9879694ecc67-kube-api-access-6hk4w\") pod \"maas-api-revoke-keys-e2e-extmod-a-10f364-nmjt8\" (UID: \"39a5d567-2188-4af8-9171-9879694ecc67\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-10f364-nmjt8" Oct 30 13:06:59.734388 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:59.734227 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39a5d567-2188-4af8-9171-9879694ecc67-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-10f364-nmjt8\" (UID: \"39a5d567-2188-4af8-9171-9879694ecc67\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-10f364-nmjt8" Oct 30 13:06:59.735945 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:59.735921 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39a5d567-2188-4af8-9171-9879694ecc67-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-10f364-nmjt8\" (UID: \"39a5d567-2188-4af8-9171-9879694ecc67\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-10f364-nmjt8" Oct 30 13:06:59.758591 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:59.758552 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hk4w\" (UniqueName: \"kubernetes.io/projected/39a5d567-2188-4af8-9171-9879694ecc67-kube-api-access-6hk4w\") pod \"maas-api-revoke-keys-e2e-extmod-a-10f364-nmjt8\" (UID: \"39a5d567-2188-4af8-9171-9879694ecc67\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-10f364-nmjt8" Oct 30 13:06:59.831648 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:59.831565 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-10f364-nmjt8" Oct 30 13:06:59.834620 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:59.834580 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d4f38b5-6492-406c-80e1-cbd779f9fd73-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-f920f7-kfk24\" (UID: \"1d4f38b5-6492-406c-80e1-cbd779f9fd73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f920f7-kfk24" Oct 30 13:06:59.834942 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:59.834641 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4x4cb\" (UniqueName: \"kubernetes.io/projected/1d4f38b5-6492-406c-80e1-cbd779f9fd73-kube-api-access-4x4cb\") pod \"maas-api-revoke-keys-e2e-worker-w0-f920f7-kfk24\" (UID: \"1d4f38b5-6492-406c-80e1-cbd779f9fd73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f920f7-kfk24" Oct 30 13:06:59.834942 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:59.834690 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d4f38b5-6492-406c-80e1-cbd779f9fd73-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-f920f7-kfk24\" (UID: \"1d4f38b5-6492-406c-80e1-cbd779f9fd73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f920f7-kfk24" Oct 30 13:06:59.835606 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:59.835580 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d4f38b5-6492-406c-80e1-cbd779f9fd73-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-f920f7-kfk24\" (UID: \"1d4f38b5-6492-406c-80e1-cbd779f9fd73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f920f7-kfk24" Oct 30 13:06:59.837037 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:59.837005 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d4f38b5-6492-406c-80e1-cbd779f9fd73-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-f920f7-kfk24\" (UID: \"1d4f38b5-6492-406c-80e1-cbd779f9fd73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f920f7-kfk24" Oct 30 13:06:59.859245 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:59.859205 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4x4cb\" (UniqueName: \"kubernetes.io/projected/1d4f38b5-6492-406c-80e1-cbd779f9fd73-kube-api-access-4x4cb\") pod \"maas-api-revoke-keys-e2e-worker-w0-f920f7-kfk24\" (UID: \"1d4f38b5-6492-406c-80e1-cbd779f9fd73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f920f7-kfk24" Oct 30 13:06:59.999461 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:06:59.999419 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f920f7-kfk24" Oct 30 13:07:00.146693 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:00.146656 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-10f364-nmjt8"] Oct 30 13:07:00.342089 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:00.342051 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f920f7-kfk24"] Oct 30 13:07:00.344059 ip-10-0-141-251 kubenswrapper[4908]: W1030 13:07:00.344019 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d4f38b5_6492_406c_80e1_cbd779f9fd73.slice/crio-8193a0b778574f8b2065707cff2712a952e131d6b353f8fbe0683b5f41cd792f WatchSource:0}: Error finding container 8193a0b778574f8b2065707cff2712a952e131d6b353f8fbe0683b5f41cd792f: Status 404 returned error can't find the container with id 8193a0b778574f8b2065707cff2712a952e131d6b353f8fbe0683b5f41cd792f Oct 30 13:07:00.648476 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:00.648427 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-10f364-nmjt8" event={"ID":"39a5d567-2188-4af8-9171-9879694ecc67","Type":"ContainerStarted","Data":"5a0f6ff9e13f686d0a04e33bc58a9efc390994ad96130fa5b2176b3833c7c281"} Oct 30 13:07:00.648683 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:00.648501 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-10f364-nmjt8" event={"ID":"39a5d567-2188-4af8-9171-9879694ecc67","Type":"ContainerStarted","Data":"d27c77a17ab42a0974ad7a55e758fb7d6bfc45033549a10065973e7b68772678"} Oct 30 13:07:00.649884 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:00.649853 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f920f7-kfk24" event={"ID":"1d4f38b5-6492-406c-80e1-cbd779f9fd73","Type":"ContainerStarted","Data":"8193a0b778574f8b2065707cff2712a952e131d6b353f8fbe0683b5f41cd792f"} Oct 30 13:07:00.694779 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:00.694702 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-10f364-nmjt8" podStartSLOduration=1.694686253 podStartE2EDuration="1.694686253s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:06:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:07:00.69112974 +0000 UTC m=+4377.205558487" watchObservedRunningTime="2026-10-30 13:07:00.694686253 +0000 UTC m=+4377.209114999" Oct 30 13:07:00.705859 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:00.705777 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0f7d14-fdln9"] Oct 30 13:07:00.709177 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:00.709136 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0f7d14-fdln9"] Oct 30 13:07:01.482646 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:01.482605 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9wwfc"] Oct 30 13:07:01.606182 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:01.606148 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2335eb21-f2c9-49ab-b060-932461ba8b38" path="/var/lib/kubelet/pods/2335eb21-f2c9-49ab-b060-932461ba8b38/volumes" Oct 30 13:07:01.659418 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:01.659372 4908 generic.go:356] "Generic (PLEG): container finished" podID="39a5d567-2188-4af8-9171-9879694ecc67" containerID="5a0f6ff9e13f686d0a04e33bc58a9efc390994ad96130fa5b2176b3833c7c281" exitCode=0 Oct 30 13:07:01.659600 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:01.659457 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-10f364-nmjt8" event={"ID":"39a5d567-2188-4af8-9171-9879694ecc67","Type":"ContainerDied","Data":"5a0f6ff9e13f686d0a04e33bc58a9efc390994ad96130fa5b2176b3833c7c281"} Oct 30 13:07:01.661405 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:01.661374 4908 generic.go:356] "Generic (PLEG): container finished" podID="1d4f38b5-6492-406c-80e1-cbd779f9fd73" containerID="dcceb32dbe45b42bcaee76639a1991eaccb538cce0845dd53b20cf7f531aaf07" exitCode=0 Oct 30 13:07:01.661565 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:01.661453 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f920f7-kfk24" event={"ID":"1d4f38b5-6492-406c-80e1-cbd779f9fd73","Type":"ContainerDied","Data":"dcceb32dbe45b42bcaee76639a1991eaccb538cce0845dd53b20cf7f531aaf07"} Oct 30 13:07:02.831354 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:02.831321 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-10f364-nmjt8" Oct 30 13:07:02.866413 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:02.866387 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f920f7-kfk24" Oct 30 13:07:02.979951 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:02.979915 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/39a5d567-2188-4af8-9171-9879694ecc67-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39a5d567-2188-4af8-9171-9879694ecc67-maas-api-xks-ca\") pod \"39a5d567-2188-4af8-9171-9879694ecc67\" (UID: \"39a5d567-2188-4af8-9171-9879694ecc67\") " Oct 30 13:07:02.980113 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:02.979996 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1d4f38b5-6492-406c-80e1-cbd779f9fd73-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d4f38b5-6492-406c-80e1-cbd779f9fd73-maas-api-xks-ca\") pod \"1d4f38b5-6492-406c-80e1-cbd779f9fd73\" (UID: \"1d4f38b5-6492-406c-80e1-cbd779f9fd73\") " Oct 30 13:07:02.980113 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:02.980067 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d4f38b5-6492-406c-80e1-cbd779f9fd73-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d4f38b5-6492-406c-80e1-cbd779f9fd73-maas-api-openshift-ca\") pod \"1d4f38b5-6492-406c-80e1-cbd779f9fd73\" (UID: \"1d4f38b5-6492-406c-80e1-cbd779f9fd73\") " Oct 30 13:07:02.980223 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:02.980135 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d4f38b5-6492-406c-80e1-cbd779f9fd73-kube-api-access-4x4cb\" (UniqueName: \"kubernetes.io/projected/1d4f38b5-6492-406c-80e1-cbd779f9fd73-kube-api-access-4x4cb\") pod \"1d4f38b5-6492-406c-80e1-cbd779f9fd73\" (UID: \"1d4f38b5-6492-406c-80e1-cbd779f9fd73\") " Oct 30 13:07:02.980223 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:02.980205 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39a5d567-2188-4af8-9171-9879694ecc67-kube-api-access-6hk4w\" (UniqueName: \"kubernetes.io/projected/39a5d567-2188-4af8-9171-9879694ecc67-kube-api-access-6hk4w\") pod \"39a5d567-2188-4af8-9171-9879694ecc67\" (UID: \"39a5d567-2188-4af8-9171-9879694ecc67\") " Oct 30 13:07:02.980347 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:02.980235 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39a5d567-2188-4af8-9171-9879694ecc67-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39a5d567-2188-4af8-9171-9879694ecc67-maas-api-openshift-ca\") pod \"39a5d567-2188-4af8-9171-9879694ecc67\" (UID: \"39a5d567-2188-4af8-9171-9879694ecc67\") " Oct 30 13:07:02.980961 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:02.980477 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d4f38b5-6492-406c-80e1-cbd779f9fd73-maas-api-openshift-ca" pod "1d4f38b5-6492-406c-80e1-cbd779f9fd73" (UID: "1d4f38b5-6492-406c-80e1-cbd779f9fd73"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:07:02.980961 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:02.980897 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39a5d567-2188-4af8-9171-9879694ecc67-maas-api-openshift-ca" pod "39a5d567-2188-4af8-9171-9879694ecc67" (UID: "39a5d567-2188-4af8-9171-9879694ecc67"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:07:02.980961 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:02.980919 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d4f38b5-6492-406c-80e1-cbd779f9fd73-maas-api-openshift-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:07:02.982565 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:02.982504 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39a5d567-2188-4af8-9171-9879694ecc67-maas-api-xks-ca" pod "39a5d567-2188-4af8-9171-9879694ecc67" (UID: "39a5d567-2188-4af8-9171-9879694ecc67"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:07:02.982707 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:02.982642 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d4f38b5-6492-406c-80e1-cbd779f9fd73-kube-api-access-4x4cb" pod "1d4f38b5-6492-406c-80e1-cbd779f9fd73" (UID: "1d4f38b5-6492-406c-80e1-cbd779f9fd73"). InnerVolumeSpecName "kube-api-access-4x4cb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:07:02.982863 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:02.982822 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d4f38b5-6492-406c-80e1-cbd779f9fd73-maas-api-xks-ca" pod "1d4f38b5-6492-406c-80e1-cbd779f9fd73" (UID: "1d4f38b5-6492-406c-80e1-cbd779f9fd73"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:07:02.983286 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:02.983260 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39a5d567-2188-4af8-9171-9879694ecc67-kube-api-access-6hk4w" pod "39a5d567-2188-4af8-9171-9879694ecc67" (UID: "39a5d567-2188-4af8-9171-9879694ecc67"). InnerVolumeSpecName "kube-api-access-6hk4w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:07:03.082289 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:03.082244 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4x4cb\" (UniqueName: \"kubernetes.io/projected/1d4f38b5-6492-406c-80e1-cbd779f9fd73-kube-api-access-4x4cb\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:07:03.082289 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:03.082288 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hk4w\" (UniqueName: \"kubernetes.io/projected/39a5d567-2188-4af8-9171-9879694ecc67-kube-api-access-6hk4w\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:07:03.082512 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:03.082303 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39a5d567-2188-4af8-9171-9879694ecc67-maas-api-openshift-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:07:03.082512 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:03.082320 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39a5d567-2188-4af8-9171-9879694ecc67-maas-api-xks-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:07:03.082512 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:03.082336 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d4f38b5-6492-406c-80e1-cbd779f9fd73-maas-api-xks-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:07:03.647001 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:03.646963 4908 patch_prober.go:29] interesting pod/e2e-inf-b-9c3f12-openshift-default-78b944cc9c-fdpqh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 13:07:03.647154 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:03.647021 4908 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-9c3f12-openshift-default-78b944cc9c-fdpqh" podUID="a2029481-e8ad-40ef-97b7-81140abedb5c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 13:07:03.680054 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:03.680023 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-10f364-nmjt8" Oct 30 13:07:03.680260 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:03.680111 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-10f364-nmjt8" event={"ID":"39a5d567-2188-4af8-9171-9879694ecc67","Type":"ContainerDied","Data":"d27c77a17ab42a0974ad7a55e758fb7d6bfc45033549a10065973e7b68772678"} Oct 30 13:07:03.680260 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:03.680171 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d27c77a17ab42a0974ad7a55e758fb7d6bfc45033549a10065973e7b68772678" Oct 30 13:07:03.691571 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:03.691531 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f920f7-kfk24" event={"ID":"1d4f38b5-6492-406c-80e1-cbd779f9fd73","Type":"ContainerDied","Data":"8193a0b778574f8b2065707cff2712a952e131d6b353f8fbe0683b5f41cd792f"} Oct 30 13:07:03.691751 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:03.691721 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8193a0b778574f8b2065707cff2712a952e131d6b353f8fbe0683b5f41cd792f" Oct 30 13:07:03.691977 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:03.691958 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f920f7-kfk24" Oct 30 13:07:04.204117 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:04.204089 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9c3f12-openshift-default-78b944cc9c-fdpqh" Oct 30 13:07:04.228395 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:04.228368 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-dd0628-openshift-default-54686d76fc-x98kl" Oct 30 13:07:04.300303 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:04.300255 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a2029481-e8ad-40ef-97b7-81140abedb5c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2029481-e8ad-40ef-97b7-81140abedb5c-istio-podinfo\") pod \"a2029481-e8ad-40ef-97b7-81140abedb5c\" (UID: \"a2029481-e8ad-40ef-97b7-81140abedb5c\") " Oct 30 13:07:04.300489 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:04.300349 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2029481-e8ad-40ef-97b7-81140abedb5c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2029481-e8ad-40ef-97b7-81140abedb5c-workload-certs\") pod \"a2029481-e8ad-40ef-97b7-81140abedb5c\" (UID: \"a2029481-e8ad-40ef-97b7-81140abedb5c\") " Oct 30 13:07:04.300489 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:04.300374 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2029481-e8ad-40ef-97b7-81140abedb5c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2029481-e8ad-40ef-97b7-81140abedb5c-istio-envoy\") pod \"a2029481-e8ad-40ef-97b7-81140abedb5c\" (UID: \"a2029481-e8ad-40ef-97b7-81140abedb5c\") " Oct 30 13:07:04.300489 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:04.300420 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2029481-e8ad-40ef-97b7-81140abedb5c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2029481-e8ad-40ef-97b7-81140abedb5c-istio-data\") pod \"a2029481-e8ad-40ef-97b7-81140abedb5c\" (UID: \"a2029481-e8ad-40ef-97b7-81140abedb5c\") " Oct 30 13:07:04.300489 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:04.300453 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2029481-e8ad-40ef-97b7-81140abedb5c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2029481-e8ad-40ef-97b7-81140abedb5c-workload-socket\") pod \"a2029481-e8ad-40ef-97b7-81140abedb5c\" (UID: \"a2029481-e8ad-40ef-97b7-81140abedb5c\") " Oct 30 13:07:04.300730 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:04.300502 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2029481-e8ad-40ef-97b7-81140abedb5c-istio-token\" (UniqueName: \"kubernetes.io/projected/a2029481-e8ad-40ef-97b7-81140abedb5c-istio-token\") pod \"a2029481-e8ad-40ef-97b7-81140abedb5c\" (UID: \"a2029481-e8ad-40ef-97b7-81140abedb5c\") " Oct 30 13:07:04.300730 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:04.300530 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2029481-e8ad-40ef-97b7-81140abedb5c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2029481-e8ad-40ef-97b7-81140abedb5c-credential-socket\") pod \"a2029481-e8ad-40ef-97b7-81140abedb5c\" (UID: \"a2029481-e8ad-40ef-97b7-81140abedb5c\") " Oct 30 13:07:04.300730 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:04.300586 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2029481-e8ad-40ef-97b7-81140abedb5c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2029481-e8ad-40ef-97b7-81140abedb5c-istiod-ca-cert\") pod \"a2029481-e8ad-40ef-97b7-81140abedb5c\" (UID: \"a2029481-e8ad-40ef-97b7-81140abedb5c\") " Oct 30 13:07:04.300730 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:04.300666 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2029481-e8ad-40ef-97b7-81140abedb5c-kube-api-access-f9fd2\" (UniqueName: \"kubernetes.io/projected/a2029481-e8ad-40ef-97b7-81140abedb5c-kube-api-access-f9fd2\") pod \"a2029481-e8ad-40ef-97b7-81140abedb5c\" (UID: \"a2029481-e8ad-40ef-97b7-81140abedb5c\") " Oct 30 13:07:04.303134 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:04.302035 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2029481-e8ad-40ef-97b7-81140abedb5c-workload-socket" pod "a2029481-e8ad-40ef-97b7-81140abedb5c" (UID: "a2029481-e8ad-40ef-97b7-81140abedb5c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:07:04.303134 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:04.302289 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2029481-e8ad-40ef-97b7-81140abedb5c-workload-certs" pod "a2029481-e8ad-40ef-97b7-81140abedb5c" (UID: "a2029481-e8ad-40ef-97b7-81140abedb5c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:07:04.303134 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:04.302450 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2029481-e8ad-40ef-97b7-81140abedb5c-istiod-ca-cert" pod "a2029481-e8ad-40ef-97b7-81140abedb5c" (UID: "a2029481-e8ad-40ef-97b7-81140abedb5c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:07:04.303134 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:04.303103 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2029481-e8ad-40ef-97b7-81140abedb5c-credential-socket" pod "a2029481-e8ad-40ef-97b7-81140abedb5c" (UID: "a2029481-e8ad-40ef-97b7-81140abedb5c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:07:04.303841 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:04.303803 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2029481-e8ad-40ef-97b7-81140abedb5c-istio-data" pod "a2029481-e8ad-40ef-97b7-81140abedb5c" (UID: "a2029481-e8ad-40ef-97b7-81140abedb5c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:07:04.307626 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:04.307596 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2029481-e8ad-40ef-97b7-81140abedb5c-istio-envoy" pod "a2029481-e8ad-40ef-97b7-81140abedb5c" (UID: "a2029481-e8ad-40ef-97b7-81140abedb5c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:07:04.307746 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:04.307619 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a2029481-e8ad-40ef-97b7-81140abedb5c-istio-podinfo" pod "a2029481-e8ad-40ef-97b7-81140abedb5c" (UID: "a2029481-e8ad-40ef-97b7-81140abedb5c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:07:04.308047 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:04.308020 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2029481-e8ad-40ef-97b7-81140abedb5c-istio-token" pod "a2029481-e8ad-40ef-97b7-81140abedb5c" (UID: "a2029481-e8ad-40ef-97b7-81140abedb5c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:07:04.309633 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:04.309599 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2029481-e8ad-40ef-97b7-81140abedb5c-kube-api-access-f9fd2" pod "a2029481-e8ad-40ef-97b7-81140abedb5c" (UID: "a2029481-e8ad-40ef-97b7-81140abedb5c"). InnerVolumeSpecName "kube-api-access-f9fd2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:07:04.401479 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:04.401440 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d070106-c79c-44d5-a7a9-b766ee82b439-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d070106-c79c-44d5-a7a9-b766ee82b439-workload-certs\") pod \"2d070106-c79c-44d5-a7a9-b766ee82b439\" (UID: \"2d070106-c79c-44d5-a7a9-b766ee82b439\") " Oct 30 13:07:04.401705 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:04.401562 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d070106-c79c-44d5-a7a9-b766ee82b439-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d070106-c79c-44d5-a7a9-b766ee82b439-istio-data\") pod \"2d070106-c79c-44d5-a7a9-b766ee82b439\" (UID: \"2d070106-c79c-44d5-a7a9-b766ee82b439\") " Oct 30 13:07:04.401705 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:04.401604 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d070106-c79c-44d5-a7a9-b766ee82b439-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d070106-c79c-44d5-a7a9-b766ee82b439-istiod-ca-cert\") pod \"2d070106-c79c-44d5-a7a9-b766ee82b439\" (UID: \"2d070106-c79c-44d5-a7a9-b766ee82b439\") " Oct 30 13:07:04.401705 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:04.401678 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2d070106-c79c-44d5-a7a9-b766ee82b439-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d070106-c79c-44d5-a7a9-b766ee82b439-istio-podinfo\") pod \"2d070106-c79c-44d5-a7a9-b766ee82b439\" (UID: \"2d070106-c79c-44d5-a7a9-b766ee82b439\") " Oct 30 13:07:04.401905 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:04.401720 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d070106-c79c-44d5-a7a9-b766ee82b439-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d070106-c79c-44d5-a7a9-b766ee82b439-workload-socket\") pod \"2d070106-c79c-44d5-a7a9-b766ee82b439\" (UID: \"2d070106-c79c-44d5-a7a9-b766ee82b439\") " Oct 30 13:07:04.401905 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:04.401750 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d070106-c79c-44d5-a7a9-b766ee82b439-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d070106-c79c-44d5-a7a9-b766ee82b439-istio-envoy\") pod \"2d070106-c79c-44d5-a7a9-b766ee82b439\" (UID: \"2d070106-c79c-44d5-a7a9-b766ee82b439\") " Oct 30 13:07:04.401905 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:04.401779 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d070106-c79c-44d5-a7a9-b766ee82b439-kube-api-access-vgchc\" (UniqueName: \"kubernetes.io/projected/2d070106-c79c-44d5-a7a9-b766ee82b439-kube-api-access-vgchc\") pod \"2d070106-c79c-44d5-a7a9-b766ee82b439\" (UID: \"2d070106-c79c-44d5-a7a9-b766ee82b439\") " Oct 30 13:07:04.401905 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:04.401857 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d070106-c79c-44d5-a7a9-b766ee82b439-istio-token\" (UniqueName: \"kubernetes.io/projected/2d070106-c79c-44d5-a7a9-b766ee82b439-istio-token\") pod \"2d070106-c79c-44d5-a7a9-b766ee82b439\" (UID: \"2d070106-c79c-44d5-a7a9-b766ee82b439\") " Oct 30 13:07:04.402123 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:04.401956 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d070106-c79c-44d5-a7a9-b766ee82b439-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d070106-c79c-44d5-a7a9-b766ee82b439-credential-socket\") pod \"2d070106-c79c-44d5-a7a9-b766ee82b439\" (UID: \"2d070106-c79c-44d5-a7a9-b766ee82b439\") " Oct 30 13:07:04.403247 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:04.402107 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d070106-c79c-44d5-a7a9-b766ee82b439-istiod-ca-cert" pod "2d070106-c79c-44d5-a7a9-b766ee82b439" (UID: "2d070106-c79c-44d5-a7a9-b766ee82b439"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:07:04.403247 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:04.402680 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d070106-c79c-44d5-a7a9-b766ee82b439-istio-data" pod "2d070106-c79c-44d5-a7a9-b766ee82b439" (UID: "2d070106-c79c-44d5-a7a9-b766ee82b439"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:07:04.403247 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:04.402782 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9fd2\" (UniqueName: \"kubernetes.io/projected/a2029481-e8ad-40ef-97b7-81140abedb5c-kube-api-access-f9fd2\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:07:04.403247 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:04.402808 4908 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2029481-e8ad-40ef-97b7-81140abedb5c-istio-podinfo\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:07:04.403247 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:04.402840 4908 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2029481-e8ad-40ef-97b7-81140abedb5c-workload-certs\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:07:04.403247 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:04.402863 4908 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2029481-e8ad-40ef-97b7-81140abedb5c-istio-envoy\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:07:04.403247 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:04.402876 4908 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2029481-e8ad-40ef-97b7-81140abedb5c-istio-data\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:07:04.403247 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:04.402890 4908 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2029481-e8ad-40ef-97b7-81140abedb5c-workload-socket\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:07:04.403247 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:04.402903 4908 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2029481-e8ad-40ef-97b7-81140abedb5c-istio-token\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:07:04.403247 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:04.402916 4908 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2029481-e8ad-40ef-97b7-81140abedb5c-credential-socket\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:07:04.403247 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:04.402928 4908 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2029481-e8ad-40ef-97b7-81140abedb5c-istiod-ca-cert\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:07:04.403247 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:04.402943 4908 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d070106-c79c-44d5-a7a9-b766ee82b439-istiod-ca-cert\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:07:04.403247 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:04.402980 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d070106-c79c-44d5-a7a9-b766ee82b439-workload-certs" pod "2d070106-c79c-44d5-a7a9-b766ee82b439" (UID: "2d070106-c79c-44d5-a7a9-b766ee82b439"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:07:04.403247 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:04.403160 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d070106-c79c-44d5-a7a9-b766ee82b439-credential-socket" pod "2d070106-c79c-44d5-a7a9-b766ee82b439" (UID: "2d070106-c79c-44d5-a7a9-b766ee82b439"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:07:04.403247 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:04.403232 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d070106-c79c-44d5-a7a9-b766ee82b439-workload-socket" pod "2d070106-c79c-44d5-a7a9-b766ee82b439" (UID: "2d070106-c79c-44d5-a7a9-b766ee82b439"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:07:04.404623 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:04.404595 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d070106-c79c-44d5-a7a9-b766ee82b439-istio-envoy" pod "2d070106-c79c-44d5-a7a9-b766ee82b439" (UID: "2d070106-c79c-44d5-a7a9-b766ee82b439"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:07:04.405332 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:04.405310 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d070106-c79c-44d5-a7a9-b766ee82b439-istio-token" pod "2d070106-c79c-44d5-a7a9-b766ee82b439" (UID: "2d070106-c79c-44d5-a7a9-b766ee82b439"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:07:04.405427 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:04.405380 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d070106-c79c-44d5-a7a9-b766ee82b439-kube-api-access-vgchc" pod "2d070106-c79c-44d5-a7a9-b766ee82b439" (UID: "2d070106-c79c-44d5-a7a9-b766ee82b439"). InnerVolumeSpecName "kube-api-access-vgchc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:07:04.405470 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:04.405435 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2d070106-c79c-44d5-a7a9-b766ee82b439-istio-podinfo" pod "2d070106-c79c-44d5-a7a9-b766ee82b439" (UID: "2d070106-c79c-44d5-a7a9-b766ee82b439"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:07:04.431315 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:04.431240 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-f920f7/simulator-w0-f920f7-kserve-599b7ff45f-g79d7"] Oct 30 13:07:04.504198 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:04.504155 4908 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d070106-c79c-44d5-a7a9-b766ee82b439-istio-token\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:07:04.504198 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:04.504200 4908 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d070106-c79c-44d5-a7a9-b766ee82b439-credential-socket\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:07:04.504480 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:04.504215 4908 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d070106-c79c-44d5-a7a9-b766ee82b439-workload-certs\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:07:04.504480 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:04.504228 4908 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d070106-c79c-44d5-a7a9-b766ee82b439-istio-data\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:07:04.504480 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:04.504239 4908 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d070106-c79c-44d5-a7a9-b766ee82b439-istio-podinfo\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:07:04.504480 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:04.504252 4908 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d070106-c79c-44d5-a7a9-b766ee82b439-workload-socket\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:07:04.504480 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:04.504265 4908 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d070106-c79c-44d5-a7a9-b766ee82b439-istio-envoy\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:07:04.504480 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:04.504280 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vgchc\" (UniqueName: \"kubernetes.io/projected/2d070106-c79c-44d5-a7a9-b766ee82b439-kube-api-access-vgchc\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:07:04.703051 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:04.702939 4908 generic.go:356] "Generic (PLEG): container finished" podID="2d070106-c79c-44d5-a7a9-b766ee82b439" containerID="5c86db12fa0656d65f0487fd1defd3ebacc6d436d4eab07243d282cfcbb509b4" exitCode=0 Oct 30 13:07:04.703051 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:04.702994 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-dd0628-openshift-default-54686d76fc-x98kl" Oct 30 13:07:04.703051 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:04.703022 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-dd0628-openshift-default-54686d76fc-x98kl" event={"ID":"2d070106-c79c-44d5-a7a9-b766ee82b439","Type":"ContainerDied","Data":"5c86db12fa0656d65f0487fd1defd3ebacc6d436d4eab07243d282cfcbb509b4"} Oct 30 13:07:04.703353 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:04.703089 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-dd0628-openshift-default-54686d76fc-x98kl" event={"ID":"2d070106-c79c-44d5-a7a9-b766ee82b439","Type":"ContainerDied","Data":"090c66c5bb70b3ba6a07eb6de03272e4a5d079128aa493a23c85db202a843c00"} Oct 30 13:07:04.703353 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:04.703127 4908 scope.go:122] "RemoveContainer" containerID="5c86db12fa0656d65f0487fd1defd3ebacc6d436d4eab07243d282cfcbb509b4" Oct 30 13:07:04.708899 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:04.705364 4908 generic.go:356] "Generic (PLEG): container finished" podID="a2029481-e8ad-40ef-97b7-81140abedb5c" containerID="8c2d59e692b8a4c74147a8d0aae249a6b34db995d531a52e86abb25c5ce863e1" exitCode=0 Oct 30 13:07:04.708899 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:04.705419 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9c3f12-openshift-default-78b944cc9c-fdpqh" event={"ID":"a2029481-e8ad-40ef-97b7-81140abedb5c","Type":"ContainerDied","Data":"8c2d59e692b8a4c74147a8d0aae249a6b34db995d531a52e86abb25c5ce863e1"} Oct 30 13:07:04.708899 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:04.705484 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9c3f12-openshift-default-78b944cc9c-fdpqh" event={"ID":"a2029481-e8ad-40ef-97b7-81140abedb5c","Type":"ContainerDied","Data":"61b3ec88d47aefd6dfbc2beded8887a5c742721ec7696e56beee771ddd47e2f8"} Oct 30 13:07:04.708899 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:04.705431 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9c3f12-openshift-default-78b944cc9c-fdpqh" Oct 30 13:07:04.715781 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:04.715728 4908 scope.go:122] "RemoveContainer" containerID="43eb1ea14c35db3a88694c94fd544c2b1516431a0afe2c55170dceb70cbbd3ba" Oct 30 13:07:04.726801 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:04.726777 4908 scope.go:122] "RemoveContainer" containerID="5c86db12fa0656d65f0487fd1defd3ebacc6d436d4eab07243d282cfcbb509b4" Oct 30 13:07:04.727135 ip-10-0-141-251 kubenswrapper[4908]: E1030 13:07:04.727110 4908 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c86db12fa0656d65f0487fd1defd3ebacc6d436d4eab07243d282cfcbb509b4\": container with ID starting with 5c86db12fa0656d65f0487fd1defd3ebacc6d436d4eab07243d282cfcbb509b4 not found: ID does not exist" containerID="5c86db12fa0656d65f0487fd1defd3ebacc6d436d4eab07243d282cfcbb509b4" Oct 30 13:07:04.727199 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:04.727147 4908 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c86db12fa0656d65f0487fd1defd3ebacc6d436d4eab07243d282cfcbb509b4"} err="failed to get container status \"5c86db12fa0656d65f0487fd1defd3ebacc6d436d4eab07243d282cfcbb509b4\": rpc error: code = NotFound desc = could not find container \"5c86db12fa0656d65f0487fd1defd3ebacc6d436d4eab07243d282cfcbb509b4\": container with ID starting with 5c86db12fa0656d65f0487fd1defd3ebacc6d436d4eab07243d282cfcbb509b4 not found: ID does not exist" Oct 30 13:07:04.727199 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:04.727167 4908 scope.go:122] "RemoveContainer" containerID="43eb1ea14c35db3a88694c94fd544c2b1516431a0afe2c55170dceb70cbbd3ba" Oct 30 13:07:04.727418 ip-10-0-141-251 kubenswrapper[4908]: E1030 13:07:04.727402 4908 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43eb1ea14c35db3a88694c94fd544c2b1516431a0afe2c55170dceb70cbbd3ba\": container with ID starting with 43eb1ea14c35db3a88694c94fd544c2b1516431a0afe2c55170dceb70cbbd3ba not found: ID does not exist" containerID="43eb1ea14c35db3a88694c94fd544c2b1516431a0afe2c55170dceb70cbbd3ba" Oct 30 13:07:04.727462 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:04.727426 4908 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43eb1ea14c35db3a88694c94fd544c2b1516431a0afe2c55170dceb70cbbd3ba"} err="failed to get container status \"43eb1ea14c35db3a88694c94fd544c2b1516431a0afe2c55170dceb70cbbd3ba\": rpc error: code = NotFound desc = could not find container \"43eb1ea14c35db3a88694c94fd544c2b1516431a0afe2c55170dceb70cbbd3ba\": container with ID starting with 43eb1ea14c35db3a88694c94fd544c2b1516431a0afe2c55170dceb70cbbd3ba not found: ID does not exist" Oct 30 13:07:04.727462 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:04.727442 4908 scope.go:122] "RemoveContainer" containerID="8c2d59e692b8a4c74147a8d0aae249a6b34db995d531a52e86abb25c5ce863e1" Oct 30 13:07:04.738431 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:04.738405 4908 scope.go:122] "RemoveContainer" containerID="77cff5c69f3faa3be287996ceaa831fc7ebd6797f0efb89476d52d6b56ce2413" Oct 30 13:07:04.750005 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:04.749983 4908 scope.go:122] "RemoveContainer" containerID="8c2d59e692b8a4c74147a8d0aae249a6b34db995d531a52e86abb25c5ce863e1" Oct 30 13:07:04.750360 ip-10-0-141-251 kubenswrapper[4908]: E1030 13:07:04.750338 4908 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c2d59e692b8a4c74147a8d0aae249a6b34db995d531a52e86abb25c5ce863e1\": container with ID starting with 8c2d59e692b8a4c74147a8d0aae249a6b34db995d531a52e86abb25c5ce863e1 not found: ID does not exist" containerID="8c2d59e692b8a4c74147a8d0aae249a6b34db995d531a52e86abb25c5ce863e1" Oct 30 13:07:04.750453 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:04.750373 4908 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c2d59e692b8a4c74147a8d0aae249a6b34db995d531a52e86abb25c5ce863e1"} err="failed to get container status \"8c2d59e692b8a4c74147a8d0aae249a6b34db995d531a52e86abb25c5ce863e1\": rpc error: code = NotFound desc = could not find container \"8c2d59e692b8a4c74147a8d0aae249a6b34db995d531a52e86abb25c5ce863e1\": container with ID starting with 8c2d59e692b8a4c74147a8d0aae249a6b34db995d531a52e86abb25c5ce863e1 not found: ID does not exist" Oct 30 13:07:04.750453 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:04.750401 4908 scope.go:122] "RemoveContainer" containerID="77cff5c69f3faa3be287996ceaa831fc7ebd6797f0efb89476d52d6b56ce2413" Oct 30 13:07:04.750712 ip-10-0-141-251 kubenswrapper[4908]: E1030 13:07:04.750692 4908 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77cff5c69f3faa3be287996ceaa831fc7ebd6797f0efb89476d52d6b56ce2413\": container with ID starting with 77cff5c69f3faa3be287996ceaa831fc7ebd6797f0efb89476d52d6b56ce2413 not found: ID does not exist" containerID="77cff5c69f3faa3be287996ceaa831fc7ebd6797f0efb89476d52d6b56ce2413" Oct 30 13:07:04.750757 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:04.750717 4908 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77cff5c69f3faa3be287996ceaa831fc7ebd6797f0efb89476d52d6b56ce2413"} err="failed to get container status \"77cff5c69f3faa3be287996ceaa831fc7ebd6797f0efb89476d52d6b56ce2413\": rpc error: code = NotFound desc = could not find container \"77cff5c69f3faa3be287996ceaa831fc7ebd6797f0efb89476d52d6b56ce2413\": container with ID starting with 77cff5c69f3faa3be287996ceaa831fc7ebd6797f0efb89476d52d6b56ce2413 not found: ID does not exist" Oct 30 13:07:04.759955 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:04.759918 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-dd0628-openshift-default-54686d76fc-x98kl"] Oct 30 13:07:04.773312 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:04.773271 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-dd0628-openshift-default-54686d76fc-x98kl"] Oct 30 13:07:04.802752 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:04.802692 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-9c3f12-openshift-default-78b944cc9c-fdpqh"] Oct 30 13:07:04.806821 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:04.806794 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-9c3f12-openshift-default-78b944cc9c-fdpqh"] Oct 30 13:07:05.607370 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:05.607329 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d070106-c79c-44d5-a7a9-b766ee82b439" path="/var/lib/kubelet/pods/2d070106-c79c-44d5-a7a9-b766ee82b439/volumes" Oct 30 13:07:05.607912 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:05.607871 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2029481-e8ad-40ef-97b7-81140abedb5c" path="/var/lib/kubelet/pods/a2029481-e8ad-40ef-97b7-81140abedb5c/volumes" Oct 30 13:07:06.359130 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:06.359091 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-12fd33-tn7wc"] Oct 30 13:07:06.374218 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:06.374177 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-12fd33-tn7wc"] Oct 30 13:07:07.605743 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:07.605708 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="978bb9d4-019d-4cf5-90de-d90a6abc4c84" path="/var/lib/kubelet/pods/978bb9d4-019d-4cf5-90de-d90a6abc4c84/volumes" Oct 30 13:07:08.940306 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:08.939415 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-0f7d14-openshift-default-cf79d4dcc-mcrqq"] Oct 30 13:07:08.940306 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:08.939777 4908 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-0f7d14-openshift-default-cf79d4dcc-mcrqq" podUID="3590e8ff-89a3-482b-b022-064bb84fe09c" containerName="istio-proxy" containerID="cri-o://7a32c67c0a333bd003cd5a52c0922d86d06dab5cab941208acab15e54b9e2919" gracePeriod=30 Oct 30 13:07:09.066860 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:09.066046 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf8e7840-pmr87"] Oct 30 13:07:09.088706 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:09.085116 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf8e7840-pmr87"] Oct 30 13:07:09.517761 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:09.517725 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-812990-cg7zp"] Oct 30 13:07:09.518251 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:09.518233 4908 cpu_manager.go:414] "containerMap: removing container" podUID="a2029481-e8ad-40ef-97b7-81140abedb5c" containerName="istio-proxy" Oct 30 13:07:09.518315 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:09.518253 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2029481-e8ad-40ef-97b7-81140abedb5c" containerName="istio-proxy" Oct 30 13:07:09.518315 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:09.518268 4908 cpu_manager.go:414] "containerMap: removing container" podUID="39a5d567-2188-4af8-9171-9879694ecc67" containerName="revoke-keys" Oct 30 13:07:09.518315 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:09.518274 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39a5d567-2188-4af8-9171-9879694ecc67" containerName="revoke-keys" Oct 30 13:07:09.518315 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:09.518291 4908 cpu_manager.go:414] "containerMap: removing container" podUID="1d4f38b5-6492-406c-80e1-cbd779f9fd73" containerName="revoke-keys" Oct 30 13:07:09.518315 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:09.518297 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d4f38b5-6492-406c-80e1-cbd779f9fd73" containerName="revoke-keys" Oct 30 13:07:09.518490 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:09.518315 4908 cpu_manager.go:414] "containerMap: removing container" podUID="2d070106-c79c-44d5-a7a9-b766ee82b439" containerName="istio-proxy" Oct 30 13:07:09.518490 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:09.518322 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d070106-c79c-44d5-a7a9-b766ee82b439" containerName="istio-proxy" Oct 30 13:07:09.518490 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:09.518333 4908 cpu_manager.go:414] "containerMap: removing container" podUID="2d070106-c79c-44d5-a7a9-b766ee82b439" containerName="istio-proxy" Oct 30 13:07:09.518490 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:09.518338 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d070106-c79c-44d5-a7a9-b766ee82b439" containerName="istio-proxy" Oct 30 13:07:09.518490 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:09.518406 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d070106-c79c-44d5-a7a9-b766ee82b439" containerName="istio-proxy" Oct 30 13:07:09.518490 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:09.518414 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2029481-e8ad-40ef-97b7-81140abedb5c" containerName="istio-proxy" Oct 30 13:07:09.518490 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:09.518421 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d070106-c79c-44d5-a7a9-b766ee82b439" containerName="istio-proxy" Oct 30 13:07:09.518490 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:09.518430 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2029481-e8ad-40ef-97b7-81140abedb5c" containerName="istio-proxy" Oct 30 13:07:09.518490 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:09.518437 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="39a5d567-2188-4af8-9171-9879694ecc67" containerName="revoke-keys" Oct 30 13:07:09.518490 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:09.518444 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d4f38b5-6492-406c-80e1-cbd779f9fd73" containerName="revoke-keys" Oct 30 13:07:09.523963 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:09.523941 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-812990-cg7zp" Oct 30 13:07:09.544386 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:09.544349 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-812990-cg7zp"] Oct 30 13:07:09.553788 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:09.553754 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:07:09.553788 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:09.553768 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 13:07:09.553788 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:09.553790 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 13:07:09.554081 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:09.553931 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s75hq\"" Oct 30 13:07:09.567894 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:09.567813 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/443b4b1e-1ff4-4c9a-8c4b-ac836c87afb6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-812990-cg7zp\" (UID: \"443b4b1e-1ff4-4c9a-8c4b-ac836c87afb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-812990-cg7zp" Oct 30 13:07:09.568096 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:09.567981 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z746h\" (UniqueName: \"kubernetes.io/projected/443b4b1e-1ff4-4c9a-8c4b-ac836c87afb6-kube-api-access-z746h\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-812990-cg7zp\" (UID: \"443b4b1e-1ff4-4c9a-8c4b-ac836c87afb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-812990-cg7zp" Oct 30 13:07:09.568096 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:09.568053 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/443b4b1e-1ff4-4c9a-8c4b-ac836c87afb6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-812990-cg7zp\" (UID: \"443b4b1e-1ff4-4c9a-8c4b-ac836c87afb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-812990-cg7zp" Oct 30 13:07:09.606807 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:09.606772 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4060653d-76ee-4df6-a077-a6c7bb2a6a5d" path="/var/lib/kubelet/pods/4060653d-76ee-4df6-a077-a6c7bb2a6a5d/volumes" Oct 30 13:07:09.668880 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:09.668751 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/443b4b1e-1ff4-4c9a-8c4b-ac836c87afb6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-812990-cg7zp\" (UID: \"443b4b1e-1ff4-4c9a-8c4b-ac836c87afb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-812990-cg7zp" Oct 30 13:07:09.669082 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:09.668909 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/443b4b1e-1ff4-4c9a-8c4b-ac836c87afb6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-812990-cg7zp\" (UID: \"443b4b1e-1ff4-4c9a-8c4b-ac836c87afb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-812990-cg7zp" Oct 30 13:07:09.669082 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:09.668947 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z746h\" (UniqueName: \"kubernetes.io/projected/443b4b1e-1ff4-4c9a-8c4b-ac836c87afb6-kube-api-access-z746h\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-812990-cg7zp\" (UID: \"443b4b1e-1ff4-4c9a-8c4b-ac836c87afb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-812990-cg7zp" Oct 30 13:07:09.669710 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:09.669676 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/443b4b1e-1ff4-4c9a-8c4b-ac836c87afb6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-812990-cg7zp\" (UID: \"443b4b1e-1ff4-4c9a-8c4b-ac836c87afb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-812990-cg7zp" Oct 30 13:07:09.671471 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:09.671448 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/443b4b1e-1ff4-4c9a-8c4b-ac836c87afb6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-812990-cg7zp\" (UID: \"443b4b1e-1ff4-4c9a-8c4b-ac836c87afb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-812990-cg7zp" Oct 30 13:07:09.698973 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:09.698934 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z746h\" (UniqueName: \"kubernetes.io/projected/443b4b1e-1ff4-4c9a-8c4b-ac836c87afb6-kube-api-access-z746h\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-812990-cg7zp\" (UID: \"443b4b1e-1ff4-4c9a-8c4b-ac836c87afb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-812990-cg7zp" Oct 30 13:07:09.837871 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:09.837816 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-812990-cg7zp" Oct 30 13:07:10.151023 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:10.150997 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-812990-cg7zp"] Oct 30 13:07:10.153236 ip-10-0-141-251 kubenswrapper[4908]: W1030 13:07:10.153195 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod443b4b1e_1ff4_4c9a_8c4b_ac836c87afb6.slice/crio-b9b9d8a6270ed206ef1ecea3dbe5818a62de4603b000210f1e73a00f1b0b223a WatchSource:0}: Error finding container b9b9d8a6270ed206ef1ecea3dbe5818a62de4603b000210f1e73a00f1b0b223a: Status 404 returned error can't find the container with id b9b9d8a6270ed206ef1ecea3dbe5818a62de4603b000210f1e73a00f1b0b223a Oct 30 13:07:10.766213 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:10.766113 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-812990-cg7zp" event={"ID":"443b4b1e-1ff4-4c9a-8c4b-ac836c87afb6","Type":"ContainerStarted","Data":"1c49d35a21996f1f9628e8381572994e1a378ddc1c7aa7bf48c1260d924e1534"} Oct 30 13:07:10.766213 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:10.766174 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-812990-cg7zp" event={"ID":"443b4b1e-1ff4-4c9a-8c4b-ac836c87afb6","Type":"ContainerStarted","Data":"b9b9d8a6270ed206ef1ecea3dbe5818a62de4603b000210f1e73a00f1b0b223a"} Oct 30 13:07:11.776907 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:11.776866 4908 generic.go:356] "Generic (PLEG): container finished" podID="443b4b1e-1ff4-4c9a-8c4b-ac836c87afb6" containerID="1c49d35a21996f1f9628e8381572994e1a378ddc1c7aa7bf48c1260d924e1534" exitCode=0 Oct 30 13:07:11.777308 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:11.776954 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-812990-cg7zp" event={"ID":"443b4b1e-1ff4-4c9a-8c4b-ac836c87afb6","Type":"ContainerDied","Data":"1c49d35a21996f1f9628e8381572994e1a378ddc1c7aa7bf48c1260d924e1534"} Oct 30 13:07:12.941706 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:12.941667 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-812990-cg7zp" Oct 30 13:07:13.012491 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:13.012449 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/443b4b1e-1ff4-4c9a-8c4b-ac836c87afb6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/443b4b1e-1ff4-4c9a-8c4b-ac836c87afb6-maas-api-openshift-ca\") pod \"443b4b1e-1ff4-4c9a-8c4b-ac836c87afb6\" (UID: \"443b4b1e-1ff4-4c9a-8c4b-ac836c87afb6\") " Oct 30 13:07:13.012704 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:13.012550 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/443b4b1e-1ff4-4c9a-8c4b-ac836c87afb6-kube-api-access-z746h\" (UniqueName: \"kubernetes.io/projected/443b4b1e-1ff4-4c9a-8c4b-ac836c87afb6-kube-api-access-z746h\") pod \"443b4b1e-1ff4-4c9a-8c4b-ac836c87afb6\" (UID: \"443b4b1e-1ff4-4c9a-8c4b-ac836c87afb6\") " Oct 30 13:07:13.012704 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:13.012593 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/443b4b1e-1ff4-4c9a-8c4b-ac836c87afb6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/443b4b1e-1ff4-4c9a-8c4b-ac836c87afb6-maas-api-xks-ca\") pod \"443b4b1e-1ff4-4c9a-8c4b-ac836c87afb6\" (UID: \"443b4b1e-1ff4-4c9a-8c4b-ac836c87afb6\") " Oct 30 13:07:13.012927 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:13.012898 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/443b4b1e-1ff4-4c9a-8c4b-ac836c87afb6-maas-api-openshift-ca" pod "443b4b1e-1ff4-4c9a-8c4b-ac836c87afb6" (UID: "443b4b1e-1ff4-4c9a-8c4b-ac836c87afb6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:07:13.013211 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:13.013182 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/443b4b1e-1ff4-4c9a-8c4b-ac836c87afb6-maas-api-openshift-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:07:13.015195 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:13.015165 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/443b4b1e-1ff4-4c9a-8c4b-ac836c87afb6-maas-api-xks-ca" pod "443b4b1e-1ff4-4c9a-8c4b-ac836c87afb6" (UID: "443b4b1e-1ff4-4c9a-8c4b-ac836c87afb6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:07:13.015535 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:13.015512 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/443b4b1e-1ff4-4c9a-8c4b-ac836c87afb6-kube-api-access-z746h" pod "443b4b1e-1ff4-4c9a-8c4b-ac836c87afb6" (UID: "443b4b1e-1ff4-4c9a-8c4b-ac836c87afb6"). InnerVolumeSpecName "kube-api-access-z746h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:07:13.115082 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:13.114996 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z746h\" (UniqueName: \"kubernetes.io/projected/443b4b1e-1ff4-4c9a-8c4b-ac836c87afb6-kube-api-access-z746h\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:07:13.115082 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:13.115037 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/443b4b1e-1ff4-4c9a-8c4b-ac836c87afb6-maas-api-xks-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:07:13.793376 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:13.793337 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-812990-cg7zp" Oct 30 13:07:13.793376 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:13.793369 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-812990-cg7zp" event={"ID":"443b4b1e-1ff4-4c9a-8c4b-ac836c87afb6","Type":"ContainerDied","Data":"b9b9d8a6270ed206ef1ecea3dbe5818a62de4603b000210f1e73a00f1b0b223a"} Oct 30 13:07:13.793593 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:13.793421 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b9b9d8a6270ed206ef1ecea3dbe5818a62de4603b000210f1e73a00f1b0b223a" Oct 30 13:07:14.059731 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:14.059699 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-12fd33-openshift-default-846b7dbb9-pt6kj"] Oct 30 13:07:14.060484 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:14.060431 4908 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-12fd33-openshift-default-846b7dbb9-pt6kj" podUID="396046e4-b596-4fed-943d-a5f42fea4f77" containerName="istio-proxy" containerID="cri-o://f3fe57816ba5c65b5b600a2b3b6c2845d5ecb71b822de4f93e925e2573fc40f6" gracePeriod=30 Oct 30 13:07:14.240510 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:14.240482 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-0f7d14-openshift-default-cf79d4dcc-mcrqq" Oct 30 13:07:14.331689 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:14.331652 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3590e8ff-89a3-482b-b022-064bb84fe09c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3590e8ff-89a3-482b-b022-064bb84fe09c-workload-certs\") pod \"3590e8ff-89a3-482b-b022-064bb84fe09c\" (UID: \"3590e8ff-89a3-482b-b022-064bb84fe09c\") " Oct 30 13:07:14.331905 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:14.331741 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3590e8ff-89a3-482b-b022-064bb84fe09c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3590e8ff-89a3-482b-b022-064bb84fe09c-workload-socket\") pod \"3590e8ff-89a3-482b-b022-064bb84fe09c\" (UID: \"3590e8ff-89a3-482b-b022-064bb84fe09c\") " Oct 30 13:07:14.331905 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:14.331801 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3590e8ff-89a3-482b-b022-064bb84fe09c-kube-api-access-6xd8k\" (UniqueName: \"kubernetes.io/projected/3590e8ff-89a3-482b-b022-064bb84fe09c-kube-api-access-6xd8k\") pod \"3590e8ff-89a3-482b-b022-064bb84fe09c\" (UID: \"3590e8ff-89a3-482b-b022-064bb84fe09c\") " Oct 30 13:07:14.331905 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:14.331878 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3590e8ff-89a3-482b-b022-064bb84fe09c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3590e8ff-89a3-482b-b022-064bb84fe09c-credential-socket\") pod \"3590e8ff-89a3-482b-b022-064bb84fe09c\" (UID: \"3590e8ff-89a3-482b-b022-064bb84fe09c\") " Oct 30 13:07:14.332089 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:14.331907 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3590e8ff-89a3-482b-b022-064bb84fe09c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3590e8ff-89a3-482b-b022-064bb84fe09c-istiod-ca-cert\") pod \"3590e8ff-89a3-482b-b022-064bb84fe09c\" (UID: \"3590e8ff-89a3-482b-b022-064bb84fe09c\") " Oct 30 13:07:14.332089 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:14.331970 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3590e8ff-89a3-482b-b022-064bb84fe09c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3590e8ff-89a3-482b-b022-064bb84fe09c-istio-envoy\") pod \"3590e8ff-89a3-482b-b022-064bb84fe09c\" (UID: \"3590e8ff-89a3-482b-b022-064bb84fe09c\") " Oct 30 13:07:14.332089 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:14.332041 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3590e8ff-89a3-482b-b022-064bb84fe09c-workload-certs" pod "3590e8ff-89a3-482b-b022-064bb84fe09c" (UID: "3590e8ff-89a3-482b-b022-064bb84fe09c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:07:14.332251 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:14.332107 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3590e8ff-89a3-482b-b022-064bb84fe09c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3590e8ff-89a3-482b-b022-064bb84fe09c-istio-data\") pod \"3590e8ff-89a3-482b-b022-064bb84fe09c\" (UID: \"3590e8ff-89a3-482b-b022-064bb84fe09c\") " Oct 30 13:07:14.332251 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:14.332107 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3590e8ff-89a3-482b-b022-064bb84fe09c-workload-socket" pod "3590e8ff-89a3-482b-b022-064bb84fe09c" (UID: "3590e8ff-89a3-482b-b022-064bb84fe09c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:07:14.332251 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:14.332167 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3590e8ff-89a3-482b-b022-064bb84fe09c-credential-socket" pod "3590e8ff-89a3-482b-b022-064bb84fe09c" (UID: "3590e8ff-89a3-482b-b022-064bb84fe09c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:07:14.332251 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:14.332187 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3590e8ff-89a3-482b-b022-064bb84fe09c-istio-token\" (UniqueName: \"kubernetes.io/projected/3590e8ff-89a3-482b-b022-064bb84fe09c-istio-token\") pod \"3590e8ff-89a3-482b-b022-064bb84fe09c\" (UID: \"3590e8ff-89a3-482b-b022-064bb84fe09c\") " Oct 30 13:07:14.332251 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:14.332229 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3590e8ff-89a3-482b-b022-064bb84fe09c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3590e8ff-89a3-482b-b022-064bb84fe09c-istio-podinfo\") pod \"3590e8ff-89a3-482b-b022-064bb84fe09c\" (UID: \"3590e8ff-89a3-482b-b022-064bb84fe09c\") " Oct 30 13:07:14.332533 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:14.332326 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3590e8ff-89a3-482b-b022-064bb84fe09c-istio-data" pod "3590e8ff-89a3-482b-b022-064bb84fe09c" (UID: "3590e8ff-89a3-482b-b022-064bb84fe09c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:07:14.332661 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:14.332633 4908 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3590e8ff-89a3-482b-b022-064bb84fe09c-credential-socket\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:07:14.332799 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:14.332668 4908 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3590e8ff-89a3-482b-b022-064bb84fe09c-istio-data\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:07:14.332799 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:14.332682 4908 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3590e8ff-89a3-482b-b022-064bb84fe09c-workload-certs\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:07:14.332799 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:14.332696 4908 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3590e8ff-89a3-482b-b022-064bb84fe09c-workload-socket\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:07:14.332799 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:14.332643 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3590e8ff-89a3-482b-b022-064bb84fe09c-istiod-ca-cert" pod "3590e8ff-89a3-482b-b022-064bb84fe09c" (UID: "3590e8ff-89a3-482b-b022-064bb84fe09c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:07:14.334419 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:14.334395 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3590e8ff-89a3-482b-b022-064bb84fe09c-istio-token" pod "3590e8ff-89a3-482b-b022-064bb84fe09c" (UID: "3590e8ff-89a3-482b-b022-064bb84fe09c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:07:14.334569 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:14.334544 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3590e8ff-89a3-482b-b022-064bb84fe09c-istio-envoy" pod "3590e8ff-89a3-482b-b022-064bb84fe09c" (UID: "3590e8ff-89a3-482b-b022-064bb84fe09c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:07:14.334675 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:14.334658 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3590e8ff-89a3-482b-b022-064bb84fe09c-kube-api-access-6xd8k" pod "3590e8ff-89a3-482b-b022-064bb84fe09c" (UID: "3590e8ff-89a3-482b-b022-064bb84fe09c"). InnerVolumeSpecName "kube-api-access-6xd8k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:07:14.334785 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:14.334764 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3590e8ff-89a3-482b-b022-064bb84fe09c-istio-podinfo" pod "3590e8ff-89a3-482b-b022-064bb84fe09c" (UID: "3590e8ff-89a3-482b-b022-064bb84fe09c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:07:14.434002 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:14.433966 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xd8k\" (UniqueName: \"kubernetes.io/projected/3590e8ff-89a3-482b-b022-064bb84fe09c-kube-api-access-6xd8k\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:07:14.434002 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:14.433999 4908 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3590e8ff-89a3-482b-b022-064bb84fe09c-istiod-ca-cert\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:07:14.434002 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:14.434009 4908 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3590e8ff-89a3-482b-b022-064bb84fe09c-istio-envoy\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:07:14.434264 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:14.434017 4908 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3590e8ff-89a3-482b-b022-064bb84fe09c-istio-token\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:07:14.434264 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:14.434026 4908 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3590e8ff-89a3-482b-b022-064bb84fe09c-istio-podinfo\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:07:14.655558 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:14.655514 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2675c0-558q9"] Oct 30 13:07:14.656089 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:14.656068 4908 cpu_manager.go:414] "containerMap: removing container" podUID="443b4b1e-1ff4-4c9a-8c4b-ac836c87afb6" containerName="revoke-keys" Oct 30 13:07:14.656218 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:14.656090 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="443b4b1e-1ff4-4c9a-8c4b-ac836c87afb6" containerName="revoke-keys" Oct 30 13:07:14.656218 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:14.656124 4908 cpu_manager.go:414] "containerMap: removing container" podUID="3590e8ff-89a3-482b-b022-064bb84fe09c" containerName="istio-proxy" Oct 30 13:07:14.656218 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:14.656133 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3590e8ff-89a3-482b-b022-064bb84fe09c" containerName="istio-proxy" Oct 30 13:07:14.656218 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:14.656149 4908 cpu_manager.go:414] "containerMap: removing container" podUID="a2029481-e8ad-40ef-97b7-81140abedb5c" containerName="istio-proxy" Oct 30 13:07:14.656218 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:14.656158 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2029481-e8ad-40ef-97b7-81140abedb5c" containerName="istio-proxy" Oct 30 13:07:14.656218 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:14.656180 4908 cpu_manager.go:414] "containerMap: removing container" podUID="3590e8ff-89a3-482b-b022-064bb84fe09c" containerName="istio-proxy" Oct 30 13:07:14.656218 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:14.656189 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3590e8ff-89a3-482b-b022-064bb84fe09c" containerName="istio-proxy" Oct 30 13:07:14.656602 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:14.656309 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="443b4b1e-1ff4-4c9a-8c4b-ac836c87afb6" containerName="revoke-keys" Oct 30 13:07:14.656602 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:14.656328 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="3590e8ff-89a3-482b-b022-064bb84fe09c" containerName="istio-proxy" Oct 30 13:07:14.660025 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:14.660004 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2675c0-558q9" Oct 30 13:07:14.676884 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:14.676851 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 13:07:14.678264 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:14.678219 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 13:07:14.681428 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:14.681404 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2675c0-558q9"] Oct 30 13:07:14.737453 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:14.737408 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8vsk\" (UniqueName: \"kubernetes.io/projected/be103a5c-91f5-4119-a4e3-8bc4b6705732-kube-api-access-m8vsk\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-2675c0-558q9\" (UID: \"be103a5c-91f5-4119-a4e3-8bc4b6705732\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2675c0-558q9" Oct 30 13:07:14.737666 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:14.737463 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be103a5c-91f5-4119-a4e3-8bc4b6705732-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-2675c0-558q9\" (UID: \"be103a5c-91f5-4119-a4e3-8bc4b6705732\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2675c0-558q9" Oct 30 13:07:14.737666 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:14.737581 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be103a5c-91f5-4119-a4e3-8bc4b6705732-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-2675c0-558q9\" (UID: \"be103a5c-91f5-4119-a4e3-8bc4b6705732\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2675c0-558q9" Oct 30 13:07:14.761617 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:14.761551 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:07:14.761935 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:14.761912 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s75hq\"" Oct 30 13:07:14.802573 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:14.802535 4908 generic.go:356] "Generic (PLEG): container finished" podID="3590e8ff-89a3-482b-b022-064bb84fe09c" containerID="7a32c67c0a333bd003cd5a52c0922d86d06dab5cab941208acab15e54b9e2919" exitCode=0 Oct 30 13:07:14.802767 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:14.802583 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-0f7d14-openshift-default-cf79d4dcc-mcrqq" Oct 30 13:07:14.802767 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:14.802619 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-0f7d14-openshift-default-cf79d4dcc-mcrqq" event={"ID":"3590e8ff-89a3-482b-b022-064bb84fe09c","Type":"ContainerDied","Data":"7a32c67c0a333bd003cd5a52c0922d86d06dab5cab941208acab15e54b9e2919"} Oct 30 13:07:14.802767 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:14.802679 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-0f7d14-openshift-default-cf79d4dcc-mcrqq" event={"ID":"3590e8ff-89a3-482b-b022-064bb84fe09c","Type":"ContainerDied","Data":"7049a1b39cdfedca988893c5706732cfbefa6743380d37bce0de7c0734546040"} Oct 30 13:07:14.802767 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:14.802706 4908 scope.go:122] "RemoveContainer" containerID="7a32c67c0a333bd003cd5a52c0922d86d06dab5cab941208acab15e54b9e2919" Oct 30 13:07:14.814134 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:14.814110 4908 scope.go:122] "RemoveContainer" containerID="e9795be9cfa47e1d277b621a66ccf1c4f0749ec3581d6f56ac16ca0e760718f7" Oct 30 13:07:14.825277 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:14.825255 4908 scope.go:122] "RemoveContainer" containerID="7a32c67c0a333bd003cd5a52c0922d86d06dab5cab941208acab15e54b9e2919" Oct 30 13:07:14.825615 ip-10-0-141-251 kubenswrapper[4908]: E1030 13:07:14.825593 4908 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a32c67c0a333bd003cd5a52c0922d86d06dab5cab941208acab15e54b9e2919\": container with ID starting with 7a32c67c0a333bd003cd5a52c0922d86d06dab5cab941208acab15e54b9e2919 not found: ID does not exist" containerID="7a32c67c0a333bd003cd5a52c0922d86d06dab5cab941208acab15e54b9e2919" Oct 30 13:07:14.825693 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:14.825628 4908 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a32c67c0a333bd003cd5a52c0922d86d06dab5cab941208acab15e54b9e2919"} err="failed to get container status \"7a32c67c0a333bd003cd5a52c0922d86d06dab5cab941208acab15e54b9e2919\": rpc error: code = NotFound desc = could not find container \"7a32c67c0a333bd003cd5a52c0922d86d06dab5cab941208acab15e54b9e2919\": container with ID starting with 7a32c67c0a333bd003cd5a52c0922d86d06dab5cab941208acab15e54b9e2919 not found: ID does not exist" Oct 30 13:07:14.825693 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:14.825652 4908 scope.go:122] "RemoveContainer" containerID="e9795be9cfa47e1d277b621a66ccf1c4f0749ec3581d6f56ac16ca0e760718f7" Oct 30 13:07:14.825933 ip-10-0-141-251 kubenswrapper[4908]: E1030 13:07:14.825911 4908 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9795be9cfa47e1d277b621a66ccf1c4f0749ec3581d6f56ac16ca0e760718f7\": container with ID starting with e9795be9cfa47e1d277b621a66ccf1c4f0749ec3581d6f56ac16ca0e760718f7 not found: ID does not exist" containerID="e9795be9cfa47e1d277b621a66ccf1c4f0749ec3581d6f56ac16ca0e760718f7" Oct 30 13:07:14.825991 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:14.825940 4908 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9795be9cfa47e1d277b621a66ccf1c4f0749ec3581d6f56ac16ca0e760718f7"} err="failed to get container status \"e9795be9cfa47e1d277b621a66ccf1c4f0749ec3581d6f56ac16ca0e760718f7\": rpc error: code = NotFound desc = could not find container \"e9795be9cfa47e1d277b621a66ccf1c4f0749ec3581d6f56ac16ca0e760718f7\": container with ID starting with e9795be9cfa47e1d277b621a66ccf1c4f0749ec3581d6f56ac16ca0e760718f7 not found: ID does not exist" Oct 30 13:07:14.839146 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:14.839116 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be103a5c-91f5-4119-a4e3-8bc4b6705732-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-2675c0-558q9\" (UID: \"be103a5c-91f5-4119-a4e3-8bc4b6705732\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2675c0-558q9" Oct 30 13:07:14.839289 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:14.839269 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8vsk\" (UniqueName: \"kubernetes.io/projected/be103a5c-91f5-4119-a4e3-8bc4b6705732-kube-api-access-m8vsk\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-2675c0-558q9\" (UID: \"be103a5c-91f5-4119-a4e3-8bc4b6705732\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2675c0-558q9" Oct 30 13:07:14.839335 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:14.839316 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be103a5c-91f5-4119-a4e3-8bc4b6705732-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-2675c0-558q9\" (UID: \"be103a5c-91f5-4119-a4e3-8bc4b6705732\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2675c0-558q9" Oct 30 13:07:14.840012 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:14.839978 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be103a5c-91f5-4119-a4e3-8bc4b6705732-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-2675c0-558q9\" (UID: \"be103a5c-91f5-4119-a4e3-8bc4b6705732\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2675c0-558q9" Oct 30 13:07:14.841600 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:14.841581 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be103a5c-91f5-4119-a4e3-8bc4b6705732-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-2675c0-558q9\" (UID: \"be103a5c-91f5-4119-a4e3-8bc4b6705732\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2675c0-558q9" Oct 30 13:07:14.859331 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:14.859291 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-0f7d14-openshift-default-cf79d4dcc-mcrqq"] Oct 30 13:07:14.866157 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:14.866110 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8vsk\" (UniqueName: \"kubernetes.io/projected/be103a5c-91f5-4119-a4e3-8bc4b6705732-kube-api-access-m8vsk\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-2675c0-558q9\" (UID: \"be103a5c-91f5-4119-a4e3-8bc4b6705732\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2675c0-558q9" Oct 30 13:07:14.871046 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:14.871013 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-0f7d14-openshift-default-cf79d4dcc-mcrqq"] Oct 30 13:07:14.974031 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:14.973944 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2675c0-558q9" Oct 30 13:07:15.335866 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:15.333414 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2675c0-558q9"] Oct 30 13:07:15.335866 ip-10-0-141-251 kubenswrapper[4908]: W1030 13:07:15.334618 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe103a5c_91f5_4119_a4e3_8bc4b6705732.slice/crio-15bd4e13d17a6250bb5c36eff06d7ba7cd36e32030c96be9e1db0c193e3abde7 WatchSource:0}: Error finding container 15bd4e13d17a6250bb5c36eff06d7ba7cd36e32030c96be9e1db0c193e3abde7: Status 404 returned error can't find the container with id 15bd4e13d17a6250bb5c36eff06d7ba7cd36e32030c96be9e1db0c193e3abde7 Oct 30 13:07:15.606762 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:15.606680 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3590e8ff-89a3-482b-b022-064bb84fe09c" path="/var/lib/kubelet/pods/3590e8ff-89a3-482b-b022-064bb84fe09c/volumes" Oct 30 13:07:15.813565 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:15.813520 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2675c0-558q9" event={"ID":"be103a5c-91f5-4119-a4e3-8bc4b6705732","Type":"ContainerStarted","Data":"20adf571753f8b6386498d6929183db30e988f6123bd8f3dcead27867839232c"} Oct 30 13:07:15.813743 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:15.813577 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2675c0-558q9" event={"ID":"be103a5c-91f5-4119-a4e3-8bc4b6705732","Type":"ContainerStarted","Data":"15bd4e13d17a6250bb5c36eff06d7ba7cd36e32030c96be9e1db0c193e3abde7"} Oct 30 13:07:15.857695 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:15.857578 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2675c0-558q9" podStartSLOduration=1.857561483 podStartE2EDuration="1.857561483s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:07:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:07:15.856331828 +0000 UTC m=+4392.370760573" watchObservedRunningTime="2026-10-30 13:07:15.857561483 +0000 UTC m=+4392.371990231" Oct 30 13:07:16.833556 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:16.833510 4908 generic.go:356] "Generic (PLEG): container finished" podID="be103a5c-91f5-4119-a4e3-8bc4b6705732" containerID="20adf571753f8b6386498d6929183db30e988f6123bd8f3dcead27867839232c" exitCode=0 Oct 30 13:07:16.834049 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:16.833596 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2675c0-558q9" event={"ID":"be103a5c-91f5-4119-a4e3-8bc4b6705732","Type":"ContainerDied","Data":"20adf571753f8b6386498d6929183db30e988f6123bd8f3dcead27867839232c"} Oct 30 13:07:18.003337 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:18.003307 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2675c0-558q9" Oct 30 13:07:18.077869 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:18.077820 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be103a5c-91f5-4119-a4e3-8bc4b6705732-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be103a5c-91f5-4119-a4e3-8bc4b6705732-maas-api-openshift-ca\") pod \"be103a5c-91f5-4119-a4e3-8bc4b6705732\" (UID: \"be103a5c-91f5-4119-a4e3-8bc4b6705732\") " Oct 30 13:07:18.078059 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:18.077970 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/be103a5c-91f5-4119-a4e3-8bc4b6705732-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be103a5c-91f5-4119-a4e3-8bc4b6705732-maas-api-xks-ca\") pod \"be103a5c-91f5-4119-a4e3-8bc4b6705732\" (UID: \"be103a5c-91f5-4119-a4e3-8bc4b6705732\") " Oct 30 13:07:18.078059 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:18.078008 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be103a5c-91f5-4119-a4e3-8bc4b6705732-kube-api-access-m8vsk\" (UniqueName: \"kubernetes.io/projected/be103a5c-91f5-4119-a4e3-8bc4b6705732-kube-api-access-m8vsk\") pod \"be103a5c-91f5-4119-a4e3-8bc4b6705732\" (UID: \"be103a5c-91f5-4119-a4e3-8bc4b6705732\") " Oct 30 13:07:18.078252 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:18.078225 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be103a5c-91f5-4119-a4e3-8bc4b6705732-maas-api-openshift-ca" pod "be103a5c-91f5-4119-a4e3-8bc4b6705732" (UID: "be103a5c-91f5-4119-a4e3-8bc4b6705732"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:07:18.078466 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:18.078448 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be103a5c-91f5-4119-a4e3-8bc4b6705732-maas-api-openshift-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:07:18.080187 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:18.080154 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be103a5c-91f5-4119-a4e3-8bc4b6705732-maas-api-xks-ca" pod "be103a5c-91f5-4119-a4e3-8bc4b6705732" (UID: "be103a5c-91f5-4119-a4e3-8bc4b6705732"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:07:18.080609 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:18.080574 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be103a5c-91f5-4119-a4e3-8bc4b6705732-kube-api-access-m8vsk" pod "be103a5c-91f5-4119-a4e3-8bc4b6705732" (UID: "be103a5c-91f5-4119-a4e3-8bc4b6705732"). InnerVolumeSpecName "kube-api-access-m8vsk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:07:18.179232 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:18.179138 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be103a5c-91f5-4119-a4e3-8bc4b6705732-maas-api-xks-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:07:18.179232 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:18.179170 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8vsk\" (UniqueName: \"kubernetes.io/projected/be103a5c-91f5-4119-a4e3-8bc4b6705732-kube-api-access-m8vsk\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:07:18.851132 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:18.851098 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2675c0-558q9" Oct 30 13:07:18.851346 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:18.851134 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2675c0-558q9" event={"ID":"be103a5c-91f5-4119-a4e3-8bc4b6705732","Type":"ContainerDied","Data":"15bd4e13d17a6250bb5c36eff06d7ba7cd36e32030c96be9e1db0c193e3abde7"} Oct 30 13:07:18.851346 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:18.851184 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="15bd4e13d17a6250bb5c36eff06d7ba7cd36e32030c96be9e1db0c193e3abde7" Oct 30 13:07:19.451911 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:19.451862 4908 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-f920f7/simulator-w0-f920f7-kserve-599b7ff45f-g79d7" podUID="a85e562f-d655-4c22-b4f4-a299a4e494a0" containerName="main" containerID="cri-o://d904798341eedab6a9fa053fc321f05682c35fccd03374ef779c718d44358680" gracePeriod=45 Oct 30 13:07:19.707130 ip-10-0-141-251 kubenswrapper[4908]: E1030 13:07:19.707006 4908 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vn6vv" containerName="extractor" Oct 30 13:07:19.762562 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:19.762528 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f920f7/simulator-w0-f920f7-kserve-599b7ff45f-g79d7" Oct 30 13:07:19.766701 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:19.766674 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-12fd33-openshift-default-846b7dbb9-pt6kj" Oct 30 13:07:19.861203 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:19.861169 4908 generic.go:356] "Generic (PLEG): container finished" podID="396046e4-b596-4fed-943d-a5f42fea4f77" containerID="f3fe57816ba5c65b5b600a2b3b6c2845d5ecb71b822de4f93e925e2573fc40f6" exitCode=0 Oct 30 13:07:19.861422 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:19.861228 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-12fd33-openshift-default-846b7dbb9-pt6kj" Oct 30 13:07:19.861422 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:19.861253 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-12fd33-openshift-default-846b7dbb9-pt6kj" event={"ID":"396046e4-b596-4fed-943d-a5f42fea4f77","Type":"ContainerDied","Data":"f3fe57816ba5c65b5b600a2b3b6c2845d5ecb71b822de4f93e925e2573fc40f6"} Oct 30 13:07:19.861422 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:19.861323 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-12fd33-openshift-default-846b7dbb9-pt6kj" event={"ID":"396046e4-b596-4fed-943d-a5f42fea4f77","Type":"ContainerDied","Data":"f4d4c26cc4fd1588e9f9eae07216cf8e550ba28bd4ceeb3a9fa53177cb789ec6"} Oct 30 13:07:19.861422 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:19.861351 4908 scope.go:122] "RemoveContainer" containerID="f3fe57816ba5c65b5b600a2b3b6c2845d5ecb71b822de4f93e925e2573fc40f6" Oct 30 13:07:19.863386 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:19.863363 4908 generic.go:356] "Generic (PLEG): container finished" podID="a85e562f-d655-4c22-b4f4-a299a4e494a0" containerID="d904798341eedab6a9fa053fc321f05682c35fccd03374ef779c718d44358680" exitCode=0 Oct 30 13:07:19.863737 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:19.863417 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f920f7/simulator-w0-f920f7-kserve-599b7ff45f-g79d7" Oct 30 13:07:19.863737 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:19.863435 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f920f7/simulator-w0-f920f7-kserve-599b7ff45f-g79d7" event={"ID":"a85e562f-d655-4c22-b4f4-a299a4e494a0","Type":"ContainerDied","Data":"d904798341eedab6a9fa053fc321f05682c35fccd03374ef779c718d44358680"} Oct 30 13:07:19.863737 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:19.863682 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f920f7/simulator-w0-f920f7-kserve-599b7ff45f-g79d7" event={"ID":"a85e562f-d655-4c22-b4f4-a299a4e494a0","Type":"ContainerDied","Data":"f89ae38dbad086be146586a7ae696320fd88fed35b6cc9b084f56de79e996028"} Oct 30 13:07:19.879396 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:19.879366 4908 scope.go:122] "RemoveContainer" containerID="5c019e2dea5de51003c7ab7c190f0cc27089c921e8f8515c1f3935a5ef128412" Oct 30 13:07:19.894190 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:19.894158 4908 scope.go:122] "RemoveContainer" containerID="f3fe57816ba5c65b5b600a2b3b6c2845d5ecb71b822de4f93e925e2573fc40f6" Oct 30 13:07:19.894586 ip-10-0-141-251 kubenswrapper[4908]: E1030 13:07:19.894561 4908 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3fe57816ba5c65b5b600a2b3b6c2845d5ecb71b822de4f93e925e2573fc40f6\": container with ID starting with f3fe57816ba5c65b5b600a2b3b6c2845d5ecb71b822de4f93e925e2573fc40f6 not found: ID does not exist" containerID="f3fe57816ba5c65b5b600a2b3b6c2845d5ecb71b822de4f93e925e2573fc40f6" Oct 30 13:07:19.894718 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:19.894596 4908 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3fe57816ba5c65b5b600a2b3b6c2845d5ecb71b822de4f93e925e2573fc40f6"} err="failed to get container status \"f3fe57816ba5c65b5b600a2b3b6c2845d5ecb71b822de4f93e925e2573fc40f6\": rpc error: code = NotFound desc = could not find container \"f3fe57816ba5c65b5b600a2b3b6c2845d5ecb71b822de4f93e925e2573fc40f6\": container with ID starting with f3fe57816ba5c65b5b600a2b3b6c2845d5ecb71b822de4f93e925e2573fc40f6 not found: ID does not exist" Oct 30 13:07:19.894718 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:19.894616 4908 scope.go:122] "RemoveContainer" containerID="5c019e2dea5de51003c7ab7c190f0cc27089c921e8f8515c1f3935a5ef128412" Oct 30 13:07:19.894902 ip-10-0-141-251 kubenswrapper[4908]: E1030 13:07:19.894879 4908 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c019e2dea5de51003c7ab7c190f0cc27089c921e8f8515c1f3935a5ef128412\": container with ID starting with 5c019e2dea5de51003c7ab7c190f0cc27089c921e8f8515c1f3935a5ef128412 not found: ID does not exist" containerID="5c019e2dea5de51003c7ab7c190f0cc27089c921e8f8515c1f3935a5ef128412" Oct 30 13:07:19.894992 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:19.894906 4908 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c019e2dea5de51003c7ab7c190f0cc27089c921e8f8515c1f3935a5ef128412"} err="failed to get container status \"5c019e2dea5de51003c7ab7c190f0cc27089c921e8f8515c1f3935a5ef128412\": rpc error: code = NotFound desc = could not find container \"5c019e2dea5de51003c7ab7c190f0cc27089c921e8f8515c1f3935a5ef128412\": container with ID starting with 5c019e2dea5de51003c7ab7c190f0cc27089c921e8f8515c1f3935a5ef128412 not found: ID does not exist" Oct 30 13:07:19.894992 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:19.894921 4908 scope.go:122] "RemoveContainer" containerID="d904798341eedab6a9fa053fc321f05682c35fccd03374ef779c718d44358680" Oct 30 13:07:19.900712 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:19.900680 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a85e562f-d655-4c22-b4f4-a299a4e494a0-home\" (UniqueName: \"kubernetes.io/empty-dir/a85e562f-d655-4c22-b4f4-a299a4e494a0-home\") pod \"a85e562f-d655-4c22-b4f4-a299a4e494a0\" (UID: \"a85e562f-d655-4c22-b4f4-a299a4e494a0\") " Oct 30 13:07:19.900883 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:19.900743 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a85e562f-d655-4c22-b4f4-a299a4e494a0-kube-api-access-cqbsz\" (UniqueName: \"kubernetes.io/projected/a85e562f-d655-4c22-b4f4-a299a4e494a0-kube-api-access-cqbsz\") pod \"a85e562f-d655-4c22-b4f4-a299a4e494a0\" (UID: \"a85e562f-d655-4c22-b4f4-a299a4e494a0\") " Oct 30 13:07:19.900883 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:19.900777 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/396046e4-b596-4fed-943d-a5f42fea4f77-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/396046e4-b596-4fed-943d-a5f42fea4f77-istio-podinfo\") pod \"396046e4-b596-4fed-943d-a5f42fea4f77\" (UID: \"396046e4-b596-4fed-943d-a5f42fea4f77\") " Oct 30 13:07:19.900883 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:19.900872 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/396046e4-b596-4fed-943d-a5f42fea4f77-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/396046e4-b596-4fed-943d-a5f42fea4f77-istiod-ca-cert\") pod \"396046e4-b596-4fed-943d-a5f42fea4f77\" (UID: \"396046e4-b596-4fed-943d-a5f42fea4f77\") " Oct 30 13:07:19.901087 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:19.900909 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/396046e4-b596-4fed-943d-a5f42fea4f77-kube-api-access-69z66\" (UniqueName: \"kubernetes.io/projected/396046e4-b596-4fed-943d-a5f42fea4f77-kube-api-access-69z66\") pod \"396046e4-b596-4fed-943d-a5f42fea4f77\" (UID: \"396046e4-b596-4fed-943d-a5f42fea4f77\") " Oct 30 13:07:19.901087 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:19.900973 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a85e562f-d655-4c22-b4f4-a299a4e494a0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a85e562f-d655-4c22-b4f4-a299a4e494a0-model-cache\") pod \"a85e562f-d655-4c22-b4f4-a299a4e494a0\" (UID: \"a85e562f-d655-4c22-b4f4-a299a4e494a0\") " Oct 30 13:07:19.901087 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:19.901000 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/396046e4-b596-4fed-943d-a5f42fea4f77-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/396046e4-b596-4fed-943d-a5f42fea4f77-istio-envoy\") pod \"396046e4-b596-4fed-943d-a5f42fea4f77\" (UID: \"396046e4-b596-4fed-943d-a5f42fea4f77\") " Oct 30 13:07:19.901087 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:19.901033 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a85e562f-d655-4c22-b4f4-a299a4e494a0-home" pod "a85e562f-d655-4c22-b4f4-a299a4e494a0" (UID: "a85e562f-d655-4c22-b4f4-a299a4e494a0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:07:19.901087 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:19.901066 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a85e562f-d655-4c22-b4f4-a299a4e494a0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a85e562f-d655-4c22-b4f4-a299a4e494a0-tmp-dir\") pod \"a85e562f-d655-4c22-b4f4-a299a4e494a0\" (UID: \"a85e562f-d655-4c22-b4f4-a299a4e494a0\") " Oct 30 13:07:19.901461 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:19.901381 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a85e562f-d655-4c22-b4f4-a299a4e494a0-tmp-dir" pod "a85e562f-d655-4c22-b4f4-a299a4e494a0" (UID: "a85e562f-d655-4c22-b4f4-a299a4e494a0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:07:19.901461 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:19.901404 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/396046e4-b596-4fed-943d-a5f42fea4f77-istiod-ca-cert" pod "396046e4-b596-4fed-943d-a5f42fea4f77" (UID: "396046e4-b596-4fed-943d-a5f42fea4f77"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:07:19.901583 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:19.901462 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a85e562f-d655-4c22-b4f4-a299a4e494a0-tls-certs\" (UniqueName: \"kubernetes.io/secret/a85e562f-d655-4c22-b4f4-a299a4e494a0-tls-certs\") pod \"a85e562f-d655-4c22-b4f4-a299a4e494a0\" (UID: \"a85e562f-d655-4c22-b4f4-a299a4e494a0\") " Oct 30 13:07:19.901583 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:19.901540 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/396046e4-b596-4fed-943d-a5f42fea4f77-istio-token\" (UniqueName: \"kubernetes.io/projected/396046e4-b596-4fed-943d-a5f42fea4f77-istio-token\") pod \"396046e4-b596-4fed-943d-a5f42fea4f77\" (UID: \"396046e4-b596-4fed-943d-a5f42fea4f77\") " Oct 30 13:07:19.901691 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:19.901625 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/396046e4-b596-4fed-943d-a5f42fea4f77-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/396046e4-b596-4fed-943d-a5f42fea4f77-workload-socket\") pod \"396046e4-b596-4fed-943d-a5f42fea4f77\" (UID: \"396046e4-b596-4fed-943d-a5f42fea4f77\") " Oct 30 13:07:19.901750 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:19.901688 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/396046e4-b596-4fed-943d-a5f42fea4f77-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/396046e4-b596-4fed-943d-a5f42fea4f77-workload-certs\") pod \"396046e4-b596-4fed-943d-a5f42fea4f77\" (UID: \"396046e4-b596-4fed-943d-a5f42fea4f77\") " Oct 30 13:07:19.901750 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:19.901741 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a85e562f-d655-4c22-b4f4-a299a4e494a0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a85e562f-d655-4c22-b4f4-a299a4e494a0-dshm\") pod \"a85e562f-d655-4c22-b4f4-a299a4e494a0\" (UID: \"a85e562f-d655-4c22-b4f4-a299a4e494a0\") " Oct 30 13:07:19.901896 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:19.901790 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/396046e4-b596-4fed-943d-a5f42fea4f77-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/396046e4-b596-4fed-943d-a5f42fea4f77-credential-socket\") pod \"396046e4-b596-4fed-943d-a5f42fea4f77\" (UID: \"396046e4-b596-4fed-943d-a5f42fea4f77\") " Oct 30 13:07:19.901896 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:19.901860 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/396046e4-b596-4fed-943d-a5f42fea4f77-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/396046e4-b596-4fed-943d-a5f42fea4f77-istio-data\") pod \"396046e4-b596-4fed-943d-a5f42fea4f77\" (UID: \"396046e4-b596-4fed-943d-a5f42fea4f77\") " Oct 30 13:07:19.902575 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:19.902297 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a85e562f-d655-4c22-b4f4-a299a4e494a0-model-cache" pod "a85e562f-d655-4c22-b4f4-a299a4e494a0" (UID: "a85e562f-d655-4c22-b4f4-a299a4e494a0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:07:19.909893 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:19.902886 4908 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/396046e4-b596-4fed-943d-a5f42fea4f77-istiod-ca-cert\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:07:19.909893 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:19.902917 4908 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a85e562f-d655-4c22-b4f4-a299a4e494a0-model-cache\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:07:19.909893 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:19.902930 4908 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a85e562f-d655-4c22-b4f4-a299a4e494a0-tmp-dir\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:07:19.909893 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:19.902955 4908 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a85e562f-d655-4c22-b4f4-a299a4e494a0-home\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:07:19.909893 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:19.903194 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/396046e4-b596-4fed-943d-a5f42fea4f77-workload-socket" pod "396046e4-b596-4fed-943d-a5f42fea4f77" (UID: "396046e4-b596-4fed-943d-a5f42fea4f77"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:07:19.909893 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:19.903387 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/396046e4-b596-4fed-943d-a5f42fea4f77-workload-certs" pod "396046e4-b596-4fed-943d-a5f42fea4f77" (UID: "396046e4-b596-4fed-943d-a5f42fea4f77"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:07:19.909893 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:19.904657 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/396046e4-b596-4fed-943d-a5f42fea4f77-istio-data" pod "396046e4-b596-4fed-943d-a5f42fea4f77" (UID: "396046e4-b596-4fed-943d-a5f42fea4f77"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:07:19.909893 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:19.904725 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/396046e4-b596-4fed-943d-a5f42fea4f77-credential-socket" pod "396046e4-b596-4fed-943d-a5f42fea4f77" (UID: "396046e4-b596-4fed-943d-a5f42fea4f77"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:07:19.909893 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:19.905203 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/396046e4-b596-4fed-943d-a5f42fea4f77-kube-api-access-69z66" pod "396046e4-b596-4fed-943d-a5f42fea4f77" (UID: "396046e4-b596-4fed-943d-a5f42fea4f77"). InnerVolumeSpecName "kube-api-access-69z66". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:07:19.909893 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:19.909691 4908 scope.go:122] "RemoveContainer" containerID="d904798341eedab6a9fa053fc321f05682c35fccd03374ef779c718d44358680" Oct 30 13:07:19.910629 ip-10-0-141-251 kubenswrapper[4908]: E1030 13:07:19.910144 4908 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d904798341eedab6a9fa053fc321f05682c35fccd03374ef779c718d44358680\": container with ID starting with d904798341eedab6a9fa053fc321f05682c35fccd03374ef779c718d44358680 not found: ID does not exist" containerID="d904798341eedab6a9fa053fc321f05682c35fccd03374ef779c718d44358680" Oct 30 13:07:19.910629 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:19.910186 4908 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d904798341eedab6a9fa053fc321f05682c35fccd03374ef779c718d44358680"} err="failed to get container status \"d904798341eedab6a9fa053fc321f05682c35fccd03374ef779c718d44358680\": rpc error: code = NotFound desc = could not find container \"d904798341eedab6a9fa053fc321f05682c35fccd03374ef779c718d44358680\": container with ID starting with d904798341eedab6a9fa053fc321f05682c35fccd03374ef779c718d44358680 not found: ID does not exist" Oct 30 13:07:19.910629 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:19.910452 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/396046e4-b596-4fed-943d-a5f42fea4f77-istio-podinfo" pod "396046e4-b596-4fed-943d-a5f42fea4f77" (UID: "396046e4-b596-4fed-943d-a5f42fea4f77"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:07:19.911718 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:19.911682 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/396046e4-b596-4fed-943d-a5f42fea4f77-istio-envoy" pod "396046e4-b596-4fed-943d-a5f42fea4f77" (UID: "396046e4-b596-4fed-943d-a5f42fea4f77"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:07:19.911718 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:19.911700 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/396046e4-b596-4fed-943d-a5f42fea4f77-istio-token" pod "396046e4-b596-4fed-943d-a5f42fea4f77" (UID: "396046e4-b596-4fed-943d-a5f42fea4f77"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:07:19.912388 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:19.912360 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a85e562f-d655-4c22-b4f4-a299a4e494a0-dshm" pod "a85e562f-d655-4c22-b4f4-a299a4e494a0" (UID: "a85e562f-d655-4c22-b4f4-a299a4e494a0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:07:19.913124 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:19.913083 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a85e562f-d655-4c22-b4f4-a299a4e494a0-tls-certs" pod "a85e562f-d655-4c22-b4f4-a299a4e494a0" (UID: "a85e562f-d655-4c22-b4f4-a299a4e494a0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:07:19.913331 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:19.913306 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a85e562f-d655-4c22-b4f4-a299a4e494a0-kube-api-access-cqbsz" pod "a85e562f-d655-4c22-b4f4-a299a4e494a0" (UID: "a85e562f-d655-4c22-b4f4-a299a4e494a0"). InnerVolumeSpecName "kube-api-access-cqbsz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:07:20.004089 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:20.004041 4908 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/396046e4-b596-4fed-943d-a5f42fea4f77-istio-token\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:07:20.004089 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:20.004087 4908 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/396046e4-b596-4fed-943d-a5f42fea4f77-workload-socket\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:07:20.004332 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:20.004102 4908 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/396046e4-b596-4fed-943d-a5f42fea4f77-workload-certs\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:07:20.004332 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:20.004116 4908 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a85e562f-d655-4c22-b4f4-a299a4e494a0-dshm\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:07:20.004332 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:20.004131 4908 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/396046e4-b596-4fed-943d-a5f42fea4f77-credential-socket\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:07:20.004332 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:20.004144 4908 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/396046e4-b596-4fed-943d-a5f42fea4f77-istio-data\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:07:20.004332 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:20.004157 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cqbsz\" (UniqueName: \"kubernetes.io/projected/a85e562f-d655-4c22-b4f4-a299a4e494a0-kube-api-access-cqbsz\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:07:20.004332 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:20.004170 4908 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/396046e4-b596-4fed-943d-a5f42fea4f77-istio-podinfo\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:07:20.004332 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:20.004182 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69z66\" (UniqueName: \"kubernetes.io/projected/396046e4-b596-4fed-943d-a5f42fea4f77-kube-api-access-69z66\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:07:20.004332 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:20.004198 4908 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/396046e4-b596-4fed-943d-a5f42fea4f77-istio-envoy\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:07:20.004332 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:20.004211 4908 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a85e562f-d655-4c22-b4f4-a299a4e494a0-tls-certs\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:07:20.216676 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:20.216635 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-12fd33-openshift-default-846b7dbb9-pt6kj"] Oct 30 13:07:20.229899 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:20.229866 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-12fd33-openshift-default-846b7dbb9-pt6kj"] Oct 30 13:07:20.266161 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:20.266068 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-f920f7/simulator-w0-f920f7-kserve-599b7ff45f-g79d7"] Oct 30 13:07:20.269724 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:20.269689 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-f920f7/simulator-w0-f920f7-kserve-599b7ff45f-g79d7"] Oct 30 13:07:21.607982 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:21.607943 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="396046e4-b596-4fed-943d-a5f42fea4f77" path="/var/lib/kubelet/pods/396046e4-b596-4fed-943d-a5f42fea4f77/volumes" Oct 30 13:07:21.608425 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:21.608410 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a85e562f-d655-4c22-b4f4-a299a4e494a0" path="/var/lib/kubelet/pods/a85e562f-d655-4c22-b4f4-a299a4e494a0/volumes" Oct 30 13:07:22.191271 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:22.191236 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9wwfc"] Oct 30 13:07:25.182060 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:25.182019 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8e525d9-m494z"] Oct 30 13:07:25.182696 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:25.182678 4908 cpu_manager.go:414] "containerMap: removing container" podUID="396046e4-b596-4fed-943d-a5f42fea4f77" containerName="istio-proxy" Oct 30 13:07:25.182739 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:25.182698 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="396046e4-b596-4fed-943d-a5f42fea4f77" containerName="istio-proxy" Oct 30 13:07:25.182739 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:25.182734 4908 cpu_manager.go:414] "containerMap: removing container" podUID="be103a5c-91f5-4119-a4e3-8bc4b6705732" containerName="revoke-keys" Oct 30 13:07:25.182806 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:25.182744 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be103a5c-91f5-4119-a4e3-8bc4b6705732" containerName="revoke-keys" Oct 30 13:07:25.182806 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:25.182758 4908 cpu_manager.go:414] "containerMap: removing container" podUID="a85e562f-d655-4c22-b4f4-a299a4e494a0" containerName="main" Oct 30 13:07:25.182806 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:25.182766 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a85e562f-d655-4c22-b4f4-a299a4e494a0" containerName="main" Oct 30 13:07:25.182937 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:25.182901 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="3590e8ff-89a3-482b-b022-064bb84fe09c" containerName="istio-proxy" Oct 30 13:07:25.182937 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:25.182914 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="be103a5c-91f5-4119-a4e3-8bc4b6705732" containerName="revoke-keys" Oct 30 13:07:25.182937 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:25.182924 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="a85e562f-d655-4c22-b4f4-a299a4e494a0" containerName="main" Oct 30 13:07:25.182937 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:25.182934 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="396046e4-b596-4fed-943d-a5f42fea4f77" containerName="istio-proxy" Oct 30 13:07:25.188284 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:25.188259 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8e525d9-m494z" Oct 30 13:07:25.205690 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:25.205652 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e9148a5-cll94"] Oct 30 13:07:25.206405 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:25.206382 4908 cpu_manager.go:414] "containerMap: removing container" podUID="396046e4-b596-4fed-943d-a5f42fea4f77" containerName="istio-proxy" Oct 30 13:07:25.206501 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:25.206407 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="396046e4-b596-4fed-943d-a5f42fea4f77" containerName="istio-proxy" Oct 30 13:07:25.206551 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:25.206539 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="396046e4-b596-4fed-943d-a5f42fea4f77" containerName="istio-proxy" Oct 30 13:07:25.210119 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:25.210095 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8e525d9-m494z"] Oct 30 13:07:25.210247 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:25.210189 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e9148a5-cll94" Oct 30 13:07:25.214471 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:25.214440 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 13:07:25.214471 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:25.214451 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 13:07:25.214665 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:25.214493 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:07:25.230061 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:25.230016 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e9148a5-cll94"] Oct 30 13:07:25.233210 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:25.233174 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s75hq\"" Oct 30 13:07:25.266860 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:25.266689 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwwkl\" (UniqueName: \"kubernetes.io/projected/29225043-ea8e-4a6a-b259-373c827784bf-kube-api-access-pwwkl\") pod \"maas-api-revoke-sub-c8e525d9-m494z\" (UID: \"29225043-ea8e-4a6a-b259-373c827784bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8e525d9-m494z" Oct 30 13:07:25.272461 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:25.267060 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29225043-ea8e-4a6a-b259-373c827784bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c8e525d9-m494z\" (UID: \"29225043-ea8e-4a6a-b259-373c827784bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8e525d9-m494z" Oct 30 13:07:25.272461 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:25.267160 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29225043-ea8e-4a6a-b259-373c827784bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c8e525d9-m494z\" (UID: \"29225043-ea8e-4a6a-b259-373c827784bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8e525d9-m494z" Oct 30 13:07:25.368980 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:25.368936 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29225043-ea8e-4a6a-b259-373c827784bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c8e525d9-m494z\" (UID: \"29225043-ea8e-4a6a-b259-373c827784bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8e525d9-m494z" Oct 30 13:07:25.369272 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:25.369241 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r96ws\" (UniqueName: \"kubernetes.io/projected/ec981005-170b-4a49-8029-a9a93bcfa9db-kube-api-access-r96ws\") pod \"maas-api-revoke-sub-8e9148a5-cll94\" (UID: \"ec981005-170b-4a49-8029-a9a93bcfa9db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e9148a5-cll94" Oct 30 13:07:25.369413 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:25.369387 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec981005-170b-4a49-8029-a9a93bcfa9db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8e9148a5-cll94\" (UID: \"ec981005-170b-4a49-8029-a9a93bcfa9db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e9148a5-cll94" Oct 30 13:07:25.369477 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:25.369438 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29225043-ea8e-4a6a-b259-373c827784bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c8e525d9-m494z\" (UID: \"29225043-ea8e-4a6a-b259-373c827784bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8e525d9-m494z" Oct 30 13:07:25.369563 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:25.369544 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwwkl\" (UniqueName: \"kubernetes.io/projected/29225043-ea8e-4a6a-b259-373c827784bf-kube-api-access-pwwkl\") pod \"maas-api-revoke-sub-c8e525d9-m494z\" (UID: \"29225043-ea8e-4a6a-b259-373c827784bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8e525d9-m494z" Oct 30 13:07:25.369630 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:25.369617 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec981005-170b-4a49-8029-a9a93bcfa9db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8e9148a5-cll94\" (UID: \"ec981005-170b-4a49-8029-a9a93bcfa9db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e9148a5-cll94" Oct 30 13:07:25.369808 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:25.369784 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29225043-ea8e-4a6a-b259-373c827784bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c8e525d9-m494z\" (UID: \"29225043-ea8e-4a6a-b259-373c827784bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8e525d9-m494z" Oct 30 13:07:25.372760 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:25.372734 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29225043-ea8e-4a6a-b259-373c827784bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c8e525d9-m494z\" (UID: \"29225043-ea8e-4a6a-b259-373c827784bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8e525d9-m494z" Oct 30 13:07:25.396384 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:25.396346 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwwkl\" (UniqueName: \"kubernetes.io/projected/29225043-ea8e-4a6a-b259-373c827784bf-kube-api-access-pwwkl\") pod \"maas-api-revoke-sub-c8e525d9-m494z\" (UID: \"29225043-ea8e-4a6a-b259-373c827784bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8e525d9-m494z" Oct 30 13:07:25.470467 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:25.470362 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec981005-170b-4a49-8029-a9a93bcfa9db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8e9148a5-cll94\" (UID: \"ec981005-170b-4a49-8029-a9a93bcfa9db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e9148a5-cll94" Oct 30 13:07:25.470666 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:25.470483 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r96ws\" (UniqueName: \"kubernetes.io/projected/ec981005-170b-4a49-8029-a9a93bcfa9db-kube-api-access-r96ws\") pod \"maas-api-revoke-sub-8e9148a5-cll94\" (UID: \"ec981005-170b-4a49-8029-a9a93bcfa9db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e9148a5-cll94" Oct 30 13:07:25.470666 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:25.470576 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec981005-170b-4a49-8029-a9a93bcfa9db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8e9148a5-cll94\" (UID: \"ec981005-170b-4a49-8029-a9a93bcfa9db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e9148a5-cll94" Oct 30 13:07:25.471457 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:25.471423 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec981005-170b-4a49-8029-a9a93bcfa9db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8e9148a5-cll94\" (UID: \"ec981005-170b-4a49-8029-a9a93bcfa9db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e9148a5-cll94" Oct 30 13:07:25.473492 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:25.473463 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec981005-170b-4a49-8029-a9a93bcfa9db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8e9148a5-cll94\" (UID: \"ec981005-170b-4a49-8029-a9a93bcfa9db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e9148a5-cll94" Oct 30 13:07:25.496316 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:25.496283 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r96ws\" (UniqueName: \"kubernetes.io/projected/ec981005-170b-4a49-8029-a9a93bcfa9db-kube-api-access-r96ws\") pod \"maas-api-revoke-sub-8e9148a5-cll94\" (UID: \"ec981005-170b-4a49-8029-a9a93bcfa9db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e9148a5-cll94" Oct 30 13:07:25.503983 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:25.503942 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8e525d9-m494z" Oct 30 13:07:25.536704 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:25.536670 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e9148a5-cll94" Oct 30 13:07:25.820429 ip-10-0-141-251 kubenswrapper[4908]: W1030 13:07:25.820392 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29225043_ea8e_4a6a_b259_373c827784bf.slice/crio-15fdf7baa196f96fd531acf839a0abd94cad1babaf10f32b08c576b50745428a WatchSource:0}: Error finding container 15fdf7baa196f96fd531acf839a0abd94cad1babaf10f32b08c576b50745428a: Status 404 returned error can't find the container with id 15fdf7baa196f96fd531acf839a0abd94cad1babaf10f32b08c576b50745428a Oct 30 13:07:25.820730 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:25.820654 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8e525d9-m494z"] Oct 30 13:07:25.909686 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:25.907848 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e9148a5-cll94"] Oct 30 13:07:25.930703 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:25.930623 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8e525d9-m494z" event={"ID":"29225043-ea8e-4a6a-b259-373c827784bf","Type":"ContainerStarted","Data":"15fdf7baa196f96fd531acf839a0abd94cad1babaf10f32b08c576b50745428a"} Oct 30 13:07:26.293936 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:26.293896 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-812990-cg7zp"] Oct 30 13:07:26.314525 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:26.314481 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-812990-cg7zp"] Oct 30 13:07:26.847542 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:26.847494 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-10f364-nmjt8"] Oct 30 13:07:26.850435 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:26.850407 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-10f364-nmjt8"] Oct 30 13:07:26.939582 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:26.939461 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e9148a5-cll94" event={"ID":"ec981005-170b-4a49-8029-a9a93bcfa9db","Type":"ContainerStarted","Data":"b00d813d22efaef5f7ff5ccec641db2d7d117a1c243578be5165e968a7b9d6e5"} Oct 30 13:07:26.939582 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:26.939568 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e9148a5-cll94" event={"ID":"ec981005-170b-4a49-8029-a9a93bcfa9db","Type":"ContainerStarted","Data":"fd91d140cf22893eb1239244869d4ba97a7fc055eff33f5d472f35fd8d9a3e9e"} Oct 30 13:07:26.941236 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:26.941208 4908 generic.go:356] "Generic (PLEG): container finished" podID="29225043-ea8e-4a6a-b259-373c827784bf" containerID="b0afa5f10491c4ad53cf45b4d5cb39cbc2453531d69ebbd2168cc106cb8b2186" exitCode=0 Oct 30 13:07:26.941355 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:26.941289 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8e525d9-m494z" event={"ID":"29225043-ea8e-4a6a-b259-373c827784bf","Type":"ContainerDied","Data":"b0afa5f10491c4ad53cf45b4d5cb39cbc2453531d69ebbd2168cc106cb8b2186"} Oct 30 13:07:26.984064 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:26.984002 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e9148a5-cll94" podStartSLOduration=1.983981804 podStartE2EDuration="1.983981804s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:07:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:07:26.982003024 +0000 UTC m=+4403.496431770" watchObservedRunningTime="2026-10-30 13:07:26.983981804 +0000 UTC m=+4403.498410550" Oct 30 13:07:27.266472 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:27.266435 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2675c0-558q9"] Oct 30 13:07:27.269286 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:27.269256 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2675c0-558q9"] Oct 30 13:07:27.608737 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:27.608645 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39a5d567-2188-4af8-9171-9879694ecc67" path="/var/lib/kubelet/pods/39a5d567-2188-4af8-9171-9879694ecc67/volumes" Oct 30 13:07:27.609316 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:27.609284 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="443b4b1e-1ff4-4c9a-8c4b-ac836c87afb6" path="/var/lib/kubelet/pods/443b4b1e-1ff4-4c9a-8c4b-ac836c87afb6/volumes" Oct 30 13:07:27.609879 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:27.609861 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be103a5c-91f5-4119-a4e3-8bc4b6705732" path="/var/lib/kubelet/pods/be103a5c-91f5-4119-a4e3-8bc4b6705732/volumes" Oct 30 13:07:27.950624 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:27.950528 4908 generic.go:356] "Generic (PLEG): container finished" podID="ec981005-170b-4a49-8029-a9a93bcfa9db" containerID="b00d813d22efaef5f7ff5ccec641db2d7d117a1c243578be5165e968a7b9d6e5" exitCode=0 Oct 30 13:07:27.950799 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:27.950623 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e9148a5-cll94" event={"ID":"ec981005-170b-4a49-8029-a9a93bcfa9db","Type":"ContainerDied","Data":"b00d813d22efaef5f7ff5ccec641db2d7d117a1c243578be5165e968a7b9d6e5"} Oct 30 13:07:28.107514 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:28.107475 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8e525d9-m494z" Oct 30 13:07:28.204241 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:28.204160 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29225043-ea8e-4a6a-b259-373c827784bf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29225043-ea8e-4a6a-b259-373c827784bf-maas-api-openshift-ca\") pod \"29225043-ea8e-4a6a-b259-373c827784bf\" (UID: \"29225043-ea8e-4a6a-b259-373c827784bf\") " Oct 30 13:07:28.204241 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:28.204227 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29225043-ea8e-4a6a-b259-373c827784bf-kube-api-access-pwwkl\" (UniqueName: \"kubernetes.io/projected/29225043-ea8e-4a6a-b259-373c827784bf-kube-api-access-pwwkl\") pod \"29225043-ea8e-4a6a-b259-373c827784bf\" (UID: \"29225043-ea8e-4a6a-b259-373c827784bf\") " Oct 30 13:07:28.204448 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:28.204420 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/29225043-ea8e-4a6a-b259-373c827784bf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29225043-ea8e-4a6a-b259-373c827784bf-maas-api-xks-ca\") pod \"29225043-ea8e-4a6a-b259-373c827784bf\" (UID: \"29225043-ea8e-4a6a-b259-373c827784bf\") " Oct 30 13:07:28.204572 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:28.204549 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29225043-ea8e-4a6a-b259-373c827784bf-maas-api-openshift-ca" pod "29225043-ea8e-4a6a-b259-373c827784bf" (UID: "29225043-ea8e-4a6a-b259-373c827784bf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:07:28.204879 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:28.204856 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29225043-ea8e-4a6a-b259-373c827784bf-maas-api-openshift-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:07:28.206414 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:28.206393 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29225043-ea8e-4a6a-b259-373c827784bf-maas-api-xks-ca" pod "29225043-ea8e-4a6a-b259-373c827784bf" (UID: "29225043-ea8e-4a6a-b259-373c827784bf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:07:28.206482 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:28.206468 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29225043-ea8e-4a6a-b259-373c827784bf-kube-api-access-pwwkl" pod "29225043-ea8e-4a6a-b259-373c827784bf" (UID: "29225043-ea8e-4a6a-b259-373c827784bf"). InnerVolumeSpecName "kube-api-access-pwwkl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:07:28.305967 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:28.305926 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwwkl\" (UniqueName: \"kubernetes.io/projected/29225043-ea8e-4a6a-b259-373c827784bf-kube-api-access-pwwkl\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:07:28.305967 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:28.305964 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29225043-ea8e-4a6a-b259-373c827784bf-maas-api-xks-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:07:28.959806 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:28.959774 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8e525d9-m494z" Oct 30 13:07:28.960261 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:28.959813 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8e525d9-m494z" event={"ID":"29225043-ea8e-4a6a-b259-373c827784bf","Type":"ContainerDied","Data":"15fdf7baa196f96fd531acf839a0abd94cad1babaf10f32b08c576b50745428a"} Oct 30 13:07:28.960261 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:28.959894 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="15fdf7baa196f96fd531acf839a0abd94cad1babaf10f32b08c576b50745428a" Oct 30 13:07:29.143267 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:29.143238 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e9148a5-cll94" Oct 30 13:07:29.216715 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:29.216620 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec981005-170b-4a49-8029-a9a93bcfa9db-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec981005-170b-4a49-8029-a9a93bcfa9db-maas-api-xks-ca\") pod \"ec981005-170b-4a49-8029-a9a93bcfa9db\" (UID: \"ec981005-170b-4a49-8029-a9a93bcfa9db\") " Oct 30 13:07:29.216949 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:29.216797 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec981005-170b-4a49-8029-a9a93bcfa9db-kube-api-access-r96ws\" (UniqueName: \"kubernetes.io/projected/ec981005-170b-4a49-8029-a9a93bcfa9db-kube-api-access-r96ws\") pod \"ec981005-170b-4a49-8029-a9a93bcfa9db\" (UID: \"ec981005-170b-4a49-8029-a9a93bcfa9db\") " Oct 30 13:07:29.217039 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:29.216954 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec981005-170b-4a49-8029-a9a93bcfa9db-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec981005-170b-4a49-8029-a9a93bcfa9db-maas-api-openshift-ca\") pod \"ec981005-170b-4a49-8029-a9a93bcfa9db\" (UID: \"ec981005-170b-4a49-8029-a9a93bcfa9db\") " Oct 30 13:07:29.217415 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:29.217361 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec981005-170b-4a49-8029-a9a93bcfa9db-maas-api-openshift-ca" pod "ec981005-170b-4a49-8029-a9a93bcfa9db" (UID: "ec981005-170b-4a49-8029-a9a93bcfa9db"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:07:29.217808 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:29.217788 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec981005-170b-4a49-8029-a9a93bcfa9db-maas-api-openshift-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:07:29.219200 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:29.219177 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec981005-170b-4a49-8029-a9a93bcfa9db-kube-api-access-r96ws" pod "ec981005-170b-4a49-8029-a9a93bcfa9db" (UID: "ec981005-170b-4a49-8029-a9a93bcfa9db"). InnerVolumeSpecName "kube-api-access-r96ws". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:07:29.219334 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:29.219313 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec981005-170b-4a49-8029-a9a93bcfa9db-maas-api-xks-ca" pod "ec981005-170b-4a49-8029-a9a93bcfa9db" (UID: "ec981005-170b-4a49-8029-a9a93bcfa9db"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:07:29.318547 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:29.318507 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec981005-170b-4a49-8029-a9a93bcfa9db-maas-api-xks-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:07:29.318547 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:29.318542 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r96ws\" (UniqueName: \"kubernetes.io/projected/ec981005-170b-4a49-8029-a9a93bcfa9db-kube-api-access-r96ws\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:07:29.968427 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:29.968332 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e9148a5-cll94" Oct 30 13:07:29.968427 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:29.968357 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e9148a5-cll94" event={"ID":"ec981005-170b-4a49-8029-a9a93bcfa9db","Type":"ContainerDied","Data":"fd91d140cf22893eb1239244869d4ba97a7fc055eff33f5d472f35fd8d9a3e9e"} Oct 30 13:07:29.968427 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:29.968412 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fd91d140cf22893eb1239244869d4ba97a7fc055eff33f5d472f35fd8d9a3e9e" Oct 30 13:07:34.881791 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:34.881751 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-10f364-openshift-default-965d4bfd5-4b57q"] Oct 30 13:07:34.883356 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:34.883317 4908 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-10f364-openshift-default-965d4bfd5-4b57q" podUID="aa19be03-336a-4af4-aba5-e089b4933bf9" containerName="istio-proxy" containerID="cri-o://0ee7336c32c90ca843b02e706f558154f455e05400d519702b8ca0f83f6b8058" gracePeriod=30 Oct 30 13:07:34.887797 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:34.887765 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-42220c-jdv8n"] Oct 30 13:07:34.888652 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:34.888623 4908 cpu_manager.go:414] "containerMap: removing container" podUID="29225043-ea8e-4a6a-b259-373c827784bf" containerName="revoke-keys" Oct 30 13:07:34.888800 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:34.888785 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29225043-ea8e-4a6a-b259-373c827784bf" containerName="revoke-keys" Oct 30 13:07:34.888981 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:34.888968 4908 cpu_manager.go:414] "containerMap: removing container" podUID="ec981005-170b-4a49-8029-a9a93bcfa9db" containerName="revoke-keys" Oct 30 13:07:34.889073 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:34.889061 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec981005-170b-4a49-8029-a9a93bcfa9db" containerName="revoke-keys" Oct 30 13:07:34.889273 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:34.889256 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec981005-170b-4a49-8029-a9a93bcfa9db" containerName="revoke-keys" Oct 30 13:07:34.889391 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:34.889379 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="29225043-ea8e-4a6a-b259-373c827784bf" containerName="revoke-keys" Oct 30 13:07:34.897770 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:34.897734 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-42220c-jdv8n"] Oct 30 13:07:34.898068 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:34.897957 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-42220c-jdv8n" Oct 30 13:07:34.916920 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:34.916889 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:07:34.917103 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:34.916958 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s75hq\"" Oct 30 13:07:34.917682 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:34.917657 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 13:07:34.917781 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:34.917694 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 13:07:34.984002 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:34.983961 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ht9r9\" (UniqueName: \"kubernetes.io/projected/d9e62368-725d-4e46-88b4-bd203a349d69-kube-api-access-ht9r9\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-42220c-jdv8n\" (UID: \"d9e62368-725d-4e46-88b4-bd203a349d69\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-42220c-jdv8n" Oct 30 13:07:34.984430 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:34.984399 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9e62368-725d-4e46-88b4-bd203a349d69-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-42220c-jdv8n\" (UID: \"d9e62368-725d-4e46-88b4-bd203a349d69\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-42220c-jdv8n" Oct 30 13:07:34.984548 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:34.984470 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9e62368-725d-4e46-88b4-bd203a349d69-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-42220c-jdv8n\" (UID: \"d9e62368-725d-4e46-88b4-bd203a349d69\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-42220c-jdv8n" Oct 30 13:07:35.085331 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:35.085281 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ht9r9\" (UniqueName: \"kubernetes.io/projected/d9e62368-725d-4e46-88b4-bd203a349d69-kube-api-access-ht9r9\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-42220c-jdv8n\" (UID: \"d9e62368-725d-4e46-88b4-bd203a349d69\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-42220c-jdv8n" Oct 30 13:07:35.085549 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:35.085351 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9e62368-725d-4e46-88b4-bd203a349d69-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-42220c-jdv8n\" (UID: \"d9e62368-725d-4e46-88b4-bd203a349d69\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-42220c-jdv8n" Oct 30 13:07:35.085549 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:35.085400 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9e62368-725d-4e46-88b4-bd203a349d69-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-42220c-jdv8n\" (UID: \"d9e62368-725d-4e46-88b4-bd203a349d69\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-42220c-jdv8n" Oct 30 13:07:35.086316 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:35.086287 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9e62368-725d-4e46-88b4-bd203a349d69-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-42220c-jdv8n\" (UID: \"d9e62368-725d-4e46-88b4-bd203a349d69\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-42220c-jdv8n" Oct 30 13:07:35.088097 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:35.088071 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9e62368-725d-4e46-88b4-bd203a349d69-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-42220c-jdv8n\" (UID: \"d9e62368-725d-4e46-88b4-bd203a349d69\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-42220c-jdv8n" Oct 30 13:07:35.110840 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:35.110796 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ht9r9\" (UniqueName: \"kubernetes.io/projected/d9e62368-725d-4e46-88b4-bd203a349d69-kube-api-access-ht9r9\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-42220c-jdv8n\" (UID: \"d9e62368-725d-4e46-88b4-bd203a349d69\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-42220c-jdv8n" Oct 30 13:07:35.225435 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:35.225349 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-42220c-jdv8n" Oct 30 13:07:35.585199 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:35.585165 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-42220c-jdv8n"] Oct 30 13:07:35.586736 ip-10-0-141-251 kubenswrapper[4908]: W1030 13:07:35.586708 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9e62368_725d_4e46_88b4_bd203a349d69.slice/crio-73fcf6d418ade8ab5e7657bc80e4e0dff11dfa947419d222b4633f88824cdf3a WatchSource:0}: Error finding container 73fcf6d418ade8ab5e7657bc80e4e0dff11dfa947419d222b4633f88824cdf3a: Status 404 returned error can't find the container with id 73fcf6d418ade8ab5e7657bc80e4e0dff11dfa947419d222b4633f88824cdf3a Oct 30 13:07:35.630927 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:35.630893 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-fddaa3-wnq5m"] Oct 30 13:07:35.635821 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:35.635797 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-fddaa3-wnq5m" Oct 30 13:07:35.660528 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:35.660023 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-fddaa3-wnq5m"] Oct 30 13:07:35.803310 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:35.803250 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/435ef18d-77f4-4046-95ea-6cd0ce4b5fc6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-fddaa3-wnq5m\" (UID: \"435ef18d-77f4-4046-95ea-6cd0ce4b5fc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-fddaa3-wnq5m" Oct 30 13:07:35.803541 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:35.803377 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/435ef18d-77f4-4046-95ea-6cd0ce4b5fc6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-fddaa3-wnq5m\" (UID: \"435ef18d-77f4-4046-95ea-6cd0ce4b5fc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-fddaa3-wnq5m" Oct 30 13:07:35.803541 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:35.803426 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmc57\" (UniqueName: \"kubernetes.io/projected/435ef18d-77f4-4046-95ea-6cd0ce4b5fc6-kube-api-access-kmc57\") pod \"maas-api-revoke-keys-e2e-extmod-b-fddaa3-wnq5m\" (UID: \"435ef18d-77f4-4046-95ea-6cd0ce4b5fc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-fddaa3-wnq5m" Oct 30 13:07:35.904614 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:35.904569 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/435ef18d-77f4-4046-95ea-6cd0ce4b5fc6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-fddaa3-wnq5m\" (UID: \"435ef18d-77f4-4046-95ea-6cd0ce4b5fc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-fddaa3-wnq5m" Oct 30 13:07:35.905100 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:35.904660 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/435ef18d-77f4-4046-95ea-6cd0ce4b5fc6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-fddaa3-wnq5m\" (UID: \"435ef18d-77f4-4046-95ea-6cd0ce4b5fc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-fddaa3-wnq5m" Oct 30 13:07:35.905100 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:35.904694 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmc57\" (UniqueName: \"kubernetes.io/projected/435ef18d-77f4-4046-95ea-6cd0ce4b5fc6-kube-api-access-kmc57\") pod \"maas-api-revoke-keys-e2e-extmod-b-fddaa3-wnq5m\" (UID: \"435ef18d-77f4-4046-95ea-6cd0ce4b5fc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-fddaa3-wnq5m" Oct 30 13:07:35.905549 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:35.905522 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/435ef18d-77f4-4046-95ea-6cd0ce4b5fc6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-fddaa3-wnq5m\" (UID: \"435ef18d-77f4-4046-95ea-6cd0ce4b5fc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-fddaa3-wnq5m" Oct 30 13:07:35.907202 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:35.907178 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/435ef18d-77f4-4046-95ea-6cd0ce4b5fc6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-fddaa3-wnq5m\" (UID: \"435ef18d-77f4-4046-95ea-6cd0ce4b5fc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-fddaa3-wnq5m" Oct 30 13:07:35.929229 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:35.929201 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmc57\" (UniqueName: \"kubernetes.io/projected/435ef18d-77f4-4046-95ea-6cd0ce4b5fc6-kube-api-access-kmc57\") pod \"maas-api-revoke-keys-e2e-extmod-b-fddaa3-wnq5m\" (UID: \"435ef18d-77f4-4046-95ea-6cd0ce4b5fc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-fddaa3-wnq5m" Oct 30 13:07:35.963435 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:35.963407 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-fddaa3-wnq5m" Oct 30 13:07:36.025218 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:36.024662 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-42220c-jdv8n" event={"ID":"d9e62368-725d-4e46-88b4-bd203a349d69","Type":"ContainerStarted","Data":"4936d8746df640a4a3fe5ff0546ff6479f917831956c4a408b1f5c2095dd2b9b"} Oct 30 13:07:36.025218 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:36.024776 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-42220c-jdv8n" event={"ID":"d9e62368-725d-4e46-88b4-bd203a349d69","Type":"ContainerStarted","Data":"73fcf6d418ade8ab5e7657bc80e4e0dff11dfa947419d222b4633f88824cdf3a"} Oct 30 13:07:36.287405 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:36.287353 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-42220c-jdv8n" podStartSLOduration=2.2873363749999998 podStartE2EDuration="2.287336375s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:07:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:07:36.113816825 +0000 UTC m=+4412.628245572" watchObservedRunningTime="2026-10-30 13:07:36.287336375 +0000 UTC m=+4412.801765121" Oct 30 13:07:36.290943 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:36.290903 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-fddaa3-wnq5m"] Oct 30 13:07:36.301041 ip-10-0-141-251 kubenswrapper[4908]: W1030 13:07:36.301011 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod435ef18d_77f4_4046_95ea_6cd0ce4b5fc6.slice/crio-03f90392a762dd9b746ff05363eda6dbe4691c7f44581b3538312111f0c018d0 WatchSource:0}: Error finding container 03f90392a762dd9b746ff05363eda6dbe4691c7f44581b3538312111f0c018d0: Status 404 returned error can't find the container with id 03f90392a762dd9b746ff05363eda6dbe4691c7f44581b3538312111f0c018d0 Oct 30 13:07:36.662800 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:36.662756 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-9d45fb-p9ftj"] Oct 30 13:07:36.666893 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:36.666868 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-9d45fb-p9ftj" Oct 30 13:07:36.702205 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:36.702172 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-9d45fb-p9ftj"] Oct 30 13:07:36.815406 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:36.815370 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6df977ff-7e53-4d0e-88c2-971c615ff83e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-9d45fb-p9ftj\" (UID: \"6df977ff-7e53-4d0e-88c2-971c615ff83e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-9d45fb-p9ftj" Oct 30 13:07:36.815596 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:36.815443 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6df977ff-7e53-4d0e-88c2-971c615ff83e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-9d45fb-p9ftj\" (UID: \"6df977ff-7e53-4d0e-88c2-971c615ff83e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-9d45fb-p9ftj" Oct 30 13:07:36.815596 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:36.815569 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pp4mb\" (UniqueName: \"kubernetes.io/projected/6df977ff-7e53-4d0e-88c2-971c615ff83e-kube-api-access-pp4mb\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-9d45fb-p9ftj\" (UID: \"6df977ff-7e53-4d0e-88c2-971c615ff83e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-9d45fb-p9ftj" Oct 30 13:07:36.916266 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:36.916165 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6df977ff-7e53-4d0e-88c2-971c615ff83e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-9d45fb-p9ftj\" (UID: \"6df977ff-7e53-4d0e-88c2-971c615ff83e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-9d45fb-p9ftj" Oct 30 13:07:36.916266 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:36.916251 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pp4mb\" (UniqueName: \"kubernetes.io/projected/6df977ff-7e53-4d0e-88c2-971c615ff83e-kube-api-access-pp4mb\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-9d45fb-p9ftj\" (UID: \"6df977ff-7e53-4d0e-88c2-971c615ff83e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-9d45fb-p9ftj" Oct 30 13:07:36.916697 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:36.916387 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6df977ff-7e53-4d0e-88c2-971c615ff83e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-9d45fb-p9ftj\" (UID: \"6df977ff-7e53-4d0e-88c2-971c615ff83e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-9d45fb-p9ftj" Oct 30 13:07:36.917061 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:36.917037 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6df977ff-7e53-4d0e-88c2-971c615ff83e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-9d45fb-p9ftj\" (UID: \"6df977ff-7e53-4d0e-88c2-971c615ff83e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-9d45fb-p9ftj" Oct 30 13:07:36.918492 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:36.918471 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6df977ff-7e53-4d0e-88c2-971c615ff83e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-9d45fb-p9ftj\" (UID: \"6df977ff-7e53-4d0e-88c2-971c615ff83e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-9d45fb-p9ftj" Oct 30 13:07:36.940737 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:36.940709 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pp4mb\" (UniqueName: \"kubernetes.io/projected/6df977ff-7e53-4d0e-88c2-971c615ff83e-kube-api-access-pp4mb\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-9d45fb-p9ftj\" (UID: \"6df977ff-7e53-4d0e-88c2-971c615ff83e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-9d45fb-p9ftj" Oct 30 13:07:36.981521 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:36.981481 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-9d45fb-p9ftj" Oct 30 13:07:37.034988 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:37.034942 4908 generic.go:356] "Generic (PLEG): container finished" podID="d9e62368-725d-4e46-88b4-bd203a349d69" containerID="4936d8746df640a4a3fe5ff0546ff6479f917831956c4a408b1f5c2095dd2b9b" exitCode=0 Oct 30 13:07:37.035149 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:37.035070 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-42220c-jdv8n" event={"ID":"d9e62368-725d-4e46-88b4-bd203a349d69","Type":"ContainerDied","Data":"4936d8746df640a4a3fe5ff0546ff6479f917831956c4a408b1f5c2095dd2b9b"} Oct 30 13:07:37.037975 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:37.037576 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-fddaa3-wnq5m" event={"ID":"435ef18d-77f4-4046-95ea-6cd0ce4b5fc6","Type":"ContainerStarted","Data":"df1efc93dd64d8f482d9f3a8595fb3ec68134d7cc9cebcf93ab4ab3ebdb458ba"} Oct 30 13:07:37.037975 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:37.037637 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-fddaa3-wnq5m" event={"ID":"435ef18d-77f4-4046-95ea-6cd0ce4b5fc6","Type":"ContainerStarted","Data":"03f90392a762dd9b746ff05363eda6dbe4691c7f44581b3538312111f0c018d0"} Oct 30 13:07:37.127655 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:37.127597 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-fddaa3-wnq5m" podStartSLOduration=2.1275808720000002 podStartE2EDuration="2.127580872s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:07:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:07:37.124334797 +0000 UTC m=+4413.638763540" watchObservedRunningTime="2026-10-30 13:07:37.127580872 +0000 UTC m=+4413.642009617" Oct 30 13:07:37.289478 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:37.289440 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-9d45fb-p9ftj"] Oct 30 13:07:38.047364 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:38.047320 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-9d45fb-p9ftj" event={"ID":"6df977ff-7e53-4d0e-88c2-971c615ff83e","Type":"ContainerStarted","Data":"2fa9e7c1f7ae2aae1e93c61914abd902a4c4dafb5ab5f3779c5a86a439af4aa5"} Oct 30 13:07:38.047868 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:38.047385 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-9d45fb-p9ftj" event={"ID":"6df977ff-7e53-4d0e-88c2-971c615ff83e","Type":"ContainerStarted","Data":"8ac548c029390d0111d50cdbff26219b0fa2844220c126d1686f21ee52539518"} Oct 30 13:07:38.049049 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:38.049022 4908 generic.go:356] "Generic (PLEG): container finished" podID="435ef18d-77f4-4046-95ea-6cd0ce4b5fc6" containerID="df1efc93dd64d8f482d9f3a8595fb3ec68134d7cc9cebcf93ab4ab3ebdb458ba" exitCode=0 Oct 30 13:07:38.049132 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:38.049105 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-fddaa3-wnq5m" event={"ID":"435ef18d-77f4-4046-95ea-6cd0ce4b5fc6","Type":"ContainerDied","Data":"df1efc93dd64d8f482d9f3a8595fb3ec68134d7cc9cebcf93ab4ab3ebdb458ba"} Oct 30 13:07:38.092230 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:38.092155 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-9d45fb-p9ftj" podStartSLOduration=2.092134768 podStartE2EDuration="2.092134768s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:07:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:07:38.089975709 +0000 UTC m=+4414.604404474" watchObservedRunningTime="2026-10-30 13:07:38.092134768 +0000 UTC m=+4414.606563516" Oct 30 13:07:38.210221 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:38.210195 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-42220c-jdv8n" Oct 30 13:07:38.229611 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:38.229580 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d9e62368-725d-4e46-88b4-bd203a349d69-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9e62368-725d-4e46-88b4-bd203a349d69-maas-api-xks-ca\") pod \"d9e62368-725d-4e46-88b4-bd203a349d69\" (UID: \"d9e62368-725d-4e46-88b4-bd203a349d69\") " Oct 30 13:07:38.229768 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:38.229640 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9e62368-725d-4e46-88b4-bd203a349d69-kube-api-access-ht9r9\" (UniqueName: \"kubernetes.io/projected/d9e62368-725d-4e46-88b4-bd203a349d69-kube-api-access-ht9r9\") pod \"d9e62368-725d-4e46-88b4-bd203a349d69\" (UID: \"d9e62368-725d-4e46-88b4-bd203a349d69\") " Oct 30 13:07:38.231752 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:38.231727 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9e62368-725d-4e46-88b4-bd203a349d69-kube-api-access-ht9r9" pod "d9e62368-725d-4e46-88b4-bd203a349d69" (UID: "d9e62368-725d-4e46-88b4-bd203a349d69"). InnerVolumeSpecName "kube-api-access-ht9r9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:07:38.231882 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:38.231744 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9e62368-725d-4e46-88b4-bd203a349d69-maas-api-xks-ca" pod "d9e62368-725d-4e46-88b4-bd203a349d69" (UID: "d9e62368-725d-4e46-88b4-bd203a349d69"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:07:38.330407 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:38.330308 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9e62368-725d-4e46-88b4-bd203a349d69-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9e62368-725d-4e46-88b4-bd203a349d69-maas-api-openshift-ca\") pod \"d9e62368-725d-4e46-88b4-bd203a349d69\" (UID: \"d9e62368-725d-4e46-88b4-bd203a349d69\") " Oct 30 13:07:38.330686 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:38.330661 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9e62368-725d-4e46-88b4-bd203a349d69-maas-api-openshift-ca" pod "d9e62368-725d-4e46-88b4-bd203a349d69" (UID: "d9e62368-725d-4e46-88b4-bd203a349d69"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:07:38.330783 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:38.330681 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ht9r9\" (UniqueName: \"kubernetes.io/projected/d9e62368-725d-4e46-88b4-bd203a349d69-kube-api-access-ht9r9\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:07:38.330783 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:38.330707 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9e62368-725d-4e46-88b4-bd203a349d69-maas-api-xks-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:07:38.431194 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:38.431159 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9e62368-725d-4e46-88b4-bd203a349d69-maas-api-openshift-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:07:39.004928 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:39.004885 4908 patch_prober.go:29] interesting pod/e2e-extmod-a-10f364-openshift-default-965d4bfd5-4b57q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 13:07:39.005111 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:39.004956 4908 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-10f364-openshift-default-965d4bfd5-4b57q" podUID="aa19be03-336a-4af4-aba5-e089b4933bf9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 13:07:39.057728 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:39.057692 4908 generic.go:356] "Generic (PLEG): container finished" podID="6df977ff-7e53-4d0e-88c2-971c615ff83e" containerID="2fa9e7c1f7ae2aae1e93c61914abd902a4c4dafb5ab5f3779c5a86a439af4aa5" exitCode=0 Oct 30 13:07:39.058262 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:39.057778 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-9d45fb-p9ftj" event={"ID":"6df977ff-7e53-4d0e-88c2-971c615ff83e","Type":"ContainerDied","Data":"2fa9e7c1f7ae2aae1e93c61914abd902a4c4dafb5ab5f3779c5a86a439af4aa5"} Oct 30 13:07:39.059249 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:39.059226 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-42220c-jdv8n" Oct 30 13:07:39.059369 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:39.059257 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-42220c-jdv8n" event={"ID":"d9e62368-725d-4e46-88b4-bd203a349d69","Type":"ContainerDied","Data":"73fcf6d418ade8ab5e7657bc80e4e0dff11dfa947419d222b4633f88824cdf3a"} Oct 30 13:07:39.059369 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:39.059301 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="73fcf6d418ade8ab5e7657bc80e4e0dff11dfa947419d222b4633f88824cdf3a" Oct 30 13:07:39.233995 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:39.233966 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-fddaa3-wnq5m" Oct 30 13:07:39.238656 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:39.238625 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/435ef18d-77f4-4046-95ea-6cd0ce4b5fc6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/435ef18d-77f4-4046-95ea-6cd0ce4b5fc6-maas-api-openshift-ca\") pod \"435ef18d-77f4-4046-95ea-6cd0ce4b5fc6\" (UID: \"435ef18d-77f4-4046-95ea-6cd0ce4b5fc6\") " Oct 30 13:07:39.238876 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:39.238712 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/435ef18d-77f4-4046-95ea-6cd0ce4b5fc6-kube-api-access-kmc57\" (UniqueName: \"kubernetes.io/projected/435ef18d-77f4-4046-95ea-6cd0ce4b5fc6-kube-api-access-kmc57\") pod \"435ef18d-77f4-4046-95ea-6cd0ce4b5fc6\" (UID: \"435ef18d-77f4-4046-95ea-6cd0ce4b5fc6\") " Oct 30 13:07:39.238876 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:39.238759 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/435ef18d-77f4-4046-95ea-6cd0ce4b5fc6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/435ef18d-77f4-4046-95ea-6cd0ce4b5fc6-maas-api-xks-ca\") pod \"435ef18d-77f4-4046-95ea-6cd0ce4b5fc6\" (UID: \"435ef18d-77f4-4046-95ea-6cd0ce4b5fc6\") " Oct 30 13:07:39.239023 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:39.238985 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/435ef18d-77f4-4046-95ea-6cd0ce4b5fc6-maas-api-openshift-ca" pod "435ef18d-77f4-4046-95ea-6cd0ce4b5fc6" (UID: "435ef18d-77f4-4046-95ea-6cd0ce4b5fc6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:07:39.239162 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:39.239144 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/435ef18d-77f4-4046-95ea-6cd0ce4b5fc6-maas-api-openshift-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:07:39.241053 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:39.241026 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/435ef18d-77f4-4046-95ea-6cd0ce4b5fc6-kube-api-access-kmc57" pod "435ef18d-77f4-4046-95ea-6cd0ce4b5fc6" (UID: "435ef18d-77f4-4046-95ea-6cd0ce4b5fc6"). InnerVolumeSpecName "kube-api-access-kmc57". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:07:39.241173 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:39.241029 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/435ef18d-77f4-4046-95ea-6cd0ce4b5fc6-maas-api-xks-ca" pod "435ef18d-77f4-4046-95ea-6cd0ce4b5fc6" (UID: "435ef18d-77f4-4046-95ea-6cd0ce4b5fc6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:07:39.339972 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:39.339885 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/435ef18d-77f4-4046-95ea-6cd0ce4b5fc6-maas-api-xks-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:07:39.339972 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:39.339917 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kmc57\" (UniqueName: \"kubernetes.io/projected/435ef18d-77f4-4046-95ea-6cd0ce4b5fc6-kube-api-access-kmc57\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:07:40.071240 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:40.071210 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-fddaa3-wnq5m" Oct 30 13:07:40.071240 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:40.071226 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-fddaa3-wnq5m" event={"ID":"435ef18d-77f4-4046-95ea-6cd0ce4b5fc6","Type":"ContainerDied","Data":"03f90392a762dd9b746ff05363eda6dbe4691c7f44581b3538312111f0c018d0"} Oct 30 13:07:40.071749 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:40.071277 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="03f90392a762dd9b746ff05363eda6dbe4691c7f44581b3538312111f0c018d0" Oct 30 13:07:40.088273 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:40.088236 4908 generic.go:356] "Generic (PLEG): container finished" podID="aa19be03-336a-4af4-aba5-e089b4933bf9" containerID="0ee7336c32c90ca843b02e706f558154f455e05400d519702b8ca0f83f6b8058" exitCode=0 Oct 30 13:07:40.088508 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:40.088490 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-10f364-openshift-default-965d4bfd5-4b57q" event={"ID":"aa19be03-336a-4af4-aba5-e089b4933bf9","Type":"ContainerDied","Data":"0ee7336c32c90ca843b02e706f558154f455e05400d519702b8ca0f83f6b8058"} Oct 30 13:07:40.088577 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:40.088566 4908 scope.go:122] "RemoveContainer" containerID="240af9c35b77a33c0f8b0ff549be18a2e92490e84c801042f95eed7a94cdc084" Oct 30 13:07:40.136743 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:40.136716 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-10f364-openshift-default-965d4bfd5-4b57q" Oct 30 13:07:40.146668 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:40.146638 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa19be03-336a-4af4-aba5-e089b4933bf9-istio-token\" (UniqueName: \"kubernetes.io/projected/aa19be03-336a-4af4-aba5-e089b4933bf9-istio-token\") pod \"aa19be03-336a-4af4-aba5-e089b4933bf9\" (UID: \"aa19be03-336a-4af4-aba5-e089b4933bf9\") " Oct 30 13:07:40.146835 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:40.146697 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa19be03-336a-4af4-aba5-e089b4933bf9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa19be03-336a-4af4-aba5-e089b4933bf9-istio-envoy\") pod \"aa19be03-336a-4af4-aba5-e089b4933bf9\" (UID: \"aa19be03-336a-4af4-aba5-e089b4933bf9\") " Oct 30 13:07:40.146835 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:40.146727 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa19be03-336a-4af4-aba5-e089b4933bf9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa19be03-336a-4af4-aba5-e089b4933bf9-credential-socket\") pod \"aa19be03-336a-4af4-aba5-e089b4933bf9\" (UID: \"aa19be03-336a-4af4-aba5-e089b4933bf9\") " Oct 30 13:07:40.146835 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:40.146750 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa19be03-336a-4af4-aba5-e089b4933bf9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa19be03-336a-4af4-aba5-e089b4933bf9-istiod-ca-cert\") pod \"aa19be03-336a-4af4-aba5-e089b4933bf9\" (UID: \"aa19be03-336a-4af4-aba5-e089b4933bf9\") " Oct 30 13:07:40.146835 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:40.146813 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aa19be03-336a-4af4-aba5-e089b4933bf9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa19be03-336a-4af4-aba5-e089b4933bf9-istio-podinfo\") pod \"aa19be03-336a-4af4-aba5-e089b4933bf9\" (UID: \"aa19be03-336a-4af4-aba5-e089b4933bf9\") " Oct 30 13:07:40.147016 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:40.146986 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa19be03-336a-4af4-aba5-e089b4933bf9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa19be03-336a-4af4-aba5-e089b4933bf9-workload-certs\") pod \"aa19be03-336a-4af4-aba5-e089b4933bf9\" (UID: \"aa19be03-336a-4af4-aba5-e089b4933bf9\") " Oct 30 13:07:40.147068 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:40.147033 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa19be03-336a-4af4-aba5-e089b4933bf9-kube-api-access-k587c\" (UniqueName: \"kubernetes.io/projected/aa19be03-336a-4af4-aba5-e089b4933bf9-kube-api-access-k587c\") pod \"aa19be03-336a-4af4-aba5-e089b4933bf9\" (UID: \"aa19be03-336a-4af4-aba5-e089b4933bf9\") " Oct 30 13:07:40.147131 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:40.147079 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa19be03-336a-4af4-aba5-e089b4933bf9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa19be03-336a-4af4-aba5-e089b4933bf9-workload-socket\") pod \"aa19be03-336a-4af4-aba5-e089b4933bf9\" (UID: \"aa19be03-336a-4af4-aba5-e089b4933bf9\") " Oct 30 13:07:40.147192 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:40.147154 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa19be03-336a-4af4-aba5-e089b4933bf9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa19be03-336a-4af4-aba5-e089b4933bf9-istio-data\") pod \"aa19be03-336a-4af4-aba5-e089b4933bf9\" (UID: \"aa19be03-336a-4af4-aba5-e089b4933bf9\") " Oct 30 13:07:40.147399 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:40.147372 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa19be03-336a-4af4-aba5-e089b4933bf9-credential-socket" pod "aa19be03-336a-4af4-aba5-e089b4933bf9" (UID: "aa19be03-336a-4af4-aba5-e089b4933bf9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:07:40.147592 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:40.147566 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa19be03-336a-4af4-aba5-e089b4933bf9-workload-socket" pod "aa19be03-336a-4af4-aba5-e089b4933bf9" (UID: "aa19be03-336a-4af4-aba5-e089b4933bf9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:07:40.147720 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:40.147651 4908 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa19be03-336a-4af4-aba5-e089b4933bf9-credential-socket\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:07:40.147794 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:40.147771 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa19be03-336a-4af4-aba5-e089b4933bf9-workload-certs" pod "aa19be03-336a-4af4-aba5-e089b4933bf9" (UID: "aa19be03-336a-4af4-aba5-e089b4933bf9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:07:40.147886 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:40.147806 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa19be03-336a-4af4-aba5-e089b4933bf9-istio-data" pod "aa19be03-336a-4af4-aba5-e089b4933bf9" (UID: "aa19be03-336a-4af4-aba5-e089b4933bf9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:07:40.148191 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:40.148169 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa19be03-336a-4af4-aba5-e089b4933bf9-istiod-ca-cert" pod "aa19be03-336a-4af4-aba5-e089b4933bf9" (UID: "aa19be03-336a-4af4-aba5-e089b4933bf9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:07:40.149746 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:40.149683 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aa19be03-336a-4af4-aba5-e089b4933bf9-istio-podinfo" pod "aa19be03-336a-4af4-aba5-e089b4933bf9" (UID: "aa19be03-336a-4af4-aba5-e089b4933bf9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:07:40.149746 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:40.149704 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa19be03-336a-4af4-aba5-e089b4933bf9-kube-api-access-k587c" pod "aa19be03-336a-4af4-aba5-e089b4933bf9" (UID: "aa19be03-336a-4af4-aba5-e089b4933bf9"). InnerVolumeSpecName "kube-api-access-k587c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:07:40.150049 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:40.149888 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa19be03-336a-4af4-aba5-e089b4933bf9-istio-envoy" pod "aa19be03-336a-4af4-aba5-e089b4933bf9" (UID: "aa19be03-336a-4af4-aba5-e089b4933bf9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:07:40.150049 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:40.149965 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa19be03-336a-4af4-aba5-e089b4933bf9-istio-token" pod "aa19be03-336a-4af4-aba5-e089b4933bf9" (UID: "aa19be03-336a-4af4-aba5-e089b4933bf9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:07:40.248534 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:40.248504 4908 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa19be03-336a-4af4-aba5-e089b4933bf9-workload-certs\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:07:40.248534 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:40.248534 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k587c\" (UniqueName: \"kubernetes.io/projected/aa19be03-336a-4af4-aba5-e089b4933bf9-kube-api-access-k587c\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:07:40.248726 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:40.248545 4908 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa19be03-336a-4af4-aba5-e089b4933bf9-workload-socket\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:07:40.248726 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:40.248554 4908 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa19be03-336a-4af4-aba5-e089b4933bf9-istio-data\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:07:40.248726 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:40.248562 4908 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa19be03-336a-4af4-aba5-e089b4933bf9-istio-token\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:07:40.248726 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:40.248570 4908 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa19be03-336a-4af4-aba5-e089b4933bf9-istio-envoy\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:07:40.248726 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:40.248577 4908 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa19be03-336a-4af4-aba5-e089b4933bf9-istiod-ca-cert\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:07:40.248726 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:40.248585 4908 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa19be03-336a-4af4-aba5-e089b4933bf9-istio-podinfo\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:07:40.250526 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:40.250503 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-9d45fb-p9ftj" Oct 30 13:07:40.349889 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:40.349855 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6df977ff-7e53-4d0e-88c2-971c615ff83e-kube-api-access-pp4mb\" (UniqueName: \"kubernetes.io/projected/6df977ff-7e53-4d0e-88c2-971c615ff83e-kube-api-access-pp4mb\") pod \"6df977ff-7e53-4d0e-88c2-971c615ff83e\" (UID: \"6df977ff-7e53-4d0e-88c2-971c615ff83e\") " Oct 30 13:07:40.349889 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:40.349901 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6df977ff-7e53-4d0e-88c2-971c615ff83e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6df977ff-7e53-4d0e-88c2-971c615ff83e-maas-api-openshift-ca\") pod \"6df977ff-7e53-4d0e-88c2-971c615ff83e\" (UID: \"6df977ff-7e53-4d0e-88c2-971c615ff83e\") " Oct 30 13:07:40.350132 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:40.350040 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6df977ff-7e53-4d0e-88c2-971c615ff83e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6df977ff-7e53-4d0e-88c2-971c615ff83e-maas-api-xks-ca\") pod \"6df977ff-7e53-4d0e-88c2-971c615ff83e\" (UID: \"6df977ff-7e53-4d0e-88c2-971c615ff83e\") " Oct 30 13:07:40.350265 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:40.350246 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6df977ff-7e53-4d0e-88c2-971c615ff83e-maas-api-openshift-ca" pod "6df977ff-7e53-4d0e-88c2-971c615ff83e" (UID: "6df977ff-7e53-4d0e-88c2-971c615ff83e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:07:40.350415 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:40.350402 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6df977ff-7e53-4d0e-88c2-971c615ff83e-maas-api-openshift-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:07:40.352087 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:40.352061 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6df977ff-7e53-4d0e-88c2-971c615ff83e-maas-api-xks-ca" pod "6df977ff-7e53-4d0e-88c2-971c615ff83e" (UID: "6df977ff-7e53-4d0e-88c2-971c615ff83e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:07:40.352331 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:40.352304 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6df977ff-7e53-4d0e-88c2-971c615ff83e-kube-api-access-pp4mb" pod "6df977ff-7e53-4d0e-88c2-971c615ff83e" (UID: "6df977ff-7e53-4d0e-88c2-971c615ff83e"). InnerVolumeSpecName "kube-api-access-pp4mb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:07:40.451930 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:40.451805 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pp4mb\" (UniqueName: \"kubernetes.io/projected/6df977ff-7e53-4d0e-88c2-971c615ff83e-kube-api-access-pp4mb\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:07:40.451930 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:40.451865 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6df977ff-7e53-4d0e-88c2-971c615ff83e-maas-api-xks-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:07:41.098097 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:41.098064 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-10f364-openshift-default-965d4bfd5-4b57q" Oct 30 13:07:41.098600 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:41.098107 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-10f364-openshift-default-965d4bfd5-4b57q" event={"ID":"aa19be03-336a-4af4-aba5-e089b4933bf9","Type":"ContainerDied","Data":"247a7c751781c9cfb4fff10470dcca3e23dd7d58d500dd5348bd0f9d7e9998a1"} Oct 30 13:07:41.098600 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:41.098177 4908 scope.go:122] "RemoveContainer" containerID="0ee7336c32c90ca843b02e706f558154f455e05400d519702b8ca0f83f6b8058" Oct 30 13:07:41.099969 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:41.099942 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-9d45fb-p9ftj" event={"ID":"6df977ff-7e53-4d0e-88c2-971c615ff83e","Type":"ContainerDied","Data":"8ac548c029390d0111d50cdbff26219b0fa2844220c126d1686f21ee52539518"} Oct 30 13:07:41.100108 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:41.099974 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-9d45fb-p9ftj" Oct 30 13:07:41.100108 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:41.099991 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8ac548c029390d0111d50cdbff26219b0fa2844220c126d1686f21ee52539518" Oct 30 13:07:41.173790 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:41.173755 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-10f364-openshift-default-965d4bfd5-4b57q"] Oct 30 13:07:41.187006 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:41.186969 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-10f364-openshift-default-965d4bfd5-4b57q"] Oct 30 13:07:41.605363 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:41.605329 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa19be03-336a-4af4-aba5-e089b4933bf9" path="/var/lib/kubelet/pods/aa19be03-336a-4af4-aba5-e089b4933bf9/volumes" Oct 30 13:07:43.199045 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:43.199007 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f920f7-kfk24"] Oct 30 13:07:43.211805 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:43.211767 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f920f7-kfk24"] Oct 30 13:07:43.607065 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:43.607030 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d4f38b5-6492-406c-80e1-cbd779f9fd73" path="/var/lib/kubelet/pods/1d4f38b5-6492-406c-80e1-cbd779f9fd73/volumes" Oct 30 13:07:49.213285 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:49.213222 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-fddaa3-wnq5m"] Oct 30 13:07:49.225759 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:49.225717 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-fddaa3-wnq5m"] Oct 30 13:07:49.606579 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:49.606534 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="435ef18d-77f4-4046-95ea-6cd0ce4b5fc6" path="/var/lib/kubelet/pods/435ef18d-77f4-4046-95ea-6cd0ce4b5fc6/volumes" Oct 30 13:07:49.688214 ip-10-0-141-251 kubenswrapper[4908]: E1030 13:07:49.688168 4908 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vn6vv" containerName="extractor" Oct 30 13:07:50.684329 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:50.684287 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-tsxml"] Oct 30 13:07:50.684774 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:50.684544 4908 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-tsxml" podUID="c8dc0dee-2192-42cf-a386-1a02e5ceb4f0" containerName="istio-proxy" containerID="cri-o://e2e87b2daf616b9c7457fd765c2d51f7100f211197cd0787125d80dda02bbcdd" gracePeriod=30 Oct 30 13:07:51.089425 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:51.089385 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9wwfc"] Oct 30 13:07:51.477704 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:51.477663 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-42220c-jdv8n"] Oct 30 13:07:51.491241 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:51.491205 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-42220c-jdv8n"] Oct 30 13:07:51.613428 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:51.613385 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9e62368-725d-4e46-88b4-bd203a349d69" path="/var/lib/kubelet/pods/d9e62368-725d-4e46-88b4-bd203a349d69/volumes" Oct 30 13:07:55.955934 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:55.955864 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-tsxml" Oct 30 13:07:56.019538 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:56.019499 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8dc0dee-2192-42cf-a386-1a02e5ceb4f0-kube-api-access-tn9dn\" (UniqueName: \"kubernetes.io/projected/c8dc0dee-2192-42cf-a386-1a02e5ceb4f0-kube-api-access-tn9dn\") pod \"c8dc0dee-2192-42cf-a386-1a02e5ceb4f0\" (UID: \"c8dc0dee-2192-42cf-a386-1a02e5ceb4f0\") " Oct 30 13:07:56.019722 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:56.019569 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8dc0dee-2192-42cf-a386-1a02e5ceb4f0-istio-token\" (UniqueName: \"kubernetes.io/projected/c8dc0dee-2192-42cf-a386-1a02e5ceb4f0-istio-token\") pod \"c8dc0dee-2192-42cf-a386-1a02e5ceb4f0\" (UID: \"c8dc0dee-2192-42cf-a386-1a02e5ceb4f0\") " Oct 30 13:07:56.019722 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:56.019606 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8dc0dee-2192-42cf-a386-1a02e5ceb4f0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8dc0dee-2192-42cf-a386-1a02e5ceb4f0-workload-socket\") pod \"c8dc0dee-2192-42cf-a386-1a02e5ceb4f0\" (UID: \"c8dc0dee-2192-42cf-a386-1a02e5ceb4f0\") " Oct 30 13:07:56.019722 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:56.019655 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c8dc0dee-2192-42cf-a386-1a02e5ceb4f0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8dc0dee-2192-42cf-a386-1a02e5ceb4f0-istio-podinfo\") pod \"c8dc0dee-2192-42cf-a386-1a02e5ceb4f0\" (UID: \"c8dc0dee-2192-42cf-a386-1a02e5ceb4f0\") " Oct 30 13:07:56.019722 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:56.019681 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8dc0dee-2192-42cf-a386-1a02e5ceb4f0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8dc0dee-2192-42cf-a386-1a02e5ceb4f0-istiod-ca-cert\") pod \"c8dc0dee-2192-42cf-a386-1a02e5ceb4f0\" (UID: \"c8dc0dee-2192-42cf-a386-1a02e5ceb4f0\") " Oct 30 13:07:56.020017 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:56.019737 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8dc0dee-2192-42cf-a386-1a02e5ceb4f0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8dc0dee-2192-42cf-a386-1a02e5ceb4f0-istio-envoy\") pod \"c8dc0dee-2192-42cf-a386-1a02e5ceb4f0\" (UID: \"c8dc0dee-2192-42cf-a386-1a02e5ceb4f0\") " Oct 30 13:07:56.020017 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:56.019762 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8dc0dee-2192-42cf-a386-1a02e5ceb4f0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8dc0dee-2192-42cf-a386-1a02e5ceb4f0-workload-certs\") pod \"c8dc0dee-2192-42cf-a386-1a02e5ceb4f0\" (UID: \"c8dc0dee-2192-42cf-a386-1a02e5ceb4f0\") " Oct 30 13:07:56.020017 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:56.019797 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8dc0dee-2192-42cf-a386-1a02e5ceb4f0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8dc0dee-2192-42cf-a386-1a02e5ceb4f0-credential-socket\") pod \"c8dc0dee-2192-42cf-a386-1a02e5ceb4f0\" (UID: \"c8dc0dee-2192-42cf-a386-1a02e5ceb4f0\") " Oct 30 13:07:56.020017 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:56.019894 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8dc0dee-2192-42cf-a386-1a02e5ceb4f0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8dc0dee-2192-42cf-a386-1a02e5ceb4f0-istio-data\") pod \"c8dc0dee-2192-42cf-a386-1a02e5ceb4f0\" (UID: \"c8dc0dee-2192-42cf-a386-1a02e5ceb4f0\") " Oct 30 13:07:56.020017 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:56.019994 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8dc0dee-2192-42cf-a386-1a02e5ceb4f0-workload-socket" pod "c8dc0dee-2192-42cf-a386-1a02e5ceb4f0" (UID: "c8dc0dee-2192-42cf-a386-1a02e5ceb4f0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:07:56.020289 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:56.020193 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8dc0dee-2192-42cf-a386-1a02e5ceb4f0-istio-data" pod "c8dc0dee-2192-42cf-a386-1a02e5ceb4f0" (UID: "c8dc0dee-2192-42cf-a386-1a02e5ceb4f0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:07:56.020289 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:56.020254 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8dc0dee-2192-42cf-a386-1a02e5ceb4f0-istiod-ca-cert" pod "c8dc0dee-2192-42cf-a386-1a02e5ceb4f0" (UID: "c8dc0dee-2192-42cf-a386-1a02e5ceb4f0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:07:56.020402 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:56.020339 4908 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8dc0dee-2192-42cf-a386-1a02e5ceb4f0-workload-socket\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:07:56.020402 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:56.020391 4908 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8dc0dee-2192-42cf-a386-1a02e5ceb4f0-istio-data\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:07:56.020603 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:56.020405 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8dc0dee-2192-42cf-a386-1a02e5ceb4f0-workload-certs" pod "c8dc0dee-2192-42cf-a386-1a02e5ceb4f0" (UID: "c8dc0dee-2192-42cf-a386-1a02e5ceb4f0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:07:56.020603 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:56.020491 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8dc0dee-2192-42cf-a386-1a02e5ceb4f0-credential-socket" pod "c8dc0dee-2192-42cf-a386-1a02e5ceb4f0" (UID: "c8dc0dee-2192-42cf-a386-1a02e5ceb4f0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:07:56.022642 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:56.022590 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8dc0dee-2192-42cf-a386-1a02e5ceb4f0-kube-api-access-tn9dn" pod "c8dc0dee-2192-42cf-a386-1a02e5ceb4f0" (UID: "c8dc0dee-2192-42cf-a386-1a02e5ceb4f0"). InnerVolumeSpecName "kube-api-access-tn9dn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:07:56.022761 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:56.022714 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8dc0dee-2192-42cf-a386-1a02e5ceb4f0-istio-token" pod "c8dc0dee-2192-42cf-a386-1a02e5ceb4f0" (UID: "c8dc0dee-2192-42cf-a386-1a02e5ceb4f0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:07:56.022953 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:56.022915 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8dc0dee-2192-42cf-a386-1a02e5ceb4f0-istio-envoy" pod "c8dc0dee-2192-42cf-a386-1a02e5ceb4f0" (UID: "c8dc0dee-2192-42cf-a386-1a02e5ceb4f0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:07:56.023120 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:56.023042 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c8dc0dee-2192-42cf-a386-1a02e5ceb4f0-istio-podinfo" pod "c8dc0dee-2192-42cf-a386-1a02e5ceb4f0" (UID: "c8dc0dee-2192-42cf-a386-1a02e5ceb4f0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:07:56.121187 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:56.121147 4908 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8dc0dee-2192-42cf-a386-1a02e5ceb4f0-istio-envoy\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:07:56.121187 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:56.121181 4908 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8dc0dee-2192-42cf-a386-1a02e5ceb4f0-workload-certs\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:07:56.121187 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:56.121196 4908 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8dc0dee-2192-42cf-a386-1a02e5ceb4f0-credential-socket\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:07:56.121458 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:56.121212 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tn9dn\" (UniqueName: \"kubernetes.io/projected/c8dc0dee-2192-42cf-a386-1a02e5ceb4f0-kube-api-access-tn9dn\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:07:56.121458 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:56.121224 4908 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8dc0dee-2192-42cf-a386-1a02e5ceb4f0-istio-token\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:07:56.121458 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:56.121237 4908 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8dc0dee-2192-42cf-a386-1a02e5ceb4f0-istio-podinfo\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:07:56.121458 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:56.121249 4908 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8dc0dee-2192-42cf-a386-1a02e5ceb4f0-istiod-ca-cert\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:07:56.225261 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:56.225170 4908 generic.go:356] "Generic (PLEG): container finished" podID="c8dc0dee-2192-42cf-a386-1a02e5ceb4f0" containerID="e2e87b2daf616b9c7457fd765c2d51f7100f211197cd0787125d80dda02bbcdd" exitCode=0 Oct 30 13:07:56.225261 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:56.225235 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-tsxml" Oct 30 13:07:56.225507 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:56.225257 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-tsxml" event={"ID":"c8dc0dee-2192-42cf-a386-1a02e5ceb4f0","Type":"ContainerDied","Data":"e2e87b2daf616b9c7457fd765c2d51f7100f211197cd0787125d80dda02bbcdd"} Oct 30 13:07:56.225507 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:56.225315 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-tsxml" event={"ID":"c8dc0dee-2192-42cf-a386-1a02e5ceb4f0","Type":"ContainerDied","Data":"69088e1dee7a137067227771a770bd6ebc51c24d3ed0d1b3acff2174691c16cf"} Oct 30 13:07:56.225507 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:56.225339 4908 scope.go:122] "RemoveContainer" containerID="e2e87b2daf616b9c7457fd765c2d51f7100f211197cd0787125d80dda02bbcdd" Oct 30 13:07:56.236843 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:56.236790 4908 scope.go:122] "RemoveContainer" containerID="e2e87b2daf616b9c7457fd765c2d51f7100f211197cd0787125d80dda02bbcdd" Oct 30 13:07:56.237133 ip-10-0-141-251 kubenswrapper[4908]: E1030 13:07:56.237106 4908 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2e87b2daf616b9c7457fd765c2d51f7100f211197cd0787125d80dda02bbcdd\": container with ID starting with e2e87b2daf616b9c7457fd765c2d51f7100f211197cd0787125d80dda02bbcdd not found: ID does not exist" containerID="e2e87b2daf616b9c7457fd765c2d51f7100f211197cd0787125d80dda02bbcdd" Oct 30 13:07:56.237241 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:56.237137 4908 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2e87b2daf616b9c7457fd765c2d51f7100f211197cd0787125d80dda02bbcdd"} err="failed to get container status \"e2e87b2daf616b9c7457fd765c2d51f7100f211197cd0787125d80dda02bbcdd\": rpc error: code = NotFound desc = could not find container \"e2e87b2daf616b9c7457fd765c2d51f7100f211197cd0787125d80dda02bbcdd\": container with ID starting with e2e87b2daf616b9c7457fd765c2d51f7100f211197cd0787125d80dda02bbcdd not found: ID does not exist" Oct 30 13:07:56.285803 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:56.285758 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-tsxml"] Oct 30 13:07:56.288501 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:56.288473 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-tsxml"] Oct 30 13:07:56.922134 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:56.922091 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-fddaa3-openshift-default-668fdb7bf-z4588"] Oct 30 13:07:56.922333 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:56.922306 4908 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-fddaa3-openshift-default-668fdb7bf-z4588" podUID="ab9d645e-4df1-482d-9ee8-54947745d8b4" containerName="istio-proxy" containerID="cri-o://a949943904a4dac9925427a479fe7bae18f9fd78de78852aea359e7d10af2150" gracePeriod=30 Oct 30 13:07:57.514494 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:57.514448 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-327e31-rlczv"] Oct 30 13:07:57.515135 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:57.515110 4908 cpu_manager.go:414] "containerMap: removing container" podUID="d9e62368-725d-4e46-88b4-bd203a349d69" containerName="revoke-keys" Oct 30 13:07:57.515241 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:57.515136 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9e62368-725d-4e46-88b4-bd203a349d69" containerName="revoke-keys" Oct 30 13:07:57.515241 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:57.515151 4908 cpu_manager.go:414] "containerMap: removing container" podUID="aa19be03-336a-4af4-aba5-e089b4933bf9" containerName="istio-proxy" Oct 30 13:07:57.515241 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:57.515158 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa19be03-336a-4af4-aba5-e089b4933bf9" containerName="istio-proxy" Oct 30 13:07:57.515241 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:57.515180 4908 cpu_manager.go:414] "containerMap: removing container" podUID="c8dc0dee-2192-42cf-a386-1a02e5ceb4f0" containerName="istio-proxy" Oct 30 13:07:57.515241 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:57.515188 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8dc0dee-2192-42cf-a386-1a02e5ceb4f0" containerName="istio-proxy" Oct 30 13:07:57.515241 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:57.515201 4908 cpu_manager.go:414] "containerMap: removing container" podUID="6df977ff-7e53-4d0e-88c2-971c615ff83e" containerName="revoke-keys" Oct 30 13:07:57.515241 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:57.515209 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6df977ff-7e53-4d0e-88c2-971c615ff83e" containerName="revoke-keys" Oct 30 13:07:57.515241 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:57.515236 4908 cpu_manager.go:414] "containerMap: removing container" podUID="435ef18d-77f4-4046-95ea-6cd0ce4b5fc6" containerName="revoke-keys" Oct 30 13:07:57.515730 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:57.515243 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="435ef18d-77f4-4046-95ea-6cd0ce4b5fc6" containerName="revoke-keys" Oct 30 13:07:57.515730 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:57.515361 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9e62368-725d-4e46-88b4-bd203a349d69" containerName="revoke-keys" Oct 30 13:07:57.515730 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:57.515376 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa19be03-336a-4af4-aba5-e089b4933bf9" containerName="istio-proxy" Oct 30 13:07:57.515730 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:57.515388 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8dc0dee-2192-42cf-a386-1a02e5ceb4f0" containerName="istio-proxy" Oct 30 13:07:57.515730 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:57.515397 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="6df977ff-7e53-4d0e-88c2-971c615ff83e" containerName="revoke-keys" Oct 30 13:07:57.515730 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:57.515410 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="435ef18d-77f4-4046-95ea-6cd0ce4b5fc6" containerName="revoke-keys" Oct 30 13:07:57.520462 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:57.520437 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-327e31-rlczv" Oct 30 13:07:57.549792 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:57.549750 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 13:07:57.549997 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:57.549863 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 13:07:57.553247 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:57.553215 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-327e31-rlczv"] Oct 30 13:07:57.607106 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:57.607062 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8dc0dee-2192-42cf-a386-1a02e5ceb4f0" path="/var/lib/kubelet/pods/c8dc0dee-2192-42cf-a386-1a02e5ceb4f0/volumes" Oct 30 13:07:57.628298 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:57.628272 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:07:57.628443 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:57.628337 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s75hq\"" Oct 30 13:07:57.637739 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:57.637712 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsh55\" (UniqueName: \"kubernetes.io/projected/69ffd6a9-d5c5-4f6c-8188-00cfa376d676-kube-api-access-gsh55\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-327e31-rlczv\" (UID: \"69ffd6a9-d5c5-4f6c-8188-00cfa376d676\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-327e31-rlczv" Oct 30 13:07:57.637853 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:57.637774 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69ffd6a9-d5c5-4f6c-8188-00cfa376d676-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-327e31-rlczv\" (UID: \"69ffd6a9-d5c5-4f6c-8188-00cfa376d676\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-327e31-rlczv" Oct 30 13:07:57.637958 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:57.637937 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69ffd6a9-d5c5-4f6c-8188-00cfa376d676-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-327e31-rlczv\" (UID: \"69ffd6a9-d5c5-4f6c-8188-00cfa376d676\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-327e31-rlczv" Oct 30 13:07:57.739470 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:57.739432 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsh55\" (UniqueName: \"kubernetes.io/projected/69ffd6a9-d5c5-4f6c-8188-00cfa376d676-kube-api-access-gsh55\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-327e31-rlczv\" (UID: \"69ffd6a9-d5c5-4f6c-8188-00cfa376d676\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-327e31-rlczv" Oct 30 13:07:57.739660 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:57.739530 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69ffd6a9-d5c5-4f6c-8188-00cfa376d676-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-327e31-rlczv\" (UID: \"69ffd6a9-d5c5-4f6c-8188-00cfa376d676\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-327e31-rlczv" Oct 30 13:07:57.739660 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:57.739597 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69ffd6a9-d5c5-4f6c-8188-00cfa376d676-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-327e31-rlczv\" (UID: \"69ffd6a9-d5c5-4f6c-8188-00cfa376d676\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-327e31-rlczv" Oct 30 13:07:57.740359 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:57.740336 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69ffd6a9-d5c5-4f6c-8188-00cfa376d676-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-327e31-rlczv\" (UID: \"69ffd6a9-d5c5-4f6c-8188-00cfa376d676\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-327e31-rlczv" Oct 30 13:07:57.742021 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:57.742004 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69ffd6a9-d5c5-4f6c-8188-00cfa376d676-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-327e31-rlczv\" (UID: \"69ffd6a9-d5c5-4f6c-8188-00cfa376d676\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-327e31-rlczv" Oct 30 13:07:57.764863 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:57.764777 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsh55\" (UniqueName: \"kubernetes.io/projected/69ffd6a9-d5c5-4f6c-8188-00cfa376d676-kube-api-access-gsh55\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-327e31-rlczv\" (UID: \"69ffd6a9-d5c5-4f6c-8188-00cfa376d676\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-327e31-rlczv" Oct 30 13:07:57.836282 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:57.836242 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-327e31-rlczv" Oct 30 13:07:58.148537 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:58.148505 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-327e31-rlczv"] Oct 30 13:07:58.149215 ip-10-0-141-251 kubenswrapper[4908]: W1030 13:07:58.149189 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69ffd6a9_d5c5_4f6c_8188_00cfa376d676.slice/crio-c4d9fc6e3558061ee5bef56678121e1c515698127dcf534f5f238c772effd6f5 WatchSource:0}: Error finding container c4d9fc6e3558061ee5bef56678121e1c515698127dcf534f5f238c772effd6f5: Status 404 returned error can't find the container with id c4d9fc6e3558061ee5bef56678121e1c515698127dcf534f5f238c772effd6f5 Oct 30 13:07:58.244897 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:58.244858 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-327e31-rlczv" event={"ID":"69ffd6a9-d5c5-4f6c-8188-00cfa376d676","Type":"ContainerStarted","Data":"c4d9fc6e3558061ee5bef56678121e1c515698127dcf534f5f238c772effd6f5"} Oct 30 13:07:59.256397 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:59.256299 4908 generic.go:356] "Generic (PLEG): container finished" podID="69ffd6a9-d5c5-4f6c-8188-00cfa376d676" containerID="98e6ac0ef0482f2fce66e2c74ead0f636603d72253f84ca81a6241ffd7f3ff26" exitCode=0 Oct 30 13:07:59.256397 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:07:59.256386 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-327e31-rlczv" event={"ID":"69ffd6a9-d5c5-4f6c-8188-00cfa376d676","Type":"ContainerDied","Data":"98e6ac0ef0482f2fce66e2c74ead0f636603d72253f84ca81a6241ffd7f3ff26"} Oct 30 13:08:00.478272 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:00.478240 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-327e31-rlczv" Oct 30 13:08:00.569905 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:00.569312 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69ffd6a9-d5c5-4f6c-8188-00cfa376d676-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69ffd6a9-d5c5-4f6c-8188-00cfa376d676-maas-api-openshift-ca\") pod \"69ffd6a9-d5c5-4f6c-8188-00cfa376d676\" (UID: \"69ffd6a9-d5c5-4f6c-8188-00cfa376d676\") " Oct 30 13:08:00.570268 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:00.570244 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69ffd6a9-d5c5-4f6c-8188-00cfa376d676-kube-api-access-gsh55\" (UniqueName: \"kubernetes.io/projected/69ffd6a9-d5c5-4f6c-8188-00cfa376d676-kube-api-access-gsh55\") pod \"69ffd6a9-d5c5-4f6c-8188-00cfa376d676\" (UID: \"69ffd6a9-d5c5-4f6c-8188-00cfa376d676\") " Oct 30 13:08:00.570859 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:00.569780 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69ffd6a9-d5c5-4f6c-8188-00cfa376d676-maas-api-openshift-ca" pod "69ffd6a9-d5c5-4f6c-8188-00cfa376d676" (UID: "69ffd6a9-d5c5-4f6c-8188-00cfa376d676"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:08:00.571077 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:00.571055 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/69ffd6a9-d5c5-4f6c-8188-00cfa376d676-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69ffd6a9-d5c5-4f6c-8188-00cfa376d676-maas-api-xks-ca\") pod \"69ffd6a9-d5c5-4f6c-8188-00cfa376d676\" (UID: \"69ffd6a9-d5c5-4f6c-8188-00cfa376d676\") " Oct 30 13:08:00.571976 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:00.571956 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69ffd6a9-d5c5-4f6c-8188-00cfa376d676-maas-api-openshift-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:08:00.582169 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:00.573183 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69ffd6a9-d5c5-4f6c-8188-00cfa376d676-kube-api-access-gsh55" pod "69ffd6a9-d5c5-4f6c-8188-00cfa376d676" (UID: "69ffd6a9-d5c5-4f6c-8188-00cfa376d676"). InnerVolumeSpecName "kube-api-access-gsh55". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:08:00.586610 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:00.586339 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69ffd6a9-d5c5-4f6c-8188-00cfa376d676-maas-api-xks-ca" pod "69ffd6a9-d5c5-4f6c-8188-00cfa376d676" (UID: "69ffd6a9-d5c5-4f6c-8188-00cfa376d676"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:08:00.673941 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:00.673864 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gsh55\" (UniqueName: \"kubernetes.io/projected/69ffd6a9-d5c5-4f6c-8188-00cfa376d676-kube-api-access-gsh55\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:08:00.673941 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:00.673910 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69ffd6a9-d5c5-4f6c-8188-00cfa376d676-maas-api-xks-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:08:01.178716 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:01.178675 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-9d45fb-p9ftj"] Oct 30 13:08:01.187541 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:01.187505 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-9d45fb-p9ftj"] Oct 30 13:08:01.283994 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:01.283961 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-327e31-rlczv" Oct 30 13:08:01.284184 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:01.283991 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-327e31-rlczv" event={"ID":"69ffd6a9-d5c5-4f6c-8188-00cfa376d676","Type":"ContainerDied","Data":"c4d9fc6e3558061ee5bef56678121e1c515698127dcf534f5f238c772effd6f5"} Oct 30 13:08:01.284184 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:01.284046 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c4d9fc6e3558061ee5bef56678121e1c515698127dcf534f5f238c772effd6f5" Oct 30 13:08:01.300171 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:01.300134 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-47fd99-nwtr6"] Oct 30 13:08:01.300588 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:01.300572 4908 cpu_manager.go:414] "containerMap: removing container" podUID="69ffd6a9-d5c5-4f6c-8188-00cfa376d676" containerName="revoke-keys" Oct 30 13:08:01.300674 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:01.300590 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69ffd6a9-d5c5-4f6c-8188-00cfa376d676" containerName="revoke-keys" Oct 30 13:08:01.300674 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:01.300617 4908 cpu_manager.go:414] "containerMap: removing container" podUID="aa19be03-336a-4af4-aba5-e089b4933bf9" containerName="istio-proxy" Oct 30 13:08:01.300674 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:01.300623 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa19be03-336a-4af4-aba5-e089b4933bf9" containerName="istio-proxy" Oct 30 13:08:01.300861 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:01.300692 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="69ffd6a9-d5c5-4f6c-8188-00cfa376d676" containerName="revoke-keys" Oct 30 13:08:01.300861 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:01.300706 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa19be03-336a-4af4-aba5-e089b4933bf9" containerName="istio-proxy" Oct 30 13:08:01.304040 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:01.304020 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-47fd99-nwtr6" Oct 30 13:08:01.322587 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:01.322558 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 13:08:01.322587 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:01.322578 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 13:08:01.322781 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:01.322567 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:08:01.322781 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:01.322765 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s75hq\"" Oct 30 13:08:01.326017 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:01.325991 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-47fd99-nwtr6"] Oct 30 13:08:01.383866 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:01.383815 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bql5\" (UniqueName: \"kubernetes.io/projected/e5e36bcb-b201-4c99-be6c-1d632ba46503-kube-api-access-9bql5\") pod \"maas-api-revoke-keys-e2e-worker-w2-47fd99-nwtr6\" (UID: \"e5e36bcb-b201-4c99-be6c-1d632ba46503\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-47fd99-nwtr6" Oct 30 13:08:01.384045 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:01.384012 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5e36bcb-b201-4c99-be6c-1d632ba46503-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-47fd99-nwtr6\" (UID: \"e5e36bcb-b201-4c99-be6c-1d632ba46503\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-47fd99-nwtr6" Oct 30 13:08:01.384097 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:01.384055 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5e36bcb-b201-4c99-be6c-1d632ba46503-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-47fd99-nwtr6\" (UID: \"e5e36bcb-b201-4c99-be6c-1d632ba46503\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-47fd99-nwtr6" Oct 30 13:08:01.485362 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:01.485320 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5e36bcb-b201-4c99-be6c-1d632ba46503-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-47fd99-nwtr6\" (UID: \"e5e36bcb-b201-4c99-be6c-1d632ba46503\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-47fd99-nwtr6" Oct 30 13:08:01.485362 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:01.485366 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5e36bcb-b201-4c99-be6c-1d632ba46503-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-47fd99-nwtr6\" (UID: \"e5e36bcb-b201-4c99-be6c-1d632ba46503\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-47fd99-nwtr6" Oct 30 13:08:01.485955 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:01.485484 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bql5\" (UniqueName: \"kubernetes.io/projected/e5e36bcb-b201-4c99-be6c-1d632ba46503-kube-api-access-9bql5\") pod \"maas-api-revoke-keys-e2e-worker-w2-47fd99-nwtr6\" (UID: \"e5e36bcb-b201-4c99-be6c-1d632ba46503\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-47fd99-nwtr6" Oct 30 13:08:01.486055 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:01.486031 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5e36bcb-b201-4c99-be6c-1d632ba46503-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-47fd99-nwtr6\" (UID: \"e5e36bcb-b201-4c99-be6c-1d632ba46503\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-47fd99-nwtr6" Oct 30 13:08:01.487890 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:01.487867 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5e36bcb-b201-4c99-be6c-1d632ba46503-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-47fd99-nwtr6\" (UID: \"e5e36bcb-b201-4c99-be6c-1d632ba46503\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-47fd99-nwtr6" Oct 30 13:08:01.510236 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:01.510205 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bql5\" (UniqueName: \"kubernetes.io/projected/e5e36bcb-b201-4c99-be6c-1d632ba46503-kube-api-access-9bql5\") pod \"maas-api-revoke-keys-e2e-worker-w2-47fd99-nwtr6\" (UID: \"e5e36bcb-b201-4c99-be6c-1d632ba46503\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-47fd99-nwtr6" Oct 30 13:08:01.606396 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:01.606364 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6df977ff-7e53-4d0e-88c2-971c615ff83e" path="/var/lib/kubelet/pods/6df977ff-7e53-4d0e-88c2-971c615ff83e/volumes" Oct 30 13:08:01.618174 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:01.618144 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-47fd99-nwtr6" Oct 30 13:08:01.923988 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:01.923948 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-47fd99-nwtr6"] Oct 30 13:08:01.924764 ip-10-0-141-251 kubenswrapper[4908]: W1030 13:08:01.924735 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5e36bcb_b201_4c99_be6c_1d632ba46503.slice/crio-f5eab60f97317c7719c67cfcc8d732b66afcb70278143ffed897bb61b93418fe WatchSource:0}: Error finding container f5eab60f97317c7719c67cfcc8d732b66afcb70278143ffed897bb61b93418fe: Status 404 returned error can't find the container with id f5eab60f97317c7719c67cfcc8d732b66afcb70278143ffed897bb61b93418fe Oct 30 13:08:02.167686 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:02.167654 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-fddaa3-openshift-default-668fdb7bf-z4588" Oct 30 13:08:02.294928 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:02.294896 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab9d645e-4df1-482d-9ee8-54947745d8b4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab9d645e-4df1-482d-9ee8-54947745d8b4-istio-data\") pod \"ab9d645e-4df1-482d-9ee8-54947745d8b4\" (UID: \"ab9d645e-4df1-482d-9ee8-54947745d8b4\") " Oct 30 13:08:02.295100 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:02.294976 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab9d645e-4df1-482d-9ee8-54947745d8b4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab9d645e-4df1-482d-9ee8-54947745d8b4-workload-certs\") pod \"ab9d645e-4df1-482d-9ee8-54947745d8b4\" (UID: \"ab9d645e-4df1-482d-9ee8-54947745d8b4\") " Oct 30 13:08:02.295100 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:02.295041 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab9d645e-4df1-482d-9ee8-54947745d8b4-kube-api-access-dchfm\" (UniqueName: \"kubernetes.io/projected/ab9d645e-4df1-482d-9ee8-54947745d8b4-kube-api-access-dchfm\") pod \"ab9d645e-4df1-482d-9ee8-54947745d8b4\" (UID: \"ab9d645e-4df1-482d-9ee8-54947745d8b4\") " Oct 30 13:08:02.295100 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:02.295073 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab9d645e-4df1-482d-9ee8-54947745d8b4-istio-token\" (UniqueName: \"kubernetes.io/projected/ab9d645e-4df1-482d-9ee8-54947745d8b4-istio-token\") pod \"ab9d645e-4df1-482d-9ee8-54947745d8b4\" (UID: \"ab9d645e-4df1-482d-9ee8-54947745d8b4\") " Oct 30 13:08:02.295293 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:02.295190 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab9d645e-4df1-482d-9ee8-54947745d8b4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab9d645e-4df1-482d-9ee8-54947745d8b4-istiod-ca-cert\") pod \"ab9d645e-4df1-482d-9ee8-54947745d8b4\" (UID: \"ab9d645e-4df1-482d-9ee8-54947745d8b4\") " Oct 30 13:08:02.295293 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:02.295213 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab9d645e-4df1-482d-9ee8-54947745d8b4-istio-data" pod "ab9d645e-4df1-482d-9ee8-54947745d8b4" (UID: "ab9d645e-4df1-482d-9ee8-54947745d8b4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:08:02.295293 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:02.295229 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab9d645e-4df1-482d-9ee8-54947745d8b4-workload-certs" pod "ab9d645e-4df1-482d-9ee8-54947745d8b4" (UID: "ab9d645e-4df1-482d-9ee8-54947745d8b4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:08:02.295293 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:02.295225 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab9d645e-4df1-482d-9ee8-54947745d8b4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab9d645e-4df1-482d-9ee8-54947745d8b4-credential-socket\") pod \"ab9d645e-4df1-482d-9ee8-54947745d8b4\" (UID: \"ab9d645e-4df1-482d-9ee8-54947745d8b4\") " Oct 30 13:08:02.295539 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:02.295305 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab9d645e-4df1-482d-9ee8-54947745d8b4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab9d645e-4df1-482d-9ee8-54947745d8b4-istio-envoy\") pod \"ab9d645e-4df1-482d-9ee8-54947745d8b4\" (UID: \"ab9d645e-4df1-482d-9ee8-54947745d8b4\") " Oct 30 13:08:02.295539 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:02.295337 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab9d645e-4df1-482d-9ee8-54947745d8b4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab9d645e-4df1-482d-9ee8-54947745d8b4-workload-socket\") pod \"ab9d645e-4df1-482d-9ee8-54947745d8b4\" (UID: \"ab9d645e-4df1-482d-9ee8-54947745d8b4\") " Oct 30 13:08:02.295539 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:02.295372 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ab9d645e-4df1-482d-9ee8-54947745d8b4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab9d645e-4df1-482d-9ee8-54947745d8b4-istio-podinfo\") pod \"ab9d645e-4df1-482d-9ee8-54947745d8b4\" (UID: \"ab9d645e-4df1-482d-9ee8-54947745d8b4\") " Oct 30 13:08:02.295539 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:02.295474 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab9d645e-4df1-482d-9ee8-54947745d8b4-credential-socket" pod "ab9d645e-4df1-482d-9ee8-54947745d8b4" (UID: "ab9d645e-4df1-482d-9ee8-54947745d8b4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:08:02.295859 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:02.295578 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab9d645e-4df1-482d-9ee8-54947745d8b4-istiod-ca-cert" pod "ab9d645e-4df1-482d-9ee8-54947745d8b4" (UID: "ab9d645e-4df1-482d-9ee8-54947745d8b4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:08:02.295859 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:02.295694 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab9d645e-4df1-482d-9ee8-54947745d8b4-workload-socket" pod "ab9d645e-4df1-482d-9ee8-54947745d8b4" (UID: "ab9d645e-4df1-482d-9ee8-54947745d8b4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:08:02.296000 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:02.295929 4908 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab9d645e-4df1-482d-9ee8-54947745d8b4-istiod-ca-cert\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:08:02.296000 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:02.295948 4908 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab9d645e-4df1-482d-9ee8-54947745d8b4-credential-socket\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:08:02.296000 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:02.295962 4908 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab9d645e-4df1-482d-9ee8-54947745d8b4-workload-socket\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:08:02.296000 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:02.295975 4908 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab9d645e-4df1-482d-9ee8-54947745d8b4-istio-data\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:08:02.296000 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:02.295989 4908 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab9d645e-4df1-482d-9ee8-54947745d8b4-workload-certs\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:08:02.296287 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:02.296261 4908 generic.go:356] "Generic (PLEG): container finished" podID="ab9d645e-4df1-482d-9ee8-54947745d8b4" containerID="a949943904a4dac9925427a479fe7bae18f9fd78de78852aea359e7d10af2150" exitCode=0 Oct 30 13:08:02.296350 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:02.296333 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-fddaa3-openshift-default-668fdb7bf-z4588" event={"ID":"ab9d645e-4df1-482d-9ee8-54947745d8b4","Type":"ContainerDied","Data":"a949943904a4dac9925427a479fe7bae18f9fd78de78852aea359e7d10af2150"} Oct 30 13:08:02.296414 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:02.296385 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-fddaa3-openshift-default-668fdb7bf-z4588" event={"ID":"ab9d645e-4df1-482d-9ee8-54947745d8b4","Type":"ContainerDied","Data":"54162980ea7dc99721dcbc57c5fa3de5fc46b51a99e2f298f309cea9306fdccf"} Oct 30 13:08:02.296477 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:02.296414 4908 scope.go:122] "RemoveContainer" containerID="a949943904a4dac9925427a479fe7bae18f9fd78de78852aea359e7d10af2150" Oct 30 13:08:02.296599 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:02.296580 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-fddaa3-openshift-default-668fdb7bf-z4588" Oct 30 13:08:02.297965 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:02.297883 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab9d645e-4df1-482d-9ee8-54947745d8b4-istio-envoy" pod "ab9d645e-4df1-482d-9ee8-54947745d8b4" (UID: "ab9d645e-4df1-482d-9ee8-54947745d8b4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:08:02.297965 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:02.297902 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab9d645e-4df1-482d-9ee8-54947745d8b4-istio-token" pod "ab9d645e-4df1-482d-9ee8-54947745d8b4" (UID: "ab9d645e-4df1-482d-9ee8-54947745d8b4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:08:02.298317 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:02.298244 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ab9d645e-4df1-482d-9ee8-54947745d8b4-istio-podinfo" pod "ab9d645e-4df1-482d-9ee8-54947745d8b4" (UID: "ab9d645e-4df1-482d-9ee8-54947745d8b4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:08:02.298673 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:02.298648 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab9d645e-4df1-482d-9ee8-54947745d8b4-kube-api-access-dchfm" pod "ab9d645e-4df1-482d-9ee8-54947745d8b4" (UID: "ab9d645e-4df1-482d-9ee8-54947745d8b4"). InnerVolumeSpecName "kube-api-access-dchfm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:08:02.299420 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:02.299391 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-47fd99-nwtr6" event={"ID":"e5e36bcb-b201-4c99-be6c-1d632ba46503","Type":"ContainerStarted","Data":"2f263d48ab994080b5441081c9fe54a8b285dc65d85f8b0ad3c5170dba81a4d9"} Oct 30 13:08:02.299538 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:02.299445 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-47fd99-nwtr6" event={"ID":"e5e36bcb-b201-4c99-be6c-1d632ba46503","Type":"ContainerStarted","Data":"f5eab60f97317c7719c67cfcc8d732b66afcb70278143ffed897bb61b93418fe"} Oct 30 13:08:02.308739 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:02.308716 4908 scope.go:122] "RemoveContainer" containerID="b4b52188906f02662aa52c0d5c332567b35ee8b165722696d6808bcc37db32ec" Oct 30 13:08:02.322950 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:02.322929 4908 scope.go:122] "RemoveContainer" containerID="a949943904a4dac9925427a479fe7bae18f9fd78de78852aea359e7d10af2150" Oct 30 13:08:02.323280 ip-10-0-141-251 kubenswrapper[4908]: E1030 13:08:02.323257 4908 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a949943904a4dac9925427a479fe7bae18f9fd78de78852aea359e7d10af2150\": container with ID starting with a949943904a4dac9925427a479fe7bae18f9fd78de78852aea359e7d10af2150 not found: ID does not exist" containerID="a949943904a4dac9925427a479fe7bae18f9fd78de78852aea359e7d10af2150" Oct 30 13:08:02.323356 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:02.323292 4908 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a949943904a4dac9925427a479fe7bae18f9fd78de78852aea359e7d10af2150"} err="failed to get container status \"a949943904a4dac9925427a479fe7bae18f9fd78de78852aea359e7d10af2150\": rpc error: code = NotFound desc = could not find container \"a949943904a4dac9925427a479fe7bae18f9fd78de78852aea359e7d10af2150\": container with ID starting with a949943904a4dac9925427a479fe7bae18f9fd78de78852aea359e7d10af2150 not found: ID does not exist" Oct 30 13:08:02.323356 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:02.323311 4908 scope.go:122] "RemoveContainer" containerID="b4b52188906f02662aa52c0d5c332567b35ee8b165722696d6808bcc37db32ec" Oct 30 13:08:02.323603 ip-10-0-141-251 kubenswrapper[4908]: E1030 13:08:02.323581 4908 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4b52188906f02662aa52c0d5c332567b35ee8b165722696d6808bcc37db32ec\": container with ID starting with b4b52188906f02662aa52c0d5c332567b35ee8b165722696d6808bcc37db32ec not found: ID does not exist" containerID="b4b52188906f02662aa52c0d5c332567b35ee8b165722696d6808bcc37db32ec" Oct 30 13:08:02.323664 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:02.323610 4908 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4b52188906f02662aa52c0d5c332567b35ee8b165722696d6808bcc37db32ec"} err="failed to get container status \"b4b52188906f02662aa52c0d5c332567b35ee8b165722696d6808bcc37db32ec\": rpc error: code = NotFound desc = could not find container \"b4b52188906f02662aa52c0d5c332567b35ee8b165722696d6808bcc37db32ec\": container with ID starting with b4b52188906f02662aa52c0d5c332567b35ee8b165722696d6808bcc37db32ec not found: ID does not exist" Oct 30 13:08:02.367320 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:02.367265 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-47fd99-nwtr6" podStartSLOduration=1.3672494130000001 podStartE2EDuration="1.367249413s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:08:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:08:02.363520876 +0000 UTC m=+4438.877949623" watchObservedRunningTime="2026-10-30 13:08:02.367249413 +0000 UTC m=+4438.881678159" Oct 30 13:08:02.396778 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:02.396745 4908 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab9d645e-4df1-482d-9ee8-54947745d8b4-istio-envoy\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:08:02.396778 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:02.396780 4908 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab9d645e-4df1-482d-9ee8-54947745d8b4-istio-podinfo\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:08:02.396963 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:02.396794 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dchfm\" (UniqueName: \"kubernetes.io/projected/ab9d645e-4df1-482d-9ee8-54947745d8b4-kube-api-access-dchfm\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:08:02.396963 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:02.396810 4908 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab9d645e-4df1-482d-9ee8-54947745d8b4-istio-token\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:08:02.654413 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:02.654372 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-fddaa3-openshift-default-668fdb7bf-z4588"] Oct 30 13:08:02.657062 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:02.657035 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-fddaa3-openshift-default-668fdb7bf-z4588"] Oct 30 13:08:03.312864 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:03.312808 4908 generic.go:356] "Generic (PLEG): container finished" podID="e5e36bcb-b201-4c99-be6c-1d632ba46503" containerID="2f263d48ab994080b5441081c9fe54a8b285dc65d85f8b0ad3c5170dba81a4d9" exitCode=0 Oct 30 13:08:03.313088 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:03.312887 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-47fd99-nwtr6" event={"ID":"e5e36bcb-b201-4c99-be6c-1d632ba46503","Type":"ContainerDied","Data":"2f263d48ab994080b5441081c9fe54a8b285dc65d85f8b0ad3c5170dba81a4d9"} Oct 30 13:08:03.606381 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:03.606303 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab9d645e-4df1-482d-9ee8-54947745d8b4" path="/var/lib/kubelet/pods/ab9d645e-4df1-482d-9ee8-54947745d8b4/volumes" Oct 30 13:08:04.475623 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:04.475595 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-47fd99-nwtr6" Oct 30 13:08:04.619083 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:04.618988 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5e36bcb-b201-4c99-be6c-1d632ba46503-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5e36bcb-b201-4c99-be6c-1d632ba46503-maas-api-openshift-ca\") pod \"e5e36bcb-b201-4c99-be6c-1d632ba46503\" (UID: \"e5e36bcb-b201-4c99-be6c-1d632ba46503\") " Oct 30 13:08:04.619083 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:04.619046 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e5e36bcb-b201-4c99-be6c-1d632ba46503-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5e36bcb-b201-4c99-be6c-1d632ba46503-maas-api-xks-ca\") pod \"e5e36bcb-b201-4c99-be6c-1d632ba46503\" (UID: \"e5e36bcb-b201-4c99-be6c-1d632ba46503\") " Oct 30 13:08:04.619354 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:04.619090 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5e36bcb-b201-4c99-be6c-1d632ba46503-kube-api-access-9bql5\" (UniqueName: \"kubernetes.io/projected/e5e36bcb-b201-4c99-be6c-1d632ba46503-kube-api-access-9bql5\") pod \"e5e36bcb-b201-4c99-be6c-1d632ba46503\" (UID: \"e5e36bcb-b201-4c99-be6c-1d632ba46503\") " Oct 30 13:08:04.619456 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:04.619431 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5e36bcb-b201-4c99-be6c-1d632ba46503-maas-api-openshift-ca" pod "e5e36bcb-b201-4c99-be6c-1d632ba46503" (UID: "e5e36bcb-b201-4c99-be6c-1d632ba46503"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:08:04.621300 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:04.621267 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5e36bcb-b201-4c99-be6c-1d632ba46503-maas-api-xks-ca" pod "e5e36bcb-b201-4c99-be6c-1d632ba46503" (UID: "e5e36bcb-b201-4c99-be6c-1d632ba46503"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:08:04.621407 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:04.621327 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5e36bcb-b201-4c99-be6c-1d632ba46503-kube-api-access-9bql5" pod "e5e36bcb-b201-4c99-be6c-1d632ba46503" (UID: "e5e36bcb-b201-4c99-be6c-1d632ba46503"). InnerVolumeSpecName "kube-api-access-9bql5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:08:04.720370 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:04.720328 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9bql5\" (UniqueName: \"kubernetes.io/projected/e5e36bcb-b201-4c99-be6c-1d632ba46503-kube-api-access-9bql5\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:08:04.720370 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:04.720369 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5e36bcb-b201-4c99-be6c-1d632ba46503-maas-api-openshift-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:08:04.720370 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:04.720385 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5e36bcb-b201-4c99-be6c-1d632ba46503-maas-api-xks-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:08:05.330130 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:05.330094 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-47fd99-nwtr6" Oct 30 13:08:05.330302 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:05.330125 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-47fd99-nwtr6" event={"ID":"e5e36bcb-b201-4c99-be6c-1d632ba46503","Type":"ContainerDied","Data":"f5eab60f97317c7719c67cfcc8d732b66afcb70278143ffed897bb61b93418fe"} Oct 30 13:08:05.330302 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:05.330208 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f5eab60f97317c7719c67cfcc8d732b66afcb70278143ffed897bb61b93418fe" Oct 30 13:08:05.745177 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:05.745136 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-47fd99/distinct-2-w2-47fd99-kserve-77cf84b469-zxs8s"] Oct 30 13:08:05.760843 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:05.760799 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-47fd99/premium-w2-47fd99-kserve-5d85f9bb95-hqkmf"] Oct 30 13:08:09.440767 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:09.440726 4908 scope.go:122] "RemoveContainer" containerID="a6e4c454d8461d86913f355f19f9349918e1ce555eaedebeaea1433a2f2dc9ca" Oct 30 13:08:09.451697 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:09.451671 4908 scope.go:122] "RemoveContainer" containerID="4613f11c31a199df39f3648785fb5bf1e75c5a0267566c84f9ce8b29a7ed3729" Oct 30 13:08:09.463145 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:09.463118 4908 scope.go:122] "RemoveContainer" containerID="9588ea39ea27c5eb940a07541f4f948f1e0e9b26ef7e14aa44fc76498b6dc401" Oct 30 13:08:09.475045 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:09.475019 4908 scope.go:122] "RemoveContainer" containerID="16631b86a037458a316965cd02edf6aa3ebe390b449f42844320b9c40349c909" Oct 30 13:08:09.486422 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:09.486393 4908 scope.go:122] "RemoveContainer" containerID="63f32d8866d4780cb8d688d273620c93e7f1c8a99a12774686a02713c32591ec" Oct 30 13:08:10.031179 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:10.031130 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-6a7148-jnbbj"] Oct 30 13:08:10.031655 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:10.031634 4908 cpu_manager.go:414] "containerMap: removing container" podUID="ab9d645e-4df1-482d-9ee8-54947745d8b4" containerName="istio-proxy" Oct 30 13:08:10.031776 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:10.031655 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab9d645e-4df1-482d-9ee8-54947745d8b4" containerName="istio-proxy" Oct 30 13:08:10.031776 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:10.031706 4908 cpu_manager.go:414] "containerMap: removing container" podUID="e5e36bcb-b201-4c99-be6c-1d632ba46503" containerName="revoke-keys" Oct 30 13:08:10.031776 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:10.031714 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5e36bcb-b201-4c99-be6c-1d632ba46503" containerName="revoke-keys" Oct 30 13:08:10.031971 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:10.031849 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5e36bcb-b201-4c99-be6c-1d632ba46503" containerName="revoke-keys" Oct 30 13:08:10.031971 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:10.031864 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab9d645e-4df1-482d-9ee8-54947745d8b4" containerName="istio-proxy" Oct 30 13:08:10.031971 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:10.031878 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab9d645e-4df1-482d-9ee8-54947745d8b4" containerName="istio-proxy" Oct 30 13:08:10.037339 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:10.037312 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-6a7148-jnbbj" Oct 30 13:08:10.054071 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:10.054036 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-6a7148-jnbbj"] Oct 30 13:08:10.064487 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:10.064146 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:08:10.064487 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:10.064197 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s75hq\"" Oct 30 13:08:10.064487 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:10.064335 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 13:08:10.067593 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:10.067553 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 13:08:10.087304 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:10.087169 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hp5m6\" (UniqueName: \"kubernetes.io/projected/0a0aaa55-6e3a-4791-8800-dbb7d27f8809-kube-api-access-hp5m6\") pod \"maas-api-revoke-keys-e2e-worker-w3-6a7148-jnbbj\" (UID: \"0a0aaa55-6e3a-4791-8800-dbb7d27f8809\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-6a7148-jnbbj" Oct 30 13:08:10.087304 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:10.087249 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a0aaa55-6e3a-4791-8800-dbb7d27f8809-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-6a7148-jnbbj\" (UID: \"0a0aaa55-6e3a-4791-8800-dbb7d27f8809\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-6a7148-jnbbj" Oct 30 13:08:10.087602 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:10.087355 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a0aaa55-6e3a-4791-8800-dbb7d27f8809-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-6a7148-jnbbj\" (UID: \"0a0aaa55-6e3a-4791-8800-dbb7d27f8809\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-6a7148-jnbbj" Oct 30 13:08:10.187982 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:10.187942 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hp5m6\" (UniqueName: \"kubernetes.io/projected/0a0aaa55-6e3a-4791-8800-dbb7d27f8809-kube-api-access-hp5m6\") pod \"maas-api-revoke-keys-e2e-worker-w3-6a7148-jnbbj\" (UID: \"0a0aaa55-6e3a-4791-8800-dbb7d27f8809\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-6a7148-jnbbj" Oct 30 13:08:10.187982 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:10.187994 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a0aaa55-6e3a-4791-8800-dbb7d27f8809-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-6a7148-jnbbj\" (UID: \"0a0aaa55-6e3a-4791-8800-dbb7d27f8809\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-6a7148-jnbbj" Oct 30 13:08:10.188356 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:10.188022 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a0aaa55-6e3a-4791-8800-dbb7d27f8809-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-6a7148-jnbbj\" (UID: \"0a0aaa55-6e3a-4791-8800-dbb7d27f8809\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-6a7148-jnbbj" Oct 30 13:08:10.188880 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:10.188822 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a0aaa55-6e3a-4791-8800-dbb7d27f8809-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-6a7148-jnbbj\" (UID: \"0a0aaa55-6e3a-4791-8800-dbb7d27f8809\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-6a7148-jnbbj" Oct 30 13:08:10.190503 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:10.190468 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a0aaa55-6e3a-4791-8800-dbb7d27f8809-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-6a7148-jnbbj\" (UID: \"0a0aaa55-6e3a-4791-8800-dbb7d27f8809\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-6a7148-jnbbj" Oct 30 13:08:10.213062 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:10.213030 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hp5m6\" (UniqueName: \"kubernetes.io/projected/0a0aaa55-6e3a-4791-8800-dbb7d27f8809-kube-api-access-hp5m6\") pod \"maas-api-revoke-keys-e2e-worker-w3-6a7148-jnbbj\" (UID: \"0a0aaa55-6e3a-4791-8800-dbb7d27f8809\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-6a7148-jnbbj" Oct 30 13:08:10.350617 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:10.350521 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-6a7148-jnbbj" Oct 30 13:08:10.725904 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:10.723737 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-6a7148-jnbbj"] Oct 30 13:08:10.725904 ip-10-0-141-251 kubenswrapper[4908]: W1030 13:08:10.725862 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a0aaa55_6e3a_4791_8800_dbb7d27f8809.slice/crio-acc6c2d0bd902a4ad0560a9a3c23c4feb0d6860c5afe6ddabcbe1ff5f964c129 WatchSource:0}: Error finding container acc6c2d0bd902a4ad0560a9a3c23c4feb0d6860c5afe6ddabcbe1ff5f964c129: Status 404 returned error can't find the container with id acc6c2d0bd902a4ad0560a9a3c23c4feb0d6860c5afe6ddabcbe1ff5f964c129 Oct 30 13:08:11.383350 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:11.383255 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-6a7148-jnbbj" event={"ID":"0a0aaa55-6e3a-4791-8800-dbb7d27f8809","Type":"ContainerStarted","Data":"6a89909d988e8f6ca1197a54e39aada1c546520fd23735b43212c4af9448766a"} Oct 30 13:08:11.383350 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:11.383322 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-6a7148-jnbbj" event={"ID":"0a0aaa55-6e3a-4791-8800-dbb7d27f8809","Type":"ContainerStarted","Data":"acc6c2d0bd902a4ad0560a9a3c23c4feb0d6860c5afe6ddabcbe1ff5f964c129"} Oct 30 13:08:11.430787 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:11.430728 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-6a7148-jnbbj" podStartSLOduration=1.4307155200000001 podStartE2EDuration="1.43071552s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:08:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:08:11.42742123 +0000 UTC m=+4447.941849975" watchObservedRunningTime="2026-10-30 13:08:11.43071552 +0000 UTC m=+4447.945144265" Oct 30 13:08:12.016286 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:12.016239 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c60d35c2-9qk7g"] Oct 30 13:08:12.016913 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:12.016881 4908 cpu_manager.go:414] "containerMap: removing container" podUID="ab9d645e-4df1-482d-9ee8-54947745d8b4" containerName="istio-proxy" Oct 30 13:08:12.016913 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:12.016913 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab9d645e-4df1-482d-9ee8-54947745d8b4" containerName="istio-proxy" Oct 30 13:08:12.020879 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:12.020852 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c60d35c2-9qk7g" Oct 30 13:08:12.050397 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:12.050357 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c60d35c2-9qk7g"] Oct 30 13:08:12.107883 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:12.107848 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4v7x\" (UniqueName: \"kubernetes.io/projected/c4dc85ad-0955-4880-994e-857e19ad4db6-kube-api-access-h4v7x\") pod \"maas-api-revoke-sub-c60d35c2-9qk7g\" (UID: \"c4dc85ad-0955-4880-994e-857e19ad4db6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c60d35c2-9qk7g" Oct 30 13:08:12.108087 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:12.107899 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4dc85ad-0955-4880-994e-857e19ad4db6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c60d35c2-9qk7g\" (UID: \"c4dc85ad-0955-4880-994e-857e19ad4db6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c60d35c2-9qk7g" Oct 30 13:08:12.108167 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:12.108136 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4dc85ad-0955-4880-994e-857e19ad4db6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c60d35c2-9qk7g\" (UID: \"c4dc85ad-0955-4880-994e-857e19ad4db6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c60d35c2-9qk7g" Oct 30 13:08:12.209350 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:12.209315 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4v7x\" (UniqueName: \"kubernetes.io/projected/c4dc85ad-0955-4880-994e-857e19ad4db6-kube-api-access-h4v7x\") pod \"maas-api-revoke-sub-c60d35c2-9qk7g\" (UID: \"c4dc85ad-0955-4880-994e-857e19ad4db6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c60d35c2-9qk7g" Oct 30 13:08:12.209531 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:12.209368 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4dc85ad-0955-4880-994e-857e19ad4db6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c60d35c2-9qk7g\" (UID: \"c4dc85ad-0955-4880-994e-857e19ad4db6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c60d35c2-9qk7g" Oct 30 13:08:12.209531 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:12.209448 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4dc85ad-0955-4880-994e-857e19ad4db6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c60d35c2-9qk7g\" (UID: \"c4dc85ad-0955-4880-994e-857e19ad4db6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c60d35c2-9qk7g" Oct 30 13:08:12.210262 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:12.210234 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4dc85ad-0955-4880-994e-857e19ad4db6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c60d35c2-9qk7g\" (UID: \"c4dc85ad-0955-4880-994e-857e19ad4db6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c60d35c2-9qk7g" Oct 30 13:08:12.211882 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:12.211859 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4dc85ad-0955-4880-994e-857e19ad4db6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c60d35c2-9qk7g\" (UID: \"c4dc85ad-0955-4880-994e-857e19ad4db6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c60d35c2-9qk7g" Oct 30 13:08:12.237997 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:12.237968 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4v7x\" (UniqueName: \"kubernetes.io/projected/c4dc85ad-0955-4880-994e-857e19ad4db6-kube-api-access-h4v7x\") pod \"maas-api-revoke-sub-c60d35c2-9qk7g\" (UID: \"c4dc85ad-0955-4880-994e-857e19ad4db6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c60d35c2-9qk7g" Oct 30 13:08:12.325484 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:12.325382 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-327e31-rlczv"] Oct 30 13:08:12.329041 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:12.329009 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-327e31-rlczv"] Oct 30 13:08:12.336741 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:12.336714 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c60d35c2-9qk7g" Oct 30 13:08:12.394121 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:12.394077 4908 generic.go:356] "Generic (PLEG): container finished" podID="0a0aaa55-6e3a-4791-8800-dbb7d27f8809" containerID="6a89909d988e8f6ca1197a54e39aada1c546520fd23735b43212c4af9448766a" exitCode=0 Oct 30 13:08:12.394364 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:12.394210 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-6a7148-jnbbj" event={"ID":"0a0aaa55-6e3a-4791-8800-dbb7d27f8809","Type":"ContainerDied","Data":"6a89909d988e8f6ca1197a54e39aada1c546520fd23735b43212c4af9448766a"} Oct 30 13:08:12.653705 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:12.653624 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c60d35c2-9qk7g"] Oct 30 13:08:12.656997 ip-10-0-141-251 kubenswrapper[4908]: W1030 13:08:12.656964 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4dc85ad_0955_4880_994e_857e19ad4db6.slice/crio-f7d6909f124743acf8a18963f9cb2ec6183093da26e41f3b9e0d5b7230fc29d5 WatchSource:0}: Error finding container f7d6909f124743acf8a18963f9cb2ec6183093da26e41f3b9e0d5b7230fc29d5: Status 404 returned error can't find the container with id f7d6909f124743acf8a18963f9cb2ec6183093da26e41f3b9e0d5b7230fc29d5 Oct 30 13:08:13.403848 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:13.403784 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c60d35c2-9qk7g" event={"ID":"c4dc85ad-0955-4880-994e-857e19ad4db6","Type":"ContainerStarted","Data":"2236e1f732ca9960f67bcc285cc1a98a9f84477a2e3f4065223290de999a946e"} Oct 30 13:08:13.404308 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:13.403867 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c60d35c2-9qk7g" event={"ID":"c4dc85ad-0955-4880-994e-857e19ad4db6","Type":"ContainerStarted","Data":"f7d6909f124743acf8a18963f9cb2ec6183093da26e41f3b9e0d5b7230fc29d5"} Oct 30 13:08:13.452502 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:13.452441 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c60d35c2-9qk7g" podStartSLOduration=2.452427414 podStartE2EDuration="2.452427414s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:08:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:08:13.449767825 +0000 UTC m=+4449.964196571" watchObservedRunningTime="2026-10-30 13:08:13.452427414 +0000 UTC m=+4449.966856159" Oct 30 13:08:13.550217 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:13.550189 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-6a7148-jnbbj" Oct 30 13:08:13.609909 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:13.609813 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69ffd6a9-d5c5-4f6c-8188-00cfa376d676" path="/var/lib/kubelet/pods/69ffd6a9-d5c5-4f6c-8188-00cfa376d676/volumes" Oct 30 13:08:13.635091 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:13.635050 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a0aaa55-6e3a-4791-8800-dbb7d27f8809-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a0aaa55-6e3a-4791-8800-dbb7d27f8809-maas-api-xks-ca\") pod \"0a0aaa55-6e3a-4791-8800-dbb7d27f8809\" (UID: \"0a0aaa55-6e3a-4791-8800-dbb7d27f8809\") " Oct 30 13:08:13.635278 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:13.635203 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a0aaa55-6e3a-4791-8800-dbb7d27f8809-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a0aaa55-6e3a-4791-8800-dbb7d27f8809-maas-api-openshift-ca\") pod \"0a0aaa55-6e3a-4791-8800-dbb7d27f8809\" (UID: \"0a0aaa55-6e3a-4791-8800-dbb7d27f8809\") " Oct 30 13:08:13.635352 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:13.635282 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a0aaa55-6e3a-4791-8800-dbb7d27f8809-kube-api-access-hp5m6\" (UniqueName: \"kubernetes.io/projected/0a0aaa55-6e3a-4791-8800-dbb7d27f8809-kube-api-access-hp5m6\") pod \"0a0aaa55-6e3a-4791-8800-dbb7d27f8809\" (UID: \"0a0aaa55-6e3a-4791-8800-dbb7d27f8809\") " Oct 30 13:08:13.635665 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:13.635627 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a0aaa55-6e3a-4791-8800-dbb7d27f8809-maas-api-openshift-ca" pod "0a0aaa55-6e3a-4791-8800-dbb7d27f8809" (UID: "0a0aaa55-6e3a-4791-8800-dbb7d27f8809"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:08:13.635947 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:13.635924 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a0aaa55-6e3a-4791-8800-dbb7d27f8809-maas-api-openshift-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:08:13.637663 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:13.637634 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a0aaa55-6e3a-4791-8800-dbb7d27f8809-maas-api-xks-ca" pod "0a0aaa55-6e3a-4791-8800-dbb7d27f8809" (UID: "0a0aaa55-6e3a-4791-8800-dbb7d27f8809"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:08:13.637759 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:13.637701 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a0aaa55-6e3a-4791-8800-dbb7d27f8809-kube-api-access-hp5m6" pod "0a0aaa55-6e3a-4791-8800-dbb7d27f8809" (UID: "0a0aaa55-6e3a-4791-8800-dbb7d27f8809"). InnerVolumeSpecName "kube-api-access-hp5m6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:08:13.736904 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:13.736871 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hp5m6\" (UniqueName: \"kubernetes.io/projected/0a0aaa55-6e3a-4791-8800-dbb7d27f8809-kube-api-access-hp5m6\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:08:13.736904 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:13.736904 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a0aaa55-6e3a-4791-8800-dbb7d27f8809-maas-api-xks-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:08:14.425896 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:14.425855 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-6a7148-jnbbj" Oct 30 13:08:14.425896 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:14.425868 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-6a7148-jnbbj" event={"ID":"0a0aaa55-6e3a-4791-8800-dbb7d27f8809","Type":"ContainerDied","Data":"acc6c2d0bd902a4ad0560a9a3c23c4feb0d6860c5afe6ddabcbe1ff5f964c129"} Oct 30 13:08:14.426500 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:14.425966 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="acc6c2d0bd902a4ad0560a9a3c23c4feb0d6860c5afe6ddabcbe1ff5f964c129" Oct 30 13:08:14.428418 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:14.428380 4908 generic.go:356] "Generic (PLEG): container finished" podID="c4dc85ad-0955-4880-994e-857e19ad4db6" containerID="2236e1f732ca9960f67bcc285cc1a98a9f84477a2e3f4065223290de999a946e" exitCode=0 Oct 30 13:08:14.428553 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:14.428474 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c60d35c2-9qk7g" event={"ID":"c4dc85ad-0955-4880-994e-857e19ad4db6","Type":"ContainerDied","Data":"2236e1f732ca9960f67bcc285cc1a98a9f84477a2e3f4065223290de999a946e"} Oct 30 13:08:14.738723 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:14.738679 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-6a7148/distinct-2-w3-6a7148-kserve-67bd5fb976-9nfff"] Oct 30 13:08:14.757414 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:14.757371 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-6a7148/premium-w3-6a7148-kserve-67b976c888-tgd6c"] Oct 30 13:08:15.579757 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:15.579721 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c60d35c2-9qk7g" Oct 30 13:08:15.656049 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:15.656009 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4dc85ad-0955-4880-994e-857e19ad4db6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4dc85ad-0955-4880-994e-857e19ad4db6-maas-api-openshift-ca\") pod \"c4dc85ad-0955-4880-994e-857e19ad4db6\" (UID: \"c4dc85ad-0955-4880-994e-857e19ad4db6\") " Oct 30 13:08:15.656252 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:15.656092 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c4dc85ad-0955-4880-994e-857e19ad4db6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4dc85ad-0955-4880-994e-857e19ad4db6-maas-api-xks-ca\") pod \"c4dc85ad-0955-4880-994e-857e19ad4db6\" (UID: \"c4dc85ad-0955-4880-994e-857e19ad4db6\") " Oct 30 13:08:15.656252 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:15.656167 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4dc85ad-0955-4880-994e-857e19ad4db6-kube-api-access-h4v7x\" (UniqueName: \"kubernetes.io/projected/c4dc85ad-0955-4880-994e-857e19ad4db6-kube-api-access-h4v7x\") pod \"c4dc85ad-0955-4880-994e-857e19ad4db6\" (UID: \"c4dc85ad-0955-4880-994e-857e19ad4db6\") " Oct 30 13:08:15.656476 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:15.656446 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4dc85ad-0955-4880-994e-857e19ad4db6-maas-api-openshift-ca" pod "c4dc85ad-0955-4880-994e-857e19ad4db6" (UID: "c4dc85ad-0955-4880-994e-857e19ad4db6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:08:15.658236 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:15.658210 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4dc85ad-0955-4880-994e-857e19ad4db6-maas-api-xks-ca" pod "c4dc85ad-0955-4880-994e-857e19ad4db6" (UID: "c4dc85ad-0955-4880-994e-857e19ad4db6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:08:15.658350 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:15.658319 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4dc85ad-0955-4880-994e-857e19ad4db6-kube-api-access-h4v7x" pod "c4dc85ad-0955-4880-994e-857e19ad4db6" (UID: "c4dc85ad-0955-4880-994e-857e19ad4db6"). InnerVolumeSpecName "kube-api-access-h4v7x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:08:15.757759 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:15.757723 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4dc85ad-0955-4880-994e-857e19ad4db6-maas-api-openshift-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:08:15.757759 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:15.757759 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4dc85ad-0955-4880-994e-857e19ad4db6-maas-api-xks-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:08:15.757994 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:15.757772 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4v7x\" (UniqueName: \"kubernetes.io/projected/c4dc85ad-0955-4880-994e-857e19ad4db6-kube-api-access-h4v7x\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:08:16.444295 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:16.444258 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c60d35c2-9qk7g" Oct 30 13:08:16.444295 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:16.444272 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c60d35c2-9qk7g" event={"ID":"c4dc85ad-0955-4880-994e-857e19ad4db6","Type":"ContainerDied","Data":"f7d6909f124743acf8a18963f9cb2ec6183093da26e41f3b9e0d5b7230fc29d5"} Oct 30 13:08:16.444524 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:16.444330 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f7d6909f124743acf8a18963f9cb2ec6183093da26e41f3b9e0d5b7230fc29d5" Oct 30 13:08:18.067625 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:18.067580 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9wwfc"] Oct 30 13:08:19.684965 ip-10-0-141-251 kubenswrapper[4908]: E1030 13:08:19.684927 4908 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vn6vv" containerName="extractor" Oct 30 13:08:20.766541 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:20.766502 4908 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-47fd99/distinct-2-w2-47fd99-kserve-77cf84b469-zxs8s" podUID="baf44d3d-42e6-43f7-a0ad-f0f9932320c1" containerName="main" containerID="cri-o://cac60264305a95c80c5af5bb81022523e41a873fe60c98433574b0b96b20db41" gracePeriod=45 Oct 30 13:08:20.781990 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:20.781949 4908 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-47fd99/premium-w2-47fd99-kserve-5d85f9bb95-hqkmf" podUID="fd2cbeda-dbc4-41f6-a730-e1c6f9343c5e" containerName="main" containerID="cri-o://2a87eadca80163145063eb0f8e3c3dd7d99f1798fee7cc2c36b29a26e00a86d2" gracePeriod=45 Oct 30 13:08:20.809725 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:20.809690 4908 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47fd99/premium-w2-47fd99-kserve-5d85f9bb95-hqkmf" podUID="fd2cbeda-dbc4-41f6-a730-e1c6f9343c5e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": dial tcp 10.132.0.63:8000: connect: connection refused" Oct 30 13:08:21.108246 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:21.108209 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-47fd99/distinct-2-w2-47fd99-kserve-77cf84b469-zxs8s" Oct 30 13:08:21.112239 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:21.112214 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-47fd99/premium-w2-47fd99-kserve-5d85f9bb95-hqkmf" Oct 30 13:08:21.223917 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:21.223876 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/baf44d3d-42e6-43f7-a0ad-f0f9932320c1-kube-api-access-cg89m\" (UniqueName: \"kubernetes.io/projected/baf44d3d-42e6-43f7-a0ad-f0f9932320c1-kube-api-access-cg89m\") pod \"baf44d3d-42e6-43f7-a0ad-f0f9932320c1\" (UID: \"baf44d3d-42e6-43f7-a0ad-f0f9932320c1\") " Oct 30 13:08:21.223917 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:21.223921 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd2cbeda-dbc4-41f6-a730-e1c6f9343c5e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd2cbeda-dbc4-41f6-a730-e1c6f9343c5e-tmp-dir\") pod \"fd2cbeda-dbc4-41f6-a730-e1c6f9343c5e\" (UID: \"fd2cbeda-dbc4-41f6-a730-e1c6f9343c5e\") " Oct 30 13:08:21.224205 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:21.223953 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/baf44d3d-42e6-43f7-a0ad-f0f9932320c1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/baf44d3d-42e6-43f7-a0ad-f0f9932320c1-tmp-dir\") pod \"baf44d3d-42e6-43f7-a0ad-f0f9932320c1\" (UID: \"baf44d3d-42e6-43f7-a0ad-f0f9932320c1\") " Oct 30 13:08:21.224205 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:21.223976 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/baf44d3d-42e6-43f7-a0ad-f0f9932320c1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/baf44d3d-42e6-43f7-a0ad-f0f9932320c1-dshm\") pod \"baf44d3d-42e6-43f7-a0ad-f0f9932320c1\" (UID: \"baf44d3d-42e6-43f7-a0ad-f0f9932320c1\") " Oct 30 13:08:21.224205 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:21.224025 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd2cbeda-dbc4-41f6-a730-e1c6f9343c5e-home\" (UniqueName: \"kubernetes.io/empty-dir/fd2cbeda-dbc4-41f6-a730-e1c6f9343c5e-home\") pod \"fd2cbeda-dbc4-41f6-a730-e1c6f9343c5e\" (UID: \"fd2cbeda-dbc4-41f6-a730-e1c6f9343c5e\") " Oct 30 13:08:21.224205 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:21.224085 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd2cbeda-dbc4-41f6-a730-e1c6f9343c5e-tls-certs\" (UniqueName: \"kubernetes.io/secret/fd2cbeda-dbc4-41f6-a730-e1c6f9343c5e-tls-certs\") pod \"fd2cbeda-dbc4-41f6-a730-e1c6f9343c5e\" (UID: \"fd2cbeda-dbc4-41f6-a730-e1c6f9343c5e\") " Oct 30 13:08:21.224436 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:21.224287 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd2cbeda-dbc4-41f6-a730-e1c6f9343c5e-tmp-dir" pod "fd2cbeda-dbc4-41f6-a730-e1c6f9343c5e" (UID: "fd2cbeda-dbc4-41f6-a730-e1c6f9343c5e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:08:21.224436 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:21.224383 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd2cbeda-dbc4-41f6-a730-e1c6f9343c5e-home" pod "fd2cbeda-dbc4-41f6-a730-e1c6f9343c5e" (UID: "fd2cbeda-dbc4-41f6-a730-e1c6f9343c5e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:08:21.224547 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:21.224495 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd2cbeda-dbc4-41f6-a730-e1c6f9343c5e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd2cbeda-dbc4-41f6-a730-e1c6f9343c5e-dshm\") pod \"fd2cbeda-dbc4-41f6-a730-e1c6f9343c5e\" (UID: \"fd2cbeda-dbc4-41f6-a730-e1c6f9343c5e\") " Oct 30 13:08:21.224547 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:21.224536 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/baf44d3d-42e6-43f7-a0ad-f0f9932320c1-home\" (UniqueName: \"kubernetes.io/empty-dir/baf44d3d-42e6-43f7-a0ad-f0f9932320c1-home\") pod \"baf44d3d-42e6-43f7-a0ad-f0f9932320c1\" (UID: \"baf44d3d-42e6-43f7-a0ad-f0f9932320c1\") " Oct 30 13:08:21.224657 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:21.224628 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baf44d3d-42e6-43f7-a0ad-f0f9932320c1-tmp-dir" pod "baf44d3d-42e6-43f7-a0ad-f0f9932320c1" (UID: "baf44d3d-42e6-43f7-a0ad-f0f9932320c1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:08:21.224657 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:21.224633 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/baf44d3d-42e6-43f7-a0ad-f0f9932320c1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/baf44d3d-42e6-43f7-a0ad-f0f9932320c1-model-cache\") pod \"baf44d3d-42e6-43f7-a0ad-f0f9932320c1\" (UID: \"baf44d3d-42e6-43f7-a0ad-f0f9932320c1\") " Oct 30 13:08:21.224781 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:21.224726 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd2cbeda-dbc4-41f6-a730-e1c6f9343c5e-kube-api-access-l6z4w\" (UniqueName: \"kubernetes.io/projected/fd2cbeda-dbc4-41f6-a730-e1c6f9343c5e-kube-api-access-l6z4w\") pod \"fd2cbeda-dbc4-41f6-a730-e1c6f9343c5e\" (UID: \"fd2cbeda-dbc4-41f6-a730-e1c6f9343c5e\") " Oct 30 13:08:21.224876 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:21.224817 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baf44d3d-42e6-43f7-a0ad-f0f9932320c1-home" pod "baf44d3d-42e6-43f7-a0ad-f0f9932320c1" (UID: "baf44d3d-42e6-43f7-a0ad-f0f9932320c1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:08:21.224876 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:21.224848 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baf44d3d-42e6-43f7-a0ad-f0f9932320c1-model-cache" pod "baf44d3d-42e6-43f7-a0ad-f0f9932320c1" (UID: "baf44d3d-42e6-43f7-a0ad-f0f9932320c1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:08:21.224991 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:21.224969 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/baf44d3d-42e6-43f7-a0ad-f0f9932320c1-tls-certs\" (UniqueName: \"kubernetes.io/secret/baf44d3d-42e6-43f7-a0ad-f0f9932320c1-tls-certs\") pod \"baf44d3d-42e6-43f7-a0ad-f0f9932320c1\" (UID: \"baf44d3d-42e6-43f7-a0ad-f0f9932320c1\") " Oct 30 13:08:21.225044 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:21.225030 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd2cbeda-dbc4-41f6-a730-e1c6f9343c5e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd2cbeda-dbc4-41f6-a730-e1c6f9343c5e-model-cache\") pod \"fd2cbeda-dbc4-41f6-a730-e1c6f9343c5e\" (UID: \"fd2cbeda-dbc4-41f6-a730-e1c6f9343c5e\") " Oct 30 13:08:21.225546 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:21.225451 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd2cbeda-dbc4-41f6-a730-e1c6f9343c5e-model-cache" pod "fd2cbeda-dbc4-41f6-a730-e1c6f9343c5e" (UID: "fd2cbeda-dbc4-41f6-a730-e1c6f9343c5e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:08:21.225546 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:21.225472 4908 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd2cbeda-dbc4-41f6-a730-e1c6f9343c5e-home\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:08:21.225546 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:21.225495 4908 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/baf44d3d-42e6-43f7-a0ad-f0f9932320c1-home\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:08:21.225546 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:21.225508 4908 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/baf44d3d-42e6-43f7-a0ad-f0f9932320c1-model-cache\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:08:21.225546 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:21.225524 4908 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd2cbeda-dbc4-41f6-a730-e1c6f9343c5e-tmp-dir\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:08:21.225546 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:21.225537 4908 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/baf44d3d-42e6-43f7-a0ad-f0f9932320c1-tmp-dir\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:08:21.226936 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:21.226907 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/baf44d3d-42e6-43f7-a0ad-f0f9932320c1-kube-api-access-cg89m" pod "baf44d3d-42e6-43f7-a0ad-f0f9932320c1" (UID: "baf44d3d-42e6-43f7-a0ad-f0f9932320c1"). InnerVolumeSpecName "kube-api-access-cg89m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:08:21.227072 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:21.227047 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd2cbeda-dbc4-41f6-a730-e1c6f9343c5e-tls-certs" pod "fd2cbeda-dbc4-41f6-a730-e1c6f9343c5e" (UID: "fd2cbeda-dbc4-41f6-a730-e1c6f9343c5e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:08:21.227132 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:21.227056 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd2cbeda-dbc4-41f6-a730-e1c6f9343c5e-dshm" pod "fd2cbeda-dbc4-41f6-a730-e1c6f9343c5e" (UID: "fd2cbeda-dbc4-41f6-a730-e1c6f9343c5e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:08:21.227318 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:21.227300 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd2cbeda-dbc4-41f6-a730-e1c6f9343c5e-kube-api-access-l6z4w" pod "fd2cbeda-dbc4-41f6-a730-e1c6f9343c5e" (UID: "fd2cbeda-dbc4-41f6-a730-e1c6f9343c5e"). InnerVolumeSpecName "kube-api-access-l6z4w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:08:21.227384 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:21.227355 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baf44d3d-42e6-43f7-a0ad-f0f9932320c1-dshm" pod "baf44d3d-42e6-43f7-a0ad-f0f9932320c1" (UID: "baf44d3d-42e6-43f7-a0ad-f0f9932320c1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:08:21.227468 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:21.227447 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/baf44d3d-42e6-43f7-a0ad-f0f9932320c1-tls-certs" pod "baf44d3d-42e6-43f7-a0ad-f0f9932320c1" (UID: "baf44d3d-42e6-43f7-a0ad-f0f9932320c1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:08:21.326351 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:21.326259 4908 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd2cbeda-dbc4-41f6-a730-e1c6f9343c5e-tls-certs\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:08:21.326351 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:21.326293 4908 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd2cbeda-dbc4-41f6-a730-e1c6f9343c5e-dshm\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:08:21.326351 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:21.326307 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l6z4w\" (UniqueName: \"kubernetes.io/projected/fd2cbeda-dbc4-41f6-a730-e1c6f9343c5e-kube-api-access-l6z4w\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:08:21.326351 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:21.326320 4908 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/baf44d3d-42e6-43f7-a0ad-f0f9932320c1-tls-certs\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:08:21.326351 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:21.326335 4908 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd2cbeda-dbc4-41f6-a730-e1c6f9343c5e-model-cache\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:08:21.326351 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:21.326347 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cg89m\" (UniqueName: \"kubernetes.io/projected/baf44d3d-42e6-43f7-a0ad-f0f9932320c1-kube-api-access-cg89m\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:08:21.326663 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:21.326359 4908 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/baf44d3d-42e6-43f7-a0ad-f0f9932320c1-dshm\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:08:21.483462 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:21.483415 4908 generic.go:356] "Generic (PLEG): container finished" podID="baf44d3d-42e6-43f7-a0ad-f0f9932320c1" containerID="cac60264305a95c80c5af5bb81022523e41a873fe60c98433574b0b96b20db41" exitCode=0 Oct 30 13:08:21.483658 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:21.483476 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-47fd99/distinct-2-w2-47fd99-kserve-77cf84b469-zxs8s" Oct 30 13:08:21.483747 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:21.483699 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-47fd99/distinct-2-w2-47fd99-kserve-77cf84b469-zxs8s" event={"ID":"baf44d3d-42e6-43f7-a0ad-f0f9932320c1","Type":"ContainerDied","Data":"cac60264305a95c80c5af5bb81022523e41a873fe60c98433574b0b96b20db41"} Oct 30 13:08:21.483747 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:21.483737 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-47fd99/distinct-2-w2-47fd99-kserve-77cf84b469-zxs8s" event={"ID":"baf44d3d-42e6-43f7-a0ad-f0f9932320c1","Type":"ContainerDied","Data":"28aa7de2180d2cc503528a49466f6b8bd98739b4cf69252cfcc9f4587501da4f"} Oct 30 13:08:21.483902 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:21.483755 4908 scope.go:122] "RemoveContainer" containerID="cac60264305a95c80c5af5bb81022523e41a873fe60c98433574b0b96b20db41" Oct 30 13:08:21.491159 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:21.491119 4908 generic.go:356] "Generic (PLEG): container finished" podID="fd2cbeda-dbc4-41f6-a730-e1c6f9343c5e" containerID="2a87eadca80163145063eb0f8e3c3dd7d99f1798fee7cc2c36b29a26e00a86d2" exitCode=0 Oct 30 13:08:21.491376 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:21.491168 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-47fd99/premium-w2-47fd99-kserve-5d85f9bb95-hqkmf" event={"ID":"fd2cbeda-dbc4-41f6-a730-e1c6f9343c5e","Type":"ContainerDied","Data":"2a87eadca80163145063eb0f8e3c3dd7d99f1798fee7cc2c36b29a26e00a86d2"} Oct 30 13:08:21.491376 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:21.491203 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-47fd99/premium-w2-47fd99-kserve-5d85f9bb95-hqkmf" event={"ID":"fd2cbeda-dbc4-41f6-a730-e1c6f9343c5e","Type":"ContainerDied","Data":"e7dbc7689dbc250d757eec337519d4af08683b8918cd9c5d9655764bef4b0813"} Oct 30 13:08:21.491376 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:21.491213 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-47fd99/premium-w2-47fd99-kserve-5d85f9bb95-hqkmf" Oct 30 13:08:21.500986 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:21.500956 4908 scope.go:122] "RemoveContainer" containerID="cac60264305a95c80c5af5bb81022523e41a873fe60c98433574b0b96b20db41" Oct 30 13:08:21.503350 ip-10-0-141-251 kubenswrapper[4908]: E1030 13:08:21.503308 4908 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cac60264305a95c80c5af5bb81022523e41a873fe60c98433574b0b96b20db41\": container with ID starting with cac60264305a95c80c5af5bb81022523e41a873fe60c98433574b0b96b20db41 not found: ID does not exist" containerID="cac60264305a95c80c5af5bb81022523e41a873fe60c98433574b0b96b20db41" Oct 30 13:08:21.503449 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:21.503350 4908 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cac60264305a95c80c5af5bb81022523e41a873fe60c98433574b0b96b20db41"} err="failed to get container status \"cac60264305a95c80c5af5bb81022523e41a873fe60c98433574b0b96b20db41\": rpc error: code = NotFound desc = could not find container \"cac60264305a95c80c5af5bb81022523e41a873fe60c98433574b0b96b20db41\": container with ID starting with cac60264305a95c80c5af5bb81022523e41a873fe60c98433574b0b96b20db41 not found: ID does not exist" Oct 30 13:08:21.503449 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:21.503372 4908 scope.go:122] "RemoveContainer" containerID="2a87eadca80163145063eb0f8e3c3dd7d99f1798fee7cc2c36b29a26e00a86d2" Oct 30 13:08:21.514280 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:21.514259 4908 scope.go:122] "RemoveContainer" containerID="2a87eadca80163145063eb0f8e3c3dd7d99f1798fee7cc2c36b29a26e00a86d2" Oct 30 13:08:21.514569 ip-10-0-141-251 kubenswrapper[4908]: E1030 13:08:21.514547 4908 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a87eadca80163145063eb0f8e3c3dd7d99f1798fee7cc2c36b29a26e00a86d2\": container with ID starting with 2a87eadca80163145063eb0f8e3c3dd7d99f1798fee7cc2c36b29a26e00a86d2 not found: ID does not exist" containerID="2a87eadca80163145063eb0f8e3c3dd7d99f1798fee7cc2c36b29a26e00a86d2" Oct 30 13:08:21.514636 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:21.514577 4908 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a87eadca80163145063eb0f8e3c3dd7d99f1798fee7cc2c36b29a26e00a86d2"} err="failed to get container status \"2a87eadca80163145063eb0f8e3c3dd7d99f1798fee7cc2c36b29a26e00a86d2\": rpc error: code = NotFound desc = could not find container \"2a87eadca80163145063eb0f8e3c3dd7d99f1798fee7cc2c36b29a26e00a86d2\": container with ID starting with 2a87eadca80163145063eb0f8e3c3dd7d99f1798fee7cc2c36b29a26e00a86d2 not found: ID does not exist" Oct 30 13:08:21.523965 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:21.523935 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-8d2170-qglwj"] Oct 30 13:08:21.524331 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:21.524319 4908 cpu_manager.go:414] "containerMap: removing container" podUID="c4dc85ad-0955-4880-994e-857e19ad4db6" containerName="revoke-keys" Oct 30 13:08:21.524372 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:21.524333 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4dc85ad-0955-4880-994e-857e19ad4db6" containerName="revoke-keys" Oct 30 13:08:21.524372 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:21.524346 4908 cpu_manager.go:414] "containerMap: removing container" podUID="baf44d3d-42e6-43f7-a0ad-f0f9932320c1" containerName="main" Oct 30 13:08:21.524372 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:21.524351 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="baf44d3d-42e6-43f7-a0ad-f0f9932320c1" containerName="main" Oct 30 13:08:21.524372 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:21.524365 4908 cpu_manager.go:414] "containerMap: removing container" podUID="0a0aaa55-6e3a-4791-8800-dbb7d27f8809" containerName="revoke-keys" Oct 30 13:08:21.524372 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:21.524371 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a0aaa55-6e3a-4791-8800-dbb7d27f8809" containerName="revoke-keys" Oct 30 13:08:21.524537 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:21.524383 4908 cpu_manager.go:414] "containerMap: removing container" podUID="fd2cbeda-dbc4-41f6-a730-e1c6f9343c5e" containerName="main" Oct 30 13:08:21.524537 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:21.524388 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd2cbeda-dbc4-41f6-a730-e1c6f9343c5e" containerName="main" Oct 30 13:08:21.524537 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:21.524455 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4dc85ad-0955-4880-994e-857e19ad4db6" containerName="revoke-keys" Oct 30 13:08:21.524537 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:21.524465 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="baf44d3d-42e6-43f7-a0ad-f0f9932320c1" containerName="main" Oct 30 13:08:21.524537 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:21.524473 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a0aaa55-6e3a-4791-8800-dbb7d27f8809" containerName="revoke-keys" Oct 30 13:08:21.524537 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:21.524479 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd2cbeda-dbc4-41f6-a730-e1c6f9343c5e" containerName="main" Oct 30 13:08:21.529269 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:21.529251 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-8d2170-qglwj" Oct 30 13:08:21.547946 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:21.547918 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:08:21.548275 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:21.548249 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s75hq\"" Oct 30 13:08:21.549347 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:21.549323 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 13:08:21.549449 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:21.549371 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 13:08:21.552180 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:21.552155 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-8d2170-qglwj"] Oct 30 13:08:21.567932 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:21.567892 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-47fd99/distinct-2-w2-47fd99-kserve-77cf84b469-zxs8s"] Oct 30 13:08:21.573230 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:21.573193 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-47fd99/distinct-2-w2-47fd99-kserve-77cf84b469-zxs8s"] Oct 30 13:08:21.608359 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:21.608276 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="baf44d3d-42e6-43f7-a0ad-f0f9932320c1" path="/var/lib/kubelet/pods/baf44d3d-42e6-43f7-a0ad-f0f9932320c1/volumes" Oct 30 13:08:21.608627 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:21.608611 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-47fd99/premium-w2-47fd99-kserve-5d85f9bb95-hqkmf"] Oct 30 13:08:21.608666 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:21.608640 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-47fd99/premium-w2-47fd99-kserve-5d85f9bb95-hqkmf"] Oct 30 13:08:21.628971 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:21.628939 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d230415a-fe71-476a-bf03-2dcc45ade8e3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-8d2170-qglwj\" (UID: \"d230415a-fe71-476a-bf03-2dcc45ade8e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-8d2170-qglwj" Oct 30 13:08:21.629101 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:21.628995 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jl2k2\" (UniqueName: \"kubernetes.io/projected/d230415a-fe71-476a-bf03-2dcc45ade8e3-kube-api-access-jl2k2\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-8d2170-qglwj\" (UID: \"d230415a-fe71-476a-bf03-2dcc45ade8e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-8d2170-qglwj" Oct 30 13:08:21.629162 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:21.629108 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d230415a-fe71-476a-bf03-2dcc45ade8e3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-8d2170-qglwj\" (UID: \"d230415a-fe71-476a-bf03-2dcc45ade8e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-8d2170-qglwj" Oct 30 13:08:21.730484 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:21.730445 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d230415a-fe71-476a-bf03-2dcc45ade8e3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-8d2170-qglwj\" (UID: \"d230415a-fe71-476a-bf03-2dcc45ade8e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-8d2170-qglwj" Oct 30 13:08:21.730670 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:21.730536 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d230415a-fe71-476a-bf03-2dcc45ade8e3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-8d2170-qglwj\" (UID: \"d230415a-fe71-476a-bf03-2dcc45ade8e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-8d2170-qglwj" Oct 30 13:08:21.730670 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:21.730572 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jl2k2\" (UniqueName: \"kubernetes.io/projected/d230415a-fe71-476a-bf03-2dcc45ade8e3-kube-api-access-jl2k2\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-8d2170-qglwj\" (UID: \"d230415a-fe71-476a-bf03-2dcc45ade8e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-8d2170-qglwj" Oct 30 13:08:21.731354 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:21.731329 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d230415a-fe71-476a-bf03-2dcc45ade8e3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-8d2170-qglwj\" (UID: \"d230415a-fe71-476a-bf03-2dcc45ade8e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-8d2170-qglwj" Oct 30 13:08:21.733190 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:21.733170 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d230415a-fe71-476a-bf03-2dcc45ade8e3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-8d2170-qglwj\" (UID: \"d230415a-fe71-476a-bf03-2dcc45ade8e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-8d2170-qglwj" Oct 30 13:08:21.755278 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:21.755242 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jl2k2\" (UniqueName: \"kubernetes.io/projected/d230415a-fe71-476a-bf03-2dcc45ade8e3-kube-api-access-jl2k2\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-8d2170-qglwj\" (UID: \"d230415a-fe71-476a-bf03-2dcc45ade8e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-8d2170-qglwj" Oct 30 13:08:21.843020 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:21.842976 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-8d2170-qglwj" Oct 30 13:08:22.152046 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:22.152007 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-8d2170-qglwj"] Oct 30 13:08:22.153617 ip-10-0-141-251 kubenswrapper[4908]: W1030 13:08:22.153587 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd230415a_fe71_476a_bf03_2dcc45ade8e3.slice/crio-8e4d82a5a56647ac72030b592f0ec4db1c5d48f237953694140045840efa0b9c WatchSource:0}: Error finding container 8e4d82a5a56647ac72030b592f0ec4db1c5d48f237953694140045840efa0b9c: Status 404 returned error can't find the container with id 8e4d82a5a56647ac72030b592f0ec4db1c5d48f237953694140045840efa0b9c Oct 30 13:08:22.502795 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:22.502759 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-8d2170-qglwj" event={"ID":"d230415a-fe71-476a-bf03-2dcc45ade8e3","Type":"ContainerStarted","Data":"ac434971ef41724fb515a04cc7e9e1e359ff18d513d91b370efc64fad66cec8d"} Oct 30 13:08:22.502986 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:22.502853 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-8d2170-qglwj" event={"ID":"d230415a-fe71-476a-bf03-2dcc45ade8e3","Type":"ContainerStarted","Data":"8e4d82a5a56647ac72030b592f0ec4db1c5d48f237953694140045840efa0b9c"} Oct 30 13:08:22.546644 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:22.546595 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-8d2170-qglwj" podStartSLOduration=1.5465816989999999 podStartE2EDuration="1.546581699s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:08:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:08:22.545368756 +0000 UTC m=+4459.059797501" watchObservedRunningTime="2026-10-30 13:08:22.546581699 +0000 UTC m=+4459.061010445" Oct 30 13:08:23.167504 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:23.167455 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e4a0f82-fsmv4"] Oct 30 13:08:23.171296 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:23.171269 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e4a0f82-fsmv4" Oct 30 13:08:23.193502 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:23.193461 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e4a0f82-fsmv4"] Oct 30 13:08:23.245602 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:23.245567 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/185877e0-bfbb-48fa-9c23-ebf709195402-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1e4a0f82-fsmv4\" (UID: \"185877e0-bfbb-48fa-9c23-ebf709195402\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e4a0f82-fsmv4" Oct 30 13:08:23.245789 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:23.245617 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mxk4\" (UniqueName: \"kubernetes.io/projected/185877e0-bfbb-48fa-9c23-ebf709195402-kube-api-access-2mxk4\") pod \"maas-api-revoke-sub-1e4a0f82-fsmv4\" (UID: \"185877e0-bfbb-48fa-9c23-ebf709195402\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e4a0f82-fsmv4" Oct 30 13:08:23.245789 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:23.245699 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/185877e0-bfbb-48fa-9c23-ebf709195402-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1e4a0f82-fsmv4\" (UID: \"185877e0-bfbb-48fa-9c23-ebf709195402\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e4a0f82-fsmv4" Oct 30 13:08:23.347036 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:23.346955 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mxk4\" (UniqueName: \"kubernetes.io/projected/185877e0-bfbb-48fa-9c23-ebf709195402-kube-api-access-2mxk4\") pod \"maas-api-revoke-sub-1e4a0f82-fsmv4\" (UID: \"185877e0-bfbb-48fa-9c23-ebf709195402\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e4a0f82-fsmv4" Oct 30 13:08:23.347036 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:23.346998 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/185877e0-bfbb-48fa-9c23-ebf709195402-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1e4a0f82-fsmv4\" (UID: \"185877e0-bfbb-48fa-9c23-ebf709195402\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e4a0f82-fsmv4" Oct 30 13:08:23.347228 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:23.347087 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/185877e0-bfbb-48fa-9c23-ebf709195402-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1e4a0f82-fsmv4\" (UID: \"185877e0-bfbb-48fa-9c23-ebf709195402\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e4a0f82-fsmv4" Oct 30 13:08:23.347759 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:23.347737 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/185877e0-bfbb-48fa-9c23-ebf709195402-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1e4a0f82-fsmv4\" (UID: \"185877e0-bfbb-48fa-9c23-ebf709195402\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e4a0f82-fsmv4" Oct 30 13:08:23.349482 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:23.349459 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/185877e0-bfbb-48fa-9c23-ebf709195402-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1e4a0f82-fsmv4\" (UID: \"185877e0-bfbb-48fa-9c23-ebf709195402\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e4a0f82-fsmv4" Oct 30 13:08:23.371868 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:23.371810 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mxk4\" (UniqueName: \"kubernetes.io/projected/185877e0-bfbb-48fa-9c23-ebf709195402-kube-api-access-2mxk4\") pod \"maas-api-revoke-sub-1e4a0f82-fsmv4\" (UID: \"185877e0-bfbb-48fa-9c23-ebf709195402\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e4a0f82-fsmv4" Oct 30 13:08:23.487331 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:23.487290 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e4a0f82-fsmv4" Oct 30 13:08:23.517495 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:23.517461 4908 generic.go:356] "Generic (PLEG): container finished" podID="d230415a-fe71-476a-bf03-2dcc45ade8e3" containerID="ac434971ef41724fb515a04cc7e9e1e359ff18d513d91b370efc64fad66cec8d" exitCode=0 Oct 30 13:08:23.517630 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:23.517554 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-8d2170-qglwj" event={"ID":"d230415a-fe71-476a-bf03-2dcc45ade8e3","Type":"ContainerDied","Data":"ac434971ef41724fb515a04cc7e9e1e359ff18d513d91b370efc64fad66cec8d"} Oct 30 13:08:23.607406 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:23.607327 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd2cbeda-dbc4-41f6-a730-e1c6f9343c5e" path="/var/lib/kubelet/pods/fd2cbeda-dbc4-41f6-a730-e1c6f9343c5e/volumes" Oct 30 13:08:23.803856 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:23.803804 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e4a0f82-fsmv4"] Oct 30 13:08:23.805959 ip-10-0-141-251 kubenswrapper[4908]: W1030 13:08:23.805932 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod185877e0_bfbb_48fa_9c23_ebf709195402.slice/crio-a3e1f2f1f3de238389e696fcf3f57e80f6017b00e2a4f43dc6635144dbcf6182 WatchSource:0}: Error finding container a3e1f2f1f3de238389e696fcf3f57e80f6017b00e2a4f43dc6635144dbcf6182: Status 404 returned error can't find the container with id a3e1f2f1f3de238389e696fcf3f57e80f6017b00e2a4f43dc6635144dbcf6182 Oct 30 13:08:24.525925 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:24.525882 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e4a0f82-fsmv4" event={"ID":"185877e0-bfbb-48fa-9c23-ebf709195402","Type":"ContainerStarted","Data":"42b19d742b9a402434bd879c2798105b11c666fbf11b45a5a036be65c18632ad"} Oct 30 13:08:24.526388 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:24.525944 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e4a0f82-fsmv4" event={"ID":"185877e0-bfbb-48fa-9c23-ebf709195402","Type":"ContainerStarted","Data":"a3e1f2f1f3de238389e696fcf3f57e80f6017b00e2a4f43dc6635144dbcf6182"} Oct 30 13:08:24.571379 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:24.571320 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e4a0f82-fsmv4" podStartSLOduration=1.5713043880000002 podStartE2EDuration="1.571304388s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:08:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:08:24.569260207 +0000 UTC m=+4461.083688954" watchObservedRunningTime="2026-10-30 13:08:24.571304388 +0000 UTC m=+4461.085733135" Oct 30 13:08:24.681718 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:24.681689 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-8d2170-qglwj" Oct 30 13:08:24.759113 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:24.759077 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d230415a-fe71-476a-bf03-2dcc45ade8e3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d230415a-fe71-476a-bf03-2dcc45ade8e3-maas-api-openshift-ca\") pod \"d230415a-fe71-476a-bf03-2dcc45ade8e3\" (UID: \"d230415a-fe71-476a-bf03-2dcc45ade8e3\") " Oct 30 13:08:24.759315 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:24.759218 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d230415a-fe71-476a-bf03-2dcc45ade8e3-kube-api-access-jl2k2\" (UniqueName: \"kubernetes.io/projected/d230415a-fe71-476a-bf03-2dcc45ade8e3-kube-api-access-jl2k2\") pod \"d230415a-fe71-476a-bf03-2dcc45ade8e3\" (UID: \"d230415a-fe71-476a-bf03-2dcc45ade8e3\") " Oct 30 13:08:24.759315 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:24.759272 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d230415a-fe71-476a-bf03-2dcc45ade8e3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d230415a-fe71-476a-bf03-2dcc45ade8e3-maas-api-xks-ca\") pod \"d230415a-fe71-476a-bf03-2dcc45ade8e3\" (UID: \"d230415a-fe71-476a-bf03-2dcc45ade8e3\") " Oct 30 13:08:24.759521 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:24.759492 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d230415a-fe71-476a-bf03-2dcc45ade8e3-maas-api-openshift-ca" pod "d230415a-fe71-476a-bf03-2dcc45ade8e3" (UID: "d230415a-fe71-476a-bf03-2dcc45ade8e3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:08:24.761496 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:24.761468 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d230415a-fe71-476a-bf03-2dcc45ade8e3-maas-api-xks-ca" pod "d230415a-fe71-476a-bf03-2dcc45ade8e3" (UID: "d230415a-fe71-476a-bf03-2dcc45ade8e3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:08:24.761642 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:24.761491 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d230415a-fe71-476a-bf03-2dcc45ade8e3-kube-api-access-jl2k2" pod "d230415a-fe71-476a-bf03-2dcc45ade8e3" (UID: "d230415a-fe71-476a-bf03-2dcc45ade8e3"). InnerVolumeSpecName "kube-api-access-jl2k2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:08:24.861091 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:24.860991 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d230415a-fe71-476a-bf03-2dcc45ade8e3-maas-api-xks-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:08:24.861091 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:24.861028 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d230415a-fe71-476a-bf03-2dcc45ade8e3-maas-api-openshift-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:08:24.861091 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:24.861041 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jl2k2\" (UniqueName: \"kubernetes.io/projected/d230415a-fe71-476a-bf03-2dcc45ade8e3-kube-api-access-jl2k2\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:08:25.535165 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:25.535124 4908 generic.go:356] "Generic (PLEG): container finished" podID="185877e0-bfbb-48fa-9c23-ebf709195402" containerID="42b19d742b9a402434bd879c2798105b11c666fbf11b45a5a036be65c18632ad" exitCode=0 Oct 30 13:08:25.535673 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:25.535207 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e4a0f82-fsmv4" event={"ID":"185877e0-bfbb-48fa-9c23-ebf709195402","Type":"ContainerDied","Data":"42b19d742b9a402434bd879c2798105b11c666fbf11b45a5a036be65c18632ad"} Oct 30 13:08:25.536704 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:25.536682 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-8d2170-qglwj" Oct 30 13:08:25.536822 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:25.536714 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-8d2170-qglwj" event={"ID":"d230415a-fe71-476a-bf03-2dcc45ade8e3","Type":"ContainerDied","Data":"8e4d82a5a56647ac72030b592f0ec4db1c5d48f237953694140045840efa0b9c"} Oct 30 13:08:25.536822 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:25.536750 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8e4d82a5a56647ac72030b592f0ec4db1c5d48f237953694140045840efa0b9c" Oct 30 13:08:26.027725 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:26.027685 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-47fd99-nwtr6"] Oct 30 13:08:26.031293 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:26.031270 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-47fd99-nwtr6"] Oct 30 13:08:26.696295 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:26.696268 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e4a0f82-fsmv4" Oct 30 13:08:26.778095 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:26.778059 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/185877e0-bfbb-48fa-9c23-ebf709195402-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/185877e0-bfbb-48fa-9c23-ebf709195402-maas-api-openshift-ca\") pod \"185877e0-bfbb-48fa-9c23-ebf709195402\" (UID: \"185877e0-bfbb-48fa-9c23-ebf709195402\") " Oct 30 13:08:26.778316 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:26.778161 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/185877e0-bfbb-48fa-9c23-ebf709195402-kube-api-access-2mxk4\" (UniqueName: \"kubernetes.io/projected/185877e0-bfbb-48fa-9c23-ebf709195402-kube-api-access-2mxk4\") pod \"185877e0-bfbb-48fa-9c23-ebf709195402\" (UID: \"185877e0-bfbb-48fa-9c23-ebf709195402\") " Oct 30 13:08:26.778316 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:26.778224 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/185877e0-bfbb-48fa-9c23-ebf709195402-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/185877e0-bfbb-48fa-9c23-ebf709195402-maas-api-xks-ca\") pod \"185877e0-bfbb-48fa-9c23-ebf709195402\" (UID: \"185877e0-bfbb-48fa-9c23-ebf709195402\") " Oct 30 13:08:26.778473 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:26.778447 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/185877e0-bfbb-48fa-9c23-ebf709195402-maas-api-openshift-ca" pod "185877e0-bfbb-48fa-9c23-ebf709195402" (UID: "185877e0-bfbb-48fa-9c23-ebf709195402"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:08:26.778596 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:26.778578 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/185877e0-bfbb-48fa-9c23-ebf709195402-maas-api-openshift-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:08:26.780401 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:26.780375 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/185877e0-bfbb-48fa-9c23-ebf709195402-maas-api-xks-ca" pod "185877e0-bfbb-48fa-9c23-ebf709195402" (UID: "185877e0-bfbb-48fa-9c23-ebf709195402"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:08:26.780526 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:26.780467 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/185877e0-bfbb-48fa-9c23-ebf709195402-kube-api-access-2mxk4" pod "185877e0-bfbb-48fa-9c23-ebf709195402" (UID: "185877e0-bfbb-48fa-9c23-ebf709195402"). InnerVolumeSpecName "kube-api-access-2mxk4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:08:26.879347 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:26.879263 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/185877e0-bfbb-48fa-9c23-ebf709195402-maas-api-xks-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:08:26.879347 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:26.879295 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2mxk4\" (UniqueName: \"kubernetes.io/projected/185877e0-bfbb-48fa-9c23-ebf709195402-kube-api-access-2mxk4\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:08:27.552878 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:27.552840 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e4a0f82-fsmv4" Oct 30 13:08:27.553039 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:27.552873 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e4a0f82-fsmv4" event={"ID":"185877e0-bfbb-48fa-9c23-ebf709195402","Type":"ContainerDied","Data":"a3e1f2f1f3de238389e696fcf3f57e80f6017b00e2a4f43dc6635144dbcf6182"} Oct 30 13:08:27.553039 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:27.552934 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a3e1f2f1f3de238389e696fcf3f57e80f6017b00e2a4f43dc6635144dbcf6182" Oct 30 13:08:27.607539 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:27.607506 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5e36bcb-b201-4c99-be6c-1d632ba46503" path="/var/lib/kubelet/pods/e5e36bcb-b201-4c99-be6c-1d632ba46503/volumes" Oct 30 13:08:29.760124 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:29.760074 4908 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-6a7148/distinct-2-w3-6a7148-kserve-67bd5fb976-9nfff" podUID="6046cc02-29fe-4f59-b522-8f57f4557277" containerName="main" containerID="cri-o://c0e007f741bf504d35e94c7a7d5ff0af3d7c5505c8fd05bad82dca1037226ad6" gracePeriod=45 Oct 30 13:08:29.778687 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:29.778651 4908 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-6a7148/premium-w3-6a7148-kserve-67b976c888-tgd6c" podUID="13de4441-e7f6-4cbd-97d9-0f079d11cb8e" containerName="main" containerID="cri-o://0c2b0c30219fc8431eaeece2d6b59fba5f7f81e9b1b3409e606e6f926c591388" gracePeriod=45 Oct 30 13:08:29.940288 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:29.939986 4908 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6a7148/distinct-2-w3-6a7148-kserve-67bd5fb976-9nfff" podUID="6046cc02-29fe-4f59-b522-8f57f4557277" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": dial tcp 10.132.0.97:8000: connect: connection refused" Oct 30 13:08:30.088677 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:30.088651 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-6a7148/distinct-2-w3-6a7148-kserve-67bd5fb976-9nfff" Oct 30 13:08:30.092012 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:30.091985 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-6a7148/premium-w3-6a7148-kserve-67b976c888-tgd6c" Oct 30 13:08:30.221130 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:30.221081 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13de4441-e7f6-4cbd-97d9-0f079d11cb8e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13de4441-e7f6-4cbd-97d9-0f079d11cb8e-model-cache\") pod \"13de4441-e7f6-4cbd-97d9-0f079d11cb8e\" (UID: \"13de4441-e7f6-4cbd-97d9-0f079d11cb8e\") " Oct 30 13:08:30.221313 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:30.221142 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13de4441-e7f6-4cbd-97d9-0f079d11cb8e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/13de4441-e7f6-4cbd-97d9-0f079d11cb8e-dshm\") pod \"13de4441-e7f6-4cbd-97d9-0f079d11cb8e\" (UID: \"13de4441-e7f6-4cbd-97d9-0f079d11cb8e\") " Oct 30 13:08:30.221313 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:30.221181 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6046cc02-29fe-4f59-b522-8f57f4557277-home\" (UniqueName: \"kubernetes.io/empty-dir/6046cc02-29fe-4f59-b522-8f57f4557277-home\") pod \"6046cc02-29fe-4f59-b522-8f57f4557277\" (UID: \"6046cc02-29fe-4f59-b522-8f57f4557277\") " Oct 30 13:08:30.221313 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:30.221231 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13de4441-e7f6-4cbd-97d9-0f079d11cb8e-kube-api-access-tbwmq\" (UniqueName: \"kubernetes.io/projected/13de4441-e7f6-4cbd-97d9-0f079d11cb8e-kube-api-access-tbwmq\") pod \"13de4441-e7f6-4cbd-97d9-0f079d11cb8e\" (UID: \"13de4441-e7f6-4cbd-97d9-0f079d11cb8e\") " Oct 30 13:08:30.221313 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:30.221264 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6046cc02-29fe-4f59-b522-8f57f4557277-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6046cc02-29fe-4f59-b522-8f57f4557277-tmp-dir\") pod \"6046cc02-29fe-4f59-b522-8f57f4557277\" (UID: \"6046cc02-29fe-4f59-b522-8f57f4557277\") " Oct 30 13:08:30.221313 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:30.221297 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/13de4441-e7f6-4cbd-97d9-0f079d11cb8e-tls-certs\" (UniqueName: \"kubernetes.io/secret/13de4441-e7f6-4cbd-97d9-0f079d11cb8e-tls-certs\") pod \"13de4441-e7f6-4cbd-97d9-0f079d11cb8e\" (UID: \"13de4441-e7f6-4cbd-97d9-0f079d11cb8e\") " Oct 30 13:08:30.221489 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:30.221332 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6046cc02-29fe-4f59-b522-8f57f4557277-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6046cc02-29fe-4f59-b522-8f57f4557277-model-cache\") pod \"6046cc02-29fe-4f59-b522-8f57f4557277\" (UID: \"6046cc02-29fe-4f59-b522-8f57f4557277\") " Oct 30 13:08:30.221489 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:30.221364 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13de4441-e7f6-4cbd-97d9-0f079d11cb8e-home\" (UniqueName: \"kubernetes.io/empty-dir/13de4441-e7f6-4cbd-97d9-0f079d11cb8e-home\") pod \"13de4441-e7f6-4cbd-97d9-0f079d11cb8e\" (UID: \"13de4441-e7f6-4cbd-97d9-0f079d11cb8e\") " Oct 30 13:08:30.221489 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:30.221390 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13de4441-e7f6-4cbd-97d9-0f079d11cb8e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13de4441-e7f6-4cbd-97d9-0f079d11cb8e-tmp-dir\") pod \"13de4441-e7f6-4cbd-97d9-0f079d11cb8e\" (UID: \"13de4441-e7f6-4cbd-97d9-0f079d11cb8e\") " Oct 30 13:08:30.221489 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:30.221393 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13de4441-e7f6-4cbd-97d9-0f079d11cb8e-model-cache" pod "13de4441-e7f6-4cbd-97d9-0f079d11cb8e" (UID: "13de4441-e7f6-4cbd-97d9-0f079d11cb8e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:08:30.221489 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:30.221444 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6046cc02-29fe-4f59-b522-8f57f4557277-tls-certs\" (UniqueName: \"kubernetes.io/secret/6046cc02-29fe-4f59-b522-8f57f4557277-tls-certs\") pod \"6046cc02-29fe-4f59-b522-8f57f4557277\" (UID: \"6046cc02-29fe-4f59-b522-8f57f4557277\") " Oct 30 13:08:30.221694 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:30.221637 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6046cc02-29fe-4f59-b522-8f57f4557277-tmp-dir" pod "6046cc02-29fe-4f59-b522-8f57f4557277" (UID: "6046cc02-29fe-4f59-b522-8f57f4557277"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:08:30.221819 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:30.221790 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13de4441-e7f6-4cbd-97d9-0f079d11cb8e-tmp-dir" pod "13de4441-e7f6-4cbd-97d9-0f079d11cb8e" (UID: "13de4441-e7f6-4cbd-97d9-0f079d11cb8e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:08:30.222020 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:30.221844 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6046cc02-29fe-4f59-b522-8f57f4557277-model-cache" pod "6046cc02-29fe-4f59-b522-8f57f4557277" (UID: "6046cc02-29fe-4f59-b522-8f57f4557277"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:08:30.222020 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:30.221901 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13de4441-e7f6-4cbd-97d9-0f079d11cb8e-home" pod "13de4441-e7f6-4cbd-97d9-0f079d11cb8e" (UID: "13de4441-e7f6-4cbd-97d9-0f079d11cb8e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:08:30.222269 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:30.222226 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6046cc02-29fe-4f59-b522-8f57f4557277-home" pod "6046cc02-29fe-4f59-b522-8f57f4557277" (UID: "6046cc02-29fe-4f59-b522-8f57f4557277"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:08:30.222639 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:30.222588 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6046cc02-29fe-4f59-b522-8f57f4557277-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6046cc02-29fe-4f59-b522-8f57f4557277-dshm\") pod \"6046cc02-29fe-4f59-b522-8f57f4557277\" (UID: \"6046cc02-29fe-4f59-b522-8f57f4557277\") " Oct 30 13:08:30.223325 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:30.222888 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6046cc02-29fe-4f59-b522-8f57f4557277-kube-api-access-mbwbp\" (UniqueName: \"kubernetes.io/projected/6046cc02-29fe-4f59-b522-8f57f4557277-kube-api-access-mbwbp\") pod \"6046cc02-29fe-4f59-b522-8f57f4557277\" (UID: \"6046cc02-29fe-4f59-b522-8f57f4557277\") " Oct 30 13:08:30.223325 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:30.223211 4908 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13de4441-e7f6-4cbd-97d9-0f079d11cb8e-model-cache\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:08:30.223325 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:30.223229 4908 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6046cc02-29fe-4f59-b522-8f57f4557277-home\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:08:30.223325 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:30.223242 4908 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6046cc02-29fe-4f59-b522-8f57f4557277-tmp-dir\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:08:30.223325 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:30.223255 4908 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6046cc02-29fe-4f59-b522-8f57f4557277-model-cache\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:08:30.223325 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:30.223267 4908 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13de4441-e7f6-4cbd-97d9-0f079d11cb8e-home\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:08:30.223325 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:30.223280 4908 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13de4441-e7f6-4cbd-97d9-0f079d11cb8e-tmp-dir\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:08:30.224492 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:30.224170 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13de4441-e7f6-4cbd-97d9-0f079d11cb8e-dshm" pod "13de4441-e7f6-4cbd-97d9-0f079d11cb8e" (UID: "13de4441-e7f6-4cbd-97d9-0f079d11cb8e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:08:30.224492 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:30.224262 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13de4441-e7f6-4cbd-97d9-0f079d11cb8e-kube-api-access-tbwmq" pod "13de4441-e7f6-4cbd-97d9-0f079d11cb8e" (UID: "13de4441-e7f6-4cbd-97d9-0f079d11cb8e"). InnerVolumeSpecName "kube-api-access-tbwmq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:08:30.224645 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:30.224517 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13de4441-e7f6-4cbd-97d9-0f079d11cb8e-tls-certs" pod "13de4441-e7f6-4cbd-97d9-0f079d11cb8e" (UID: "13de4441-e7f6-4cbd-97d9-0f079d11cb8e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:08:30.224911 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:30.224883 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6046cc02-29fe-4f59-b522-8f57f4557277-dshm" pod "6046cc02-29fe-4f59-b522-8f57f4557277" (UID: "6046cc02-29fe-4f59-b522-8f57f4557277"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:08:30.225205 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:30.225186 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6046cc02-29fe-4f59-b522-8f57f4557277-tls-certs" pod "6046cc02-29fe-4f59-b522-8f57f4557277" (UID: "6046cc02-29fe-4f59-b522-8f57f4557277"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:08:30.225863 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:30.225807 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6046cc02-29fe-4f59-b522-8f57f4557277-kube-api-access-mbwbp" pod "6046cc02-29fe-4f59-b522-8f57f4557277" (UID: "6046cc02-29fe-4f59-b522-8f57f4557277"). InnerVolumeSpecName "kube-api-access-mbwbp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:08:30.324107 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:30.324015 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbwmq\" (UniqueName: \"kubernetes.io/projected/13de4441-e7f6-4cbd-97d9-0f079d11cb8e-kube-api-access-tbwmq\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:08:30.324107 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:30.324051 4908 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13de4441-e7f6-4cbd-97d9-0f079d11cb8e-tls-certs\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:08:30.324107 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:30.324063 4908 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6046cc02-29fe-4f59-b522-8f57f4557277-tls-certs\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:08:30.324107 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:30.324071 4908 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6046cc02-29fe-4f59-b522-8f57f4557277-dshm\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:08:30.324107 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:30.324079 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mbwbp\" (UniqueName: \"kubernetes.io/projected/6046cc02-29fe-4f59-b522-8f57f4557277-kube-api-access-mbwbp\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:08:30.324107 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:30.324089 4908 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13de4441-e7f6-4cbd-97d9-0f079d11cb8e-dshm\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:08:30.578047 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:30.577943 4908 generic.go:356] "Generic (PLEG): container finished" podID="6046cc02-29fe-4f59-b522-8f57f4557277" containerID="c0e007f741bf504d35e94c7a7d5ff0af3d7c5505c8fd05bad82dca1037226ad6" exitCode=0 Oct 30 13:08:30.578047 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:30.577988 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-6a7148/distinct-2-w3-6a7148-kserve-67bd5fb976-9nfff" Oct 30 13:08:30.578047 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:30.578028 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-6a7148/distinct-2-w3-6a7148-kserve-67bd5fb976-9nfff" event={"ID":"6046cc02-29fe-4f59-b522-8f57f4557277","Type":"ContainerDied","Data":"c0e007f741bf504d35e94c7a7d5ff0af3d7c5505c8fd05bad82dca1037226ad6"} Oct 30 13:08:30.578327 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:30.578075 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-6a7148/distinct-2-w3-6a7148-kserve-67bd5fb976-9nfff" event={"ID":"6046cc02-29fe-4f59-b522-8f57f4557277","Type":"ContainerDied","Data":"53a13d2aa52225f36269c9fde43848eea68c017bbefdb9a46b030177c8e73a7f"} Oct 30 13:08:30.578327 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:30.578098 4908 scope.go:122] "RemoveContainer" containerID="c0e007f741bf504d35e94c7a7d5ff0af3d7c5505c8fd05bad82dca1037226ad6" Oct 30 13:08:30.580208 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:30.580183 4908 generic.go:356] "Generic (PLEG): container finished" podID="13de4441-e7f6-4cbd-97d9-0f079d11cb8e" containerID="0c2b0c30219fc8431eaeece2d6b59fba5f7f81e9b1b3409e606e6f926c591388" exitCode=0 Oct 30 13:08:30.580388 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:30.580215 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-6a7148/premium-w3-6a7148-kserve-67b976c888-tgd6c" event={"ID":"13de4441-e7f6-4cbd-97d9-0f079d11cb8e","Type":"ContainerDied","Data":"0c2b0c30219fc8431eaeece2d6b59fba5f7f81e9b1b3409e606e6f926c591388"} Oct 30 13:08:30.580388 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:30.580240 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-6a7148/premium-w3-6a7148-kserve-67b976c888-tgd6c" Oct 30 13:08:30.580388 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:30.580254 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-6a7148/premium-w3-6a7148-kserve-67b976c888-tgd6c" event={"ID":"13de4441-e7f6-4cbd-97d9-0f079d11cb8e","Type":"ContainerDied","Data":"30a5a68a1d364f0a126a56f395a30865b8f4c48d5be11457c3d2e60d32a77dc1"} Oct 30 13:08:30.592261 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:30.592235 4908 scope.go:122] "RemoveContainer" containerID="c0e007f741bf504d35e94c7a7d5ff0af3d7c5505c8fd05bad82dca1037226ad6" Oct 30 13:08:30.592598 ip-10-0-141-251 kubenswrapper[4908]: E1030 13:08:30.592570 4908 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0e007f741bf504d35e94c7a7d5ff0af3d7c5505c8fd05bad82dca1037226ad6\": container with ID starting with c0e007f741bf504d35e94c7a7d5ff0af3d7c5505c8fd05bad82dca1037226ad6 not found: ID does not exist" containerID="c0e007f741bf504d35e94c7a7d5ff0af3d7c5505c8fd05bad82dca1037226ad6" Oct 30 13:08:30.592673 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:30.592611 4908 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0e007f741bf504d35e94c7a7d5ff0af3d7c5505c8fd05bad82dca1037226ad6"} err="failed to get container status \"c0e007f741bf504d35e94c7a7d5ff0af3d7c5505c8fd05bad82dca1037226ad6\": rpc error: code = NotFound desc = could not find container \"c0e007f741bf504d35e94c7a7d5ff0af3d7c5505c8fd05bad82dca1037226ad6\": container with ID starting with c0e007f741bf504d35e94c7a7d5ff0af3d7c5505c8fd05bad82dca1037226ad6 not found: ID does not exist" Oct 30 13:08:30.592673 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:30.592637 4908 scope.go:122] "RemoveContainer" containerID="0c2b0c30219fc8431eaeece2d6b59fba5f7f81e9b1b3409e606e6f926c591388" Oct 30 13:08:30.603250 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:30.603225 4908 scope.go:122] "RemoveContainer" containerID="0c2b0c30219fc8431eaeece2d6b59fba5f7f81e9b1b3409e606e6f926c591388" Oct 30 13:08:30.613371 ip-10-0-141-251 kubenswrapper[4908]: E1030 13:08:30.613330 4908 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c2b0c30219fc8431eaeece2d6b59fba5f7f81e9b1b3409e606e6f926c591388\": container with ID starting with 0c2b0c30219fc8431eaeece2d6b59fba5f7f81e9b1b3409e606e6f926c591388 not found: ID does not exist" containerID="0c2b0c30219fc8431eaeece2d6b59fba5f7f81e9b1b3409e606e6f926c591388" Oct 30 13:08:30.613531 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:30.613375 4908 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c2b0c30219fc8431eaeece2d6b59fba5f7f81e9b1b3409e606e6f926c591388"} err="failed to get container status \"0c2b0c30219fc8431eaeece2d6b59fba5f7f81e9b1b3409e606e6f926c591388\": rpc error: code = NotFound desc = could not find container \"0c2b0c30219fc8431eaeece2d6b59fba5f7f81e9b1b3409e606e6f926c591388\": container with ID starting with 0c2b0c30219fc8431eaeece2d6b59fba5f7f81e9b1b3409e606e6f926c591388 not found: ID does not exist" Oct 30 13:08:30.639202 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:30.639100 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-6a7148/premium-w3-6a7148-kserve-67b976c888-tgd6c"] Oct 30 13:08:30.644337 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:30.644306 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-6a7148/premium-w3-6a7148-kserve-67b976c888-tgd6c"] Oct 30 13:08:30.684171 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:30.684114 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-6a7148/distinct-2-w3-6a7148-kserve-67bd5fb976-9nfff"] Oct 30 13:08:30.688439 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:30.688407 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-6a7148/distinct-2-w3-6a7148-kserve-67bd5fb976-9nfff"] Oct 30 13:08:31.604525 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:31.604489 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13de4441-e7f6-4cbd-97d9-0f079d11cb8e" path="/var/lib/kubelet/pods/13de4441-e7f6-4cbd-97d9-0f079d11cb8e/volumes" Oct 30 13:08:31.605088 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:31.604935 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6046cc02-29fe-4f59-b522-8f57f4557277" path="/var/lib/kubelet/pods/6046cc02-29fe-4f59-b522-8f57f4557277/volumes" Oct 30 13:08:33.733414 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:33.733377 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-8d2170-qglwj"] Oct 30 13:08:33.740412 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:33.740377 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-8d2170-qglwj"] Oct 30 13:08:35.606497 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:35.606448 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d230415a-fe71-476a-bf03-2dcc45ade8e3" path="/var/lib/kubelet/pods/d230415a-fe71-476a-bf03-2dcc45ade8e3/volumes" Oct 30 13:08:39.478639 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:39.478604 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-6a7148-jnbbj"] Oct 30 13:08:39.491262 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:39.491223 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-6a7148-jnbbj"] Oct 30 13:08:39.604552 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:39.604515 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a0aaa55-6e3a-4791-8800-dbb7d27f8809" path="/var/lib/kubelet/pods/0a0aaa55-6e3a-4791-8800-dbb7d27f8809/volumes" Oct 30 13:08:45.181453 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:45.181408 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9wwfc"] Oct 30 13:08:46.838371 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:46.838235 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-512ec5/simulator-w4-512ec5-kserve-7df7674f8-8rqwb"] Oct 30 13:08:49.685857 ip-10-0-141-251 kubenswrapper[4908]: E1030 13:08:49.685777 4908 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vn6vv" containerName="extractor" Oct 30 13:08:55.746746 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:55.746703 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64a0fec2-r5hzg"] Oct 30 13:08:55.747365 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:55.747339 4908 cpu_manager.go:414] "containerMap: removing container" podUID="d230415a-fe71-476a-bf03-2dcc45ade8e3" containerName="revoke-keys" Oct 30 13:08:55.747453 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:55.747368 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d230415a-fe71-476a-bf03-2dcc45ade8e3" containerName="revoke-keys" Oct 30 13:08:55.747453 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:55.747400 4908 cpu_manager.go:414] "containerMap: removing container" podUID="6046cc02-29fe-4f59-b522-8f57f4557277" containerName="main" Oct 30 13:08:55.747453 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:55.747409 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6046cc02-29fe-4f59-b522-8f57f4557277" containerName="main" Oct 30 13:08:55.747453 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:55.747432 4908 cpu_manager.go:414] "containerMap: removing container" podUID="185877e0-bfbb-48fa-9c23-ebf709195402" containerName="revoke-keys" Oct 30 13:08:55.747453 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:55.747441 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="185877e0-bfbb-48fa-9c23-ebf709195402" containerName="revoke-keys" Oct 30 13:08:55.747453 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:55.747453 4908 cpu_manager.go:414] "containerMap: removing container" podUID="13de4441-e7f6-4cbd-97d9-0f079d11cb8e" containerName="main" Oct 30 13:08:55.747675 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:55.747461 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13de4441-e7f6-4cbd-97d9-0f079d11cb8e" containerName="main" Oct 30 13:08:55.747675 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:55.747621 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="d230415a-fe71-476a-bf03-2dcc45ade8e3" containerName="revoke-keys" Oct 30 13:08:55.747675 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:55.747638 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="6046cc02-29fe-4f59-b522-8f57f4557277" containerName="main" Oct 30 13:08:55.747675 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:55.747650 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="185877e0-bfbb-48fa-9c23-ebf709195402" containerName="revoke-keys" Oct 30 13:08:55.747675 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:55.747658 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="13de4441-e7f6-4cbd-97d9-0f079d11cb8e" containerName="main" Oct 30 13:08:55.751768 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:55.751748 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64a0fec2-r5hzg" Oct 30 13:08:55.769950 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:55.768851 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64a0fec2-r5hzg"] Oct 30 13:08:55.779100 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:55.779066 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 13:08:55.779244 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:55.779149 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 13:08:55.786538 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:55.786505 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2qbb\" (UniqueName: \"kubernetes.io/projected/8a632b40-348a-4fcb-b5ed-fc8d1b6d5cd4-kube-api-access-d2qbb\") pod \"maas-api-revoke-sub-64a0fec2-r5hzg\" (UID: \"8a632b40-348a-4fcb-b5ed-fc8d1b6d5cd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64a0fec2-r5hzg" Oct 30 13:08:55.786666 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:55.786565 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a632b40-348a-4fcb-b5ed-fc8d1b6d5cd4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-64a0fec2-r5hzg\" (UID: \"8a632b40-348a-4fcb-b5ed-fc8d1b6d5cd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64a0fec2-r5hzg" Oct 30 13:08:55.786727 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:55.786701 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a632b40-348a-4fcb-b5ed-fc8d1b6d5cd4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-64a0fec2-r5hzg\" (UID: \"8a632b40-348a-4fcb-b5ed-fc8d1b6d5cd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64a0fec2-r5hzg" Oct 30 13:08:55.842027 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:55.841998 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:08:55.842174 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:55.842112 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s75hq\"" Oct 30 13:08:55.887964 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:55.887927 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2qbb\" (UniqueName: \"kubernetes.io/projected/8a632b40-348a-4fcb-b5ed-fc8d1b6d5cd4-kube-api-access-d2qbb\") pod \"maas-api-revoke-sub-64a0fec2-r5hzg\" (UID: \"8a632b40-348a-4fcb-b5ed-fc8d1b6d5cd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64a0fec2-r5hzg" Oct 30 13:08:55.888128 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:55.887988 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a632b40-348a-4fcb-b5ed-fc8d1b6d5cd4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-64a0fec2-r5hzg\" (UID: \"8a632b40-348a-4fcb-b5ed-fc8d1b6d5cd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64a0fec2-r5hzg" Oct 30 13:08:55.888128 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:55.888108 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a632b40-348a-4fcb-b5ed-fc8d1b6d5cd4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-64a0fec2-r5hzg\" (UID: \"8a632b40-348a-4fcb-b5ed-fc8d1b6d5cd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64a0fec2-r5hzg" Oct 30 13:08:55.889012 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:55.888983 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a632b40-348a-4fcb-b5ed-fc8d1b6d5cd4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-64a0fec2-r5hzg\" (UID: \"8a632b40-348a-4fcb-b5ed-fc8d1b6d5cd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64a0fec2-r5hzg" Oct 30 13:08:55.890657 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:55.890632 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a632b40-348a-4fcb-b5ed-fc8d1b6d5cd4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-64a0fec2-r5hzg\" (UID: \"8a632b40-348a-4fcb-b5ed-fc8d1b6d5cd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64a0fec2-r5hzg" Oct 30 13:08:55.912526 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:55.912495 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2qbb\" (UniqueName: \"kubernetes.io/projected/8a632b40-348a-4fcb-b5ed-fc8d1b6d5cd4-kube-api-access-d2qbb\") pod \"maas-api-revoke-sub-64a0fec2-r5hzg\" (UID: \"8a632b40-348a-4fcb-b5ed-fc8d1b6d5cd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64a0fec2-r5hzg" Oct 30 13:08:56.073261 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:56.073182 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64a0fec2-r5hzg" Oct 30 13:08:56.378306 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:56.378262 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64a0fec2-r5hzg"] Oct 30 13:08:56.379644 ip-10-0-141-251 kubenswrapper[4908]: W1030 13:08:56.379613 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a632b40_348a_4fcb_b5ed_fc8d1b6d5cd4.slice/crio-a1a64a7bca0147c18024aa6ba49412e0528f2124d34199117001e6e6030606c5 WatchSource:0}: Error finding container a1a64a7bca0147c18024aa6ba49412e0528f2124d34199117001e6e6030606c5: Status 404 returned error can't find the container with id a1a64a7bca0147c18024aa6ba49412e0528f2124d34199117001e6e6030606c5 Oct 30 13:08:56.795139 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:56.795096 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64a0fec2-r5hzg" event={"ID":"8a632b40-348a-4fcb-b5ed-fc8d1b6d5cd4","Type":"ContainerStarted","Data":"f80a3f6bc7ef2e6649d3dc212b25a013444315f23f669cc462230c9a0d4d9b52"} Oct 30 13:08:56.795588 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:56.795165 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64a0fec2-r5hzg" event={"ID":"8a632b40-348a-4fcb-b5ed-fc8d1b6d5cd4","Type":"ContainerStarted","Data":"a1a64a7bca0147c18024aa6ba49412e0528f2124d34199117001e6e6030606c5"} Oct 30 13:08:56.840399 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:56.840332 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64a0fec2-r5hzg" podStartSLOduration=1.84031093 podStartE2EDuration="1.84031093s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:08:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:08:56.838009808 +0000 UTC m=+4493.352438573" watchObservedRunningTime="2026-10-30 13:08:56.84031093 +0000 UTC m=+4493.354739679" Oct 30 13:08:57.040330 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:57.040292 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c8c8523-lwhzc"] Oct 30 13:08:57.045191 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:57.045154 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c8c8523-lwhzc"] Oct 30 13:08:57.615623 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:57.615578 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00b2ae7c-2c85-4f84-b5a3-a1d3ef7ca4e6" path="/var/lib/kubelet/pods/00b2ae7c-2c85-4f84-b5a3-a1d3ef7ca4e6/volumes" Oct 30 13:08:57.805742 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:57.805698 4908 generic.go:356] "Generic (PLEG): container finished" podID="8a632b40-348a-4fcb-b5ed-fc8d1b6d5cd4" containerID="f80a3f6bc7ef2e6649d3dc212b25a013444315f23f669cc462230c9a0d4d9b52" exitCode=0 Oct 30 13:08:57.806248 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:57.805783 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64a0fec2-r5hzg" event={"ID":"8a632b40-348a-4fcb-b5ed-fc8d1b6d5cd4","Type":"ContainerDied","Data":"f80a3f6bc7ef2e6649d3dc212b25a013444315f23f669cc462230c9a0d4d9b52"} Oct 30 13:08:58.952476 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:58.952447 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64a0fec2-r5hzg" Oct 30 13:08:59.017381 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:59.017334 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a632b40-348a-4fcb-b5ed-fc8d1b6d5cd4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a632b40-348a-4fcb-b5ed-fc8d1b6d5cd4-maas-api-openshift-ca\") pod \"8a632b40-348a-4fcb-b5ed-fc8d1b6d5cd4\" (UID: \"8a632b40-348a-4fcb-b5ed-fc8d1b6d5cd4\") " Oct 30 13:08:59.017602 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:59.017421 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a632b40-348a-4fcb-b5ed-fc8d1b6d5cd4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a632b40-348a-4fcb-b5ed-fc8d1b6d5cd4-maas-api-xks-ca\") pod \"8a632b40-348a-4fcb-b5ed-fc8d1b6d5cd4\" (UID: \"8a632b40-348a-4fcb-b5ed-fc8d1b6d5cd4\") " Oct 30 13:08:59.017602 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:59.017543 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a632b40-348a-4fcb-b5ed-fc8d1b6d5cd4-kube-api-access-d2qbb\" (UniqueName: \"kubernetes.io/projected/8a632b40-348a-4fcb-b5ed-fc8d1b6d5cd4-kube-api-access-d2qbb\") pod \"8a632b40-348a-4fcb-b5ed-fc8d1b6d5cd4\" (UID: \"8a632b40-348a-4fcb-b5ed-fc8d1b6d5cd4\") " Oct 30 13:08:59.017751 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:59.017730 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a632b40-348a-4fcb-b5ed-fc8d1b6d5cd4-maas-api-openshift-ca" pod "8a632b40-348a-4fcb-b5ed-fc8d1b6d5cd4" (UID: "8a632b40-348a-4fcb-b5ed-fc8d1b6d5cd4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:08:59.017869 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:59.017848 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a632b40-348a-4fcb-b5ed-fc8d1b6d5cd4-maas-api-openshift-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:08:59.019672 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:59.019643 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a632b40-348a-4fcb-b5ed-fc8d1b6d5cd4-maas-api-xks-ca" pod "8a632b40-348a-4fcb-b5ed-fc8d1b6d5cd4" (UID: "8a632b40-348a-4fcb-b5ed-fc8d1b6d5cd4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:08:59.019798 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:59.019695 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a632b40-348a-4fcb-b5ed-fc8d1b6d5cd4-kube-api-access-d2qbb" pod "8a632b40-348a-4fcb-b5ed-fc8d1b6d5cd4" (UID: "8a632b40-348a-4fcb-b5ed-fc8d1b6d5cd4"). InnerVolumeSpecName "kube-api-access-d2qbb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:08:59.119452 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:59.119371 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2qbb\" (UniqueName: \"kubernetes.io/projected/8a632b40-348a-4fcb-b5ed-fc8d1b6d5cd4-kube-api-access-d2qbb\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:08:59.119452 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:59.119404 4908 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a632b40-348a-4fcb-b5ed-fc8d1b6d5cd4-maas-api-xks-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:08:59.823354 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:59.823319 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64a0fec2-r5hzg" Oct 30 13:08:59.823546 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:59.823348 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64a0fec2-r5hzg" event={"ID":"8a632b40-348a-4fcb-b5ed-fc8d1b6d5cd4","Type":"ContainerDied","Data":"a1a64a7bca0147c18024aa6ba49412e0528f2124d34199117001e6e6030606c5"} Oct 30 13:08:59.823546 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:08:59.823404 4908 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a1a64a7bca0147c18024aa6ba49412e0528f2124d34199117001e6e6030606c5" Oct 30 13:09:01.859112 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:09:01.859075 4908 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-512ec5/simulator-w4-512ec5-kserve-7df7674f8-8rqwb" podUID="8a5dfcc9-dec3-4bc6-b0a8-c928350b3596" containerName="main" containerID="cri-o://6d87953e0148ef1a9ab936760bbceec78ad43bbf2d54319c94c7cbded807b647" gracePeriod=45 Oct 30 13:09:02.105923 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:09:02.105894 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-512ec5/simulator-w4-512ec5-kserve-7df7674f8-8rqwb" Oct 30 13:09:02.251561 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:09:02.251521 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a5dfcc9-dec3-4bc6-b0a8-c928350b3596-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a5dfcc9-dec3-4bc6-b0a8-c928350b3596-tmp-dir\") pod \"8a5dfcc9-dec3-4bc6-b0a8-c928350b3596\" (UID: \"8a5dfcc9-dec3-4bc6-b0a8-c928350b3596\") " Oct 30 13:09:02.251740 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:09:02.251583 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a5dfcc9-dec3-4bc6-b0a8-c928350b3596-home\" (UniqueName: \"kubernetes.io/empty-dir/8a5dfcc9-dec3-4bc6-b0a8-c928350b3596-home\") pod \"8a5dfcc9-dec3-4bc6-b0a8-c928350b3596\" (UID: \"8a5dfcc9-dec3-4bc6-b0a8-c928350b3596\") " Oct 30 13:09:02.251740 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:09:02.251641 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a5dfcc9-dec3-4bc6-b0a8-c928350b3596-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a5dfcc9-dec3-4bc6-b0a8-c928350b3596-model-cache\") pod \"8a5dfcc9-dec3-4bc6-b0a8-c928350b3596\" (UID: \"8a5dfcc9-dec3-4bc6-b0a8-c928350b3596\") " Oct 30 13:09:02.251740 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:09:02.251673 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a5dfcc9-dec3-4bc6-b0a8-c928350b3596-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a5dfcc9-dec3-4bc6-b0a8-c928350b3596-dshm\") pod \"8a5dfcc9-dec3-4bc6-b0a8-c928350b3596\" (UID: \"8a5dfcc9-dec3-4bc6-b0a8-c928350b3596\") " Oct 30 13:09:02.251740 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:09:02.251701 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a5dfcc9-dec3-4bc6-b0a8-c928350b3596-kube-api-access-xp6wz\" (UniqueName: \"kubernetes.io/projected/8a5dfcc9-dec3-4bc6-b0a8-c928350b3596-kube-api-access-xp6wz\") pod \"8a5dfcc9-dec3-4bc6-b0a8-c928350b3596\" (UID: \"8a5dfcc9-dec3-4bc6-b0a8-c928350b3596\") " Oct 30 13:09:02.251740 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:09:02.251735 4908 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a5dfcc9-dec3-4bc6-b0a8-c928350b3596-tls-certs\" (UniqueName: \"kubernetes.io/secret/8a5dfcc9-dec3-4bc6-b0a8-c928350b3596-tls-certs\") pod \"8a5dfcc9-dec3-4bc6-b0a8-c928350b3596\" (UID: \"8a5dfcc9-dec3-4bc6-b0a8-c928350b3596\") " Oct 30 13:09:02.252067 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:09:02.251925 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a5dfcc9-dec3-4bc6-b0a8-c928350b3596-tmp-dir" pod "8a5dfcc9-dec3-4bc6-b0a8-c928350b3596" (UID: "8a5dfcc9-dec3-4bc6-b0a8-c928350b3596"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:09:02.252067 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:09:02.251941 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a5dfcc9-dec3-4bc6-b0a8-c928350b3596-model-cache" pod "8a5dfcc9-dec3-4bc6-b0a8-c928350b3596" (UID: "8a5dfcc9-dec3-4bc6-b0a8-c928350b3596"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:09:02.252198 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:09:02.252180 4908 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a5dfcc9-dec3-4bc6-b0a8-c928350b3596-tmp-dir\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:09:02.252266 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:09:02.252203 4908 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a5dfcc9-dec3-4bc6-b0a8-c928350b3596-model-cache\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:09:02.252318 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:09:02.252258 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a5dfcc9-dec3-4bc6-b0a8-c928350b3596-home" pod "8a5dfcc9-dec3-4bc6-b0a8-c928350b3596" (UID: "8a5dfcc9-dec3-4bc6-b0a8-c928350b3596"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:09:02.254236 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:09:02.254209 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a5dfcc9-dec3-4bc6-b0a8-c928350b3596-dshm" pod "8a5dfcc9-dec3-4bc6-b0a8-c928350b3596" (UID: "8a5dfcc9-dec3-4bc6-b0a8-c928350b3596"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:09:02.254383 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:09:02.254327 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a5dfcc9-dec3-4bc6-b0a8-c928350b3596-kube-api-access-xp6wz" pod "8a5dfcc9-dec3-4bc6-b0a8-c928350b3596" (UID: "8a5dfcc9-dec3-4bc6-b0a8-c928350b3596"). InnerVolumeSpecName "kube-api-access-xp6wz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:09:02.254481 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:09:02.254460 4908 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a5dfcc9-dec3-4bc6-b0a8-c928350b3596-tls-certs" pod "8a5dfcc9-dec3-4bc6-b0a8-c928350b3596" (UID: "8a5dfcc9-dec3-4bc6-b0a8-c928350b3596"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:09:02.353389 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:09:02.353348 4908 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a5dfcc9-dec3-4bc6-b0a8-c928350b3596-home\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:09:02.353389 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:09:02.353382 4908 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a5dfcc9-dec3-4bc6-b0a8-c928350b3596-dshm\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:09:02.353389 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:09:02.353396 4908 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xp6wz\" (UniqueName: \"kubernetes.io/projected/8a5dfcc9-dec3-4bc6-b0a8-c928350b3596-kube-api-access-xp6wz\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:09:02.353635 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:09:02.353409 4908 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a5dfcc9-dec3-4bc6-b0a8-c928350b3596-tls-certs\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\"" Oct 30 13:09:02.848639 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:09:02.848597 4908 generic.go:356] "Generic (PLEG): container finished" podID="8a5dfcc9-dec3-4bc6-b0a8-c928350b3596" containerID="6d87953e0148ef1a9ab936760bbceec78ad43bbf2d54319c94c7cbded807b647" exitCode=0 Oct 30 13:09:02.848981 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:09:02.848646 4908 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-512ec5/simulator-w4-512ec5-kserve-7df7674f8-8rqwb" Oct 30 13:09:02.848981 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:09:02.848680 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-512ec5/simulator-w4-512ec5-kserve-7df7674f8-8rqwb" event={"ID":"8a5dfcc9-dec3-4bc6-b0a8-c928350b3596","Type":"ContainerDied","Data":"6d87953e0148ef1a9ab936760bbceec78ad43bbf2d54319c94c7cbded807b647"} Oct 30 13:09:02.848981 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:09:02.848751 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-512ec5/simulator-w4-512ec5-kserve-7df7674f8-8rqwb" event={"ID":"8a5dfcc9-dec3-4bc6-b0a8-c928350b3596","Type":"ContainerDied","Data":"21b33880b3e1d77d6ef4ea777af938cfcb78718c8c5a43e345fd1e70a62e0e95"} Oct 30 13:09:02.848981 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:09:02.848777 4908 scope.go:122] "RemoveContainer" containerID="6d87953e0148ef1a9ab936760bbceec78ad43bbf2d54319c94c7cbded807b647" Oct 30 13:09:02.864105 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:09:02.864002 4908 scope.go:122] "RemoveContainer" containerID="6d87953e0148ef1a9ab936760bbceec78ad43bbf2d54319c94c7cbded807b647" Oct 30 13:09:02.864408 ip-10-0-141-251 kubenswrapper[4908]: E1030 13:09:02.864330 4908 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d87953e0148ef1a9ab936760bbceec78ad43bbf2d54319c94c7cbded807b647\": container with ID starting with 6d87953e0148ef1a9ab936760bbceec78ad43bbf2d54319c94c7cbded807b647 not found: ID does not exist" containerID="6d87953e0148ef1a9ab936760bbceec78ad43bbf2d54319c94c7cbded807b647" Oct 30 13:09:02.864408 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:09:02.864356 4908 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d87953e0148ef1a9ab936760bbceec78ad43bbf2d54319c94c7cbded807b647"} err="failed to get container status \"6d87953e0148ef1a9ab936760bbceec78ad43bbf2d54319c94c7cbded807b647\": rpc error: code = NotFound desc = could not find container \"6d87953e0148ef1a9ab936760bbceec78ad43bbf2d54319c94c7cbded807b647\": container with ID starting with 6d87953e0148ef1a9ab936760bbceec78ad43bbf2d54319c94c7cbded807b647 not found: ID does not exist" Oct 30 13:09:02.905570 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:09:02.905532 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-512ec5/simulator-w4-512ec5-kserve-7df7674f8-8rqwb"] Oct 30 13:09:02.918925 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:09:02.918890 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-512ec5/simulator-w4-512ec5-kserve-7df7674f8-8rqwb"] Oct 30 13:09:03.604957 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:09:03.604918 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a5dfcc9-dec3-4bc6-b0a8-c928350b3596" path="/var/lib/kubelet/pods/8a5dfcc9-dec3-4bc6-b0a8-c928350b3596/volumes" Oct 30 13:09:04.031398 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:09:04.031353 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-992edf68-5x8qs"] Oct 30 13:09:04.046282 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:09:04.046243 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-992edf68-5x8qs"] Oct 30 13:09:05.603932 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:09:05.603896 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c19d2e0-b5e4-4e44-a2cf-6724f0f1e0b1" path="/var/lib/kubelet/pods/1c19d2e0-b5e4-4e44-a2cf-6724f0f1e0b1/volumes" Oct 30 13:09:06.609545 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:09:06.609512 4908 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-plczt_a70ee11c-c20e-4a4f-8476-ac5eb02816d0/console-operator/3.log" Oct 30 13:09:06.615272 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:09:06.615245 4908 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b8bxz_644e19b3-10ad-4145-9ed3-595d19ed9b07/ovn-acl-logging/0.log" Oct 30 13:09:07.389412 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:09:07.389379 4908 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-plczt_a70ee11c-c20e-4a4f-8476-ac5eb02816d0/console-operator/3.log" Oct 30 13:09:07.395488 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:09:07.395457 4908 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b8bxz_644e19b3-10ad-4145-9ed3-595d19ed9b07/ovn-acl-logging/0.log" Oct 30 13:09:08.035052 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:09:08.035010 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd8413a1-9x4km"] Oct 30 13:09:08.047843 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:09:08.047794 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd8413a1-9x4km"] Oct 30 13:09:09.122905 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:09:09.122867 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9wwfc"] Oct 30 13:09:09.571396 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:09:09.571354 4908 scope.go:122] "RemoveContainer" containerID="152a06bad4922828997c7c83c521980706205440d1023dd02f0c3597eeec3b07" Oct 30 13:09:09.585346 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:09:09.585318 4908 scope.go:122] "RemoveContainer" containerID="51d13c6c039e1f4f79a365fe51d17708279d7eb42ee276ca110cf5ff7075f5d6" Oct 30 13:09:09.605222 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:09:09.605194 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a88b3635-eba4-41bf-b9d3-a1a14c5305f9" path="/var/lib/kubelet/pods/a88b3635-eba4-41bf-b9d3-a1a14c5305f9/volumes" Oct 30 13:09:19.048229 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:09:19.048182 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc7cac0f-6mqbg"] Oct 30 13:09:19.059508 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:09:19.059465 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc7cac0f-6mqbg"] Oct 30 13:09:19.606278 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:09:19.606236 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7838da57-6c71-4e7a-a830-a2a69759e28d" path="/var/lib/kubelet/pods/7838da57-6c71-4e7a-a830-a2a69759e28d/volumes" Oct 30 13:09:19.684874 ip-10-0-141-251 kubenswrapper[4908]: E1030 13:09:19.684817 4908 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vn6vv" containerName="extractor" Oct 30 13:09:20.358624 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:09:20.358589 4908 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-7m749_ca1be587-5f3e-4004-8d90-573130f6e9ed/manager/0.log" Oct 30 13:09:20.522449 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:09:20.522415 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9wwfc"] Oct 30 13:09:20.576995 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:09:20.576962 4908 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-lnqqz_1398a43d-5477-4f46-bfad-6e8dba0af13a/manager/1.log" Oct 30 13:09:21.066585 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:09:21.066547 4908 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-5z7d8_e2bcf742-b9a1-46e1-865d-e7e062786c05/manager/0.log" Oct 30 13:09:21.296362 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:09:21.296327 4908 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-9wwfc_f01cbe07-9124-4bc8-80fd-902b23e57f67/limitador/0.log" Oct 30 13:09:21.481800 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:09:21.481765 4908 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-wgm7f_15039390-67e6-48c7-ba22-8704aa2cc777/manager/0.log" Oct 30 13:09:21.903515 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:09:21.903419 4908 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-lnqqz_1398a43d-5477-4f46-bfad-6e8dba0af13a/manager/0.log" Oct 30 13:09:22.694449 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:09:22.694408 4908 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-7q7p5_4c36eb48-c2b2-4500-a1eb-26a7611bc2b6/manager/0.log" Oct 30 13:09:22.893592 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:09:22.893511 4908 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-vjg48_706593d5-4331-41a7-a9d6-b66f2596e81b/manager/0.log" Oct 30 13:09:23.124887 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:09:23.124855 4908 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-nsldm_37fef44c-01b0-430e-a657-e9f22e60b580/manager/0.log" Oct 30 13:09:36.320302 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:09:36.320271 4908 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-7q7p5_4c36eb48-c2b2-4500-a1eb-26a7611bc2b6/manager/0.log" Oct 30 13:09:36.519814 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:09:36.519779 4908 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-vjg48_706593d5-4331-41a7-a9d6-b66f2596e81b/manager/0.log" Oct 30 13:09:36.739146 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:09:36.739114 4908 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-nsldm_37fef44c-01b0-430e-a657-e9f22e60b580/manager/0.log" Oct 30 13:09:39.147084 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:09:39.147049 4908 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-1e4a0f82-fsmv4_185877e0-bfbb-48fa-9c23-ebf709195402/revoke-keys/0.log" Oct 30 13:09:39.523045 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:09:39.523008 4908 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-2eac5b06-dd7vs_4558943a-c9e0-4649-896c-e78d2fe1d25f/revoke-keys/0.log" Oct 30 13:09:39.717662 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:09:39.717620 4908 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-479ba8c0-pg9qf_e5c2ee3e-6917-4773-8bab-76eaa9f1d9ed/revoke-keys/0.log" Oct 30 13:09:40.092211 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:09:40.092170 4908 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-64a0fec2-r5hzg_8a632b40-348a-4fcb-b5ed-fc8d1b6d5cd4/revoke-keys/0.log" Oct 30 13:09:40.285361 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:09:40.285334 4908 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6c55470c-scdtx_ff9664e3-b790-4a5e-b96a-5553ef1189d8/revoke-keys/0.log" Oct 30 13:09:40.671034 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:09:40.671006 4908 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-8e9148a5-cll94_ec981005-170b-4a49-8029-a9a93bcfa9db/revoke-keys/0.log" Oct 30 13:09:40.865553 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:09:40.865515 4908 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-c60d35c2-9qk7g_c4dc85ad-0955-4880-994e-857e19ad4db6/revoke-keys/0.log" Oct 30 13:09:41.091272 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:09:41.091239 4908 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-c8e525d9-m494z_29225043-ea8e-4a6a-b259-373c827784bf/revoke-keys/0.log" Oct 30 13:09:41.282119 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:09:41.282083 4908 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-cfba1837-8pwmd_8e77e666-f695-41d9-9243-d845872f235c/revoke-keys/0.log" Oct 30 13:09:43.252333 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:09:43.252302 4908 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-7m749_ca1be587-5f3e-4004-8d90-573130f6e9ed/manager/0.log" Oct 30 13:09:43.463978 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:09:43.463941 4908 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-lnqqz_1398a43d-5477-4f46-bfad-6e8dba0af13a/manager/1.log" Oct 30 13:09:43.927528 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:09:43.927474 4908 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-5z7d8_e2bcf742-b9a1-46e1-865d-e7e062786c05/manager/0.log" Oct 30 13:09:44.148578 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:09:44.148546 4908 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-9wwfc_f01cbe07-9124-4bc8-80fd-902b23e57f67/limitador/0.log" Oct 30 13:09:44.346890 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:09:44.346857 4908 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-wgm7f_15039390-67e6-48c7-ba22-8704aa2cc777/manager/0.log" Oct 30 13:09:44.783469 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:09:44.783440 4908 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-lnqqz_1398a43d-5477-4f46-bfad-6e8dba0af13a/manager/0.log" Oct 30 13:09:45.308631 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:09:45.308599 4908 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-85d774877-zkcdx_6faa552b-c9a4-49a8-bc98-4d5a2cabf3e8/istio-proxy/0.log" Oct 30 13:09:45.506199 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:09:45.506162 4908 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-vmxrp_8c130b2b-0577-4020-82e0-2c6c1e48f653/discovery/0.log" Oct 30 13:09:46.701422 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:09:46.701346 4908 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-bt2fx_b3c15d0a-6612-444e-9e5f-af10d8bc367d/istio-proxy/0.log" Oct 30 13:09:47.300251 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:09:47.300216 4908 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5bfdf864c-tzv67_6acb13a1-cb2e-45e1-897b-3ef89e6d1b84/router/0.log" Oct 30 13:09:49.685508 ip-10-0-141-251 kubenswrapper[4908]: E1030 13:09:49.685469 4908 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vn6vv" containerName="extractor" Oct 30 13:10:06.500378 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:10:06.500345 4908 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-fxjlt_6b350710-c48c-4c7e-bce9-e021099e98ca/global-pull-secret-syncer/0.log" Oct 30 13:10:07.083470 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:10:07.083436 4908 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-ccctl_1f5667a1-a773-4520-b2fb-c69148df632c/connection-checker/0.log" Oct 30 13:10:07.203855 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:10:07.203808 4908 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-k8ldb_8e44595b-0bdf-407c-bf22-33eb06272cc1/connection-checker/0.log" Oct 30 13:10:07.329063 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:10:07.329035 4908 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-kg9td_e497352b-2348-4fc7-888f-5b68f116394f/connection-checker/0.log" Oct 30 13:10:07.859594 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:10:07.859562 4908 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-tdf8d_4e18f771-273b-49e6-872c-6cbe47ab68c4/konnectivity-agent/0.log" Oct 30 13:10:08.299255 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:10:08.299226 4908 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-141-251.ec2.internal_a73cae35bcc0a4b145486ce8e9e460c1/haproxy/0.log" Oct 30 13:10:09.652330 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:10:09.652289 4908 scope.go:122] "RemoveContainer" containerID="394b5d61c097f9b712a7522773ee65dda6be9ad76415ad0079a3504a679aac7f" Oct 30 13:10:09.662957 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:10:09.662932 4908 scope.go:122] "RemoveContainer" containerID="453f3a1ce0c37b291835f57325d6789c0977328bac829442c8d6a273e12ddbe4" Oct 30 13:10:09.673029 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:10:09.673001 4908 scope.go:122] "RemoveContainer" containerID="610e9f6e3570a524674d74893bfc0ef5a8996a025ca0fb0327714d799eda674a" Oct 30 13:10:09.685789 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:10:09.685763 4908 scope.go:122] "RemoveContainer" containerID="3d2cf01d94a04429b1efa5eb7b77e903288f114bce1ac251e446dbe445501071" Oct 30 13:10:09.696466 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:10:09.696440 4908 scope.go:122] "RemoveContainer" containerID="b45da0a61ff596744b6b70fa4ca7c566b2793b22603fe65ac2d3b633a6ec44e7" Oct 30 13:10:16.049394 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:10:16.049352 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cfba1837-8pwmd"] Oct 30 13:10:16.062469 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:10:16.062435 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cfba1837-8pwmd"] Oct 30 13:10:16.377991 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:10:16.377911 4908 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-7m749_ca1be587-5f3e-4004-8d90-573130f6e9ed/manager/0.log" Oct 30 13:10:16.511665 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:10:16.511635 4908 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-lnqqz_1398a43d-5477-4f46-bfad-6e8dba0af13a/manager/1.log" Oct 30 13:10:16.511867 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:10:16.511730 4908 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-lnqqz_1398a43d-5477-4f46-bfad-6e8dba0af13a/manager/0.log" Oct 30 13:10:16.795399 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:10:16.795361 4908 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-5z7d8_e2bcf742-b9a1-46e1-865d-e7e062786c05/manager/0.log" Oct 30 13:10:17.604945 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:10:17.604904 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e77e666-f695-41d9-9243-d845872f235c" path="/var/lib/kubelet/pods/8e77e666-f695-41d9-9243-d845872f235c/volumes" Oct 30 13:10:17.610900 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:10:17.610875 4908 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-9wwfc_f01cbe07-9124-4bc8-80fd-902b23e57f67/limitador/0.log" Oct 30 13:10:17.766936 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:10:17.766905 4908 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-wgm7f_15039390-67e6-48c7-ba22-8704aa2cc777/manager/0.log" Oct 30 13:10:18.409976 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:10:18.409939 4908 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-7q7p5_4c36eb48-c2b2-4500-a1eb-26a7611bc2b6/manager/0.log" Oct 30 13:10:18.496921 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:10:18.496888 4908 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-47mpn/perf-node-gather-daemonset-pmkrj"] Oct 30 13:10:18.497419 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:10:18.497403 4908 cpu_manager.go:414] "containerMap: removing container" podUID="8a5dfcc9-dec3-4bc6-b0a8-c928350b3596" containerName="main" Oct 30 13:10:18.497494 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:10:18.497423 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a5dfcc9-dec3-4bc6-b0a8-c928350b3596" containerName="main" Oct 30 13:10:18.497494 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:10:18.497452 4908 cpu_manager.go:414] "containerMap: removing container" podUID="8a632b40-348a-4fcb-b5ed-fc8d1b6d5cd4" containerName="revoke-keys" Oct 30 13:10:18.497494 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:10:18.497461 4908 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a632b40-348a-4fcb-b5ed-fc8d1b6d5cd4" containerName="revoke-keys" Oct 30 13:10:18.497679 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:10:18.497547 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a5dfcc9-dec3-4bc6-b0a8-c928350b3596" containerName="main" Oct 30 13:10:18.497679 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:10:18.497559 4908 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a632b40-348a-4fcb-b5ed-fc8d1b6d5cd4" containerName="revoke-keys" Oct 30 13:10:18.501019 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:10:18.500998 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-47mpn/perf-node-gather-daemonset-pmkrj" Oct 30 13:10:18.521603 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:10:18.521576 4908 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-vjg48_706593d5-4331-41a7-a9d6-b66f2596e81b/manager/0.log" Oct 30 13:10:18.523187 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:10:18.523165 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-47mpn/perf-node-gather-daemonset-pmkrj"] Oct 30 13:10:18.533015 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:10:18.532985 4908 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-47mpn\"/\"default-dockercfg-xw6qs\"" Oct 30 13:10:18.561753 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:10:18.561724 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-47mpn\"/\"openshift-service-ca.crt\"" Oct 30 13:10:18.561953 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:10:18.561893 4908 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-47mpn\"/\"kube-root-ca.crt\"" Oct 30 13:10:18.635622 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:10:18.635576 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c0d3cfe8-ed9c-4aeb-ada2-a05823f28692-lib-modules\") pod \"perf-node-gather-daemonset-pmkrj\" (UID: \"c0d3cfe8-ed9c-4aeb-ada2-a05823f28692\") " pod="openshift-must-gather-47mpn/perf-node-gather-daemonset-pmkrj" Oct 30 13:10:18.636133 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:10:18.635647 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64r45\" (UniqueName: \"kubernetes.io/projected/c0d3cfe8-ed9c-4aeb-ada2-a05823f28692-kube-api-access-64r45\") pod \"perf-node-gather-daemonset-pmkrj\" (UID: \"c0d3cfe8-ed9c-4aeb-ada2-a05823f28692\") " pod="openshift-must-gather-47mpn/perf-node-gather-daemonset-pmkrj" Oct 30 13:10:18.636133 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:10:18.635682 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c0d3cfe8-ed9c-4aeb-ada2-a05823f28692-sys\") pod \"perf-node-gather-daemonset-pmkrj\" (UID: \"c0d3cfe8-ed9c-4aeb-ada2-a05823f28692\") " pod="openshift-must-gather-47mpn/perf-node-gather-daemonset-pmkrj" Oct 30 13:10:18.636133 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:10:18.635799 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c0d3cfe8-ed9c-4aeb-ada2-a05823f28692-proc\") pod \"perf-node-gather-daemonset-pmkrj\" (UID: \"c0d3cfe8-ed9c-4aeb-ada2-a05823f28692\") " pod="openshift-must-gather-47mpn/perf-node-gather-daemonset-pmkrj" Oct 30 13:10:18.636133 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:10:18.635860 4908 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c0d3cfe8-ed9c-4aeb-ada2-a05823f28692-podres\") pod \"perf-node-gather-daemonset-pmkrj\" (UID: \"c0d3cfe8-ed9c-4aeb-ada2-a05823f28692\") " pod="openshift-must-gather-47mpn/perf-node-gather-daemonset-pmkrj" Oct 30 13:10:18.736412 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:10:18.736371 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c0d3cfe8-ed9c-4aeb-ada2-a05823f28692-lib-modules\") pod \"perf-node-gather-daemonset-pmkrj\" (UID: \"c0d3cfe8-ed9c-4aeb-ada2-a05823f28692\") " pod="openshift-must-gather-47mpn/perf-node-gather-daemonset-pmkrj" Oct 30 13:10:18.736630 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:10:18.736441 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64r45\" (UniqueName: \"kubernetes.io/projected/c0d3cfe8-ed9c-4aeb-ada2-a05823f28692-kube-api-access-64r45\") pod \"perf-node-gather-daemonset-pmkrj\" (UID: \"c0d3cfe8-ed9c-4aeb-ada2-a05823f28692\") " pod="openshift-must-gather-47mpn/perf-node-gather-daemonset-pmkrj" Oct 30 13:10:18.736630 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:10:18.736475 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c0d3cfe8-ed9c-4aeb-ada2-a05823f28692-sys\") pod \"perf-node-gather-daemonset-pmkrj\" (UID: \"c0d3cfe8-ed9c-4aeb-ada2-a05823f28692\") " pod="openshift-must-gather-47mpn/perf-node-gather-daemonset-pmkrj" Oct 30 13:10:18.736630 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:10:18.736521 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c0d3cfe8-ed9c-4aeb-ada2-a05823f28692-lib-modules\") pod \"perf-node-gather-daemonset-pmkrj\" (UID: \"c0d3cfe8-ed9c-4aeb-ada2-a05823f28692\") " pod="openshift-must-gather-47mpn/perf-node-gather-daemonset-pmkrj" Oct 30 13:10:18.736630 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:10:18.736585 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c0d3cfe8-ed9c-4aeb-ada2-a05823f28692-sys\") pod \"perf-node-gather-daemonset-pmkrj\" (UID: \"c0d3cfe8-ed9c-4aeb-ada2-a05823f28692\") " pod="openshift-must-gather-47mpn/perf-node-gather-daemonset-pmkrj" Oct 30 13:10:18.736630 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:10:18.736629 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c0d3cfe8-ed9c-4aeb-ada2-a05823f28692-proc\") pod \"perf-node-gather-daemonset-pmkrj\" (UID: \"c0d3cfe8-ed9c-4aeb-ada2-a05823f28692\") " pod="openshift-must-gather-47mpn/perf-node-gather-daemonset-pmkrj" Oct 30 13:10:18.736988 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:10:18.736674 4908 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c0d3cfe8-ed9c-4aeb-ada2-a05823f28692-podres\") pod \"perf-node-gather-daemonset-pmkrj\" (UID: \"c0d3cfe8-ed9c-4aeb-ada2-a05823f28692\") " pod="openshift-must-gather-47mpn/perf-node-gather-daemonset-pmkrj" Oct 30 13:10:18.736988 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:10:18.736765 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c0d3cfe8-ed9c-4aeb-ada2-a05823f28692-podres\") pod \"perf-node-gather-daemonset-pmkrj\" (UID: \"c0d3cfe8-ed9c-4aeb-ada2-a05823f28692\") " pod="openshift-must-gather-47mpn/perf-node-gather-daemonset-pmkrj" Oct 30 13:10:18.736988 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:10:18.736764 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c0d3cfe8-ed9c-4aeb-ada2-a05823f28692-proc\") pod \"perf-node-gather-daemonset-pmkrj\" (UID: \"c0d3cfe8-ed9c-4aeb-ada2-a05823f28692\") " pod="openshift-must-gather-47mpn/perf-node-gather-daemonset-pmkrj" Oct 30 13:10:18.761225 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:10:18.761189 4908 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64r45\" (UniqueName: \"kubernetes.io/projected/c0d3cfe8-ed9c-4aeb-ada2-a05823f28692-kube-api-access-64r45\") pod \"perf-node-gather-daemonset-pmkrj\" (UID: \"c0d3cfe8-ed9c-4aeb-ada2-a05823f28692\") " pod="openshift-must-gather-47mpn/perf-node-gather-daemonset-pmkrj" Oct 30 13:10:18.815648 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:10:18.815612 4908 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-47mpn/perf-node-gather-daemonset-pmkrj" Oct 30 13:10:19.045878 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:10:19.045763 4908 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-nsldm_37fef44c-01b0-430e-a657-e9f22e60b580/manager/0.log" Oct 30 13:10:19.129597 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:10:19.129568 4908 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-47mpn/perf-node-gather-daemonset-pmkrj"] Oct 30 13:10:19.131543 ip-10-0-141-251 kubenswrapper[4908]: W1030 13:10:19.131513 4908 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podc0d3cfe8_ed9c_4aeb_ada2_a05823f28692.slice/crio-e7387116063c9d5be87fc48e620b6dc63956a16b79a9f98df0603052d61fdf9f WatchSource:0}: Error finding container e7387116063c9d5be87fc48e620b6dc63956a16b79a9f98df0603052d61fdf9f: Status 404 returned error can't find the container with id e7387116063c9d5be87fc48e620b6dc63956a16b79a9f98df0603052d61fdf9f Oct 30 13:10:19.410094 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:10:19.409995 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-47mpn/perf-node-gather-daemonset-pmkrj" event={"ID":"c0d3cfe8-ed9c-4aeb-ada2-a05823f28692","Type":"ContainerStarted","Data":"e2b94e66fdf8087a589f3de0981e7bd750e45d5ec679609ec24d2be3777dd87b"} Oct 30 13:10:19.410094 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:10:19.410059 4908 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-47mpn/perf-node-gather-daemonset-pmkrj" event={"ID":"c0d3cfe8-ed9c-4aeb-ada2-a05823f28692","Type":"ContainerStarted","Data":"e7387116063c9d5be87fc48e620b6dc63956a16b79a9f98df0603052d61fdf9f"} Oct 30 13:10:19.410316 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:10:19.410185 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-47mpn/perf-node-gather-daemonset-pmkrj" Oct 30 13:10:19.460793 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:10:19.460735 4908 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-47mpn/perf-node-gather-daemonset-pmkrj" podStartSLOduration=1.460720715 podStartE2EDuration="1.460720715s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 13:10:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:10:19.459571282 +0000 UTC m=+4575.974000039" watchObservedRunningTime="2026-10-30 13:10:19.460720715 +0000 UTC m=+4575.975149460" Oct 30 13:10:19.685811 ip-10-0-141-251 kubenswrapper[4908]: E1030 13:10:19.685719 4908 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vn6vv" containerName="extractor" Oct 30 13:10:19.705473 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:10:19.705433 4908 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-7q7p5_4c36eb48-c2b2-4500-a1eb-26a7611bc2b6/manager/0.log" Oct 30 13:10:19.834082 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:10:19.834046 4908 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-vjg48_706593d5-4331-41a7-a9d6-b66f2596e81b/manager/0.log" Oct 30 13:10:20.513309 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:10:20.513271 4908 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-nsldm_37fef44c-01b0-430e-a657-e9f22e60b580/manager/0.log" Oct 30 13:10:20.690607 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:10:20.690569 4908 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-mvhwk_6d976900-ecd0-469c-a9e2-d60708361f1a/cluster-monitoring-operator/0.log" Oct 30 13:10:21.915496 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:10:21.915452 4908 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-h7xsz_c22cb98a-c283-4b39-af4d-4a4edf388b16/node-exporter/0.log" Oct 30 13:10:22.037586 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:10:22.037554 4908 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-h7xsz_c22cb98a-c283-4b39-af4d-4a4edf388b16/kube-rbac-proxy/0.log" Oct 30 13:10:22.161251 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:10:22.161218 4908 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-h7xsz_c22cb98a-c283-4b39-af4d-4a4edf388b16/init-textfile/0.log" Oct 30 13:10:22.883538 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:10:22.883505 4908 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-7q7p5_4c36eb48-c2b2-4500-a1eb-26a7611bc2b6/manager/0.log" Oct 30 13:10:22.991062 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:10:22.991026 4908 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-vjg48_706593d5-4331-41a7-a9d6-b66f2596e81b/manager/0.log" Oct 30 13:10:23.599067 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:10:23.599023 4908 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-nsldm_37fef44c-01b0-430e-a657-e9f22e60b580/manager/0.log" Oct 30 13:10:24.230492 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:10:24.230451 4908 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-7q7p5_4c36eb48-c2b2-4500-a1eb-26a7611bc2b6/manager/0.log" Oct 30 13:10:24.351328 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:10:24.351282 4908 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-vjg48_706593d5-4331-41a7-a9d6-b66f2596e81b/manager/0.log" Oct 30 13:10:25.025575 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:10:25.025534 4908 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-nsldm_37fef44c-01b0-430e-a657-e9f22e60b580/manager/0.log" Oct 30 13:10:25.433395 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:10:25.433290 4908 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-47mpn/perf-node-gather-daemonset-pmkrj" Oct 30 13:10:27.046694 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:10:27.046655 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2eac5b06-dd7vs"] Oct 30 13:10:27.049251 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:10:27.049216 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c55470c-scdtx"] Oct 30 13:10:27.051381 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:10:27.051354 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c55470c-scdtx"] Oct 30 13:10:27.059039 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:10:27.059016 4908 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-479ba8c0-pg9qf"] Oct 30 13:10:27.060928 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:10:27.060907 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2eac5b06-dd7vs"] Oct 30 13:10:27.062737 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:10:27.062714 4908 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-479ba8c0-pg9qf"] Oct 30 13:10:27.480115 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:10:27.480089 4908 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-7q7p5_4c36eb48-c2b2-4500-a1eb-26a7611bc2b6/manager/0.log" Oct 30 13:10:27.552886 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:10:27.552855 4908 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-86c478bb88-zzxhv_179d55d3-7958-4928-a98c-68984fca9837/networking-console-plugin/0.log" Oct 30 13:10:27.588019 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:10:27.587990 4908 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-vjg48_706593d5-4331-41a7-a9d6-b66f2596e81b/manager/0.log" Oct 30 13:10:27.604785 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:10:27.604748 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4558943a-c9e0-4649-896c-e78d2fe1d25f" path="/var/lib/kubelet/pods/4558943a-c9e0-4649-896c-e78d2fe1d25f/volumes" Oct 30 13:10:27.605445 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:10:27.605425 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5c2ee3e-6917-4773-8bab-76eaa9f1d9ed" path="/var/lib/kubelet/pods/e5c2ee3e-6917-4773-8bab-76eaa9f1d9ed/volumes" Oct 30 13:10:27.605981 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:10:27.605961 4908 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff9664e3-b790-4a5e-b96a-5553ef1189d8" path="/var/lib/kubelet/pods/ff9664e3-b790-4a5e-b96a-5553ef1189d8/volumes" Oct 30 13:10:28.316405 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:10:28.316365 4908 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-nsldm_37fef44c-01b0-430e-a657-e9f22e60b580/manager/0.log" Oct 30 13:10:28.875876 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:10:28.875809 4908 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-7q7p5_4c36eb48-c2b2-4500-a1eb-26a7611bc2b6/manager/0.log" Oct 30 13:10:28.982719 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:10:28.982679 4908 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-vjg48_706593d5-4331-41a7-a9d6-b66f2596e81b/manager/0.log" Oct 30 13:10:29.136353 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:10:29.136261 4908 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-plczt_a70ee11c-c20e-4a4f-8476-ac5eb02816d0/console-operator/3.log" Oct 30 13:10:29.142065 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:10:29.142034 4908 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-plczt_a70ee11c-c20e-4a4f-8476-ac5eb02816d0/console-operator/4.log" Oct 30 13:10:29.581085 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:10:29.581041 4908 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-nsldm_37fef44c-01b0-430e-a657-e9f22e60b580/manager/0.log" Oct 30 13:10:31.697244 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:10:31.697214 4908 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-58c9f5f46d-ksxpw_d0e0cec2-f16a-4bae-bc70-6c08da4bda2e/volume-data-source-validator/0.log" Oct 30 13:10:32.999036 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:10:32.999002 4908 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-85d774877-zkcdx_6faa552b-c9a4-49a8-bc98-4d5a2cabf3e8/istio-proxy/0.log" Oct 30 13:10:33.713626 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:10:33.713581 4908 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-85d774877-zkcdx_6faa552b-c9a4-49a8-bc98-4d5a2cabf3e8/istio-proxy/0.log" Oct 30 13:10:33.823216 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:10:33.823182 4908 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-85d774877-zkcdx_6faa552b-c9a4-49a8-bc98-4d5a2cabf3e8/istio-proxy/0.log" Oct 30 13:10:34.799583 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:10:34.799544 4908 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-xxcbn_221ced85-cc15-4e07-82d9-a1a1f3e958a5/dns/0.log" Oct 30 13:10:34.915698 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:10:34.915667 4908 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-xxcbn_221ced85-cc15-4e07-82d9-a1a1f3e958a5/kube-rbac-proxy/0.log" Oct 30 13:10:35.289917 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:10:35.289868 4908 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-xmrdj_1b735438-5947-4374-bd4c-66ac3a299770/dns-node-resolver/0.log" Oct 30 13:10:36.996489 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:10:36.996448 4908 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-hb8j7_5aee536a-8942-4b48-9b2c-3e36089f3ac5/node-ca/0.log" Oct 30 13:10:37.055976 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:10:37.055934 4908 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-vmxrp_8c130b2b-0577-4020-82e0-2c6c1e48f653/discovery/0.log" Oct 30 13:10:37.391407 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:10:37.391287 4908 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-bt2fx_b3c15d0a-6612-444e-9e5f-af10d8bc367d/istio-proxy/0.log" Oct 30 13:10:37.730573 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:10:37.730532 4908 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5bfdf864c-tzv67_6acb13a1-cb2e-45e1-897b-3ef89e6d1b84/router/0.log" Oct 30 13:10:38.143968 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:10:38.143876 4908 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-vmxrp_8c130b2b-0577-4020-82e0-2c6c1e48f653/discovery/0.log" Oct 30 13:10:38.223566 ip-10-0-141-251 kubenswrapper[4908]: I1030 13:10:38.223510 4908 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-vmxrp_8c130b2b-0577-4020-82e0-2c6c1e48f653/discovery/0.log"